[python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2620

johnkerl · 2024-05-22T13:52:22Z

Problem

Our current package-release CI is broken. It requires manual wheel-publish as at
https://github.com/single-cell-data/TileDB-SOMA/wiki/PyPI-packaging-notes

CI failures are like
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9189052757
following the well-intentioned but fateful #2491.

Also nightly wheel-build failures such as #2630.

Cause

Debugging reveals:

[ci] Move Python wheel-builds from macos-11,macos-12 to macos-12,macos-latest #2491 moved Python wheel-builds from macos 11 and 12 to 14 and 12
- Why we did that: GitHub Actions is deprecating macos 11 runners
Printing python -c 'from distutils import util; print(util.get_platform())' at build-wheel time and smoke-test-wheel time
- Build-wheel time on macos 14: this reports (!) macosx-10.9-universal2
  - https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25283290847#step:4:20
- Smoke-test-wheel time on macos 14: this reports (!) macosx-10.9-universal2
- https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25284241544#step:4:93
However: the wheel that's created has the substring macosx_11 in its filename :(
- https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472/job/25283290847#step:5:1005
- Created wheel for tiledbsoma: filename=tiledbsoma-1.11.2.post0.dev4180921687-cp38-cp38-macosx_11_0_x86_64.whl
As a result, at smoke-test time, util.get_platform() says macosx-10.9-universal2 and yet there is no wheel for 10.9 -- only 11.0
- On MacOS 11, pip install of a wheel with either macosx_10_9 or macosx_11_0 in its wheel-file name succeeds
- Not so on MacOS 14
- So this all worked fine when the smoke-test runner was macos 11; not fine now that it's macos 12 or 14

Attempted solutions

❌ Setting the CMAKE_OSX_DEPLOYMENT_TARGET: "10.9" in our CI YAML's env does not help since in [c++] Refactor SOMAArray and SOMAGroup to inherit from SOMAObject #2124 we set -mmacosx-version-min=11.0 to support usage of C++17 feature std::filesystem::path, which overrides the env
❌ Having our CI YAML copy files like tiledbsoma-VERSIONHERE-cp311-cp311-macosx_11_0_arm64.whl to tiledbsoma-VERSIONHERE-cp311-cp311-macosx_14_0_arm64.whl does not work, and in fact is a step in the wrong direction: since macos14 self-reports as 10.9 within pip install as shown above.
❌ Having our CI YAML rename files like tiledbsoma-VERSIONHERE-cp311-cp311-macosx_11_0_arm64.whl to tiledbsoma-VERSIONHERE-cp311-cp311-macosx_10_9_arm64.whl does work, since macos 14 self-reports as 10.9 at pip install time.
- ✅ Works on my laptop with macos14 and Python 3.9
- ❌ Fails in CI with macos14 and Python 3.11
- ❌ Fails in macos-on-macos Docker on my laptop with Python 3.11 (see below)

Laptop details

With macos 14 and Python 3.9:

laptop$ sw_vers
ProductName:		macOS
ProductVersion:		14.5
BuildVersion:		23F79

laptop$ python --version
Python 3.9.13

laptop$ ll | grep macos | grep x86
-rw-r--r--@ 1 johnkerl  staff  24547997 May 21 22:50 tiledbsoma-1.11.2-cp310-cp310-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24549656 May 21 22:50 tiledbsoma-1.11.2-cp311-cp311-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24547829 May 21 22:50 tiledbsoma-1.11.2-cp38-cp38-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  24548029 May 21 22:50 tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl
-rw-r--r--@ 1 johnkerl  staff  97615624 May 21 18:53 wheels-macos-x86_64.zip

laptop$ pip install tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl
ERROR: tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl is not a supported wheel on this platform.

laptop$ cp tiledbsoma-1.11.2-cp39-cp39-macosx_11_0_x86_64.whl tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl

laptop$ pip install tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl
Processing ./tiledbsoma-1.11.2-cp39-cp39-macosx_10_9_x86_64.whl

Note this exact logic -- successful on laptop -- fails as of
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9198109199/job/25300662543
for commit d793416

Using https://hub.docker.com/r/sickcodes/docker-osx -- macos-on-macos Docker -- with Python 3.11 I found that the trick which worked on my laptop (macos 14 with Python 3.9), and which fails in CI here with macos 14 and Python 3.11, also fails in sickcodes/docker-osx with Python 3.11.

In short it seems that the renaming trick from https://stackoverflow.com/questions/65888506/error-wheel-whl-is-not-a-supported-wheel-on-this-platform works in Python 3.9 but not 3.11.

Also note: supposedly-macos-on-macos sickcodes/docker-osx:latest isn't really:

[arch@041a7b1338f2 OSX-KVM]$ uname -a
Linux 041a7b1338f2 5.10.104-linuxkit #1 SMP Wed Mar 9 19:05:23 UTC 2022 x86_64 GNU/Linux

Also, why doesn't pip install tiledbsoma (from PyPI) fail on my Mac?

johnkerl@exade[prod][][~]$ pip install tiledbsoma
Collecting tiledbsoma
  Downloading tiledbsoma-1.11.1.tar.gz (291 kB)
...
Building wheels for collected packages: tiledbsoma
  Building wheel for tiledbsoma (pyproject.toml) ... |
...

-- it doesn't appear to be using the .whl file ...

Also note: on my arm64 mac with macos-14:

$ python -c 'from distutils import util; print(util.get_platform())'
macosx-14-arm64

However even macos12+arm64 runners in GHA misbehave:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9198915278/job/25302688619

distutil.util.get_platform: macosx-10.9-universal2

https://stackoverflow.com/questions/65888506/error-wheel-whl-is-not-a-supported-wheel-on-this-platform -- for the rename trick
MacOS producing wheel for 10.9 even though 11.0 is specified pypa/cibuildwheel#952 -- nice to know, but this turns out not to help here
Also note our CI YAML currently says pypa/[email protected] whereas newer versions exist https://github.com/pypa/cibuildwheel/releases/tag/v2.18.1 👀 🤔 ... this turns out not to help here

johnkerl · 2024-05-22T13:56:31Z

Manual debug runs:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9192805473
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193024559
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193078472

johnkerl · 2024-05-22T14:55:13Z

Manual debug run:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9193642019

johnkerl · 2024-05-22T15:43:28Z

Manual debug run with the copy strategy from 0d85de6:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9194374208

johnkerl · 2024-05-22T16:13:41Z

Manual debug run:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9194821897

johnkerl · 2024-05-22T16:45:37Z

Manual debug runs:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9195067641
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9196146568
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9196460727
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9196626582
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9196916723
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9197080504

johnkerl · 2024-05-22T19:35:40Z

Debug run with new verbosities/clarities, but, building and smoke-testing on macos 11:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9197526045

Same but without even the rename:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9197642927

johnkerl · 2024-05-22T21:59:17Z

Manual debug run trying macos with arm64 as there are (AFAIK) runners now:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9198915278
-- same failure as in the description section of this PR.

johnkerl · 2024-05-23T02:03:06Z

Experiment pip install -> python -m pip install:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9201085806

johnkerl · 2024-05-23T14:27:08Z

09f2481 manual run
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9209822187

johnkerl · 2024-05-23T16:22:59Z

Commit b39b87d revealing platform.platform() per advice of @eric-hughes-tiledb:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9211800457

johnkerl · 2024-05-23T16:58:04Z

I ran an EC2 macos14 instance with the help of @nguyenv -- it installed the wheel without problem.

Also note:

>>> platform.platform()
'macOS-14.4.1-x86_64-i386-64bit'

>>> util.get_platform()
'macosx-14.0-x86_64'

where the first one is per @eric-hughes-tiledb and the second is what I've been using.

So this EC2 MacOS instance does not self-report as macosx-10.9-universal2 whereas our tiledbsoma CI macos runner does.

So, useful info, but, does not allow me to interact with a failing macos system:

CI: failing -- can't interact -- i've been pushing print-debug PRs the last day and a half
My x86 work mac: no repro
My personal arm64 mac: no repro
EC2 mac x86 instance: no repro

johnkerl · 2024-05-23T17:08:23Z

At this point I've spent a day and a half of solid debug trying to find out why pip install of wheel-files doesn't work in GHA on MacOS unless MacOS v11 is being used (which GHA is deprecating).

The smoke test -- direct pip install of wheel file -- works on macos11 in GHA but GHA is deprecating that
The smoke test -- direct pip install of wheel file -- fails on macos > 11 GHA, with no resolution after extensive debug
Direct pip install of the wheel file succeeds on both my macs (x86 and arm64) without problem
Direct pip install of the wheel file succeeds on Mac EC2 instance
pip install tiledbsoma works fine for recent releases

Proving that this is a CI-only issue. I cannot justify further loss of valuable time chasing into this GHA-infra issue at present.

I've filed #2634 to follow up.

johnkerl · 2024-05-23T17:12:22Z

Manual run for commit 9ef7e16:
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/9212122517

* [python/ci] Debug MacOS 14 wheel-smoke-test issue * no ticket-cut for experiments * more * more * try `CMAKE_OSX_DEPLOYMENT_TARGET` * try the cp strategy * try again * more * narrow list of installables at smoke-test time * debug fail in last * neaten * abandon non-functional idea * typofox * neaten * iterating * experimenting with the old way to test narrative * typofix * debug * macos 11 experiment over; back to macos 14 * pip install -vv for smoke test * attempt macos arm64 * try `pip install` -> `python -m pip install` * try macos12 x86_64 and macos14 arm64 * debug * follow-up issue 2634

johnkerl · 2024-05-23T19:32:44Z

[sc-48079]

) * [python/ci] Debug MacOS 14 wheel-smoke-test issue * no ticket-cut for experiments * more * more * try `CMAKE_OSX_DEPLOYMENT_TARGET` * try the cp strategy * try again * more * narrow list of installables at smoke-test time * debug fail in last * neaten * abandon non-functional idea * typofox * neaten * iterating * experimenting with the old way to test narrative * typofix * debug * macos 11 experiment over; back to macos 14 * pip install -vv for smoke test * attempt macos arm64 * try `pip install` -> `python -m pip install` * try macos12 x86_64 and macos14 arm64 * debug * follow-up issue 2634 Co-authored-by: John Kerl <[email protected]>

[python/ci] Debug MacOS 14 wheel-smoke-test issue

3edc8d9

johnkerl force-pushed the kerl/macosx-14-wheel-smoke-test-debug branch from 7da2348 to 3edc8d9 Compare May 22, 2024 14:03

johnkerl added 2 commits May 22, 2024 10:05

no ticket-cut for experiments

78179d6

more

6ffc04b

johnkerl mentioned this pull request May 22, 2024

Daily GitHub Actions Build Fail on Wed, May 22nd 2024 #2622

Closed

more

f933cd5

johnkerl force-pushed the kerl/macosx-14-wheel-smoke-test-debug branch from 07b905a to f933cd5 Compare May 22, 2024 14:19

try CMAKE_OSX_DEPLOYMENT_TARGET

dee4de0

johnkerl mentioned this pull request May 22, 2024

[ci] Move Python wheel-builds from macos-11,macos-12 to macos-12,macos-latest #2491

Merged

try the cp strategy

0d85de6

johnkerl mentioned this pull request May 22, 2024

Daily GitHub Actions Build Fail on Wed, May 22nd 2024 #2623

Closed

try again

3c8f1a7

more

41dc4e3

johnkerl force-pushed the kerl/macosx-14-wheel-smoke-test-debug branch from a671bc6 to 41dc4e3 Compare May 22, 2024 16:31

johnkerl added 2 commits May 22, 2024 12:54

narrow list of installables at smoke-test time

716c37f

debug fail in last

ed04b21

johnkerl mentioned this pull request May 22, 2024

Daily GitHub Actions Build Fail on Wed, May 22nd 2024 #2625

Closed

johnkerl added 2 commits May 22, 2024 14:20

neaten

f18f004

abandon non-functional idea

c519900

johnkerl changed the title ~~[python/ci] Debug MacOS 14 wheel-smoke-test issue~~ [python/ci] Fix MacOS 14 wheel-smoke-test issue [WIP] May 22, 2024

johnkerl added 4 commits May 22, 2024 14:59

typofox

73500a0

neaten

c900354

iterating

8953076

experimenting with the old way to test narrative

b8172c0

try pip install -> python -m pip install

dd73b9d

johnkerl mentioned this pull request May 23, 2024

Daily GitHub Actions Build Fail on Thu, May 23rd 2024 #2630

Closed

try macos12 x86_64 and macos14 arm64

09f2481

debug

b39b87d

johnkerl force-pushed the kerl/macosx-14-wheel-smoke-test-debug branch from a7f7032 to b39b87d Compare May 23, 2024 16:44

johnkerl changed the title ~~[python/ci] Fix MacOS 14 wheel-smoke-test issue [WIP]~~ [python/ci] Diagnose MacOS 14 wheel-smoke-test issue May 23, 2024

johnkerl mentioned this pull request May 23, 2024

[ci] Resolve issues with MacOS wheel-file smoke test #2634

Closed

follow-up issue 2634

9ef7e16

johnkerl marked this pull request as ready for review May 23, 2024 17:09

johnkerl requested review from nguyenv and ryan-williams May 23, 2024 17:10

johnkerl changed the title ~~[python/ci] Diagnose MacOS 14 wheel-smoke-test issue~~ [python/ci] Stop MacOS 14 wheel-smoke-test failures May 23, 2024

johnkerl changed the title ~~[python/ci] Stop MacOS 14 wheel-smoke-test failures~~ [python/ci] Stop MacOS > 11 wheel-smoke-test failures May 23, 2024

johnkerl changed the title ~~[python/ci] Stop MacOS > 11 wheel-smoke-test failures~~ [python/ci] Stop wheel-smoke-test failures for MacOS > 11 May 23, 2024

nguyenv approved these changes May 23, 2024

View reviewed changes

johnkerl merged commit 768d8ab into main May 23, 2024
10 checks passed

johnkerl deleted the kerl/macosx-14-wheel-smoke-test-debug branch May 23, 2024 19:29

johnkerl added the backport release-1.11 label May 23, 2024

github-actions bot mentioned this pull request May 23, 2024

[Backport release-1.11] [python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2635

Merged

ryan-williams mentioned this pull request May 24, 2024

Daily GitHub Actions Build Fail on Fri, May 24th 2024 #2639

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2620

[python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2620

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024 •

edited

Loading

johnkerl commented May 23, 2024 •

edited

Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024 •

edited

Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024

[python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2620

[python/ci] Stop wheel-smoke-test failures for MacOS > 11 #2620

Conversation

johnkerl commented May 22, 2024 • edited Loading

Problem

Cause

Attempted solutions

Laptop details

See also

johnkerl commented May 22, 2024 • edited Loading

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024

johnkerl commented May 22, 2024 • edited Loading

johnkerl commented May 22, 2024 • edited Loading

johnkerl commented May 22, 2024 • edited Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024 • edited Loading

johnkerl commented May 23, 2024 • edited Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024 • edited Loading

johnkerl commented May 23, 2024

johnkerl commented May 23, 2024

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 22, 2024 •

edited

Loading

johnkerl commented May 23, 2024 •

edited

Loading

johnkerl commented May 23, 2024 •

edited

Loading

johnkerl commented May 23, 2024 •

edited

Loading