Python hyperqueue package for ARM

[theely]

Hi there,
Would it be possible to release the hyperqueue API client also for the ARM architecture?
Or alternatively, I am open to suggestions on how to enable the development that makes use of your API client on Apple silicon.

Cheers,
Elia

[Kobzol]

Hi, just to clarify, you are looking for precompiled HyperQueue Python API package that would work on ARM (Apple Silicon)?

[theely]

Yes exactly, a package that can be installed via pip install ...

…

On Tue, Jul 16, 2024, 14:50 Jakub Beránek ***@***.***> wrote: Hi, just to clarify, you are looking for precompiled HyperQueue Python API package that would work on ARM (Apple Silicon)? — Reply to this email directly, view it on GitHub <#726 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAPOUDYMFGN3YBWQT7LB663ZMUJJJAVCNFSM6AAAAABK6QZ4ZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZQHAYTANZXHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Kobzol]

I see. So, currently we offer pre-built binary HyperQueue packages for x64, PowerPC64 and ARM64 Linux, and a pre-built Python HyperQueue package for x64 Linux.

We could in theory expand the number of packages that we offer, but a larger issue is that HyperQueue doesn't really support macOS (you can find more information here: #688). While it can be currently compiled for it, we do not offer any support for it, and some functionality will simply not work on other operating systems than Linux, at the moment.

Since macOS is not supported, and the Python API itself is still very experimental, we probably will not support prebuilt macOS Python packages anytime soon. I'm not even sure how can we build such packages, but if you investigate and find some solution, I'd be glad if you can post it here.

[spirali]

I see. So, currently we offer pre-built binary HyperQueue packages for x64, PowerPC64 and ARM64 Linux, and a pre-built Python HyperQueue package for x64 Linux.

We could in theory expand the number of packages that we offer, but a larger issue is that HyperQueue doesn't really support macOS (you can find more information here: #688). While it can be currently compiled for it, we do not offer any support for it, and some functionality will simply not work on other operating systems than Linux, at the moment.

Since macOS is not supported, and the Python API itself is still very experimental, we probably will not support prebuilt macOS Python packages anytime soon. I'm not even sure how can we build such packages, but if you investigate and find some solution, I'd be glad if you can post it here.

On the other hand, our problems with MacOS is mostly on the worker side. So theoretically it should not be not that hard to support MacOS client. However, I am no volunteering to do it :)

[theely]

Thanks for the fast responses.
I followed your advice and built my own binary. I was able to compile the Python package on MacOS with no issues.
If it helps, I am happy to share the binary (hyperqueue-0.19.0-cp36-abi3-macosx_11_0_arm64.whl)?

[Kobzol]

I'm glad that it worked for you! Providing a single wheel is not enough to make this robust for us, though. We'd need a way to produce the wheels on our CI, using maturin. This is the CI job that builds our Linux Python package. It would probably need to run in a macOS container so that it could generate a macOS ARM Python wheel.

[theely]

I used maturin. Everything worked out of the box (no custom changes needed).
Github does provide macOS arm images: https://github.com/actions/runner-images?tab=readme-ov-file#available-images

[Kobzol]

Okay, if maturin supports it out of the box, then it shouldn't be that hard. I don't have bandwidth to work on this right now, but I'd happy to review a PR :)