Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Porting to Allegro CL 11 / MacOS 14 / Apple silicon (M2) #13

Open
LEHunter opened this issue Apr 21, 2024 · 63 comments
Open

Porting to Allegro CL 11 / MacOS 14 / Apple silicon (M2) #13

LEHunter opened this issue Apr 21, 2024 · 63 comments

Comments

@LEHunter
Copy link

As a happy user of py4cl2 who wants to move some large arrays around, I thought I might try porting py4cl2-cffi to Allegro CL 11 / MacOS 14 / Apple silicon (M2). I've made some progress, but also hit a dead end. I'm trying to install via quicklisp.

First issue was this error:

While file-compiling #'"An Anonymous Function"
in
#P"/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/numpy-installed-p.lisp"
; between file character positions 0 (inclusively) and 3 (exclusively):
Error: Package "CL" not found.

The file just contains "CL:T". ACL doesn't define CL, and T is defined in every package. I'm using a case sensitive lisp, so I just replaced the contents of the file with "t". Works, although I'm not sure how that file gets created, so can't completely fix things.

Next up was

Subprocess with command "gcc -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/' -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o"
exited with error code 1

Not sure what configuration changes would be necessary to run this in the correct directory, but compiling manually after substituting the full pathname "~/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/py4cl-utils.c" worked.

Next problem was that modern MacOS doesn't provide endian.h So, following https://stackoverflow.com/questions/20813028/endian-h-not-found-on-mac-osx/52904079#52904079 I put https://gist.github.com/dendisuhubdy/19482135d26da86cdcf442b3724e0728 into /usr/local/include/endian.h and that worked.

Next problem was that the includes of numpy/ndarraytypes.h and numpy/arrayobject.h in py4cl-utils.c failed. I used (py4cl2:raw-pyeval "np.get_include()") to discover that the files were in ~/mambaforge/lib/python3.10/site-packages/numpy/core/include" so I could fix that manually, but didn't actually fix the configuration issue.

Finally, the linker complained about a bunch of missing symbols for the arm64 architecture, and here I am out of my depth. Error is below, any suggestions much appreciated.

gcc -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I/opt/homebrew/opt/[email protected]/Frameworks/Python.framework/Versions/3.12/include/python3.12 -I'/Users/hunter/quicklisp/dists/quicklisp/software/py4cl2-cffi-20231021-git/src/' -c -Wall -Werror -fpic py4cl-utils.c && gcc -shared -o libpy4cl-utils.so py4cl-utils.o
Undefined symbols for architecture arm64:
"_PyCapsule_GetPointer", referenced from:
__import_array in py4cl-utils.o
"_PyCapsule_Type", referenced from:
__import_array in py4cl-utils.o
"_PyErr_Format", referenced from:
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
"_PyErr_Print", referenced from:
_import_numpy in py4cl-utils.o
"_PyErr_SetString", referenced from:
_import_numpy in py4cl-utils.o
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
"_PyExc_AttributeError", referenced from:
__import_array in py4cl-utils.o
"_PyExc_ImportError", referenced from:
_import_numpy in py4cl-utils.o
"_PyExc_RuntimeError", referenced from:
__import_array in py4cl-utils.o
"_PyImport_ImportModule", referenced from:
__import_array in py4cl-utils.o
"_PyObject_GetAttrString", referenced from:
__import_array in py4cl-utils.o
"__Py_Dealloc", referenced from:
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
__import_array in py4cl-utils.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
src %
@LEHunter
Copy link
Author

Quick followup. The problems with numpy/ndarraytypes.h and numpy/arrayobject.h were because of a configuration error on my part (conda and brew not playing well together). Everything else remains.

@digikar99
Copy link
Owner

digikar99 commented Apr 22, 2024 •
edited
Loading

As a happy user of py4cl2 who wants to move some large arrays around.

py4cl2 pickles the arrays and then transfers py4cl2 sends and receives the larger arrays as npy files, so it could be sufficient. But I understand, it can still be too slow.

I'm using a case sensitive lisp.

Ah, I'd love to support a case sensitive readtable! I will check and fix the issues case-sensitivity creates in about 1-2 days.

I'm not sure how that file gets created, so can't completely fix things.

The file src/numpy-installed-p is created by src/shared-objects.lisp. It uses it as a auto-setup permanent configuration file to check and store whether numpy is installed. Depending on whether numpy is installed, the py4cl-utils.so (or equivalent) might need to be recompiled. The path to the numpy header files itself changes. This actually needs to be improved to handle different numpy versions in different environments.

For the linker error, I'm hoping to find a cloud service to test py4cl2-cffi on Apple systems. As a guess, you can try dumping the py4cl-utils.o using objdump or equivalent to see if the missing symbols it complains about are actually missing. If they are, that makes me curious why things are working on linux. Perhaps, on linux, there are no complaints because the other libraries are already loaded on SBCL through (load-python-and-libraries). If so, we might need to supply the information obtained through the python-config program in the py4cl2-cffi/config package to the step involved in the compilation of libpy4cl-utils.so itself. This would be the compile-utils-shared-object in shared-objects.lisp. Also, shouldn't that be libpy4cl-utils.dylib? Or do both libpy4cl-utils.so and libpy4cl-utils.dylib mean the same on mac?

@LEHunter
Copy link
Author

LEHunter commented Apr 22, 2024 •
edited
Loading

Thanks for the quick response!

I'll take a look at the linker issue tomorrow, but for now, I think you should change

(format nil "CL:~A" numpy-installed-p)

in src/shared-objects.lisp

to be
(write-to-string numpy-installed-p :readably t)

the CL:~A is not portable.

digikar99 added a commit that referenced this issue Apr 23, 2024
@digikar99
[fix] slightly improve handling of numpy-installed-p (#13)
0962e5f
@digikar99
Copy link
Owner

Hi! The issue about numpy-installed-p should be fixed now.

@digikar99
Copy link
Owner

digikar99 commented Apr 23, 2024 •
edited
Loading

I have made some progress in the macos branch - the CI uses macos-14, which according to github is M1. Let me know if you can get it working to a slightly better extent. It still needs some fixes though.

@digikar99
Copy link
Owner

I got hold of a MacOS at 5 USD for a week from hostmyapple. I can confirm that all-but-one tests pass on this, if you pass --dynamic-space-size 2560 or better option to SBCL.

User-6914:~ User$ sw_vers
ProductName:		macOS
ProductVersion:		14.4
BuildVersion:		23E214
User-6914:~ User$ uname -a
Darwin User-6914 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:31 PST 2024; root:xnu-10063.101.15~2/RELEASE_X86_64 x86_64

For reasons I'm unable to debug, on github actions:

  • macos-12 (x86_64) freezes while compiling the tests.
  • macos-14 (arm64) crashes. I wonder if I'm making some assumptions about pointers that are true on x86_64 but not on arm64. I might also check if I can get py4cl2-cffi to work on an arm64 linux device. Beyond that, an M1 mac in the cloud looks a bit too expensive.

@LEHunter
Copy link
Author

LEHunter commented Apr 25, 2024 via email

@digikar99
Copy link
Owner

The cheapest I see on HostMyApple is 69.99 USD/month. At those price points, I might as well buy myself an M1 or M2 in the upcoming year once I benchmark against my current i7-8750H to see if it's worth an upgrade :).

I'm certain there are other options out there, but I haven't checked which turns out to be the cheapest. There are other projects I'd like to support on MacOS and not just py4cl2-cffi. Setting up a sponsor might be the best going forward.

@LEHunter
Copy link
Author

LEHunter commented Apr 25, 2024 via email

@digikar99
Copy link
Owner

Alright, I'll set up a sponsor over the upcoming week or two. And will also check if py4cl2-cffi works on linux arm64 before that.

@LEHunter
Copy link
Author

LEHunter commented Apr 25, 2024 via email

@digikar99
Copy link
Owner

I just tested this on aarch64 via AnLinux (proot) on Termux on Android:

root@localhost:~# uname -a
Linux localhost 4.14.190-26178195-abT865XXS6DWH1 #2 SMP PREEMPT Thu Aug 3 10:33:12 KST 2023 aarch64 aarch64 aarch64 GNU/Linux

The good news is that all the tests from py4cl2-cffi-tests pass out of the box without any crashes. But that raises the concern that the crashing behavior on M1 MacOS 14 on Github Actions might be harder to debug than I initially thought.

The tests themselves pass on amd64 MacOS 14 as I tried on a HostMyMac instance the other day. Consistent freezing on github actions is reported to be not very uncommon.

M1 MacOS and their successors are the specific concerns then. Could you check on both ECL and SBCL if you obtain the crashing behavior on your machine too?

Also, running an arm64 linux VM on M* MacOS can be another option in the meanwhile.

@LEHunter
Copy link
Author

LEHunter commented Apr 28, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented Apr 28, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented Apr 28, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented Apr 28, 2024 via email

@digikar99
Copy link
Owner

Okay, great! And thanks for your reports, I will issue the fixes soon.

Well, this is interesting. I installed SBCL, and py4cl2-cffi loads without a hitch. I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message. However, this suggests a problem with Allegro CL, not py4cl2-cffi. I’ll file a bug report with them and see if they have anything to say…

Oh right, py4cl2-cffi-tests has a number of lisp dependencies that need more recent versions than the one's in quicklisp. Sidestepping them, you can manually compile the defpymodule forms in the py4cl2-cffi-tests/package.lisp file and see if it produces the appropriate lisp packages and functions. I will also work on modularizing the tests further to see if these dependencies can be separated out.

Until we get a new quicklisp dist, you can either use ultralisp or download the dependencies using download-dependencies. /path/to/dependencies can be either quicklisp local projects or a more temporary directory.

@LEHunter
Copy link
Author

LEHunter commented Apr 28, 2024 via email

@digikar99
Copy link
Owner

I couldn’t run the tests though. I tried git clone https://github.com/digikar99/py4cl2-cffi-tests in the quicklsp local-distributions directory, and got the result at the end of this message.

I have split the tests now. You should be able to (ql:quickload "py4cl2-cffi-tests-lite") against the current quicklisp dist. If you want to run the full tests, check out ultralisp or download-dependencies. I'll probably add clpm support some day.

(Uiop:operating-system) returns :macosx.

Same behavior on amd64 Mac. And thanks for pointing to this one! I have replaced (software-type) with (uiop:operating-system).

I just discovered -framework CoreFoundation flags from github actions' python3-config --libs and python3-config --ldflags. But, adding these flags didn't change anything --- github actions still crashes on M1 and freezes on non-M1. The non-M1 Mac I have access to passes the tests without crashing though!

@digikar99
Copy link
Owner

digikar99 commented Apr 29, 2024 •
edited
Loading

Okay, a very simple cause and fix.

44f0784: Mask all the float traps while loading numpy.

Tests pass now! (This is on the macos branch. I will merge into master soon.)

@LEHunter
Copy link
Author

LEHunter commented Apr 30, 2024 via email

@digikar99
Copy link
Owner

digikar99 commented Apr 30, 2024 •
edited
Loading

Still getting problems with the compile of py4cl-numpy-utils.c not finding the right (arm64) libraries. I think this is a configuration problem.

Is the config-darwin.lisp actually loading? It should be performing the configuration correctly. Is non-NIL (member :darwin cl:*features*) on your system? In py4cl2-cffi.asd, "py4cl2-cffi" depends on "py4cl2-cffi/config-darwin" only when the :darwin feature is present. May be this should be changed?

Otherwise, I will make the dependence unconditional. And leave it up to the (uiop:operating-system) to handle it correctly at the end of config-darwin.lisp.

(when (eq :macosx (uiop:operating-system))
  (configure))

(shared-library-from-ldflag python-ldflags (intern (uiop:operating-system)
:keyword))

load-python-and-libraries in shared-objects.lisp calls %shared-library-from-ldflag only when the flag name starts with -l. So, these errors should not be arising, could you confirm?

@LEHunter
Copy link
Author

LEHunter commented Apr 30, 2024 via email

@digikar99
Copy link
Owner

This does appear to be executing. NB: I think this (and any other lop level form) should be wrapped in a (eval-when (:compile-toplevel :load-toplevel :execute) … )

Ah, this should be it!

There may be a case problem here. Sometimes the flag is -L and sometime -l. Maybe that’s the problem?

The -l flag is for the library, the -L flag is for the search paths.

@digikar99
Copy link
Owner

This does appear to be executing. NB: I think this (and any other lop level form) should be wrapped in a (eval-when (:compile-toplevel :load-toplevel :execute) … )

Ah, this should be it!

But, may be no, compile and load of config-darwin.lisp should both take place before shared-objects.lisp itself. I will try but don't expect it to change anything.

@digikar99
Copy link
Owner

I believe config-darwin is loading.

Could you manually evaluate (py4cl2-cffi/config-darwin::configure) to see if the configuration variables are incorrect. (I assume you are at the latest commit of macos branch.)

@digikar99
Copy link
Owner

You might need to pull with the -f flag though.

@LEHunter
Copy link
Author

LEHunter commented Apr 30, 2024 via email

@digikar99
Copy link
Owner

Maybe not, actually: :Darwin is on features but the config-darrwin file does not seem to be getting loaded…

Okay, with 58bfb34, config-darwin should load unconditionally now.

There are other errors now which I'm looking into. Between 61cc8ef and 53402f3, something changed which is leading to memory fault during (py4cl2-cffi:pystart) in the presence of numpy.

@digikar99
Copy link
Owner

It appears that the (py4cl2-cffi/config-darwin::configure) function is still not being called.

Let me know the output of this:

(in-package :py4cl2-cffi/config-darwin)
(when (print (eq :macosx (print (uiop:operating-system))))
  (configure))

@LEHunter
Copy link
Author

LEHunter commented Apr 30, 2024 via email

@digikar99
Copy link
Owner

Alright, test it whenever you are free next. Hoping it works for you!

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@digikar99
Copy link
Owner

(find-class 'foreign-pointer)

pythonize defined in pythonizers.lisp is a generic function. One of its methods dispatches over the class corresponding to foreign-pointer. On SBCL, this is sb-sys:system-area-pointer. Could you find an equivalent class on Allegro CL? typexpand or type-expand or equivalent should be useful for this, or the Allegro CL documentation, or their support channels.

@digikar99
Copy link
Owner

digikar99 commented May 1, 2024 •
edited
Loading

Otherwise, I could merge the pythonize method dispatching over foreign-pointer into its default method. This would be a much portable solution.

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@digikar99
Copy link
Owner

I still get a warning when compiling numpy-utils

This still remains.

Fix this by wrapping the defconstant with eval-when:

(eval-when (:load-toplevel :compile-toplevel :execute)
...)

Done.

I could merge the pythonize method dispatching over foreign-pointer into its default method. This would be a much portable solution.

Done.

I have pushed these changes to the macos branch. Let me know if this works.

@digikar99
Copy link
Owner

I think there might be other places which require wrapping in eval-when too. I will check with asdf:compile-system and get back.

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@digikar99
Copy link
Owner

Turns out their license is permissive enough to adapt their code to our needs with appropriate acknowledgements.

@LEHunter
Copy link
Author

LEHunter commented May 1, 2024 via email

@digikar99
Copy link
Owner

I wonder what an appropriate configuration for ASDF would be to let it use the case insensitive readtable by default, and any other custom readtable only when a user explicitly specifies so.

I have also wanted package local readtables, but I don't see any good way other that dirty patching the lisp system to redefine certain core functions like read and compile. Or creating a shadowing CL package which acknowledges this. Well, I guess I can include that in peltadot - I already have a shadowing CL package there, including a shadowing compile function.

About the system::ff-funcall, can you check if it's possible to cffi:foreign-funcall for any other foreign function? If that does not work, may be CFFI might need to be updated for Allegro CL. But I doubt it could be as grave as that! Certainly, other Allegro CL users must be using CFFI, so it should be up to date.

PS: It will be another 2-3 days before I continue development. Some non-lisp tasks require my attention.

@LEHunter
Copy link
Author

LEHunter commented May 2, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented May 2, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented May 5, 2024 via email

@digikar99
Copy link
Owner

Sorry, it took me longer than expected!

I think it’s mostly a matter of testing, rather than trying for package-local read tables (which seem heavy duty, but doable).

To me, PLRT seem cleaner. Plus, they also allow one to use their favourite syntax modifications without bothering others. But yes, I do agree they seem heavy.

I’ve gotten the compiles of libpy4cl-utils and libpy4cl-numpy-utils to work from the command line. Here’s what seems to work (note -dynamiclib instead of -shared, and .dylib suffixes)

So, they did matter! Thanks a lot. I have pushed the fix in 9da26b1.

At this point, I think I need your help to figure out how to modify your config, loading and startup code to reproduce these within your system. If I load those dylibs, set python-libraries-loaded-p to t, and call (pystart), it fails. it calls your (load-python-and-libraries),which goes looking for the wrong files.

Is 9da26b1 the fix you were thinking about, or were you pointing to something else?

@digikar99
Copy link
Owner

The CI fix for macos/amd64 couldn't have been sillier. I had reported that I was able to get the tests to pass on macos on the amd64 VPS from HostMyMac, but the CI kept failing for amd64 macos on github actions. I have been able to get the CI pass on amd64 macos too. 9dbf4cb

With that, I think the changes and fixes in the macos branch can be merged into master. I can see that the CI passes. Could you test it locally on SBCL and confirm?

I'll create a separate branch for case sensitive lisp. Allegro CL can only proceed once CFFI gets fixed, right?

@LEHunter
Copy link
Author

LEHunter commented May 8, 2024 via email

@LEHunter
Copy link
Author

LEHunter commented May 9, 2024 via email

@digikar99
Copy link
Owner

It didn’t work, still had issues using the wrong architecture libraries

...

I tested on SBCL and (aside from not having networkx, which I just commented out), all the py4cl2-cffi-tests-lite pass.

Do you mean SBCL too requires lib-dynload instead of lib? Or did it pass without the lib-dynload fix?

I tried replacing lib with lib-dynload in de3dde8 but both SBCL / Mac OS 14 / arm64 and SBCL / Mac OS 13 / amd64 fail on Github Actions.

In a0e015b, I'm now looking up lib-dynload first and then lib after that, expecting gcc to mind the order. Even if this does pass for you, that makes me curious when or why some systems might use lib-dynload and some others only lib?

@LEHunter
Copy link
Author

LEHunter commented May 9, 2024 via email

@digikar99
Copy link
Owner

So, lib-dynload is unnecessary and we can delete it?

And does it work on Allegro now after you make the CFFI fix?

@digikar99
Copy link
Owner

With 65111b3, MacOS specific fixes have been merged into master. Thanks a lot for your tests and reports!
In the asd file, I'm expecting trivial-features to do the right thing and push :darwin into cl:features on Allegro. This should load config-darwin.lisp on Allegro too. But, in case there's any issue, feel free to ping.

I have started a separate branch readtable-case with fixes pertaining to case-sensitive readtable. Merging this requires figuring out the best way to handle readtable-case and setting up the CI.

@LEHunter
Copy link
Author

LEHunter commented May 13, 2024 via email

@digikar99
Copy link
Owner

  1. What you did to try to get Allegro to push :darwin onto features didn’t quite work. Doesn’t load config-darwin the way you have it. If I push :darwin onto features before quickloading py4cl2-cffi, it works.

Should be fixed by 307a121.

  1. Config-darwin is still trying to add the lib-dynload libraries, which causes an error. When I remove that (and the associated format argument), everything compiles.

I assume you are referring to the lib-dynload path not found warning. 18349a1 should fix it.

Although I note it still shows up in *python-site-libraries*

*python-site-packages-path* - this variable is being set using the information provided by python3 in the environment itself. I certainly expect there are no newlines in the pathnames.

The error message in lisp is

opening #P"/tmp/py4cl2-cffi-output-2322" resulted in
error: Interrupted system call [errno=4].

The '#P' indicates the error must stem from a lisp code. I suspect it is from these lines of code. This is called from the (python-output-thread) call just above the line of code you pointed out. The named pipe itself is created quite early. So, this is indeed mysterious.

Let me know if changing the value of *py-output-stream-pipe* and *py-error-output-stream-pipe* to some other locations help. They are set here. If so, I could move them to the config system/package.

The hanging probably occurs because, normally:

  1. The main thread asks python to open the named pipe for writing and waits.
  2. The "py4cl2-cffi-output-thread" opens the named pipe in read-mode. This resolves the waiting of the main thread, which can then continue executing normally.

So, if 2 fails, the main thread will keep waiting and that is, perhaps, what causes the lock-up.

@digikar99
Copy link
Owner

I just noted that cl-travis supports AllegroCL for CI. I guess I could try that here too.

@digikar99
Copy link
Owner

I just noted that cl-travis supports AllegroCL for CI. I guess I could try that here too.

Nope, haven't been able to debug why Allegro freezes on Github Actions - most likely, I'm passing the wrong command line options, but I haven't been able to figure out what the correct ones are. I wonder if it differs from platform to platform.

About m1 mac in cloud, Amazon EC2 pricing for M1 is off the charts. Scaleway, Hetzner, and HostMyApple all seem reasonable, but with their own peculiarities. The first two also offer on-demand hourly (or rather min. 24 hours) pricing at about 0.11 euro/hour (= 2.64 euro/day). HostMyApple only offers monthly plans, starting at 70 USD/month for M1 Mac. But that seems wasteful because I would never use it for an entire month, not even 15 days.

The bad news might be about sponsor platforms.

  • Github Sponsor requires waiting atleast 60 days, and there are some who report not receiving their payouts even after 5ish months.
  • Patreon: I'm from India, and due to government regulations, Indian payment methods are rejected on Patreon. I wonder how the case is with the payout from Patreon to any Indian bank account.
  • I'd be glad to hear any other options. But lacking that, I might be open to UPI based payments for within-India users. I gotta appreciate the system here for that. Getting UPI to work outside India seems a work in progress. Barring that, Paypal might be another alternative. However, Paypal does not allow India-to-India payments, so, potentially, it only allows India to outside-India and vice versa payments. If you are still looking to get py4cl2-cffi to work, we can try a 1 USD payment to see if it actually works.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@LEHunter @digikar99