Building OpenBLAS 0.3.24 fails on MacPorts running macOS 13.5.2

[kickingvegas]

Description

Updating to OpenBLAS 0.3.24 on MacPorts seg faults due to invalid memory reference. Issue reported to MacPorts with response from MacPorts maintainers to escalate upstream to OpenBLAS.

Error message below:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./cblat2 < ./cblat2.dat

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGTRAP: Trace/breakpoint trap.

Backtrace for this error:
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./zblat2 < ./zblat2.dat

Environment

Apple M1 Pro, macOS 13.5.2 (22G91)

Steps to Reproduce

Upgrade OpenBLAS from 0.3.23_0 to 0.3.24_0

Expected Result

OpenBLAS 0.3.24 is built and installed.

Actual Result

Build hangs, no backtrace is emitted in build log.

openblas-upgrade.log

[martin-frbg]

Is that Apple's clang you used, or something from macports as well ? I see the Fortran part seems to have been compiled with gfortran from GCC12, but I can't seem to find the clang version in your log. (No problems are/were seen in CI with clang14.0.6 and gfortran 12.2 (both from homebrew) under macOS 12)

[michaellass]

I am facing the same issue and was able to reproduce the issue just running make in a checked-out git master. It seems to be related to Apple Clang. There was very recently an XCode update (15.0) which could play a role here. By default, on my system the following compilers are used:

% cc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

% gfortran --version
GNU Fortran (MacPorts gcc12 12.3.0_0+stdlib_flag) 12.3.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

When building with CC=clang make, I can force the use of macport's clang instead of the system one:

% clang --version
clang version 16.0.6
Target: arm64-apple-darwin22.6.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-16/bin

With that, the segfault is gone. However, the tests fails at a later point in time:

TEST 38/38 kernel_regress:skx_avx dyld[9121]: missing symbol called
make[1]: *** [run_test] Abort trap: 6
make: *** [tests] Error 2

[catap]

@michaellass out of curiosity may you try to test MacPorts Clang-15 and possible Clang-16?

[michaellass]

@michaellass out of curiosity may you try to test MacPorts Clang-15 and possible Clang-16?

Sure:

• CC=/opt/local/bin/clang-mp-15 make NO_AVX=1: segfault
• CC=/opt/local/bin/clang-mp-16 make NO_AVX=1: only failing in the skx_avx test (any hint on how I can disable that test?)

So it looks like it's not an Apple clang issue but a clang 15 issue. I guess I should also test with clang 14 but I first need to install that.

[catap]

@michaellass good. MacPorts has clang-17 now (it was added yesterday), if you have time I suggest to check it at it also, and if it fails => seems like a strong indicator that it is regression in clang/llvm which should be backported to upstream as well.

After your test I'll handle blacklisting of the bad clang at MacPorts to make life of users easy :)

Thanks!

[michaellass]

Update on more versions:

• CC=/opt/local/bin/clang-mp-17 make NO_AVX=1: does not compile (see below)
• CC=/opt/local/bin/clang-mp-16 make NO_AVX=1: only failing in the skx_avx test
• CC=/opt/local/bin/clang-mp-15 make NO_AVX=1: segfault
• CC=/opt/local/bin/clang-mp-14 make NO_AVX=1: segfault
• CC=/opt/local/bin/clang-mp-13 make NO_AVX=1: segfault
• CC=/opt/local/bin/clang-mp-12 make NO_AVX=1: only failing in the skx_avx test

So clang versions 13, 14 and 15 show the error for me. I am a bit surprised that this issue only now popped up as I would think that XCode 14 came with a newer clang than version 12. And it also contradicts the CI that successfully builds with clang 14.

With clang 17, I get errors during compilation:

In file included from ../include/lapacke_config.h:43:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdlib.h:66:10: fatal error: 'sys/wait.h' file not found
   66 | #include <sys/wait.h>
      |          ^~~~~~~~~~~~
In file included from lapacke_cheev_2stage.c:33:
In file included from ../include/lapacke_utils.h:36:
In file included from ../include/lapacke.h:36:
In file included from ../include/lapack.h:8:
In file included from ../include/lapacke_config.h:43:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdlib.h:128:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/malloc/_malloc.h:38:10: fatal error: 'malloc/_malloc_type.h' file not found
   38 | #include <malloc/_malloc_type.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~

Those files are present though, so I think this may be an issue with clang 17 from macports.

[martin-frbg]

• CC=/opt/local/bin/clang-mp-16 make NO_AVX=1: only failing in the skx_avx test (any hint on how I can disable that test?)

you would need to edit utest/Makefile and remove all references to test_kernel_regress. The "missing symbol" error is very strange - and btw that test is neither specific to skylakex,nor to avx - it compares DGEMM results to their non-DGEMM counterpart. The similarity to the other failing tests is that it contains calls from C to Fortran.
And I take it you're compiling on&for Intel Mac ?

[michaellass]

And I take it you're compiling on&for Intel Mac ?

No, I have an M1 processor. Probably I'm missing some compilation flags. When building OpenBLAS via macports, that test is not an issue.

[martin-frbg]

ok, so the NO_AVX=1 in your make commands is entirely spurious. unfortunately I do not have M1 locally so I'm limited to whats available or reasonably installable on Cirrus

[martin-frbg]

I wonder if there is anything that MacPorts does fundamentally differently from homebrew - there are still no errors in the CI job when I update to the "brew" builds of LLVM 16 or 17 and gfortran from GCC13.2 .

[michaellass]

Please note that the segfault occurs with LLVM 13, 14 and 15 and not with 16 and 17. Could you please do a run similar to e15d717 but with llvm-15 and clang-15?

[catap]

@michaellass inside MacPorts issue we're discovered that it is seem to be well known issue with Xcode SDK, see: https://trac.macports.org/ticket/68225#comment:23

[martin-frbg]

Thanks for this update - and I have realized that Cirrus CI still gives me the same old MacOS Monterey when I request their "Ventura-xcode-latest", so my test have been with xcode <15 all along

[michaellass]

@michaellass inside MacPorts issue we're discovered that it is seem to be well known issue with Xcode SDK, see: https://trac.macports.org/ticket/68225#comment:23

Oh, thanks for the hint. I forgot to push the "CC me" button in macport's trac and missed that development entirely. So right now, this issue looks unrelated to OpenBLAS itself and instead it is just caused by GCC-generated object code being linked with the new XCode 15 linker.

[catap]

@martin-frbg base on macports/macports-ports#21354 (comment) I'd like to assume that this issue wasn't fixed on 0.3.25.

Cc: @szhorvat

[martin-frbg]

Can you please tell me if the log for your failed build contains the -Wl,-ld_classic that #4253 was supposed to add ? If the build seemed to pass on Intel Mac but failed on Silicon, maybe the xcode version check isn't working as intended there...

[catap]

@martin-frbg I suggest to ask @szhorvat because I haven't able to reproduce that issue.

[lepus2589]

@martin-frbg I see the following line in my build log multiple times (Ventura 13.6.2, Silicon M2 ARM64, XCode 15, MacPorts GCC 13.2, OpenBlas 0.3.25):

clang: warning: -Wl,-ld_classic: 'linker' input unused [-Wunused-command-line-argument]

Did Apple Clang 15 deprecate/remove the ld_classic linker option?

[catap]

@lepus2589 yes, it is. Soon ld-classic will be removed as well 🥲

[lepus2589]

But for XCode 15, they are officially suggesting it as a workaround (https://developer.apple.com/documentation/xcode-release-notes/xcode-15-release-notes#Linking). Why does my XCode Clang 15 not accept this?

[martin-frbg]

From their changelogs it appears that they assume the issue is fixed in XCode 15.1 (or one of its betas) - no idea what to make of that. I see that they also mention using -ld64 to invoke the old linker (if I read that correctly), could you try if substituting that in Makefile.system helps ?

[szhorvat]

Here's the logs from the MacPorts install:

https://gist.githubusercontent.com/szhorvat/5bcbf95b418d81b6309694334fc93c95/raw/1f61915de52c33d77ba9ecbf413eb09b3f9a6ed3/gistfile1.txt

The flag selecting the classic linker is present.

For some reason, tests are run during build (is this normal with the MacPorts setup, @catap ?) It is the tests that fail, not the build itself. Some tests crash, and some hang.

Here are the last few lines of output (not present in the above log) when I killed the tests after hanging for a while.

/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/fc/opt/local/bin/gfortran-mp-13 -m64 -Os -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -fno-tree-vectorize -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -o cblat3 cblat3.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran 
ld: warning: duplicate -rpath '/opt/local/lib/libgcc' ignored
ld: warning: ignoring duplicate libraries: '-lemutls_w', '-lgcc', '-lgfortran'
/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/fc/opt/local/bin/gfortran-mp-13 -m64 -Os -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8-a -fno-tree-vectorize -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -o zblat3 zblat3.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran 
ld: warning: duplicate -rpath '/opt/local/lib/libgcc' ignored
ld: warning: ignoring duplicate libraries: '-lemutls_w', '-lgcc', '-lgfortran'
rm -f ?BLAT3.SUMM
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./sblat3 < ./sblat3.dat
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./dblat3 < ./dblat3.dat

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGBUS: Access to an undefined portion of a memory object.

Backtrace for this error:
^Cmake[1]: *** [level3] Interrupt: 2
make: *** [tests] Interrupt: 2
Command failed:  cd "/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/OpenBLAS-0.3.25" && /usr/bin/make -j12 -w all AR=/opt/local/bin/ar RANLIB=/opt/local/bin/ranlib OPENBLAS_INCLUDE_DIR=/opt/local/include/openblas CC="/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/cc/usr/bin/clang" CXX="/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/cxx/usr/bin/clang++" OBJC="/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/objc/usr/bin/clang" OBJCXX="/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/objcxx/usr/bin/clang++" FC="/opt/local/var/macports/build/_Users_szhorvat_macports-ports_math_OpenBLAS/OpenBLAS/work/compwrap/fc/opt/local/bin/gfortran-mp-13" INSTALL="/usr/bin/install -c" CC+="-arch arm64" CXX+="-arch arm64" OBJC+="-arch arm64" OBJCXX+="-arch arm64" LD+="-arch arm64" F77+="-m64" F90+="-m64" FC+="-m64" 
Killed by signal: 2
Error: Aborted: SIGINT received.

I am not really familiar with the OpenBLAS build process. I did try to build it using cmake, directly from this repo, and choosing the v0.3.25 tag. The build completes, but make test leads to some segfaults and some of the tests will hang. It was picking up the system's C compile (Apple Clang) and MacPorts's gfortran 13.

My system:

macOS 13.6 22G120 arm64
Xcode 15.0.1 15A507

[martin-frbg]

Thanks, that looks as if the build ran as intended but the problem is (still/again) there. Running some tests with the build is normal (unless cross-compiling) and the library is very likely unusable if tests fail.No idea at present apart from trying ld64 instead of ld_classic as mentioned, or trying the other option(s) related to the handling of weak symbols that are mentioned in the release notes for xcode 15.x

[catap]

@martin-frbg before I'd like to reproduce an issue on this machine. Anyway, MacPorts builds everything with actual MACOSX_DEPLOYMENT_TARGET and on my case it is MACOSX_DEPLOYMENT_TARGET=13; that means that I'm testing your idea right now because as far as I see the code that logic is ignored when env contains MACOSX_DEPLOYMENT_TARGET.

[szhorvat]

Here's the output from running the tests after building OpenBLAS 0.3.25 from the release tarball using CMake. See the first line for how testing was set up. Lots of tests hang.

https://gist.github.com/szhorvat/a1f848b239d059d787f56ed00e91f5e1

[szhorvat]

Same thing (sans the segfault, which I think I know why it happens) if I build using gcc 13.2.0 instead of Apple's Clang. CMAKE_OSX_DEPLOYMENT_TARGET was left at the default of 13.6 in both cases.

[martin-frbg]

My log is here - the CirrusCI job runs in a VM that has both XCode and gfortran-13 preinstalled, I suspect "their" gfortran is
from Homebrew
https://gist.github.com/martin-frbg/ad33b7fd56c4568dd49dcd1a3a69cbf4#file-gistfile1-txt

[szhorvat]

This is the build log, but do the tests hang when you run them?

[martin-frbg]

The file is truncated in github's web view, I think you should get a "view the full file" link at the top ? All tests pass for me, only difference in the ctest setup is that mine does not have the -j 4 but I doubt it matters

[catap]

BTW I was able to reproduce it on my machine as well:

OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./cblat2 < ./cblat2.dat

Program received signal SIGSEGV: Segmentation fault - invalid memory reference.

Backtrace for this error:

Program received signal SIGTRAP: Trace/breakpoint trap.

Backtrace for this error:
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 ./zblat2 < ./zblat2.dat

[catap]

The log line to build zblat2 was:

:info:build /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_math_OpenBLAS/OpenBLAS/work/compwrap/fc/opt/local/bin/gfortran-mp-13 -m64 -Os -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8.3-a -O3 -Wall -frecursive -fno-optimize-sibling-calls  -march=armv8.3-a -fno-tree-vectorize -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc -Wl,-syslibroot,/Library/Developer/CommandLineTools/SDKs/MacOSX13.sdk -o zblat2 zblat2.o ../libopenblas-r1.a -lpthread -lgfortran -lpthread -lgfortran 

no ld-classic :(

[martin-frbg]

Do you see it applied during the build of the library ? CCOMMON_OPT should carry over to the Fortran flags eventually, but you could try adding the -Wl option to the FLDFLAGS in test/Makefile just to see if this fixes anything.

[catap]

@martin-frbg FLDFLAGS = -Wl,-ld_classic $(FFLAGS:-fPIC=) $(LDFLAGS) helps:

The following ports are currently installed:
  OpenBLAS @0.3.25_1+gcc13+lapack+native (active) requested_variants='+native' platform='darwin 22' archs='arm64' date='2023-11-18T19:59:13+0100'

[martin-frbg]

Thanks. Just noticed myself that the make build fails where the cmake build succeeded. So option propagation indeed an issue.

[catap]

@martin-frbg I've started to migrate Portfile to cmake, but haven't figured out how to make NO_AVX=1 and similar tricks via new cmake build.

[szhorvat]

@catap The segfault in the test gets resolved if I use -Wl,-rpath /opt/local/lib/gcc13 in the linker flags. Not the hangs though.

[catap]

@szhorvat in my test system were enough to add FLDFLAGS to switch to classic ld; I'm waiting the patch to propagate LDFLAGS, as soon as it ready I'll prepare PR to MacPorts with bugfix ;)

[szhorvat]

Just to confirm, did you see the hang at all?

[catap]

@szhorvat yes, I did.

I had upgraded my test mini and it had booted (I’m in few thousand kilometers), and was able to reproduce that issue after update

[martin-frbg]

@martin-frbg I've started to migrate Portfile to cmake, but haven't figured out how to make NO_AVX=1 and similar tricks via new cmake build.

cmake -DNO_AVX=1 etc. is supposed to work (even if it is probably not really advertised right now, being hidden in cmake scripts rather than the toplevel CMakeLists.txt).
Testing now if simply repeating the CCOMMON_OPT addition as FCOMMON_OPT in Makefile.system can be the "official patch".

[catap]

Testing now if simply repeating the CCOMMON_OPT addition as FCOMMON_OPT in Makefile.system can be the "official patch".

I've tried it on my end and it works as well

[martin-frbg]

Curiously it does not appear to end up in the gfortran command line when I try it on cirrus (or cirrus is borked and has not picked up that change yet), and consequently the tests still segfault. Did you remember to remove the FLDFLAGS addition again ?

[catap]

Curiously it does not appear to end up in the gfortran command line when I try it on cirrus (or cirrus is borked and has not picked up that change yet), and consequently the tests still segfault. Did you remember to remove the FLDFLAGS addition again ?

Let me double check it. I've run: sudo port clean --all OpenBLAS to cleanup any trace of build. After that I've run sudo port patch OpenBLAS +native to extract and apply all patches from MacPorts. After that I run sudo vi $(port work OpenBLAS)/OpenBLAS-0.3.25/Makefile.system to hack that file by hand to adjust only one thing to look like:

ifeq (x$(XCVER), x 15)
CCOMMON_OPT += -Wl,-ld_classic
FCOMMON_OPT += -Wl,-ld_classic
endif

and the last step was sudo port destroot OpenBLAS +native to prepare everyhting but not install it.

I just repeated all that steps and it works.

[martin-frbg]

Thank you - then Cirrus CI is somehow not applying the patch although the changes are shown correctly in github. I have seen (or at least suspected) this happen with them before. Annoying as they are the only semi-free provider of CI for the M1...

[catap]

@martin-frbg if you create a branch I may prepare an update for -devel subport, and we may ask @szhorvat to test it also.

[martin-frbg]

PR #4328 now

[rgommers]

Thank you - then Cirrus CI is somehow not applying the patch although the changes are shown correctly in github.

Cirrus is doing what multiple other CI systems are also doing - taking a PR branch as-is, and not automatically merging the default branch into the PR branch. It's a little annoying to convince it to do the merge: https://github.com/numpy/numpy/blob/18c6157f5da5736575b6d8d492aacca3b5d69551/tools/ci/cirrus_arm.yml#L1-L34

[martin-frbg]

@rgommers I'm not sure I understand - surely cloning the default branch and merging the PR would be the standard operation for any CI setup here, and it does work "most of the time" on Cirrus and all the time everywhere else. What Cirrus seems to be doing occasionally is merging an outdated version of the PR in question

[rgommers]

surely cloning the default branch and merging the PR would be the standard operation for any CI setup here

It's not actually, e.g. Circle CI does the same, you have to do the merge in your job if you want it: https://github.com/scipy/scipy/blob/f3cda9db9ad67cc897953fc2d1786edc651c1d4e/.circleci/config.yml#L38-L50.

I agree that it is pretty annoying coming from GitHub Actions; I am unsure if that Cirrus/Circle behavior is due to a philosophy on workflows (e.g., "will still run when there are merge conflicts") or just due to a not implemented feature and then backwards compat.

[catap]

@martin-frbg seems that this isn't enough. You may read discussion with @szhorvat here macports/macports-ports#21452 ; long story short: -Wl,-ld_classic should be added into .cmake files somehow, otherwise it had random failures.

[catap]

@martin-frbg and I've found one more issue with Xcode 15. Since 15 it drop support of -Wl,-noall_load.