Odd meson behavior under CMake with f2py

[mathomp4]

I'm hoping someone here can help me out with this. I have a code under CMake that tries to run f2py3 on a very exciting code:

subroutine test
   real x
end subroutine test

mainly to determine what the suffix f2py3 will use will be. As I have Python 3.12, f2py3 means meson!

Now, by hand running:

f2py3 -m test_ -c test.F90

f2py3 is happy, meson does its thing, all is well.

Now, under CMake I do:

   execute_process(
      COMMAND ${F2PY3_EXECUTABLE} -m test_ -c ${file}
      WORKING_DIRECTORY ${_f2py3_check_bindir}
      RESULT_VARIABLE result
      )

which should run the same command (I've checked and I'm using the same f2py3 for both.

But here's the thing, the first time I run CMake I see:

-- F2PY3_EXECUTABLE: /Users/mathomp4/installed/Core/GEOSpyD/24.9.0-0/2024-11-05/envs/py3.12/bin/f2py3
The Meson build system
Version: 1.6.0
Source dir: /private/tmp/tmpwybc06l3
Build dir: /private/tmp/tmpwybc06l3/bbdir
Build type: native build
Project name: test_
Project version: 0.1
Fortran compiler for the host machine: /Users/mathomp4/.homebrew/brew/bin/gfortran-14 (gcc 14.2.0 "GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0")
Fortran linker for the host machine: /Users/mathomp4/.homebrew/brew/bin/gfortran-14 ld64 1115.7.3

meson.build:1:0: ERROR: Compiler /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang cannot compile programs.

A full log can be found at /private/tmp/tmpwybc06l3/bbdir/meson-logs/meson-log.txt
...

I've attached the file here:

meson-log.txt

So meson failed saying clang could not compile programs.

But if I then rerun CMake changing absolutely nothing:

-- F2PY3_EXECUTABLE: /Users/mathomp4/installed/Core/GEOSpyD/24.9.0-0/2024-11-05/envs/py3.12/bin/f2py3
The Meson build system
Version: 1.6.0
Source dir: /private/tmp/tmph93ewa6t
Build dir: /private/tmp/tmph93ewa6t/bbdir
Build type: native build
Project name: test_
Project version: 0.1
Fortran compiler for the host machine: /Users/mathomp4/.homebrew/brew/bin/gfortran-14 (gcc 14.2.0 "GNU Fortran (Homebrew GCC 14.2.0_1) 14.2.0")
Fortran linker for the host machine: /Users/mathomp4/.homebrew/brew/bin/gfortran-14 ld64 1115.7.3
C compiler for the host machine: /usr/bin/clang (clang 16.0.0 "Apple clang version 16.0.0 (clang-1600.0.26.4)")
C linker for the host machine: /usr/bin/clang ld64 1115.7.3
Host machine cpu family: aarch64
Host machine cpu: aarch64
Program /Users/mathomp4/installed/Core/GEOSpyD/24.9.0-0/2024-11-05/envs/py3.12/bin/python3 found: YES (/Users/mathomp4/installed/Core/GEOSpyD/24.9.0-0/2024-11-05/envs/py3.12/bin/python3)
Found pkg-config: YES (/Users/mathomp4/.homebrew/brew/bin/pkg-config) 0.29.2
...

I'm a bit baffled. Why would meson say clang can't compile programs the first time, but it can the second time?

Has anyone ever seen this behavior? I'm hoping maybe someone can see something in that meson-log.txt that I can't. I mean I see:

Sanity check compile stderr:
ld: library 'System' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)

but if clang can't see System the first time...why would it the second time?

[eli-schwartz]

It would be interesting to see the log file for the second (successful) time...

[mathomp4]

It would be interesting to see the log file for the second (successful) time...

@eli-schwartz Hmm. Is there a way to see where meson puts it? I only knew about the crash one because meson nicely told me where to find it.

[eli-schwartz]

Build dir: /private/tmp/tmpwybc06l3/bbdir

[...]

meson.build:1:0: ERROR: Compiler /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang cannot compile programs.

A full log can be found at /private/tmp/tmpwybc06l3/bbdir/meson-logs/meson-log.txt

Notice that it always logs the source and build directories. The log file is always in the same place in the build directory -- meson-logs/meson-log.txt. We only print where it is as part of the data collection advice emitted when an error occurred, but it always exists even when an error doesn't occur.

[mathomp4]

Well I feel dumb. Okay, did a complete clean. Here is the first log:

first-log.txt

and here is the second:

second-log.txt

I see some clang oddity, but I will note I have:

❯ echo $CC
/usr/bin/clang

but the two clangs are different in size:

❯ ls -l /usr/bin/clang
Permissions Size User Group Date Modified Name
.rwxr-xr-x@ 119k root wheel 15 Oct 06:22   /usr/bin/clang*

❯ ls -l /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang
Permissions Size User Group Date Modified Name
.rwxr-xr-x@ 257M root wheel 16 Oct 00:24   /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang*

but not in behavior (maybe):

❯ /usr/bin/clang --version
Apple clang version 16.0.0 (clang-1600.0.26.4)
Target: arm64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

❯ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang --version
Apple clang version 16.0.0 (clang-1600.0.26.4)
Target: arm64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[eli-schwartz]

So the moral here is that the one inside a nested subdirectory of Xcode.app doesn't work even to compile a simple program:

sanitycheckc.c

int main(void) { int class=0; return class; }

compiled via:

$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang sanitycheckc.c -o sanitycheckc.exe

ld: library 'System' not found
clang: error: linker command failed with exit code 1 (use -v to see invocation)

$ /usr/bin/clang sanitycheckc.c -o sanitycheckc.exe
# succeeds

The size could indicate that the version in /usr/bin is a wrapper shell script that sets paths up?

I wonder why rerunning cmake changes anything here. Does it e.g. export CC down to f2py and therefore meson?

Is the version in Xcode.app in $PATH anyways?

[mathomp4]

@eli-schwartz

The only clang I see is the /usr/bin one:

❯ which -a clang
/usr/bin/clang

I definitely don't add the SDKROOT one to my path!

That said, I do see that CMake seems to super-resolve it:

-- The Fortran compiler identification is GNU 14.2.0
-- The CXX compiler identification is AppleClang 16.0.0.16000026
-- The C compiler identification is AppleClang 16.0.0.16000026
-- Checking whether Fortran compiler has -isysroot
-- Checking whether Fortran compiler has -isysroot - yes
-- Checking whether Fortran compiler supports OSX deployment target flag
-- Checking whether Fortran compiler supports OSX deployment target flag - yes
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /Users/mathomp4/.homebrew/brew/bin/gfortran-14 - skipped
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang - skipped
...

And now that I know what to search for:

https://discourse.cmake.org/t/help-with-cc-clang-environment-and-cmake/2221/1

Hey hey! Look who opened that. 😄

If I then follow the chain we get to CMP0132

and if I add:

cmake_policy (SET CMP0132 NEW)

it works! (Here is the log: CMP0132.meson-log.txt)

I hate to do it, but, @bradking can you come over to GitHub land and lend your knowledge? Will this policy have any side-effects I don't know of? Not only for me, but perhaps for meson? I mean it seems to be a solution for me (and hopefully other meson-f2py-cmake users), but...

Note: I'm still not sure exactly what it does as I still see:

-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++ - skipped
...
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang - skipped
...

in the CMake, but things working are fine by me.

I won't close this yet until I can more fully test (the example I'm reporting on here is a bit...minimal), but good news it seems!

[eli-schwartz]

On subsequent CMake runs, these environment variables are not set, opening up the opportunity for different behavior between the first and subsequent CMake runs.

This is a fascinating decision...

[bradking]

Will this policy have any side-effects I don't know of?

If a CMake project invokes a third-party build system that also pays attention to CC/CXX env vars then its selection could change in the case that CC/CXX are not set in the first place, because CMake will no longer set them before launching that build system. If a project was relying on the old behavior to tell the third-party build system what compilers to use, it should start doing so another way.

This is a fascinating decision...

That's why policy CMP0132 removes the behavior. The quoted text is from the policy documentation's description of the old behavior and the motivation for changing it.

[mathomp4]

That's why policy CMP0132 removes the behavior. The quoted text is from the policy documentation's description of the old behavior and the motivation for changing it.

Ahhh. My brain was seeing it the other way. All I knew was it helped me :)

I suppose for now I'll do the finer cmake_policy(SET), though maybe it's time to just explore doing:

cmake_minimum_required(VERSION 3.24)
cmake_policy(VERSION 3.24)

I think my code also sets 0053 and 0054 but those are so old at this point setting those is probably like saying "Set CMake as of 10 years ago."

Actually, I guess just:

cmake_minimum_required(VERSION 3.24)

might solve...everything? Sets all the policies I care about...

[bradking]

@mathomp4 cmake_minimum_required(VERSION 3.24) should be fine, though if you don't want to drop support for older CMake versions you can also do cmake_minimum_required(VERSION 3.10...3.31), where 3.10 is the oldest version you support, ... is literal between the versions, and 3.31 is the latest version you've tested with. That will set policies to the newest that the running version of CMake supports, up to 3.31. See docs here.

[mathomp4]

So far, tests are positive on at least one machine. I mean, I still can't use meson with Intel ifort because of the usual meson-ifort warning bug, but I'm slowly going through my testing matrix.

But I think with some very ugly CMake I can sort of shove meson past the ifort issue. Probably be over at the CMake Discourse soon asking "how can I make this more elegant"? 😄