Recent HDF5 package in conda-forge (1.14.4) does not define HDF5_VERSION in CMake

[franzpoeschel]

Describe the bug
CMake scripts that include HDF5 can normally use HDF5_VERSION to determine the version of the found HDF5 installation. For some reason, the current package available on conda-forge does not set this variable.

To reproduce, use a minimal CMakeLists.txt:

cmake_minimum_required(VERSION 3.22)

project(hdf5-debug)

add_executable(main main.c)
find_package(HDF5 1.14.0 REQUIRED COMPONENTS C)
message(STATUS "HDF5 version: ${HDF5_VERSION}")
target_link_libraries(main PRIVATE HDF5::HDF5)

Create a test environment with Conda (I'm using miniforge3 on LinuxMint 21.3, but saw this originally in a CI run with conda-forge on Ubuntu 22.04):

. miniforge3/bin/activate
conda create -n hdf5-debug
conda activate hdf5-debug
conda install hdf5 cmake compilers pkg-config

This currently installs HDF5 1.14.4.

Then try configuring the above CMake script and see that the variable is apparently not set (ref. the line saying -- HDF5 version::

> cmake . -B build 
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /home/franz/miniforge3/envs/hdf5-debug/lib/libhdf5.so (Required is at least version "1.14.0") found components: C
-- HDF5 version: 
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/franz/workspace/hdf5-debug/build

Downgrading to HDF5 1.14.0 will fix this:

> conda install hdf5=1.14.0 cmake compilers pkg-config
> rm -r build
> cmake . -B build 
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /home/franz/miniforge3/envs/hdf5-debug/lib/libhdf5.so (found suitable version "1.14.0", minimum required is "1.14.0") found components: C
-- HDF5 version: 1.14.0
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /home/franz/workspace/hdf5-debug/build

The issue does not occur with a self-compiled version of HDF5 1.14.4:

> cmake . -B build2 
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/dxxi1l921a48zhsaxsixccaal310af29-ccache-links-wrapper-4.9.1/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/dxxi1l921a48zhsaxsixccaal310af29-ccache-links-wrapper-4.9.1/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found HDF5: hdf5-shared (found suitable version "1.14.4", minimum required is "1.14.0") found components: C
-- HDF5 version: 1.14.4
-- Configuring done (2.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/franz/workspace/hdf5-debug/build2

Expected behavior
HDF5_VERSION should be defined.

Platform (please complete the following information)

• HDF5 version 1.14.4 from conda-forge
• Linux Mint 21.3, Ubuntu 22.04
• Compiler and version: GNU 13.3.0
• Build system (e.g. CMake, Autotools) and version: CMake 3.31.1
• Any configure options you specified: none
• MPI library and version (parallel HDF5): seen with and without MPI (our CI enables MPI, my local test did not)

Additional context
Failing CI run: https://github.com/openPMD/openPMD-api/actions/runs/12012158126/job/33600997878

[byrnHDF]

I have inspected the CMake code and the config files produced and cannot find anything that would affect the VERSION string.
It would be worthwhile to see the CMake hdf5-config-*.cmake files for both - since that is what the find_package call is using.

[franzpoeschel]

I think that this might rather be an issue in the Conda packaging and not in the HDF5 CMake scripts themselves, considering that I do not have these problems with a self-compiled installation of HDF5.
The Conda packages apparently do not install hdf5-config-*.cmake files:

> find -iname '*hdf5*'
./conda-meta/hdf5-1.14.4-nompi_h2d575fe_105.json
./lib/libhdf5.settings
./lib/libhdf5.so
./lib/libhdf5.so.310
./lib/libhdf5.so.310.4.0
./lib/libhdf5_cpp.so
./lib/libhdf5_cpp.so.310
./lib/libhdf5_cpp.so.310.0.4
./lib/libhdf5_fortran.so
./lib/libhdf5_fortran.so.310
./lib/libhdf5_fortran.so.310.3.0
./lib/libhdf5_hl.so
./lib/libhdf5_hl.so.310
./lib/libhdf5_hl.so.310.0.4
./lib/libhdf5_hl_cpp.so
./lib/libhdf5_hl_cpp.so.310
./lib/libhdf5_hl_cpp.so.310.0.4
./lib/libhdf5_hl_fortran.so
./lib/libhdf5hl_fortran.so
./lib/libhdf5hl_fortran.so.310
./lib/libhdf5hl_fortran.so.310.0.4
./share/cmake-3.31/Help/module/FindHDF5.rst
./share/cmake-3.31/Modules/FindHDF5.cmake
./share/hdf5_examples
./include/hdf5.h
./include/hdf5.mod
./include/hdf5_hl.h

Find below the FindHDF5.cmake file (had to add the .txt ending for GitHub).
FindHDF5.cmake.txt

I also noticed that in the Conda environment, both in the failing CI run and in my local reproducer, there is a line -- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program., but this also appears in version 1.14.0 which does not have these issues.

The CMake variables for HDF5 in the Conda environment look as follows:

 HDF5_C_COMPILER_EXECUTABLE       /home/franz/miniforge3/envs/hdf5-debug/bin/h5cc                                                                           
 HDF5_C_INCLUDE_DIR               /home/franz/miniforge3/envs/hdf5-debug/include                                                                            
 HDF5_DIFF_EXECUTABLE             /home/franz/miniforge3/envs/hdf5-debug/bin/h5diff                                                                         
 HDF5_DIR                         HDF5_DIR-NOTFOUND                                                                                                         
 HDF5_IS_PARALLEL                 OFF                                                                                                                       
 HDF5_hdf5_LIBRARY_DEBUG          HDF5_hdf5_LIBRARY_DEBUG-NOTFOUND                                                                                          
 HDF5_hdf5_LIBRARY_RELEASE        /home/franz/miniforge3/envs/hdf5-debug/lib/libhdf5.so

For my self-compiled version, they are:

 HDF5_DIFF_EXECUTABLE             /nix/store/hriqy2nnqpxl9lg8hijhv93yf6hvwx7c-hdf5-1.14.4/bin/h5diff                                                                                                                                                                                                                      
 HDF5_DIR                         /nix/store/hriqy2nnqpxl9lg8hijhv93yf6hvwx7c-hdf5-1.14.4/cmake

For what it's worth, here are the hdf5-config-*.cmake files for the self-installed version, but those work correctly anyway cmake.zip

[hmaarrfk]

Is it because we are using the configure script instead of the cmake build system?

We could switch from one build system to the next in the next release of HDF5.

Either 1.14.5 (since we haven't released yet), or 1.14.6. Unfortunately we can't do it for releases that we have already made.

[byrnHDF]

find_package falls back to module mode (FindHDF5.cmake in the CMake repo) when it encounters an autotools build. So the missing HDF5_VERSION is a result of that CMake Find module.
Note that the next release of HDF5 will be 2.0.0 and we will be dropping autotools files in favor of the CMake build system for builds across all platforms. There will be a significant number of changes in the next release, needed for future development.

[hmaarrfk]

Note that the next release of HDF5 will be 2.0.0 and we will be dropping autotools

this makes our decision REALLY easy in 2.0.0 ^_^

So the missing HDF5_VERSION is a result of that CMake Find module.

I think I'm just misunderstanding your wording since i'm not an expert at cmake configuration and fallbacks thereof.

Given the fact that conda-forge uses autotools (for now 1.14.3 and 1.14.4 being our latest released versions), is there anything we can do to help cmake users beyond moving to a cmake build system? We could do this as part of our 1.14.5 update.

[byrnHDF]

I don't know why the HDF5_VERSION value is not being set, it must be some failure/issue in the CMake FindHDF5.cmake module file? Maybe it has an issue with asking for 1.14.0 and only having 1.14.4? A test would be to ask for 1.14.4 and see if the HDF5_VERSION is set?
Short answer is that there isn't much to be done with autotools builds and I'm not sure of the "add the cmake files" workaround to suggest that solution.

[franzpoeschel]

find_package falls back to module mode (FindHDF5.cmake in the CMake repo) when it encounters an autotools build. So the missing HDF5_VERSION is a result of that CMake Find module.

So this is not a bug, but just how that installation works? To be fair, I did not look into the CMake logic of HDF5, just tried to narrow down the conditions under which this behavior occurs.
I have now modified our CMake logic to deal with the missing version string without failure, so I don't necessarily need the bug fixed in HDF5 (except for versions that are already released maybe, that is).
I'll leave the ticket open as it's a reproducible issue that might affect other users (and I was a bit surprised to see the behavior change in a patch release). Feel free to close it as you need.

Maybe it has an issue with asking for 1.14.0 and only having 1.14.4?

I can try that out, but I think that it won't make a difference; where I first saw this issue, the find_package() call does not ask for a version at all (instead, we check HDF5_VERSION which is how we ended up here anyway..).

[franzpoeschel]

Yep, no difference:

> cat ../CMakeLists.txt 
cmake_minimum_required(VERSION 3.22)

project(hdf5-debug)

add_executable(main main.c)
find_package(HDF5 1.14.4 REQUIRED COMPONENTS C)
message(STATUS "HDF5 version: ${HDF5_VERSION}")
target_link_libraries(main PRIVATE HDF5::HDF5)(hdf5-debug)
> cmake ..
-- The C compiler identification is GNU 13.3.0
-- The CXX compiler identification is GNU 13.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/franz/miniforge3/envs/hdf5-debug/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- HDF5 C compiler wrapper is unable to compile a minimal HDF5 program.
-- Found HDF5: /home/franz/miniforge3/envs/hdf5-debug/lib/libhdf5.so (Required is at least version "1.14.4") found components: C
-- HDF5 version: 
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/franz/workspace/hdf5-debug/build