feat: Python wheels workflow and build backend

[zachlewis]

Description

Summary

This PR is the spiritual successor to #4011. It implements a scikit-build-core-based python build-backend, making it possible to use pip install . to build from source; and it adds a Github workflow for building with cibuildwheel and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.12, across major operating systems and architectures.

When you pip install OpenImageIO, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.

PEP-Compliant Packaging: pyproject.toml

The pyproject.toml file is organized in three parts:

1. Package metadata: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth.
2. scikit-build-core options: governs how pip install ... interacts with cmake.
3. cibuildwheel options: additional steps and considerations for building, packaging, "repairing", and testing relocatable wheel build artifacts in isolated environments.

Additions to __ init __.py

Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).

This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in pyproject.toml,
each item under [project.scripts] is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.

Relocatable Binary Distributions with cibuildwheel + repairwheel

cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.

Importantly, cibuildwheel allows for an arbitrary "repair-wheel-command" step, intended as a means for collecting and bundling any external shared libraries linked by the module (or the CLI tools) living outside the package, and subsequently patching RPATH entries to point to relative paths to these libraries inside the wheel. This ensures that compiled extensions have no runtime dependencies outside the pip toolchain.

In order to keep wheel file sizes to an absolute minimum, our repair-wheel-command is a multi-step process, implemented in a new tasks.py file living in the repo's root, invoked with invoke:

1. Remove everything except for the python module and the binary executables under /bin.
2. Use repairwheel to collect and bundle only and exactly the shared libraries needed -- i.e., missing dependencies built by OIIO's build system, as well as any other needed libraries found in the environment.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size.

Note: the steps taken in tasks.py remove everything except that which is absolutely necessary for end users to import the Python modules and run the CLI tools, partly in order to determine a baseline minimum file size for the wheels. However, part of the motivation for setting things up the way I have is to make it easier to add things back into the wheels in the future; e.g., should we decide to package the headers and/or the fonts, generate or modify pkgconfig .pcs and/or CMake modules and configs, etc. Also worth noting, there's pyproject.toml metadata we can add to help downstream scikit-build-core-based builds find OpenImageIO.

"Wheels" Github workflow

I straight-up copied .github/workflows/wheel.yml from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v2.6* or v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64, min deployment 10.15) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.

Note: THE PUBLISH TASK WILL FAIL until a "trusted publisher" is set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

I've submitted separate PRs for these, since they fall outside the intended scope of this PR.

Note: One of the aims for this PR is to leverage OIIO's new dependency-self-building mechanisms as much as possible, so that users who pip install from source are guaranteed a nominally working installation. However, we can also use platform-specific package managers for the wheels uploaded to pypi.org (e.g., added as cibuildwheel "before-build" commands for each platform).

Other Changes

I made some minor adjustments to pythonutils.cmake and fancy_add_executable.cmake that only affect scikit-build-core-based installs -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.

Tests

The cibuildwheel "test-command" is oiiotool --help. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to import OpenImageIO; that the actual binary executable oiiotool is properly packaged and exists in the expected location (e.g., at .../site-packages/OpenImageIO/bin); and that all runtime dependencies are found.

Inspiration, Credit, Prior Art

• @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See #3249, and #4011, as well as JCM's python_wheels and python_wheels_windows branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging.
• OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here.
• @joaovbs96's help and patience with testing stuff on Windows.

[aclark4life]

Well played 😄

[zachlewis]

ha -- we'll see... what I've been trying has apparently broken everything. I'm not sure if I have enough time to debug this myself. We may have to split off the attempt to get rid of tasks.py into its own PR. Right now, it's solving a few problems at once.

[JeanChristopheMorinPerso]

Great work with the latest changes @zachlewis! I like how the extract step to fix the wheels are not needed anymore.

I started to review the generated artifacts. I see that the versioned libraries are not yet fully removed:

$ unzip -l openimageio-3.1.0.0-cp313-cp313-linux_x86_64.whl | grep libOpenImageIO
 11355768  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO.so
 11355768  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO.so.3.1.0
  1171672  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO_Util.so
  1171672  2024-10-17 14:19   OpenImageIO/lib/libOpenImageIO_Util.so.3.1.0

I also see

    33700  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES-0.x.md
   223954  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES-1.x.md
    60831  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/CHANGES.md
    11357  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/LICENSE.md
    13418  2024-10-17 14:11   OpenImageIO/share/doc/OpenImageIO/THIRD-PARTY.md
    42480  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSans-Bold.ttf
    41028  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSans.ttf
   117072  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSansMono.ttf
    48880  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-Bold.ttf
    45652  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-BoldItalic.ttf
    40416  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif-Italic.ttf
    43648  2024-10-17 14:11   OpenImageIO/share/fonts/OpenImageIO/DroidSerif.ttf

Are these really needed?

The OIIO libs have an absolute RPATH:

[jcmorin@arch01 asd]$ readelf -d OpenImageIO/lib/libOpenImageIO.so.3.1.0

Dynamic section at offset 0x9cb738 contains 35 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libOpenImageIO_Util.so.3.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libOpenColorIO_v_2_3_2_OIIO.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000e (SONAME)             Library soname: [libOpenImageIO.so.3.1.0]
 0x000000000000000f (RPATH)              Library rpath: [/tmp/tmpmkq88nni/wheel/platlib/OpenImageIO/lib]

Right now I think the dynamic linker will do its job only because OpenImageIO.cpython-313-x86_64-linux-gnu.so (the python extension) has the correct RPATH and RPATH is recursive (as opposed to RUNPATH, because yes there is RPATH and RUNPATH and they bahave slithly differently). See https://man7.org/linux/man-pages/man8/ld.so.8.html:

(3) Using the directories specified in the DT_RUNPATH dynamic
section attribute of the binary if present. Such
directories are searched only to find those objects required
by DT_NEEDED (direct dependencies) entries and do not apply
to those objects' children, which must themselves have their
own DT_RUNPATH entries. This is unlike DT_RPATH, which is
applied to searches for all children in the dependency tree.

(emphasis mine)

Though, the tests in CI seem to say that there is a problem with RPATHs since it can't find OCIO.

Lastly, the wheel filenames are wrong on Linux.

[JeanChristopheMorinPerso]

About the rpath, if I had to take a guess, I would guess that the culprit is

OpenImageIO/src/cmake/compiler.cmake

Lines 657 to 658 in c3c6b1e

	if (NOT CMAKE_INSTALL_RPATH)
	set (CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_FULL_LIBDIR}")

[zachlewis]

Hey @JeanChristopheMorinPerso -- once again, I can't tell you how much I appreciate you having a look!

I apologize, I left things in an inconsistent state over the weekend. Good news is, I do believe I've fixed remaining issues...!

Major changes:

1. Added a NAMELINK_SKIP option to the custom install_targets macro in compiler.cmake, and I'm using that option during scikit-build-core builds for installing libOpenImageIO and libOpenImageIO_Util libraries
2. Changed the default CMAKE_INSTALL_RPATH to $ORIGIN:$ORIGIN/lib
3. Explicitly set INSTALL_RPATH to $ORIGIN/lib or $ORIGIN/../lib for certain targets
4. Re-enabled the default CIBW step. (Fixes bad filenames)

About the rpath, if I had to take a guess, I would guess that the culprit is

OpenImageIO/src/cmake/compiler.cmake

Lines 657 to 658 in c3c6b1e

	if (NOT CMAKE_INSTALL_RPATH)
	set (CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_FULL_LIBDIR}")

You were totally right about this -- now I'm doing this:

if (NOT CMAKE_INSTALL_RPATH)
        if(APPLE)
            set(BASEPOINT @loader_path)
        else()
            set(BASEPOINT $ORIGIN)
        endif()
        set (CMAKE_INSTALL_RPATH ${BASEPOINT} ${BASEPOINT}/${CMAKE_INSTALL_LIBDIR})
endif ()

And now the OIIO libs have a relative RPATH:

[zach@alma OpenImageIO]$ readelf -d lib/libOpenImageIO.so.3.1.0 

Dynamic section at offset 0xc5df10 contains 34 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libOpenImageIO_Util.so.3.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libz.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libOpenColorIO_v_2_3_2_OIIO.so]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000e (SONAME)             Library soname: [libOpenImageIO.so.3.1.0]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN:$ORIGIN/lib]

🤘

I've gotten rid of the docs. For the fonts, actually, what I'd like to do adjust __init __.py and append the relative dir to the env var "OPENIMAGEIO_FONTS". I like the idea of a cross-platform default font shipping with the module...

[JeanChristopheMorinPerso]

This is kind of reverting what was made before. You could pass -DCMAKE_INSTALL_RPATH on a per platform basis instead.

[JeanChristopheMorinPerso]

Thanks @zachlewis. I'm always happy to help!