-
Notifications
You must be signed in to change notification settings - Fork 595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Python wheels workflow and build backend #4428
base: main
Are you sure you want to change the base?
feat: Python wheels workflow and build backend #4428
Conversation
The build system has been updated to specifically detect the Python3 Development.Module meta component, as opposed to the entire Development component. This allows for better compatibility with python distributions that do not provide the Development.Embed component, which is only required for projects that ship embedded Python interpreters. The changes have been made in CMakeLists.txt and pythonutils.cmake files. Signed-off-by: Zach Lewis <[email protected]>
Scikit-build-core is used for collecting CMake and Ninja as needed, and for invoking the build. When invoked via cibuildwheels, `repairwheel` is used after each build to re-bundle and relink the shared library dependencies into properly redistributable whl archives. The command-line tools are exposed under the [project.scripts] section. This commit incorporates or is otherwise inspired by similar efforts by @aclark4life and @JeanChristopheMorinPerso, as well as @remia's work on the OpenColorIO wheels. Signed-off-by: Zach Lewis <[email protected]>
Use Apache-2.0 license identifier instead of BSD-3-Clause. Add "OpenImageIO Contributors" / "[email protected]" as author. I could not bring myself to remove Larry as an author and maintainer. Signed-off-by: Zach Lewis <[email protected]>
…tic libdeflate It's now possible to disable building of shared `libdeflate` libs. Also, we're checking for and aliasing `libdeflate` in `externalpackages.cmake`, just before checking for TIFF, as opposed to only doing so within `build_TIFF.cmake`. This change is necessary for certain build systems and pipelines that utilize cached dependency builds. Specifically, when building wheels for multiple versions of cpython, `cibuildwheel` would complete the first build, and then throw an exception on the *second* build re: not being able to find `Deflate::Deflate`. Moving the aliasing above the check for TIFF ensures that the expected aliasing always takes place, whether or not TIFF needs to be built; whereas before, we were only creating the alias when initially building TIFF. Signed-off-by: Zach Lewis <[email protected]>
Build and link missing libjpeg-turbo shared + static libs Signed-off-by: Zach Lewis <[email protected]>
Instead of creating a separate OpenImageIO.OpenImageIO.command_line module for the CLI shims, move the CLI shim logic up to a "_command_line()" method in OpenImageIO.__init__.py. (Maybe this method should still be called "main()" though?) This also means the module is technically importable from OpenImageIO.OpenImageIO, but that's an improvement over OpenImageIO.OpenImageIO.OpenImageIO...! Signed-off-by: Zach Lewis <[email protected]>
…oftwareFoundation#4358) As suggested by Moritz Moeller Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
…twareFoundation#4359) * Get rid of some obsolete cmake code. * Movement (but no change) to some parts of CMakeLists.txt, primarily to make it closer to the corresponding file in OSL to make it easy for me to diff them and port innovations back and forth between them. * Some typo/etc fixes * Remove unused OIIO_UNUSED_OK macro that's been deprecated since 2.0, and OIIO_CONSTEXPR and OIIO_CONSTEXPR14, neither of which have been needed for years. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
This seems to break builds under certain circumstances. Better to handle the problem with cached rebuilds another way, either in a FindLibdeflate.cmake, or by always locally-building libdeflate and TIFF. Signed-off-by: Zach Lewis <[email protected]>
…directory, under module root Added conditional logic to set relative RPATHs when building with scikit-build. This change ensures that the Python module and compiled cli tools correctly find all built dynamic libraries relative to a shared root, and keeps distributions self-contained and relocatable (i.e., without requiring a `repairwheel` step). Signed-off-by: Zach Lewis <[email protected]>
* The build-directory is no longer hard-coded to a local "build_wheels" path. The 'repairwheel' tool needs to know where it can find any dynamic libs compiled by the build system; and, frustratingly, doesn't seem to consider libraries already bundled in the whl. We can either point repairwheel to directory within an unzipped whl; or, we can point repairwheel back to where the compiled dependencies live inside the build-directory, under .../dist/deps/lib. There isn't a straightforward way of passing information from skbuild to repairwheel directly; but here, we're using cibw to set an environment variable dictating to where scikit-build-core builds, which we can also reference in the repair-wheel step. * Always (re)build the TIFF dependency when building local wheels. This is a workaround for an issue where "Deflate::Deflate" either can't be found, or can't be redeclared, under certain circumstances. A more robust solution might be to instead write a FindLibdeflate.cmake module that adds an alias for Deflate::Deflate as needed. * Add "wheelhouse" directory created by cibuildwheel to .gitignore. Signed-off-by: Zach Lewis <[email protected]>
We don't want to locally build TIFF when it already exists; but if we build static TIFF libs locally once, we have to rebuild every time (i.e., for subsequent builds), or else "Deflate::Deflate" is forgotten. This commit forces TIFF to be rebuilt every time, but only for cibuildwheel unix builds. Under normal circumstances, only missing dependencies will be locally built. Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Update pyproject.toml configuration The project's TOML file has been updated to reflect changes in the build system, dependencies, and licensing. The scikit-build-core version requirement has been bumped up, and new tools have been added for wheel repair and invocation. The license text has also been simplified to only include Apache-2.0. Signed-off-by: Zach Lewis <[email protected]>
The wheel repair command in the pyproject.toml file has been refactored to use an invoke task. he before-build step now also installs invoke. A new tasks.py file has been added with a 'wheel_repair' task that slims down and repairs the wheel file. Step 1: Remove `lib`, `include`, `share` directories from wheel Step 2: Let `repairwheel` fix the wheel with freshly-built libraries found in {build_dir}/lib and {build_dir}/deps/dist/lib. Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Also, tiny bit of tidying. Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
It's an OCIO dependency. Signed-off-by: Zach Lewis <[email protected]>
Apparently MSVC is having trouble linking Python3::Python otherwise... Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
- Lowered the required Python version from 3.9 to 3.7 - Added numpy as a new dependency - Refined DLL loading for Windows in Python 3.8+ - Adjusted build verbosity settings - Moved CMAKE_INSTALL_LIBDIR setting into platform-specific overrides for Linux and macOS only - On Windows, do not make adjustments to the INSTALL_RPATH. - Reorganized variables in cibuildwheel configuration for better readability - Refactored variable names in __init__._call_program function to follow PEP8 guidelines Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Stolen nearly line-for-line from OpenColorIO's wheel workflow. Signed-off-by: Zach Lewis <[email protected]>
…cademySoftwareFoundation#4365) Fixes a simple copy/paste error in a copy constructor where the y coordinate gets initialised twice instead of y and z. Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
…emySoftwareFoundation#4348) Additional stack manipulation commands: * `--popbottom` discards the bottom-of-stack image * `--stackreverse` reverses the order of the whole stack * `--stackclear` fully empties the stack * `--stackextract <index>` moves the indexed item from the stack (index 0 means the top) to the top. Make `--for` work correctly in both directions: * Correct behavior if `--for` has a negative step value. * If the end value is less than the begin value and no step is supplied, assume -1 (analogous to how we usually assueme step=1 under ordinary circumstances). * Error if step is 0 (presume it will make an infinite loop). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
…TORY (AcademySoftwareFoundation#4368) It's not just in oiiotool. This seems clearer and adheres to the env variable naming convention we chose. Reminder: This controls whether command line history gets written to output image metadata by default by oiiotool and maketx. We historically did it, but recently stopped because of security concerns. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
JPEG output configuration hint "jpeg:iptc" (default: 1), if set to 0, will suppress IPTC block output to the file. In the process, we changed the return type of utility function encode_iptc_iim() to return true if anything was successfully encoded, false otherwise. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
…ion#4347) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>
Co-authored-by: Jean-Christophe Morin <[email protected]> Signed-off-by: zachlewis <[email protected]>
27b81c0
to
9783b7f
Compare
Signed-off-by: Zach Lewis <[email protected]>
4929455
to
7b57e29
Compare
When `-DIGNORE_HOMEBREWED_DEPS=ON` is provided, ignore libraries that live under the following paths: - /opt/homebrew - /usr/local - /usr/X11 - /usr/X11R6 Also, builds invoked with `build_dependency_with_cmake` will now inherit CMAKE_IGNORE_PATH and CMAKE_IGNORE_PREFIX_PATH values set in a higher-level scope. This prevents dependencies from finding sub-dependencies where it shouldn't. For example, if we're ignoring homebrewed dependencies, even though brew-installed PNG and brew-installed freetype may be ignored when first trying to find the dependencies, we need to make sure that when we build freetype locally, we take care _not_ to link brew-installed PNG. Signed-off-by: Zach Lewis <[email protected]>
7b57e29
to
deeba08
Compare
Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
During a pip-based installation, install as few copies of OpenImageIO and OpenImageIO_Util as possible. Also, don't make versioned copies of OCIO Signed-off-by: Zach Lewis <[email protected]>
ee6e5f6
to
34d7e3a
Compare
For downstream packages using scikit-build-core, append the OpenImageIOTarget.cmake directory to CMAKE_PREFIX_PATH, so that `find_package(OpenImageIO)` works as expected, fingers crossed. (should behave akin to setting -DOpenImageIO_DIR) Signed-off-by: Zach Lewis <[email protected]>
e.g., installed by XQuartz. Signed-off-by: Zach Lewis <[email protected]>
Not only do we remove the custom repairwheel step *entirely*, we want to make sure the default repairwheel step does not run at all. For some reason, `repairwheel`, `autditwheel`, etc. have trouble recognizing that the wheel first created by the build system is fine as-is, and doesn't need repairing. In a best-case scenario, the repair-wheel step ends up creating and packaging copies of OpenImageIO_Util and OpenColorIO. We may need to reintroduce a custom repair wheel step if we cannot insulate the build from other dependencies on runner images picked up by OIIO. Signed-off-by: Zach Lewis <[email protected]>
tasks.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well played 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ha -- we'll see... what I've been trying has apparently broken everything. I'm not sure if I have enough time to debug this myself. We may have to split off the attempt to get rid of tasks.py into its own PR. Right now, it's solving a few problems at once.
I've dedicated an entire "remove_prefixes_from_variable" cmake function to omitting prefixes from environment and cmake variables to help me debug and fine-tune dependency building. I believe this would be useful for tricky build recipes; but I think it would be better to pare this down for the immediate future, pending CI checks... Signed-off-by: Zach Lewis <[email protected]>
This reverts commit 34d7e3a. Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Great work with the latest changes @zachlewis! I like how the extract step to fix the wheels are not needed anymore. I started to review the generated artifacts. I see that the versioned libraries are not yet fully removed:
I also see
Are these really needed? The OIIO libs have an absolute RPATH:
Right now I think the dynamic linker will do its job only because
(emphasis mine) Though, the tests in CI seem to say that there is a problem with RPATHs since it can't find OCIO. Lastly, the wheel filenames are wrong on Linux. |
About the rpath, if I had to take a guess, I would guess that the culprit is OpenImageIO/src/cmake/compiler.cmake Lines 657 to 658 in c3c6b1e
|
Optionally disable creation of multiple versioned copies / symlinks of installed libraries. In other words, create libOpenImageIO.so.3.1.0; but not libOpenImageIO.so or libOpenImageIO.so.3.1. Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
Hey @JeanChristopheMorinPerso -- once again, I can't tell you how much I appreciate you having a look! I apologize, I left things in an inconsistent state over the weekend. Good news is, I do believe I've fixed remaining issues...! Major changes:
You were totally right about this -- now I'm doing this: if (NOT CMAKE_INSTALL_RPATH)
if(APPLE)
set(BASEPOINT @loader_path)
else()
set(BASEPOINT $ORIGIN)
endif()
set (CMAKE_INSTALL_RPATH ${BASEPOINT} ${BASEPOINT}/${CMAKE_INSTALL_LIBDIR})
endif () And now the OIIO libs have a relative RPATH:
🤘 I've gotten rid of the docs. For the fonts, actually, what I'd like to do adjust __init __.py and append the relative dir to the env var "OPENIMAGEIO_FONTS". I like the idea of a cross-platform default font shipping with the module... |
@@ -655,7 +655,12 @@ if (CMAKE_SKIP_RPATH) | |||
unset (CMAKE_INSTALL_RPATH) | |||
else () | |||
if (NOT CMAKE_INSTALL_RPATH) | |||
set (CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_FULL_LIBDIR}") | |||
if(APPLE) | |||
set(BASEPOINT @loader_path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is kind of reverting what was made before. You could pass -DCMAKE_INSTALL_RPATH
on a per platform basis instead.
Thanks @zachlewis. I'm always happy to help! |
Assuming the wheel bundles fonts under <platlib>/OpenImageIO/share/fonts, setting the OpenImageIO_ROOT env var should help OIIO automatically find the fonts. As a bonus, pip-installed CLI tools will be aware of OpenImageIO_ROOT. Signed-off-by: Zach Lewis <[email protected]>
Description
Summary
This PR is the spiritual successor to #4011. It implements a
scikit-build-core
-based python build-backend, making it possible to usepip install .
to build from source; and it adds a Github workflow for building withcibuildwheel
and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.12, across major operating systems and architectures.When you
pip install OpenImageIO
, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.PEP-Compliant Packaging:
pyproject.toml
The
pyproject.toml
file is organized in three parts:pip install ...
interacts withcmake
.Additions to
__ init __.py
Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).
This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.
It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in
pyproject.toml
,each item under
[project.scripts]
is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.Relocatable Binary Distributions with
cibuildwheel
+repairwheel
cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.
Importantly,
cibuildwheel
allows for an arbitrary "repair-wheel-command" step, intended as a means for collecting and bundling any external shared libraries linked by the module (or the CLI tools) living outside the package, and subsequently patching RPATH entries to point to relative paths to these libraries inside the wheel. This ensures that compiled extensions have no runtime dependencies outside the pip toolchain.In order to keep wheel file sizes to an absolute minimum, our repair-wheel-command is a multi-step process, implemented in a new
tasks.py
file living in the repo's root, invoked with invoke:Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size.
Note: the steps taken in
tasks.py
remove everything except that which is absolutely necessary for end users to import the Python modules and run the CLI tools, partly in order to determine a baseline minimum file size for the wheels. However, part of the motivation for setting things up the way I have is to make it easier to add things back into the wheels in the future; e.g., should we decide to package the headers and/or the fonts, generate or modify pkgconfig .pcs and/or CMake modules and configs, etc. Also worth noting, there's pyproject.toml metadata we can add to help downstream scikit-build-core-based builds find OpenImageIO."Wheels" Github workflow
I straight-up copied
.github/workflows/wheel.yml
from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v2.6* or v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64, min deployment 10.15) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.orgNote: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.
Note: THE PUBLISH TASK WILL FAIL until a "trusted publisher" is set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/
I've submitted separate PRs for these, since they fall outside the intended scope of this PR.
Note: One of the aims for this PR is to leverage OIIO's new dependency-self-building mechanisms as much as possible, so that users who
pip install
from source are guaranteed a nominally working installation. However, we can also use platform-specific package managers for the wheels uploaded to pypi.org (e.g., added as cibuildwheel "before-build" commands for each platform).Other Changes
I made some minor adjustments to
pythonutils.cmake
andfancy_add_executable.cmake
that only affect scikit-build-core-based installs -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.Tests
The
cibuildwheel
"test-command" isoiiotool --help
. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able toimport OpenImageIO
; that the actual binary executableoiiotool
is properly packaged and exists in the expected location (e.g., at.../site-packages/OpenImageIO/bin
); and that all runtime dependencies are found.Inspiration, Credit, Prior Art