feat: Python wheels workflow and build backend #4428

The build system has been updated to specifically detect the Python3 Development.Module meta component, as opposed to the entire Development component. This allows for better compatibility with python distributions that do not provide the Development.Embed component, which is only required for projects that ship embedded Python interpreters. The changes have been made in CMakeLists.txt and pythonutils.cmake files. Signed-off-by: Zach Lewis <[email protected]>

@aclark4life

Scikit-build-core is used for collecting CMake and Ninja as needed, and for invoking the build. When invoked via cibuildwheels, `repairwheel` is used after each build to re-bundle and relink the shared library dependencies into properly redistributable whl archives. The command-line tools are exposed under the [project.scripts] section. This commit incorporates or is otherwise inspired by similar efforts by @aclark4life and @JeanChristopheMorinPerso, as well as @remia's work on the OpenColorIO wheels. Signed-off-by: Zach Lewis <[email protected]>

Use Apache-2.0 license identifier instead of BSD-3-Clause. Add "OpenImageIO Contributors" / "[email protected]" as author. I could not bring myself to remove Larry as an author and maintainer. Signed-off-by: Zach Lewis <[email protected]>

…tic libdeflate It's now possible to disable building of shared `libdeflate` libs. Also, we're checking for and aliasing `libdeflate` in `externalpackages.cmake`, just before checking for TIFF, as opposed to only doing so within `build_TIFF.cmake`. This change is necessary for certain build systems and pipelines that utilize cached dependency builds. Specifically, when building wheels for multiple versions of cpython, `cibuildwheel` would complete the first build, and then throw an exception on the *second* build re: not being able to find `Deflate::Deflate`. Moving the aliasing above the check for TIFF ensures that the expected aliasing always takes place, whether or not TIFF needs to be built; whereas before, we were only creating the alias when initially building TIFF. Signed-off-by: Zach Lewis <[email protected]>

Build and link missing libjpeg-turbo shared + static libs Signed-off-by: Zach Lewis <[email protected]>

Instead of creating a separate OpenImageIO.OpenImageIO.command_line module for the CLI shims, move the CLI shim logic up to a "_command_line()" method in OpenImageIO.__init__.py. (Maybe this method should still be called "main()" though?) This also means the module is technically importable from OpenImageIO.OpenImageIO, but that's an improvement over OpenImageIO.OpenImageIO.OpenImageIO...! Signed-off-by: Zach Lewis <[email protected]>

…oftwareFoundation#4358) As suggested by Moritz Moeller Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…twareFoundation#4359) * Get rid of some obsolete cmake code. * Movement (but no change) to some parts of CMakeLists.txt, primarily to make it closer to the corresponding file in OSL to make it easy for me to diff them and port innovations back and forth between them. * Some typo/etc fixes * Remove unused OIIO_UNUSED_OK macro that's been deprecated since 2.0, and OIIO_CONSTEXPR and OIIO_CONSTEXPR14, neither of which have been needed for years. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

This seems to break builds under certain circumstances. Better to handle the problem with cached rebuilds another way, either in a FindLibdeflate.cmake, or by always locally-building libdeflate and TIFF. Signed-off-by: Zach Lewis <[email protected]>

…directory, under module root Added conditional logic to set relative RPATHs when building with scikit-build. This change ensures that the Python module and compiled cli tools correctly find all built dynamic libraries relative to a shared root, and keeps distributions self-contained and relocatable (i.e., without requiring a `repairwheel` step). Signed-off-by: Zach Lewis <[email protected]>

* The build-directory is no longer hard-coded to a local "build_wheels" path. The 'repairwheel' tool needs to know where it can find any dynamic libs compiled by the build system; and, frustratingly, doesn't seem to consider libraries already bundled in the whl. We can either point repairwheel to directory within an unzipped whl; or, we can point repairwheel back to where the compiled dependencies live inside the build-directory, under .../dist/deps/lib. There isn't a straightforward way of passing information from skbuild to repairwheel directly; but here, we're using cibw to set an environment variable dictating to where scikit-build-core builds, which we can also reference in the repair-wheel step. * Always (re)build the TIFF dependency when building local wheels. This is a workaround for an issue where "Deflate::Deflate" either can't be found, or can't be redeclared, under certain circumstances. A more robust solution might be to instead write a FindLibdeflate.cmake module that adds an alias for Deflate::Deflate as needed. * Add "wheelhouse" directory created by cibuildwheel to .gitignore. Signed-off-by: Zach Lewis <[email protected]>

We don't want to locally build TIFF when it already exists; but if we build static TIFF libs locally once, we have to rebuild every time (i.e., for subsequent builds), or else "Deflate::Deflate" is forgotten. This commit forces TIFF to be rebuilt every time, but only for cibuildwheel unix builds. Under normal circumstances, only missing dependencies will be locally built. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Update pyproject.toml configuration The project's TOML file has been updated to reflect changes in the build system, dependencies, and licensing. The scikit-build-core version requirement has been bumped up, and new tools have been added for wheel repair and invocation. The license text has also been simplified to only include Apache-2.0. Signed-off-by: Zach Lewis <[email protected]>

The wheel repair command in the pyproject.toml file has been refactored to use an invoke task. he before-build step now also installs invoke. A new tasks.py file has been added with a 'wheel_repair' task that slims down and repairs the wheel file. Step 1: Remove `lib`, `include`, `share` directories from wheel Step 2: Let `repairwheel` fix the wheel with freshly-built libraries found in {build_dir}/lib and {build_dir}/deps/dist/lib. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Also, tiny bit of tidying. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

It's an OCIO dependency. Signed-off-by: Zach Lewis <[email protected]>

Apparently MSVC is having trouble linking Python3::Python otherwise... Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

- Lowered the required Python version from 3.9 to 3.7 - Added numpy as a new dependency - Refined DLL loading for Windows in Python 3.8+ - Adjusted build verbosity settings - Moved CMAKE_INSTALL_LIBDIR setting into platform-specific overrides for Linux and macOS only - On Windows, do not make adjustments to the INSTALL_RPATH. - Reorganized variables in cibuildwheel configuration for better readability - Refactored variable names in __init__._call_program function to follow PEP8 guidelines Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Stolen nearly line-for-line from OpenColorIO's wheel workflow. Signed-off-by: Zach Lewis <[email protected]>

…cademySoftwareFoundation#4365) Fixes a simple copy/paste error in a copy constructor where the y coordinate gets initialised twice instead of y and z. Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…emySoftwareFoundation#4348) Additional stack manipulation commands: * `--popbottom` discards the bottom-of-stack image * `--stackreverse` reverses the order of the whole stack * `--stackclear` fully empties the stack * `--stackextract <index>` moves the indexed item from the stack (index 0 means the top) to the top. Make `--for` work correctly in both directions: * Correct behavior if `--for` has a negative step value. * If the end value is less than the begin value and no step is supplied, assume -1 (analogous to how we usually assueme step=1 under ordinary circumstances). * Error if step is 0 (presume it will make an infinite loop). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…TORY (AcademySoftwareFoundation#4368) It's not just in oiiotool. This seems clearer and adheres to the env variable naming convention we chose. Reminder: This controls whether command line history gets written to output image metadata by default by oiiotool and maketx. We historically did it, but recently stopped because of security concerns. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

JPEG output configuration hint "jpeg:iptc" (default: 1), if set to 0, will suppress IPTC block output to the file. In the process, we changed the return type of utility function encode_iptc_iim() to return true if anything was successfully encoded, false otherwise. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ion#4347) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…on#4360) The main purpose of this change is to make it possible to use OIIO for reading raw files in rawtoaces instead of calling LibRaw directly. There are the changes: - add the missing hints needed to implement all combinations of white-balancing methods and matrix methods provided by rawtoaces. - add the DNG-specific attributes The change adds this functionality: - new "raw:user_black" hint to override the default black point - new "raw:use_auto_wb" hint to force LibRaw to white balance by averaging over the whole image. - new "raw:grey_box" hint to make LibRaw to white balance by averaging over the given rectange. - new "raw:dng:XXX" attributes added to the output ImageBuf if the input image is a DNG file. The attributes consist of 2 sets of [calibration illuminant; calibration matrix, XYZ to camera RGB matrix]. Note, the current DNG standard supports up to 3 calibration illuminants, but both LibRaw and rawtoaces only use 2 currently. I have manually tested all permutations of white-balancing modes and matrix methods which are currently supported by raw to aces. The images match up to a rounding error. The current unit tests pass, but they only seem to use the default conversion settings. We may want to extend those. I'm not clear on how to do that, there are multiple reference images for different versions of LibRaw, not sure if I will have to re-generate all of them. I intend to make more changes to the raw plugin soon, may come back to updating tests during/after that. There are currently no tests using DNG files, so the new DNG-specific attributes are not covered. The code relying on those in rawtoaces works fine. I have also updated the documentation to add the new hints, however, I haven't been able to build the documentation. Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ce (AcademySoftwareFoundation#4362) This is needed for the related OpenCV-related PR Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Some more updates to the relicensing. At this point, less than 0.15% of the codebase remains under the original BSD-3-clause license. Signed-off-by: Zach Lewis <[email protected]>

@lgritz

… memory tracking (AcademySoftwareFoundation#4322) First PR of two, following @lgritz and I discussions on memory tracking in the OIIO::ImageCache. - Add two template methods and their specializations for types various to help memory tracking: ```c++ // return the total heap allocated memory held by the object and its members template<typename T> inline size_t heapsize(const T& t); // return the total memory footprint including the size of the structure itself template<typename T> inline size_t footprint(const T& t); ``` - Specialized for: ParamValue, ParamValueList, ImageSpec, ImageInput, ImageOutput, std::vector, std::shared_ptr, std::unique_ptr, oiio::intrusive_ptr, ImageCacheImpl related objects. - New files: - `include/memory.h` : adds base and few types specialization of heapsize and footprint. - `libtexture/imagecache_memory_pvt.h` : adds specilializations for ImageCacheImpl and related objects. - `libtexture/imagecache_memory_print.h` : adds a helper function that print a breakdown of the ImageCacheImpl total memory usage as well, as well as per image format. Note: this is slow, but gives accurate memory estimation. Related PR from Larry : AcademySoftwareFoundation#4317 Signed-off-by: Basile Fraboni <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…#4367) Make OCIO a hard dependency, at least version 2.2. Revise and improve the OCIO "auto-build" scripts. Remove OCIO 1.x testsuite reference output. No longer need FindOpenColorIO.cmake module As part of this, I split the CI "oldest/hobbled" test into two different tests: * oldest -- test against the oldest supported versions of all dependencies. * hobbled -- test against most optional dependencies missing and optional features disabled. --------- Signed-off-by: Larry Gritz <[email protected]>

…ySoftwareFoundation#4373) Fix a bug in oiiotool wherein `-i:ch=...` (input only a subset of channels when reading from a file) was not adjusting the spec's alpha_channel and z_channel -- that is, they still contained the OLD channel numbers of those things, which not only could be wrong due to the reordering, but they might be out of range of the new number of channels. Also a related fix in the TIFF reader, to fix a possible dereference of an empty vector. This was being triggered by that out-of-range alpha_channel. Maybe that can't happen anymore with the above bug fix, but it still feels like the safer way to proceed. This was all found by the sanitizers in CI -- and only recently, despite this code having this bug for a long time. So yay for sanitizer CI! Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…cademySoftwareFoundation#4370) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

CI sanitizers in master have been failing lately. A different PR addresses the root cause, but this patch contains some improvements I made along the way while investigating it: * CMake: Change the SANIIZE build variables to use set_cache, thus allowing easier override by env variable. * Bumped the CI sanitizer test to use the 2024 container and latest compilers. * Suppress UB warnings in STL function.h, where it's claiming undefined behavior in the std::function guts called by ArgParse internals. Only seen failing on my Mac, wasn't actually failing in the GHA CI. I'm pretty sure it's a false positive. * Also, only on my Mac building against OpenVDB 11, it found some UB in OpenVDB, which I'm suppressing (I think it's also false positive). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…#4380) This version dates from early 2021 (so still supporting back 3.5 years). The reason for the bump is to simplify a bit on our part and avoid older versions known to be broken or lacking certain features. This is for master/2.6/3.0 only, will not be backported to 2.5, since we never raise minimum dependencies in already-released branches. --------- Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…4381) Microsoft's runtime library doesn't support the standard setenv() function. _putenv_s() should be used instead. Without fix iv do not compiling: `E:\GH\OpenImageIO\src\iv\ivmain.cpp(119,9): error C3861: 'setenv': identifier not found [E:\GH\OpenImageIO\build\src\iv\iv.vcxproj]` --------- Signed-off-by: Vlad (Kuzmin) Erium <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ation#4363) Fully linking OpenCV against libOpenImageIO is burdensome for people downstream, especially those who want a static version of libOpenImageIO. There were only three IBA functions that use OpenCV: to_OpenCV(), from_OpenCV(), and capture_image(), and all three are fairly short. Move them to a separate header, imagebufalgo_opencv.h, make them fully INLINE (so they don't actually produce object code in libOpenImageIO, which therefore does not need to link against OpenCV). Applications wanting these three functions should include this header and will be responsible themselves for ensuring that the include paths and linkage of their application makes provisions for finding and using OpenCV. All other applications don't need to deal with OpenCV dependency at all. Our own oiiotool (for --capture) and our python bindings still link against OpenCV to support this functionality (again, still as an optional dependency, enabled only if OpenCV is found at build time). But downstream apps using libOpenImageIO and who do not themselves need our OpenCV functioality now no longer need to link against OpenCV. We have removed capture_image() from the Python bindings -- Python scripts that need to capture live camera images can use OpenCV or any other capture API of their choice without going through OIIO. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…Foundation#4377) Change IC::create() and TS::create() to return shared_ptr instead of a raw pointer. This cleans up a lot of lingering lifetime management issue of these classes. The switch to 3.0 is an opportunity to make breaking changes to these APIs. For the sake of downstream users, we define preprocessor symbols OIIO_IMAGECACHE_CREATE_SHARED and OIIO_TEXTURESYSTEM_CREATE_SHARED to signal that the new APIs are present. Some very minor changes may be needed at the call site. --------- Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…cademySoftwareFoundation#4383) Nominally, fix four places where "it's" was used when "its" was appropriate. Realistically, force the EasyCLA dialog so that I can get an individual CLA in place, my employment with ARRI having terminated on 15 July of this year. Signed-off-by: Joseph Goldstone <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…#4382) It was disabled to make prettier formatting for getargs, but neglected to turn it back on for the rest of the file. I also changed a few old `<<` stream output to `print()`. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…wareFoundation#4379) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…es. (AcademySoftwareFoundation#4376) Don't read or store full metadata on every MIP level for every face. That can really add up! Ptex can only have one set of metadata, the format itself doesn't allow it to vary from face to face. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…reFoundation#4386) In trying to avoid doing the color conversion pointlessly for pixels with alpha=0, I accidentally also managed to skip the multiplication by alpha for those pixels. Oops. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…wareFoundation#4366) Added IBA::demosaic(). Currently supports 4 Bayer pattern layouts, "RGGB", "GRBG", "GBRG", "BGGR", and 2 algorithms: "linear", "MHC". Should be expandable in the future. --------- Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…undation#4394) Correctly detect new versions of FFmpeg (6.1 and 7.0). The FFmpeg versions page was used for reference: https://www.ffmpeg.org/download.html#releases Signed-off-by: Darby Johnston <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

AcademySoftwareFoundation#4399) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

It was documented and declared, but implementation was missing in action. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

) This change adds additional FFmpeg metadata from the video and data streams. These streams can hold the start timecode which is very useful in OpenTimelineIO applications. I also added `av_guess_frame_rate()` which might be more robust than just checking `stream->avg_frame_rate`, but I can also create a separate PR if it is better to break it up. It doesn't look like there are any FFmpeg tests in `testsuite`? I tested with the DPEL ALab trailer and some Netflix Open Content movies, and was able to see the timecode metadata in the OIIO image attributes. --------- Signed-off-by: Darby Johnston <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…cademySoftwareFoundation#4391) Because IBA::transpose changes resolution, it doesn't pass the input image to IBAPrep(), and there was a special case there that always set destination to float if there were no source images. So change transpose to specifically request a format, to be the same as the input image. And that required a tweak to IBAPrep allowing us to specify such a thing (there was already an option that meant "force it to float", but not the general case of an arbitrary type). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

* Added as_bytes and as_writable_bytes that convert spans of arbitrary type to spans of std::byte. (This mimics C++20 utilities of the same name.) * spancopy, spanset, spanzero: add parameter defaults for the common case of wanting to do the operation on the whole span, also simplify the bound-limiting logic in those functions a bit (but still performing precisely the same task). Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ion#4397) Make the 'display window' match the default crop, which is applied to in-camera JPEG. --------- Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

@lgritz

…utput (AcademySoftwareFoundation#4323) Second PR of two, following @lgritz and I discussions on memory tracking in the OIIO::ImageCache. - the memory tracking system from AcademySoftwareFoundation#4322 is not sufficient to track OIIO public objects that can be overriden. - add virtual `heapsize()` method to ImageInput and ImageOutput that return the total heap allocated memory held by the structure and its members. - [ ] **TODO**: override for every internal OIIO type (bmp, tiff, etc). Related PR from Larry : AcademySoftwareFoundation#4317 First PR: AcademySoftwareFoundation#4322 Signed-off-by: Basile Fraboni <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Background: The ImageCache and TextureSystem had historically been implemented as pure virtual abstract base classe defining an interface, then a hidden concrete subclass. The flexibility this was intended to provide was never really used; instead the imagined main client for the flexibility, OSL, has a different mechanism for customizing the texture system (via their RendererServices class). Meanwhile, this has made the interfaces to IC and TS impossible to change without breaking the ABI. So this PR does the following: * ImageCache and TextureSytem are fullly de-virtualized, and are now concrete classes, no hidden subclassing, no virtual methods. As such, removing or changing existing methods will be an ABI break, but adding new methods is not. * Each now use a PIMPL idiom, where all the data and internal methods are hidden from the public interface and do not affect the ABI. * Roughly speaking, we added the PIMPL pointers and made the ImageCacheImpl/TextureSystemImpl -- what used to be the hidden concrete subclasses -- into the hidden PIMPL classes. * Moved a lot of classes that used to be in a "pvt" namespace into the main namespace. If they are opaque types not exposed in public headers or as symbols exported from the library, it doesn't matter, so this just removes some arbitrary clutter. * Had to rearrange some of the recently added heapsize/footprint code, moved some inline functions in the headers into methods of the classes, defined in the cpp files. This itself is a huge ABI change, so will only be merged into master, to become part of the 3.0 release. This is not an API change, though! This is all about the internals, and should not require any client software to change a single line of code. I may do further simplification or refactoring of this code in the future, but that will all be smaller and have no further API/ABI changes that break compatibility. Although the structure of the class hierarchy has changed around, none of the logic about how IC and TS actually *work* has changed, so there should be no change in observable behavior. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…dation#4393) - ColorProcCacheKey: Update definition, signature, and usage to support caching NamedTransform processor handles. (Note: **ABI-breaking**) - ColorConfig: add NamedTransform convenience functions: `getNumNamedTransforms`, `getNamedTransformNameByIndex`, `getNamedTransformNames`, `getNamedTransformAliases` - ColorConfig: add `createNamedTransform` function returning new ColorProcessorHandle - Add `ImageBufAlgo::ocionamedtransform` functions - oiiotool: add `--ocionamedtransform` operator - python: add NamedTransform convenience function bindings - python: add `ImageBufAlgo.ocionamedtransform(...)` bindings - Update `oiiotool` and `python` documentation with examples ## Description OCIO-2 config authors may define a set of stand-alone "source-agnostic" `NamedTransforms`. Unlike the `colorconvert`, `ociodisplay`, and `ociolook` IBAs, which explicitly convert _from_ a color space to something else, the `ocionamedtransform` IBA behaves more like the `ociofiletransform` IBA, where the transform itself can be applied either in the forwards or inverse direction, and does not take into consideration the input encoding or image state. Quoting from the OCIO [NamedTransform documentation](https://opencolorio.readthedocs.io/en/latest/guides/authoring/authoring.html#named-transforms): > Sometimes it is helpful to include one or more transforms in a config that are essentially stand-alone transforms that do not have a fixed relationship to a reference space or a process space. An example would be a “utility curve” transform where the intent is to simply apply a LUT1D without any conversion to a reference space. In these cases, a named_transforms section may be added to the config with one or more named transforms. [...] This feature may be used to emulate older methods of color management that ignored the RGB primaries and simply applied one-dimensional transformations. Notably, the built-in OCIO-2 Studio config (`ocio://studio-config-latest`) contains NamedTransforms "sRGB - Curve" and "Rec709 - Curve", which we could implement as drop-in replacements for the existing `linear_to_sRGB`, `sRGB_to_linear`, `linear_to_Rec709`, and `Rec709_to_linear` functions in `color.h` (unless those functions are deprecated...). There are several other applications for NamedTransforms as well. They can be extremely useful as workflow-oriented transforms implemented at various points in facility pipelines (e.g., for applying shot-specific conversions, grades, gamut-compression, etc.); they can be used to provide diagnostic transforms for visualizing "exposure zones", clipped areas, NaNs, etc; for visually "tagging" problematic clips; for applying parts of ACES Metadata Files parsed by OCIO's forthcoming AMF parser; for conversion to non-RGB color models... all sorts of things. ## Tests Tests are forthcoming. I've tested everything locally, but have not yet implemented proper tests in the testsuite. --------- Signed-off-by: Zach Lewis <[email protected]>

…tion#4400) When fmt arguments don't match the format string, fmt will throw an exception, or terminate if we disable exceptions (which we do). For gcc11+, we tried intercepting these, but let's do it for all platforms, and let's not terminate ourselves, but insted print and log the error. But, oof, to do this properly, I needed to move some error recording functionality from libOpenImageIO to libOpenImageIO_Util and make it owned more properly by strutil.cpp, so that this all works even when only using the util library. The logic isn't changing, it's just moving over to the other library. This all helps to address AcademySoftwareFoundation#4388 Unfortunately, the exception thrown by fmt doesn't tell us the bad format string itself. That would have really allowed to probably zero right in on it. But at least we know it's occurring, and one could put a breakpoint on pvt::log_fmt_error to catch it in the act and see where it's being called, revealing the bad line. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

* Bump version to reflect recent ABI changes * Bring CHANGES and CREDITS up to date Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

The link points to the Apache 2.0 license, we just never changed the label here. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…#4402) span_util.h is relatively new. I'm honestly no longer sure why I thought it was necessary to make a second file, but so little is in it that I now believe its contents should be folded into span.h to keep all the span-related definitions in once place. It's just not big enough to need to be divided into several headers. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…ation#4401) The flavor of ImageBuf that wraps caller-owned memory is constructed or reset by merely passing a raw pointer and it assumes that the range of memory implied by the strides is entirely safe to read from or write to. And even if correct, there is not much within ImageBuf internals to prevent bugs from buffer overruns. This PR introduces a new constructor and reset() method to ImageBuf where instead of taking a raw pointer, it takes a span describing the extent of the safely addressible memory (and also an optional "pointer to first pixel" within the buffer, which is necessary if you're using negative strides or for some other reason are describing safe memory that starts before the location of pixel 0). That span is stored in the ImageBuf, and also it remembers if it was a cspan (which should indicate a read-only IB) or a non-const span (which should wrap writable memory). At this moment, I'm just adding this API and will start using and documenting it as the perferred method, with the raw pointer versions being regarded as unsafe for now (and maybe some day will be deprecated?). But we're not yet using the span internally for safety checks. In the future, my intent is to gradually add safeguards to do bounds checking against the span for IB operations that touch the wrapped memory. Because we'll most likely bounds check only in debug mode, it shouldn't ever impact performance, but it will make debugging, sanitizer, and fuzz tests more robust to catch and zero in on buffer overruns. I also like how it forces the caller, at the call site, to think hard about exactly what range of memory the IB is wrapping, and not merely pass a single pointer and hope it works out. I also changed all ImageBuf ctrs and resets within the OIIO codebase to use the new span-based versions. This is part of an overall trend to migrate from raw pointer based API calls to span-based API design as much as possible. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…latest (AcademySoftwareFoundation#4410) Also some other minor simplifications. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

This allows somebody who is enhancing the docs to check the results entirely through CI without having to set up the dependencies necessary for docs on their end. (Perfect for Dev Days.) It also ensures that any changes we make to the code base don't break the documentation somehow, because now a successful docs build will be part of the CI. Note that the docs building job leaves behind an artifact, which will be a zip file of the directly containing the HTML docs, just like one would find on RTD, so you can download that zip file and view the docs it made to verify that everything looks right. I made the docs builder a separate workflow, but I also changed some things in the CI workflow for the sake of making them more similar. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Set MACOSX_DEPLOYMENT_TARGET = 10.15 so builds stop screaming about C++17’s std::filesystem API features not being available Signed-off-by: Zach Lewis <[email protected]>

Had previously commented out, to see if it was aversely affecting the windows builds... but I think it's actually causing the module not to be packaged in the wheel at all! Signed-off-by: Zach Lewis <[email protected]>

… architectures. Also add clarifying comments to pyproject.toml Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

I'd like to see how much larger the build artifacts are, and test to see if windows has any less trouble. Signed-off-by: Zach Lewis <[email protected]>

…SoftwareFoundation#4039) Convert C++ and Python examples from the "Image transformations and data movement" section of the "imagebufalgo" chapter into tests within the "docs-examples" testsuites (related to issue AcademySoftwareFoundation#3992). Update 'oiiotool' tabs to use a simple 'tab' instead of 'code-tab' to prevent synchronized tab selection, which is inconsistent with the other tabs using the 'literalinclude' directive. --------- Signed-off-by: Jeremy Retailleau <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…#4418) openexr_copy and openexr-decreasingy These recently added test were writing into the ref/ directory, which they certainly should not do. Write into the build test area, compare to what's in the ref area. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

More attempts to fix windows wheel builds... Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Also removed superfluous settings. Signed-off-by: Zach Lewis <[email protected]>

* Limit to commits pushed to AcademySoftwareFoundation/OpenImageIO, tagged as v2.6.* or v3.* * Uncomment "upload_pypi" step. (NB: requires setting a secret named `PYPI_API_TOKEN`) Signed-off-by: Zach Lewis <[email protected]>

Add a new test command in the pyproject.toml cibw options. This command runs after wheels are built with `cibuildwheel`, and checks if CLI Python scripts can import the OpenImageIO module, invoke CLI binary executables, and load all shared libraries. It uses 'oiiotool --help' to output a list of dependencies with which OIIO was compiled. If this works, everything else should check out too (pending CI workflow results)... Signed-off-by: Zach Lewis <[email protected]>

No PYPI_API_TOKEN secret needed -- see https://docs.pypi.org/trusted-publishers/using-a-publisher/ Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: zachlewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Apparently the macos-14 github runners ship with PNG-1.4, which is too old; the build system notes this, but tries to link the system PNG regardless. This recipe will hopefully inspire the runner to build and use PNG-1.6+ Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Trying to get to the bottom of why OIIO is continuing to link system shared system PNG (CI images) instead of freshly-built static libs. Presumably, if I build shared PNG libs, OIIO will prefer the newer ones over the system libraries. (Which I guess would mean any found dynamic libraries are preferred over any found static libraries...) Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

*This* seems to be the key we were looking for... Also, it looks like libpng 1.6.44 (released less than a week ago) generates a proper CMake config, so prefer that if it's available.... Signed-off-by: Zach Lewis <[email protected]>

Turns out the macos-12 runners are deprecated. Also, the publish step should only occur when pushing a tag that starts with v2.6 or v3. Signed-off-by: Zach Lewis <[email protected]>

I'm not sure if this actually affects anything, since OCIO is a required dependency of OIIO; but if OCIO doesn't need to be built, then neither does yaml-cpp. Signed-off-by: Zach Lewis <[email protected]>

- Add support for Python 3.13 - Update minimum required Python version from 3.7 to 3.8 (Need to further investigate CIBW troubles with python 3.7) - Upgrade scikit-build-core dependency to >=0.10.6,<1 (up from 0.10) - Add oldest-supported-numpy as a new build dependency. I don't think numpy is actually needed at build time, but this seems to help certain builds not have to compile numpy from source. - Skip 32-bit builds in cibuildwheel configuration - Enable free-threaded support for Python 3.13 - Adjust manylinux and musllinux images versions in cibuildwheel configuration for local testing -- these are currently overridden by environment variables set in the Wheels github workflow. Specifically, use the manylinux2014 (i.e., CentOS-7) image for python 3.7 and 3.8; and use manylinux_2_28 (Rocky) for the rest. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

it was screwing everything up. Signed-off-by: Zach Lewis <[email protected]>

...to see if this fixes a problem with building on Windows. Signed-off-by: Zach Lewis <[email protected]>

The aarch64 wheels are currently failing , and they take forever to build. Moving the aarch64 wheels builds to the MacOS M1 runners will hopefully fix both problems at once, provided these mac runners have a docker engine installed... Signed-off-by: Zach Lewis <[email protected]>

Create a separate job just for building aarch64 wheels. Another attempt to get the wheels cross-compiled on the apple silicon runners. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

We have separate PRs in for dealing with PNG and yaml-cpp deps. Removing the recipes from this PR to simplify things, and to check my previous assertion that are needed for the wheels github workflow. Signed-off-by: Zach Lewis <[email protected]>

* Create a dedicated job for linux-arm builds. They take substantially longer than the x86_64 builds, because the aarch64 architecture is being emulated. Ideally, we want to build these wheels on ARM runners, which should be widely available in a few months. * Disable tests for Linux ARM wheels. Apparently, the tests are failing, I think due to trouble CIBW has with testing ARM wheels from x86_64 runners. I've checked the builds myself on my local machine (Rocky and Alma aarch64 docker containers running on a M1 Mac), and they seem totally fine. We'll have to re-enable when we switch to ARM runners. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: zachlewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Until we can figure out a better solution, having homebrew install libpng will work around the issue with Wheels checks failing due to trying to link the too-old version of PNG installed to the macos-14 gh runners Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

This should fix failing cp38-macos-arm64 builds. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

This reverts commit 25d774c. Signed-off-by: Zach Lewis <[email protected]>

Previously, the python-colorconfig tests used the "ocio://default" OCIO config, which varies across minor versions of OCIO. This caused our CI "Bleeding Edge" checks to fail whenever the OpenColorIO project updated the default builtin config in their "main" branch. This commit forces the tests to use the `cg-config-v1.0.0_aces-v1.3_ocio-v2.1` builtin config that ships with all versions of OCIO-2.2+. Signed-off-by: Zach Lewis <[email protected]>

See https://github.com/scipy/oldest-supported-numpy?tab=readme-ov-file#deprecation-notice-for-numpy-20. Co-authored-by: Jean-Christophe Morin <[email protected]> Signed-off-by: zachlewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

…O vers" This reverts commit e45ca15. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Is numpy even needed at build-time, or does pybind11 provide everything needed for the build? Signed-off-by: Zach Lewis <[email protected]>

Apparently, there are no numpy-2 bdists for python3.8 available on pypi... also: Revert "testing: temporarily disable numpy build requirement" This reverts commit 5f4caf5. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Trying to prevent x265 (GPL) dynamic libraries from getting included. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

…oundation#4459) This fixes OIIO's CI "bleeding edge" test that broke when OCIO pushed to their main the new config files that will be in OCIO 2.4. (Yay!) Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

On the macos runners, the static freetype library is linked against a x86_64 PNG lib. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

It's an old flavor not well supported by homebrew any longer, so what's happening is that most of the dependency packages we ask to install have to build from source, which means it takes a long long time. Just get rid of it, we have two other Mac test cases -- a newer MacOS 13 Intel one, and a MacOS 14 ARM one. That's enough coverage. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

…or it Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

It's an old flavor not well supported by homebrew any longer, so what's happening is that some of the dependency packages we ask to install have to build from source, which means it takes a long long time. By disabling OpenCV and Qt homebrew installation for it, we can make it fast again. Neither the minimal OpenCV interop, nor osltoy (the only thing that needs Qt) are exercised by the automated testsuite anyway, so I don't think there's any loss of testing functionality in practice. Signed-off-by: Larry Gritz <[email protected]>

OpenEXR 3.3 was just released and its main branch has bumped its advertised version to 4.0 (the latter breaking our "bleeding edge" CI test). Raise our "latest releases" test to use 3.3. Our "bleeding edge" test picking up the now-called-4.0 OpenEXR main branch required some minor logic changes to not reject the version as too new to be compatible. Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Instead of defining the brew uninstall command at the pyproject.toml config level, define using an env var during the workflow run. This makes things more predictable when testing cibuildwheel locally Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

OCIO recently made the decision to disable the "Load DLLs from $PATH" hack for Python > 3.7 that they had copied from OIIO. We should follow suit -- this behavior was removed from Python for a reason (it was a security vulnerability). This PR warns users that this behavior is deprecated. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Larry Gritz <[email protected]>

Signed-off-by: zachlewis <[email protected]>

…n_cibuildwheels Signed-off-by: Zach Lewis <[email protected]>

…wheels Signed-off-by: Zach Lewis <[email protected]>

- Remove "numpy" dependency -- technically, numpy is only needed if using `ImageBuf.get_data()` (I believe). Previously, we had set this to require numpy-2; but that caused the Python 3.8 wheels not to build. - Remove MacOS architecture CMake defines -- since we're using both x86_64 and ARM64 runners to build the ARM and intel wheels separately, we shouldn't have to worry about the ARM64 python modules trying to link x86_64 dependencies. - Test with "oiiotool --buildinfo" instead of "oiiotool --help" Signed-off-by: Zach Lewis <[email protected]>

Turns out, /opt/homebrew/lib/libfreetype.a links a libPNG that wasn't built for arm64. Signed-off-by: Zach Lewis <[email protected]>

Now that numpy-2 isn't a requirement, we can re-enable Python 3.8 wheel builds. Also, I've added top-level permissions to the Workflow, and pinned the Action dependencies by hash to pass / improve the advanced security checks / scorecard Signed-off-by: Zach Lewis <[email protected]>

Co-authored-by: Jean-Christophe Morin <[email protected]> Signed-off-by: zachlewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

When `-DIGNORE_HOMEBREWED_DEPS=ON` is provided, ignore libraries that live under the following paths: - /opt/homebrew - /usr/local - /usr/X11 - /usr/X11R6 Also, builds invoked with `build_dependency_with_cmake` will now inherit CMAKE_IGNORE_PATH and CMAKE_IGNORE_PREFIX_PATH values set in a higher-level scope. This prevents dependencies from finding sub-dependencies where it shouldn't. For example, if we're ignoring homebrewed dependencies, even though brew-installed PNG and brew-installed freetype may be ignored when first trying to find the dependencies, we need to make sure that when we build freetype locally, we take care _not_ to link brew-installed PNG. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

During a pip-based installation, install as few copies of OpenImageIO and OpenImageIO_Util as possible. Also, don't make versioned copies of OCIO Signed-off-by: Zach Lewis <[email protected]>

For downstream packages using scikit-build-core, append the OpenImageIOTarget.cmake directory to CMAKE_PREFIX_PATH, so that `find_package(OpenImageIO)` works as expected, fingers crossed. (should behave akin to setting -DOpenImageIO_DIR) Signed-off-by: Zach Lewis <[email protected]>

e.g., installed by XQuartz. Signed-off-by: Zach Lewis <[email protected]>

Not only do we remove the custom repairwheel step *entirely*, we want to make sure the default repairwheel step does not run at all. For some reason, `repairwheel`, `autditwheel`, etc. have trouble recognizing that the wheel first created by the build system is fine as-is, and doesn't need repairing. In a best-case scenario, the repair-wheel step ends up creating and packaging copies of OpenImageIO_Util and OpenColorIO. We may need to reintroduce a custom repair wheel step if we cannot insulate the build from other dependencies on runner images picked up by OIIO. Signed-off-by: Zach Lewis <[email protected]>

I've dedicated an entire "remove_prefixes_from_variable" cmake function to omitting prefixes from environment and cmake variables to help me debug and fine-tune dependency building. I believe this would be useful for tricky build recipes; but I think it would be better to pare this down for the immediate future, pending CI checks... Signed-off-by: Zach Lewis <[email protected]>

This reverts commit 34d7e3a. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Optionally disable creation of multiple versioned copies / symlinks of installed libraries. In other words, create libOpenImageIO.so.3.1.0; but not libOpenImageIO.so or libOpenImageIO.so.3.1. Signed-off-by: Zach Lewis <[email protected]>

Signed-off-by: Zach Lewis <[email protected]>

Assuming the wheel bundles fonts under <platlib>/OpenImageIO/share/fonts, setting the OpenImageIO_ROOT env var should help OIIO automatically find the fonts. As a bonus, pip-installed CLI tools will be aware of OpenImageIO_ROOT. Signed-off-by: Zach Lewis <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Python wheels workflow and build backend #4428

feat: Python wheels workflow and build backend #4428

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 23, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Sep 30, 2024

Commits on Oct 1, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 11, 2024

Commits on Oct 14, 2024

Commits on Oct 15, 2024

Commits on Oct 16, 2024

Commits on Oct 17, 2024

Commits on Oct 21, 2024

Commits on Oct 22, 2024

feat: Python wheels workflow and build backend #4428

Are you sure you want to change the base?

feat: Python wheels workflow and build backend #4428

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 18, 2024

Commits on Sep 19, 2024

Commits on Sep 23, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Sep 30, 2024

Commits on Oct 1, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 11, 2024

Commits on Oct 14, 2024

Commits on Oct 15, 2024

Commits on Oct 16, 2024

Commits on Oct 17, 2024

Commits on Oct 21, 2024

Commits on Oct 22, 2024