Skip to content

Latest commit

 

History

History
282 lines (224 loc) · 19.6 KB

README.md

File metadata and controls

282 lines (224 loc) · 19.6 KB
Raw

hatch-cython
CI/CD Build Testscodecov
Package PyPI - Version PyPI - Downloads PyPI - Python Version
Meta Hatch project Ruff

In comparison to the original hatch-cython, this fork introduces the following options/features:

  • options.include_all_compiled_src (default=True): toggle for including or excluding all source files from the build target after they were precompiled by cython (cythonized).
  • options.files.exclude_compiled_src: excluding selected source files from the build target after they were precompiled by cython (cythonized). This option is applied after options.include_all_compiled_src.
  • options.files.include_compiled_src: including selected source files from the build target after they were precompiled by cython (cythonized). This option is applied after options.files.exclude_compiled_src.
  • options.compiled_extensions_as_artifacts (default=True): Adding compiled extension files (.so, .pyd, .dll) to the artifacts for the wheel build target. To keep backward compatibility, set this option to false.
  • options.intermediate_extensions_as_artifacts (default=False): Adding intermediate extension files (.c, .cpp) to the artifacts for the wheel build target. To keep backward compatibility, set this option to true.
  • Clean works now via absolute paths which fixes the issue with the clean command not working when the current working directory is not the project root.
  • All matches are now based on git-style glob patterns instead of using i mix of regex and glob patterns. This is more consistent with the rest of the hatch configuration.

Table of Contents

Usage

The build hook name is cython.

  • pyproject.toml
# [tool.hatch.build.hooks.cython]
# or
# (hatch.toml)
# [build.targets.wheel.hooks.cython]
# or
[tool.hatch.build.targets.wheel.hooks.cython]
dependencies = ["hatch-cython"]

# [tool.hatch.build.hooks.cython.options]
# or
# (hatch.toml)
# [build.targets.wheel.hooks.cython.options]
# or
[tool.hatch.build.targets.wheel.hooks.cython.options]
# include .h or .cpp directories
includes = []
# include numpy headers
include_numpy = false
include_pyarrow = false

# include_{custom}
include_somelib = {
    # must be included in build-dependencies
    pkg = "somelib",
    # somelib.gets_include() -> str
    include = "gets_include",
    # somelib.gets_libraries() -> list[str]
    libraries = "gets_libraries",
    # somelib.gets_library_dirs() -> list[str]
    library_dirs = "gets_library_dirs",
    # somelib.some_setup_op() before build
    required_call = "some_setup_op"
}

compile_args = [
  # single string
  "-v",
  # by platform
  { platforms = ["linux", "darwin"], arg = "-Wcpp" },
  # by platform & arch
  { platforms = "darwin", arch = "x86_64", arg = "-arch x86_64" },
  { platforms = ["darwin"], arch = "arm64", arg = "-arch arm64" },
  # with pep508 markers
  { platforms = ["darwin"], arch = "x86_64", arg = "-I/usr/local/opt/llvm/include", depends_path = true, marker = "python_version <= '3.10'"  },
]

directives = { boundscheck = false, nonecheck = false, language_level = 3, binding = true }

compile_kwargs = { }
  • hatch.toml
# [build.hooks.cython]
# or
[build.targets.wheel.hooks.cython]
dependencies = ["hatch-cython"]

# [build.hooks.cython.options]
# or
[build.targets.wheel.hooks.cython.options]
directives = { boundscheck = false, nonecheck = false, language_level = 3, binding = true }
compile_args = [
    "-O3",
]
includes = []
include_numpy = false
# equivalent to include_pyarrow = true
include_somelib = { pkg = "pyarrow", include="get_include", libraries="get_libraries", library_dirs="get_library_dirs", required_call="create_library_symlinks" }
define_macros = [
    # ["ABC"] -> ["ABC", "FOO"] | ["ABC", "DEF"]
    ["NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"],
]

Configuration Options

Field Type
src str | None 
 directory within src dir or .  which aliases the package being built. e.g. package_a -> src/package_a_lib 
 src = "package_a"
directives directives to cython (see compiler-directives)
compile_args str or { platforms = ["*"] | "*", arg = str }. see extensions for what args may be relevant
extra_link_args str or { platforms = ["*"] | "*", arg = str }. see extensions for what args may be relevant
env { env = "VAR1", arg = "VALUE", platforms = ["*"], arch = ["*"] }
 if flag is one of:
 - ARFLAGS
 - LDSHARED 
 - LDFLAGS
 - CPPFLAGS 
 - CFLAGS 
 - CCSHARED
the current env vars will be merged with the value (provided platform & arch applies), separated by a space. This can be enabled by adding { env = "MYVAR" ... , merges = true } to the definition.
includes list str
include_{package} { pkg = str, include = str, libraries = str| None, library_dirs = str | None , required_call = str | None } 
where all fields, but pkg, are attributes of pkg in the type of callable() -> list[str] | str | list[str] | str. pkg is a module, or loadable module object, which may be imported through import x.y.z.
include_numpy | include_pyarrow | include_pythran bool
3rd party named imports. must have the respective opt in dependencies
parallel bool = False 
if parallel, add openmp headers
important: if using macos, you need the homebrew llvm vs apple's llvm in order to pass -fopenmp to clang compiler
compiler compiler used at build-time. if msvc (Microsoft Visual Studio), /openmp is used as argument to compile instead of -fopenmp  when parallel = true. default = false
compile_py whether to include .py files when building cython exts. note, this can be enabled & you can do per file / matched file ignores as below. default = true
files dict offering file specific adjustments exclude, targets, aliases.
define_macros list of list str (of len 1 or 2). len 1 == [KEY] == #define KEY FOO . len 2 == [KEY, VALUE] == #define KEY VALUE. see extensions
** kwargs keyword = value pair arguments to pass to the extension module when building. see extensions

Files

[build.targets.wheel.hooks.cython.options.files]
exclude = [
    # anything matching no_compile is ignored by cython
    "*/no_compile/*",
    # note - anything "*" is escaped to "([^\s]*)" (non whitespace).
    # if you need an actual * for python regex, use as below:
    # this excludes all pyd or pytempl extensions
    "([^.]\\*).(pyd$|pytempl$)",
    # only windows
    { matches = "*/windows", platforms = ["linux", "darwin", "freebsd"] },
    # only darwin
    { matches = "*/darwin", platforms = ["linux", "freebsd", "windows"] },
    # only linux
    { matches = "*/linux", platforms = ["darwin", "freebsd", "windows"] },
    # only freebsd
    { matches = "*/freebsd", platforms = ["linux", "darwin", "windows"] }
]
aliases = {"abclib._filewithoutsuffix" = "abclib.importalias"}

Explicit Build Targets

If explicit targets are required (i.e. hatch-cython only builds the files specified), use options.files.targets. Specifying this option will implicly enable compile_py, in addition to checking all c, cpp, and cc files against the specified inclusions.

[build.targets.wheel.hooks.cython.options.files]
targets = [
  # using a single string
  "*/compile.py",
  # or a match clause
  { matches = "*/windows", platforms = ["windows"] },
  { matches = "*/posix", platforms = ["darwin", "freebsd", "linux"] },
]

sdist

Sdist archives may be generated normally. hatch must be defined as the build-system build-backend in pyproject.toml. As such, hatch will automatically install hatch-cython, and perform the specified e.g. platform-specific adjustments to the compile-time arguments. This allows the full build-process to be respected, and generated following specifications of the developer.Note: If hatch-cython is specified to run outside of a wheel-step processes, the extension module is skipped. As such, the .c & .cpp, as well as templated files, may be generated and stored in the sdist should you wish. However, there is currently little purpose to this, as the extension will likely have differed compile arguments.

Templating

Cython tempita is supported for any files suffixed with .in, where the extension output is:

  • .pyx.in
  • .pyd.in
  • .pyi.in For these files, expect the output .pyx.in -> .pyx. Thus, with aliasing this would look like:
[build.targets.wheel.hooks.cython.options.files]
aliases = {"abclib._somemod" = "abclib.somemod"}
    1. Source files somemod.pyi.in, _somemod.pyx.in
    1. Processed templates somemod.pyi, _somemod.pyx
    1. Compiled module abclib.somemod{.pyi,.pyx}

An example of this is included in:

Template Arguments

Per-file matched namespaces are supported for templating. This follows the above platforms, arch, & marker formats, where if supplied & passing the condition the argument is passed to the template as a named series of keyword arguments.

If an index value is provided, and all other kwargs to templates are keywords for each index value. Follows FIFO priority for all keys except global, which is evaluated first and overriden if there are other matching index directives. The engine will attempt to merge the items of the keywords, roughly following:

args = {
    "index": [
        {"keyword": "global", ...},
        {"keyword": "thisenv", ...},
    ],
    "global": {"abc": 1, "other": 2},
    "thisenv": {"other": 3},
}

merge(args) -> {"abc": 1, "other": 3}

In hatch.toml:

[build.targets.wheel.hooks.cython.options.templates]
index = [
  {keyword = "global", matches = "*" },
  {keyword = "templated_mac", matches = "templated.*.in",  platforms = ["darwin"] },
  {keyword = "templated_mac_py38", matches = "templated.*.in",  platforms = ["darwin"], marker = "python == '3.8'" },
  {keyword = "templated_win", matches = "templated.*.in",  platforms = ["windows"] },
  {keyword = "templated_win_x86_64", matches = "templated.*.in",  platforms = ["windows"], arch = ["x86_64"] },

]

# these are passed as arguments for templating

# 'global' is a special directive reserved & overriden by all other matched values
global = { supported = ["int"] }

templated_mac = { supported = ["int", "float"] }
templated_mac_py38 = { supported = ["int", "float"] }

templated_win = { supported = ["int", "float", "complex"] }

# assuming numpy is cimported in the template
templated_win_x86_64 = { supported = ["int", "float", "np.double"]}

Notes

  • MacOS users with brew installed will have brew --prefix libs and include paths added in compilation step. Code parsing is found here

  • Github Runners now run MacOS on m1 platforms. You may have ci issues if you are using MacOS m1 runners and you do not disable macos-max-compat in hatch. e.g.

    # hatch.toml

[build.targets.wheel]
macos-max-compat = false

Development

Requirements

  • a c / c++ compiler
  • python 3.8-<=3.12
  • git-cliff (pip install git-cliff)
  • tasks

Scripts

  • test: library & coverage
    • hatch run cov
  • test: src structure example
    • task example
  • test: simple structure example
    • task simple-structure
  • commit: precommitt
    • task precommit

License

hatch-cython is distributed under the terms of the MIT license.