Support for setuptools.dynamic when using pip_install_from_pyproject

[peterroelants]

I have a local package configured by a pyproject.toml file that looks similar to:

[project]
name = "<name>"
description = "<description>"
version = "0.0.1"
readme = "README.md"
license = {text = "Proprietary/Confidential"}
classifiers = [
    "Programming Language :: Python :: 3",
]
requires-python = ">=3.9"
dynamic = [
  "dependencies",
  "optional-dependencies",
]

[build-system]
requires = ["setuptools>=67", "wheel", "pip"]
build-backend = "setuptools.build_meta"

[tool.setuptools.dynamic]
dependencies = {file = ["env/requirements.txt"]}
optional-dependencies.dev = {file = ["env/dev_requirements.txt"]}
optional-dependencies.strict_versions = {file = ["env/local_requirements.txt"]}

[tool.setuptools.packages.find]
where = ["./src"]

Note that I'm not using a [project.dependencies] section. However my dependencies are defined as dynamic, and I'm using [tool.setuptools.dynamic] to define where to get the dependencies dynamically (from a requirements.txt file)

However, when creating an image and installing this package using pip_install_from_pyproject modal complains that there is No [project.dependencies] section in pyproject.toml file:

❯ modal serve server.py
✓ Initialized. View run at ...
Stopping app - uncaught exception raised locally: ValueError('No [project.dependencies] section in pyproject.toml file. If your pyproject.toml instead declares [tool.poetry.dependencies], use `Image.poetry_install_from_file()`. See https://packaging.python.org/en/latest/guides/writing-pyproject-toml for further file format guidelines.').
╭─ Error ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No [project.dependencies] section in pyproject.toml file. If your pyproject.toml instead declares [tool.poetry.dependencies], use `Image.poetry_install_from_file()`. See          │
│ https://packaging.python.org/en/latest/guides/writing-pyproject-toml for further file format guidelines.                                                                           │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

I think this error originates here.

I'm using:

❯ modal --version
modal client version: 0.72.5

Btw, this local package builds just fine locally using either pip or uv.

[mwaskom]

Could you just point Image.pip_install_from_requirements at env/requirements.txt here?

[peterroelants]

Thanks for the suggestion, in similar spirit as a workaround I copied all the dependencies over to .pip_install.

However, as a feature request, it would be great to be able to use pip_install_from_pyproject on any pyproject.toml file that also works with pip/uv. We are working in a custom monorepo setup where we rely on linking to local packages (each defined by a pyproject.toml.by) by pip installing their respective directories.

[peterroelants]

I think this error originates here.

I was looking at how modal parses the pyproject.toml file and I noticed modal is using the toml library: config = toml.load(os.path.expanduser(pyproject_toml)).

Using setuptools's read_configuration will parse the optional depencencies automatically, e.g.

from setuptools.config.pyprojecttoml import read_configuration
from pathlib import Path
import json

# Parse config
config = read_configuration(filepath=Path("pyproject.toml"))

# Print parsed config
print(f"{json.dumps(config, indent=2)}")

# Print the correct dependencies from my `env/requirements.txt` defined in the pyproject.toml above
print(config["project"]["dependencies"])
# Prints the correct dependencies from the `env/dev_requirements.txt` defined in the pyproject.toml above
print(config["project"]["optional_dependencies"]["dev"])

However, that being said, you might be using a plain toml parser for a reason instead of setuptools?

[peterroelants]

I tried a small draft PR here: #2753 (untested)

Not sure what your contributor guidelines are, and how to test this properly.

[mwaskom]

Thanks for the suggested PR. We'll need to think about it a little bit. In your case, is using setuptools to parse the dependencies only solving the problem because your dynamic dependencies are defined in the tool.setuptools.dynamic section of the toml file?

When dynamic dependencies have come up in the past, it's often been the case that our lack of support for them is not actually the main blocker to things working the way users expect. Rather, when projects specify dynamic dependencies, it's often because they want the dependencies to vary depending on the state of the local system (machine arch, etc.). But with the Modal Image configuration, you would actually want the dynamic dependencies to reflect the state of the remote system. Making that work as expected would would require a much deeper change to the Image specification process.

BTW in your specific case I'm still not understanding why it was necessary to duplicate your requirements in an Image.pip_install command when we do support reading local requirements.txt files.

[peterroelants]

BTW in your specific case I'm still not understanding why it was necessary to duplicate your requirements in an Image.pip_install command when we do support reading local requirements.txt files.

I don't think it matters that much, however I did notice that pip_install_from_requirements only supports passing in a single requirements file and not a list. Using pip_install just allowed me to copy them all together under this command.

[peterroelants]

Thanks for the suggested PR. We'll need to think about it a little bit.

Fair enough. Just treat the PR as a suggestion. I think I can work around it for now.
Although it would be nice to be able to install from my pyproject.toml files without needing to change them due to a Modal constraint.

In your case, is using setuptools to parse the dependencies only solving the problem because your dynamic dependencies are defined in the tool.setuptools.dynamic section of the toml file?

Yes, you currently don't support using tool.setuptools.dynamic. Using setuptools should allow this in theory. I think this should be the only blocker in my use-case.

[mwaskom]

Yes, you currently don't support using tool.setuptools.dynamic. Using setuptools should allow this in theory. I think this should be the only blocker in my use-case.

Sorry to be clear, my question, my question was whether your suggestion will work for people who have pyproject.toml files written to be used with different build backends. I think promising support for "dynamic dependencies" generically would require a more complicated change.

[mwaskom]

In any case, the local / remote distinction is probably a more challenging problem for supporting dynamic dependencies.