Support pyproject.toml as an alternative to pixi.toml

[baszalmstra]

In python world it would be far from ideal to need both a pyproject.toml and a pixi.toml. Ideally pixi would also be able to read a pyproject.toml file and use that as a project file.

Some care needs to be taken in the distinction between conda and pypi packages. Since the pyproject.toml originally specifies pypi packages I think we would need to add another section for the conda packages.

[tool.pixi.dependencies]
some_conda_package = "*"

[baszalmstra]

Related to

• Add the ability to install pip dependencies #29

[abkfenris]

How about putting all Python dependencies in project.dependencies and then having a Pixi mapping that allows overriding specific dependencies with their conda sourced counterparts?

That would help a project that could be installed via PyPI and via Pixi, and allow specifying dependcies that should be installed via pip by not mapping a conda source.

# Python only dependencies
[project]
dependencies = [
  "xarray=2023.05.01",
  "requests",  # since it isn't mapped below, Pixi should try to install it from pip
]

# Up to the user to provide if installing via PyPI
[tool.pixi.dependencies]
netcdf4 = "*"

# Python dependencies that should preferably be installed via conda packages
[tool.pixi.mapped_dependencies]
xarray = "xarray=2023.05.01"

[baszalmstra]

I like that a lot!

[dhirschfeld]

having a Pixi mapping that allows overriding specific dependencies with their conda sourced counterparts?

That would be useful where the packages have different names or there are constraints specified in the pip format instead of what pixi expects.

You could theoretically automate that by parsing and converting the pip constraints and using a pypi --> conda-forge mapping but easier, and just as effective, would be to allow the pip dependencies to be manually overridden as outlined above 🚀

[majidaldo]

how about conda/pixi just serve as a 'bootstraper' (to instally python, R, julia, npm, js) where each language can have its own way of dealing with libraries?

i used to want conda to handle all the packaging but i learned that creates constant friction.

[baszalmstra]

@majidaldo we firmly believe that the conda ecosystem provides a package management solution beyond just language-specific packages. It makes it trivial to install packages for different languages and provides packages for system dependencies. Conda is an excellent replacement for docker in a lot of ways. We believe conda packages are the way forward.

Nonetheless, pixi can already easily be used as a boostrapper. Using a pixi project like this would bootstrap a python/poetry environment:

[project]
name = "python"
version = "0.1.0"
description = "Add a short description here"
authors = ["Bas Zalmstra <[email protected]>"]
channels = ["conda-forge"]
platforms = ["win-64"]

[tasks]
poetry = "poetry"

[dependencies]
poetry = "~=1.5.1"

You can do a similar thing for Rust, npm, etc.

However, we are currently investigating ways to use pypi packages with pixi as well, stay tuned!

[shughes-uk]

Installing poetry into the environment it manages is generally sadness inducing. It tends to do things like update it's own dependencies (which causes it crash mid update). We moved away from using mamba to manage poetry for this and use brew instead, just using mamba for the python binary + some other bits.

[zen-xu]

@baszalmstra
But installing poetry can pollute Python's environment dependencies. Is it possible to isolate such dev-dependencies?

For example, installing dev-dependencies in .pixi/dev-env?

[raceychan]

on updates on this idea?

[vsoch]

I'd like to +1 some kind of support (or conversion even) from pyproject.toml. I have a protect with a pyproject.toml, with the expectation to use poetry:

[tool.poetry]
name = "snakemake-executor-plugin-googlebatch"
version = "0.1.0"
description = ""
authors = [
    "Vanessa Sochat <[email protected]>",
    "Johannes Koester <[email protected]>"
]
readme = "README.md"
license = "MIT"
repository = "https://github.com/snakemake/snakemake-executor-plugin-googlebatch"
documentation = "https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/googlebatch.html"
keywords = ["snakemake", "plugin", "executor", "cloud", "google-batch"]

[tool.poetry.dependencies]
python = "^3.11"
google-cloud-batch = "^0.17.1"
requests = "^2.31.0"
google-api-core = "^2.12.0"
google-cloud-storage = "^2.12.0"
snakemake-interface-common = "^1.14.0"
snakemake-interface-executor-plugins = "^8.1.1"
jinja2 = "^3.1.2"
google-cloud-logging = "^3.8.0"

[tool.poetry.group.dev.dependencies]
black = "^23.9.1"
flake8 = "^6.1.0"
coverage = "^7.3.1"
pytest = "^7.4.2"
snakemake = {git = "https://github.com/snakemake/snakemake.git", branch="main"}
snakemake-storage-plugin-s3 = "^0.2.7"

[tool.coverage.run]
omit = [".*", "*/site-packages/*", "Snakefile"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

For one reason or another, it tries to resolve dependencies forever on one of my machines. I'm trying out replacing it with pixi, and (in practice) I'm doing a lot of:

$ pixi add --pypi google-cloud-logging
 WARN pixi::project: ALPHA feature enabled!

I'm not sure conversion needs to be a core feature, because there will be oodles of different formats (and likely impossible to support them all). But Perhaps a small library that is even separate, or a web interface to handle it?

Also a suggestion for the new docs - maybe default to dark mode with yellow to match the current site? The white/yellow is a bit jarring for someone like myself that lives in a bat cave. 🦇

If you like the mkdocs look but want sphinx, I like https://github.com/jbms/sphinx-immaterial. It comes with light/dark modes out of the box (with the choice to choose a default) and the yellow / marigold on dark looks pretty nice :)

And final question - is the expected workflow for developing a python package to do:

• pixi add pip # add pip to your environment
• pixi shell # shell in
• pip install -e . # local development install

Or is there a pixi add command that says "install this local path in development mode"?

OK I think that's all the questions I have - thanks muchly!

[ruben-arts]

Story update

We're still working hard on the pypi support, which the focus goes towards especially when you want to support pyproject.toml you don't want to use other tools like pip and poetry as they are all going to read from the same file.

We're also going through some major refactors of the pixi code in the way we interact with the toml file so supporting two formats as of right now will slow down progress.

That said this feature will most likely be one we're picking up after the refactors that are planned.

Continue giving ideas to this thread so we can start on this with a good understanding of the users needs! 💯

[daviskirk]

My main use case here as simple as it sounds is to not have a bunch of different files flying around our repositories.

Especially as pixi (and for that matter conda) are specialized tools that don't show up in python tutorials, installation examples and so on so python devs are used to seeing a pyproject.toml and looking into it to figure out what is going on.

I know that pixi is trying to also be a tool for non-python packages but I would assume that there are a huge amount of python devs that might be a pool for you to gain traction and more early adopters.

Managing a separate file might seem like a small thing but I actually think it might be quite important for adoption to convince devs to try out the tool.

[paugier]

Personally, I have nothing against having a pixi.toml file next to the standard pyproject.toml file. I even quite like it to have some specific Pixi configurations in pixi.toml because pyproject.toml files tend to be a bit too long if we include the configuration of all tools.

However, I don't think it is good to have to maintain two versions of the version string in these 2 files (same for name and description but this is less a problem since they don't change too much).

I understood (#638 (comment)) that Pixi developers believe in "one source of truth", meaning in practice one file.

For Python projects, if we'd like only one file, it has to be pyproject.toml and not pixi.toml.

Python has a Python Packaging Authority, many related recent PEPs and a nice dynamics on packaging related to what is done by PyPA (PDM is an interesting example and IMHO Pixi should also take inspiration from this project). It's counter productive not to follow the results of this standardization effort when you can.

If you (Pixi devs) think "only one file" is important:

• Pixi should support pyproject.toml (Support pyproject.toml as an alternative to pixi.toml #79)
• pixi init should not create pixi.toml for projects using pyproject.toml
• pixi.toml has to be deprecated for projects using pyproject.toml

[jdumas]

I believe you already know about Rye, but I think it's worth re-reading about Rye's philosophy and this discussion about whether Rye should exist or not.

Personally I'd love to have a single tool support both conda package and PyPi packages that is fast and supports building Python packages using the standard pyproject.toml with minimal tool-specific additions to it. Ideally the same pyproject.toml should be usable by PDM/Hatch/Rye/Pixi to build and upload to PyPi, and by Pixi to also build & upload to conda-forge.

[majidaldo]

An easier, more defined, and more stable path is for pixi to process the lock files of each programming language's "standard" way of creating them.

Let each programming language do its thing. pixi can then be some "supervisor" on top of that.

[dhirschfeld]

I'd like to use pixi but I also want to have my package installable by pip and to be able to publish functional python wheels. I'd like to be able to support that without having to specify dependencies and constraints twice in the pyproject.toml file.

To interop smoothly with pip it might be nice to have a package which could parse the pixi dependencies and dynamically specify them for pip in the same way that hatch-requirements-txt can parse requirements.txt files to dynamically specify dependencies.

[paugier]

I'd like to be able to support that without having to specify dependencies and constraints twice in the pyproject.toml file.

Pixi dependencies are about conda packages, which are really not the same than the PyPI packages written in the list pyproject.toml [project] dependencies.

Of course, for a lot of simple cases, there is a direct correspondence between a Python wheel on PyPI and a conda package. But there are also more complicated cases, for example https://pypi.org/search/?q=pyqt5-qt5 does not exists in conda-forge.

IMHO, if you want to publish functional python wheels, you need another specialized tool (PDM, Hatch, Poetry, ...) working with PyPI packages and standard Python virtual envs.

We now have two different worlds for Python packaging:

• Official: PyPI, sdist and wheels, PyPA and the PEPs about packaging (see https://packaging.python.org)
• conda / conda-forge with Conda packages

Both are nice and have their pros and cons. For maintainers of scientific Python packages it's now good to support both frameworks.

I think Pixi could become great for the conda part and should focus on this aspect. Therefore, I'm not sure Pixi should try to also work with standard Python venvs based only on PyPI packages.

If you like Pixi, I guess you like lock files. Then you might want to try PDM for the PyPA aspect.

Some data can be shared between these 2 frameworks (in particular [project], name, description, version). Other data should be kept separated, as for example the runtime, build, optional and dev dependencies.

Of course, it would be nice to also have a simple way to specify that the conda/pixi dependencies can just be taken to be the PyPI dependencies ([build-system] requires, [project] dependencies and groups in [project.optional-dependencies]), which could work for simple cases, but it is a special case.

[dhirschfeld]

Of course, for a lot of simple cases, there is a direct correspondence between a Python wheel on PyPI and a conda package. But there are also more complicated cases,

Yeah, I think I agree that it might be more trouble than it's worth 😔

You'd need pypi-specific deps, conda-specific deps, and even where there are common dependencies you'd still need to translate the package names. At that point, it's probably just easiest to maintain separate sets of dependencies.

At least having them in the same pyproject.toml file would help to keep them in-sync as it would be less likely to forget to update one set.

[maximlt]

+1 on supporting pyproject.toml in Pixi!

In the HoloViz ecosystem we are Python library authors and also heavy conda users. A long, long, long time ago (before my time) the group of HoloViz maintainers built https://github.com/holoviz-dev/pyctdev, a tooling library meant to support developing a library using either conda or pip through a unified interface. While this tool has been useful to some extent, it also has failed in many ways and in practice is now only used on our CI (holoviz-dev/pyctdev#104). We're also not super fond of maintaining a Python packaging/tooling library :D (please don't use pyctdev!!!).

While pyctdev hasn't been wildly successful, I believe there's still a lot of value in making it easier to develop a Python library using conda as the env+package manager, while still allowing contributors to only use pip if they prefer. Some of the advantages given by conda are that it makes it easy to control which Python version to use and it allows installing non-Python dependencies (some of our projects require nodejs for instance).

Some of the things I wish I could declare in a pyproject.toml and that would be understood/picked up by a tool like Pixi:

• optional deps/extras
• a mapping (or using the one from conda-forge) of pip -> conda names
• a list of packages to install from conda only (e.g. nodejs)
• a list of packages to install from pip only (e.g. one of the many Python packages available on PyPI but not on anaconda.org)
• the channels to install the conda packages from
• and basically all the arguments I may want to pass conda/mamba/pixi when it runs conda create

I would need to be able to specify the Python version I want the environment to contain. I would also need Pixi to either know how to install the library or let me declare how to do it (it sounds simple but I'm sure this is a complex topic as there are very complex packages to build out there, the ones I deal with are pretty simple).

That's a pretty long wish list I know and I'm sorry! I've been looking at the conda ecosystem for a while, waiting for the replacement of pyctdev, I don't think it's yet there but have some hope Pixi might be a good candidate! Other projects I plan to explore a little more and from which some inspiration could be drawn: https://github.com/OldGrumpyViking/hatch-conda, https://github.com/macro128/pdm-conda and https://github.com/basnijholt/unidep.

[majidaldo]

+1 on supporting pyproject.toml in Pixi!

In the HoloViz ecosystem we are Python library authors and also heavy conda users. A long, long, long time ago (before my time) the group of HoloViz maintainers built https://github.com/holoviz-dev/pyctdev, a tooling library meant to support developing a library using either conda or pip through a unified interface. While this tool has been useful to some extent, it also has failed in many ways and in practice is now only used on our CI (holoviz-dev/pyctdev#104). We're also not super fond of maintaining a Python packaging/tooling library :D (please don't use pyctdev!!!).

While pyctdev hasn't been wildly successful, I believe there's still a lot of value in making it easier to develop a Python library using conda as the env+package manager, while still allowing contributors to only use pip if they prefer. Some of the advantages given by conda are that it makes it easy to control which Python version to use and it allows installing non-Python dependencies (some of our projects require nodejs for instance).

Some of the things I wish I could declare in a pyproject.toml and that would be understood/picked up by a tool like Pixi:

• optional deps/extras
• a mapping (or using the one from conda-forge) of pip -> conda names
• a list of packages to install from conda only (e.g. nodejs)
• a list of packages to install from pip only (e.g. one of the many Python packages available on PyPI but not on anaconda.org)
• the channels to install the conda packages from
• and basically all the arguments I may want to pass conda/mamba/pixi when it runs conda create

I would need to be able to specify the Python version I want the environment to contain. I would also need Pixi to either know how to install the library or let me declare how to do it (it sounds simple but I'm sure this is a complex topic as there are very complex packages to build out there, the ones I deal with are pretty simple).

That's a pretty long wish list I know and I'm sorry! I've been looking at the conda ecosystem for a while, waiting for the replacement of pyctdev, I don't think it's yet there but have some hope Pixi might be a good candidate! Other projects I plan to explore a little more and from which some inspiration could be drawn: https://github.com/OldGrumpyViking/hatch-conda, https://github.com/macro128/pdm-conda and https://github.com/basnijholt/unidep.

Check out what I did here https://github.com/pnnl/hydraconda

[ruben-arts]

As this is more of a discussion then an issue, I moved it here. Hope this provides a better interface!

[matterhorn103]

I would also love it if pixi used pyproject.toml!

In terms of actual implementation though, discussions are ongoing as to how pip should deal with pyproject.toml, and they are in turn waiting for PEP 735 which wants to standardize specification of dependencies for projects that aren't packages or destined to be built/become wheels. In general the feeling is that project.dependencies probably shouldn't be used as a generic list of dependencies, but instead the proposed table [dependency-groups] is better suited to it. So perhaps better to wait to see what they settle on.

Interoperability between pixi and conda would be nice too, so it would be good if both used the same table to store conda dependencies. That is something that the conda team seem to be considering, see here and here. :)

[olivier-lacroix]

Hi there!

I have started to implement pyproject.toml support in #999