Possible to work in conjunction with local external and unpackaged dependencies?

[matthewfeickert]

(Apologies as that title isn't the greatest, and I should make it clear that I'm rather certain that the answer to this question is "no, because that goes against the whole idea of how reproducible dependency management works" and in my mind that would be a great answer. 👍 I'm asking here to more have a formal reference to point to internally in the future, and not that I'm suggesting the dev team spend time on this.)

Some background

I work as part of the ATLAS experiment, where most of our experiment specific tooling exists in collections of C++ releases. Without getting into the specifics, most of these releases aren't distributed with normal package management tools, but instead are distributed via the read only CernVM File System. We also distribute them in Linux container images publicly like gitlab-registry.cern.ch/atlas/athena/analysisbase:25.2.2. These releases provide specific compiler versions and things like CPython.

$ docker run --rm -ti gitlab-registry.cern.ch/atlas/athena/analysisbase:25.2.2
             _ _____ _      _   ___
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

[bash][atlas]:workdir > . /release_setup.sh
Configured GCC from: /opt/lcg/gcc/13.1.0-b3d18/x86_64-el9/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt
[bash][atlas AnalysisBase-25.2.2]:workdir > command -v gcc
/opt/lcg/gcc/13.1.0-b3d18/x86_64-el9/bin/gcc
[bash][atlas AnalysisBase-25.2.2]:workdir > command -v python
/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/bin/python
[bash][atlas AnalysisBase-25.2.2]:workdir > python --version --version
Python 3.9.12 (main, Mar  7 2024, 02:22:30)
[GCC 13.1.0]
[bash][atlas AnalysisBase-25.2.2]:workdir >

So the main point is that these are frameworks that are not going to be available on conda-forge.

Question

It is clear to me that trying to naively use pixi with these releases to extend the analysis release Python environments is going to fail as the analysis releases are built around having things tied together (and so are going to require that the Python in the release is the Python at runtime)

[bash][atlas AnalysisBase-25.2.2]:workdir > curl -fsSL https://pixi.sh/install.sh | bash
This script will automatically download and install Pixi (latest) for you.
Getting it from this url: https://github.com/prefix-dev/pixi/releases/latest/download/pixi-x86_64-unknown-linux-musl.tar.gz
The binary will be installed into '/home/atlas/.pixi/bin'
######################################################################## 100.0%
Updating '/home/atlas/.bash_profile'
Please restart or source your shell.
[bash][atlas AnalysisBase-25.2.2]:workdir > export PATH=$PATH:/home/atlas/.pixi/bin
[bash][atlas AnalysisBase-25.2.2]:workdir > eval "$(pixi completion --shell bash)"
[bash][atlas AnalysisBase-25.2.2]:workdir > mkdir example && cd example
[bash][atlas AnalysisBase-25.2.2]:example > pixi init .
✔ Initialized project in /workdir/example/.
[bash][atlas AnalysisBase-25.2.2]:example > pixi add awkward
  × Querying Python at `/workdir/example/.pixi/envs/default/bin/python3.12` failed with status exit status: 1 with exit status: 1
  │ --- stdout:
  │
  │ --- stderr:
  │ Python path configuration:
  │   PYTHONHOME = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt'
  │   PYTHONPATH = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/python:/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib:/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-
  │ el9-gcc13-opt/lib:/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/python:/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/site-packages'
  │   program name = '/workdir/example/.pixi/envs/default/bin/python3.12'
  │   isolated = 0
  │   environment = 1
  │   user site = 1
  │   safe_path = 0
  │   import site = 1
  │   is in build tree = 0
  │   stdlib dir = '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.12'
  │   sys._base_executable = '/workdir/example/.pixi/envs/default/bin/python3.12'
  │   sys.base_prefix = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt'
  │   sys.base_exec_prefix = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt'
  │   sys.platlibdir = 'lib'
  │   sys.executable = '/workdir/example/.pixi/envs/default/bin/python3.12'
  │   sys.prefix = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt'
  │   sys.exec_prefix = '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt'
  │   sys.path = [
  │     '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/python',
  │     '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/python',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/site-packages',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python312.zip',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.12',
  │     '/usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.12/lib-dynload',
  │   ]
  │ Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
  │ Python runtime state: core initialized
  │ ModuleNotFoundError: No module named 'encodings'
  │
  │ Current thread 0x00007f44f0d54740 (most recent call first):
  │   <no Python frame>
  │ ---

[bash][atlas AnalysisBase-25.2.2]:example > python -c 'import encodings; print(encodings)'
<module 'encodings' from '/usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt/../../../../AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/encodings/__init__.py'>
[bash][atlas AnalysisBase-25.2.2]:example >

So is there any feasible way hack pixi to define a "local" unpacked dependency, like the CPython provided, that would be used instead of the CPython versions on conda-forge?

Way that I get around this sort of thing now without pixi:

I have to hack back the virtual environment activate scripts to have normal virtual environment behaviors (https://github.com/matthewfeickert/cvmfs-venv).

$ docker run --rm -ti gitlab-registry.cern.ch/atlas/athena/analysisbase:25.2.2
             _ _____ _      _   ___
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

[bash][atlas]:workdir > . /release_setup.sh
Configured GCC from: /opt/lcg/gcc/13.1.0-b3d18/x86_64-el9/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt
[bash][atlas AnalysisBase-25.2.2]:workdir > mkdir -p ~/.local/bin
[bash][atlas AnalysisBase-25.2.2]:workdir > export PATH=~/.local/bin:"${PATH}"
[bash][atlas AnalysisBase-25.2.2]:workdir > curl -sL https://raw.githubusercontent.com/matthewfeickert/cvmfs-venv/main/cvmfs-venv.sh -o ~/.local/bin/cvmfs-venv
[bash][atlas AnalysisBase-25.2.2]:workdir > chmod +x ~/.local/bin/cvmfs-venv
[bash][atlas AnalysisBase-25.2.2]:workdir > cvmfs-venv .venv
WARNING: /release_setup.sh exists and it is assumed you are in a Linux container.
         Setting up environment with '. /release_setup.sh'.

. /release_setup.sh
Configured GCC from: /opt/lcg/gcc/13.1.0-b3d18/x86_64-el9/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/25.2.2/InstallArea/x86_64-el9-gcc13-opt
# Creating new Python virtual environment '.venv'
[bash][atlas AnalysisBase-25.2.2]:workdir > . .venv/bin/activate
(.venv) [bash][atlas AnalysisBase-25.2.2]:workdir > python -m pip install --upgrade awkward
Collecting awkward
  Downloading awkward-2.6.2-py3-none-any.whl.metadata (7.0 kB)
Collecting awkward-cpp==30 (from awkward)
  Downloading awkward_cpp-30-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)
Collecting fsspec>=2022.11.0 (from awkward)
  Downloading fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)
Collecting importlib-metadata>=4.13.0 (from awkward)
  Downloading importlib_metadata-7.1.0-py3-none-any.whl.metadata (4.7 kB)
Requirement already satisfied: numpy>=1.18.0 in /usr/AnalysisBaseExternals/25.2.2/InstallArea/x86_64-el9-gcc13-opt/lib/python3.9/site-packages (from awkward) (1.23.5)
Collecting packaging (from awkward)
  Downloading packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting typing-extensions>=4.1.0 (from awkward)
  Downloading typing_extensions-4.10.0-py3-none-any.whl.metadata (3.0 kB)
Collecting zipp>=0.5 (from importlib-metadata>=4.13.0->awkward)
  Downloading zipp-3.18.1-py3-none-any.whl.metadata (3.5 kB)
Downloading awkward-2.6.2-py3-none-any.whl (774 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 774.7/774.7 kB 5.6 MB/s eta 0:00:00
Downloading awkward_cpp-30-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (706 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 706.2/706.2 kB 50.8 MB/s eta 0:00:00
Downloading fsspec-2024.3.1-py3-none-any.whl (171 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 172.0/172.0 kB 18.1 MB/s eta 0:00:00
Downloading importlib_metadata-7.1.0-py3-none-any.whl (24 kB)
Downloading typing_extensions-4.10.0-py3-none-any.whl (33 kB)
Downloading packaging-24.0-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 6.1 MB/s eta 0:00:00
Downloading zipp-3.18.1-py3-none-any.whl (8.2 kB)
Installing collected packages: zipp, typing-extensions, packaging, fsspec, awkward-cpp, importlib-metadata, awkward
Successfully installed awkward-2.6.2 awkward-cpp-30 fsspec-2024.3.1 importlib-metadata-7.1.0 packaging-24.0 typing-extensions-4.10.0 zipp-3.18.1
(.venv) [bash][atlas AnalysisBase-25.2.2]:workdir > python -m pip show awkward
Name: awkward
Version: 2.6.2
Summary: Manipulate JSON-like data with NumPy-like idioms.
Home-page:
Author:
Author-email: Jim Pivarski <[email protected]>
License: BSD-3-Clause
Location: /workdir/.venv/lib/python3.9/site-packages
Requires: awkward-cpp, fsspec, importlib-metadata, numpy, packaging, typing-extensions
Required-by:
(.venv) [bash][atlas AnalysisBase-25.2.2]:workdir >