Make even more use of the package cache

[ruben-arts]

Statement

Pixi solves one extremely important issue: it doesn't break other projects when installing the requirements for one project.

Problem

TL;DR: The disk usage grows rapidly if the packages don't share the exact same versions, thus not making use of the cache

Just getting my Raspberry Pi in noticed that with low storage capabilities, running multiple projects can become a disk filling process. This has some what to do with having multiple versions of similar packages on a machine.

Using the below mentioned script I found that some of the more important libraries on my (1TB) dev-system have multiple duplicates. This is exactly what I want to ensure my projects keep working but it's not the best experience on my 32GB RPI.

qt-main: 3
torchvision: 3
aws-sdk-cpp: 3
pytorch: 3
libarrow: 5
pandas: 5
scipy: 5
numpy: 5
gstreamer: 5
aiohttp: 6
matplotlib-base: 6
pydantic-core: 6
pytest: 6
libblas: 6
matplotlib: 6
libclang13: 7
python: 14

Simple Python script to find package duplicates

Run this script in the ~/.cache/rattler/cache/pkgs dir to find the conda package duplicates

import os
import re
from collections import Counter

def count_prefixes(directory):
    # Get a list of all entries in the directory
    entries = os.listdir(directory)
    prefix_count = Counter()

    # Regular expression to match the prefix before '-<number>'
    prefix_pattern = re.compile(r"^(.*?)-\d")

    # Iterate over each entry in the directory
    for entry in entries:
        if os.path.isdir(os.path.join(directory, entry)):
            match = prefix_pattern.search(entry)
            if match:
                prefix_count[match.group(1)] += 1

    return prefix_count

current_directory = os.getcwd()

prefixes = count_prefixes(current_directory)

# Sort the results by count and print
for prefix, count in sorted(prefixes.items(), key=lambda item: item[1]):
    print(f"{prefix}: {count}")

Idea

What if we could tell the solver, with an extra option, which packages we already have locally and it should pick from this set if it can "comfortably" (what ever that means). This in contrast with the "get latest version" logic that it uses right now.

This is particularly useful for pixi global install as there you don't care about a previously locked set but you just want to quickly install packages, most-likely growing the disk usage as little as possible.

> pixi global install startship --prefer-cache

For projects we could allow the same logic, but with like a request if you set a certain config variable.

> pixi config set prefer-cache-over-gb 1

> pixi install 
Found the setting `prefer-cache-over-gb` = `1`. While this project will require you to download `3.8 GB` of data.
Do you want me to re-solve your project to figure out if you already have enough in the cache to build the project.
> yes
  no

.... pixi goes brrr

Found a solution that only requires you to download `0.2 GB`.
You want me to continue?
> yes - use new resolution
  no - install directly from pixi.lock

Simple example would be python, if a project supports >=3.8, <3.12 you don't need to download 3.11 if you already have 3.10.

Alternatives

• For a company they could solve this issue with creating a "distribution" of packages from which pixi can choose, so it only downloads and solves with a limited set of packages. Automatically solving the ever growing set of possible packages.
• Having really good support for figuring out where packages are linked to and delete unused cache with pixi clean cache --unused. This would limit the ever growing cache issue.

Acknowledgement

This potentially creates a strange experience when you simply want the latest version of everything, but misunderstood the configuration options.

For projects, this might reintroduce reproducibility issues, that we avoid with the "locked" approach. That is why this should always be a strong opt-in and interactive feature in my opinion.

⚠️ This is a personal un-discussed brain dump, and not in any way a concrete plan for implementation. Don't take on this project, expecting us to merge it. Of course anyone is free to prototype, but I wrote this down to spark ideas.

[ruben-arts]

I found that npx has a similar implementation on how to deal with chaching, where they allow the user to specify how it should use fetching of packages. https://docs.npmjs.com/cli/v8/commands/npm-exec#a-note-on-caching

For example:

• prefer-online: Always check the online options to get the latest greatest
• prefer-offline: Only fetch package (options) when the cache isn't enough for the request
• offline: Don't fetch at all. Install from local if you can.
• workspace: Take the context of a local workspace (project for pixi) and use similar configuration.

With allowing a user to set these through CLI or config.toml it would be easier to define how you want pixi to act.