user workflows and the case for env maybe unrelated to cwd

[oscar6echo]

This is discussion was started on discord channel General workflow use question.

Quoting my last line:

Indeed the inability to use envs distinct from current dir is an adoption barrier. Our typical workflow with conda/micromamba is one env per task family, and several corp packages installed in pip editable mode per env (in addition to public packages from conda-forge & pypi). I understand/respect (of course) the "env definition in current folder" and see how it fits workflows that I have met with npm and cargo. However I would argue python workflows are more varied given the range of its business (non IT specialists) users. Hence a documented and resonnably simple possibility to use env (with all its caveats, first of which is less straightforward reproducibility - important but maybe not an absolute constraint) would widen the scope of pixi - IMHO.

Quoting Ruben's last line:

Got it, so it sounds like the main goal isn't about team collaboration on projects, but rather making it easier for individual local experimentation and testing in different environments, where exact reproducibility isn't the top priority, right?

Yes, but it is not only individual experimentation. It can be a business work "setup" (using a different word from env to express a user concept, not the usual conda env everybody knows) where python packages are installed as editable because they occasionally/frequenlty need be modified (if only for extra prints). Additionally, this setup can be on a VM shared among several "business people" (again using a different word from developper to stress the fact quite few python users are not developpers).

Indeed, these use cases are not about reproducibility, where the goal is to "git clone & run & it just works".
To be clear, I do see the value of reproducibility. But to be honest I don't think this is the full spectrum of users needs.

Now, I'll think out loud: Because there is a PIXI_HOME with an ability to global install, surely there is a way to add the usual global env feature, or maybe provide it as a plugin to make it secondary, or display warning message when using such global envs. For "business people" verbosity can only help.

Food for thought.

[ruben-arts]

Related to this discussion: #1011

[ruben-arts]

This is a personal reaction.
Just to clarify some of the ideas I have for pixi and why I currently believe so strongly in the "No global envs". My biggest goal is reproducibility, not in the sense of actually being able to reproduce (download the exact same software) but the lack of influence by the global setup of a users machine.

One of my biggest problems with conda is that it tempts to take over your machine with a base environment, by design breaking a lot of local development projects. Pixi is currently having adoption problems because people have bumped their head on this workflow and don't trust anything related to conda because of it. Pixi will not and should never break existing non-pixi installations.

With much better IDE integration and possibly other tools included in pixi, I think we're able to solve most of the annoyances people encounter with not having the default conda workflow.

For the "business people", which from my background also counts for industrial engineers we should focus on good documentation, helpful messaging in the tooling and the ability to stay out of the terminal as that often gives the biggest push back in my experience.

I just wanted to get these thoughts on the matter out, this doesn't mean we would never support anything like it. We're a flexible and risk taking team but also build out of a believe, if this believe would choke the adoption we should definitely re-validate but I'm not convinced yet.

[baszalmstra]

Great! Sorry I shouldve read the entire thread before commenting 😉

[ruben-arts]

Yeah we need to fix that @baszalmstra.

[synapticarbors]

Would it be helpful if I opened an issue specific to this change (persistent manifest path in pixi shell)? I think this with tasks and other functionality defaulting to the env of the shell, would solve a bunch of usability issues.

[tdejager]

Would it be helpful if I opened an issue specific to this change (persistent manifest path in pixi shell)? I think this with tasks and other functionality defaulting to the env of the shell, would solve a bunch of usability issues.

Yes that would be great!

[synapticarbors]

Here is the new issue: #1038

[synapticarbors]

I wholly agree that not having a global default env like the conda base environment is a very good thing. I do think, however, that pixi could be made to support workflows without a global base env that would make it appealing to current conda users. Specifically the use cases/workflows that I use and I think are pretty common are:

• environments that are common to multiple projects/repos
• basic data analysis where I want access to an environment, but may be working in some directory where data lives outside of the project or a scratch directory where I'm playing around with code snippets or a jupyter notebook.

There is a hierarchy of changes that I think would be useful and could address those other workflows.

1. When activating a pixi shell and cd'ing into another directory, I should be able to run commands like pixi list (currently you get "could not find pixi.toml"). This is what @dhirschfeld was in part asking for in Allow defining a default manifest-path which is used by all pixi commands irrespective of what folder you happen to be in #1011. That is a persistent manifest path when in the subshell for the life of that subshell. If you happen to cd from one project to another, then a warning could be given that you are trying to run pixi commands in an activated pixi shell from another project.
2. The ability to set the manifest path as an environment variable. This would be a niche use case that would be entirely opt-in, but would enable using a pixi project outside of the project directory without constantly having to specify the --manifest-path flag or cd'ing into the project first. This differs from (1) in that you wouldn't necessarily have to be in a pixi shell to get the effect that you associated with a project from outside of that project's directory.

With one or both of these changes, it would allow one to write a tool that then goes a step further and get a conda-like workflow of being able to activate environments by name. What I imagine is having a mapping of env names to project directories or project manifest paths that is stored globally (like ~/.conda/environments.txt). Then the tool could run like tool activate <env name>, and you'd get a pixi env loaded up from anywhere. I wouldn't expect pixi developers to write such a tool because it's outside of the scope of pixi.

I think pixi adoption could be increased from those already using conda if it was easier to adopt a conda-like workflow. These are the users that are already using conda-forge and have buy-in in the ecosystem. Currently, switching brings noted benefits, but breaks current workflows. I know some would argue that the local project directory workflow is better, but if it throws up a barrier to current conda users, then many won't take the time to adapt and will just stick with mamba/conda.

[pavelzw]

environments that are common to multiple projects/repos

this sounds related to workspace/monorepo support (which I think is a valid use case but not trivial to implement right) #387

but may be working in some directory where data lives outside of the project

i think it's okay to run pixi shell --manifest-path in this case. If you want to automate that, you might want to take a look at integrating pixi with direnv https://pixi.sh/dev/environment/#using-pixi-with-direnv

[synapticarbors]

I don't understand how the direnv example would help with this. In the linked documentation example, as soon as you cd out of the project directory, it unloads the environment. Additionally, it's not clear to me how to make that work with multi env projects.

[pavelzw]

There are two ways you could incorporate direnv here:

eval "$(pixi shell-hook --manifest-path /path/to/pixi.toml)"

which means that every time you enter said directory which is not in your project (but belongs to it), you would be in the pixi env

The other way would be

alias pixi="pixi --manifest-path=/path/to/pixi.toml"

Which ensures every time you call pixi it behaves as if you call pixi --manifest-path=/path/to/pixi.toml

[tdejager]

So in addition, to what was said I believe we as the pixi developers want to adhere to the pixi philisophy but at the same time, make it easy for conda/mamba users to switch. So we welcome any thoughts that would help with this transition :)

[oscar6echo]

Quotes:

One of my biggest problems with conda is that it tempts to take over your machine with a base environment, by design breaking a lot of local development projects.

I wholly agree that not having a global default env like the conda base environment is a very good thing.

I agree too.
And I think everybody agrees that doing without a base env is unambiguously a good design decision.

[chawyehsu]

+1 for this. The base environment is a trap imho and before using pixi I always have auto_activate_base set to false and avoid the base env when using conda/mamba. The vision of no global env is why I was attracted by pixi at the very beginning. I feel good when there is finally a project focusing on py/conda ecosystem doing something like npm/cargo.

But for sure the lack of default(base, global or whatever named) env does cause difficulties for conda/mamba users transitioning to pixi.

[oscar6echo]

but may be working in some directory where data lives outside of the project

i think it's okay to run pixi shell --manifest-path in this case. If you want to automate that, you might want to take a look at integrating pixi with direnv https://pixi.sh/dev/environment/#using-pixi-with-direnv

So it seems the discussion boils down to: Does pixi want to package a direct way to run these scripts or should third parties devise their own - if they want to leverage pixi qualities. It is not so complicated but it is also duplicate efforts.

[tdejager]

So the case you are describing is activating a pixi environment if you want to work on another project, I think. But isn't there a problem if you decide to use a pixi project inside that activated environment?

From what I gather @oscar6echo described in the original post, there are people working in kind of a default environment with different packages they want to edit quickly. But what if you decide to use another pixi project in that environment, then with the solution that is described here: #1021 (reply in thread) you would never be able to run tasks from that new pixi.toml.

I'm not saying that works well now either, because it might require a layered environment. I just think that it would not solve your use case once you are using more than one pixi.toml in conjuction? WDYT?

EDIT:
Although I get why it would be fine if you just stick to jupyterlab experiments and ad-hoc scripts, as these do not require a pixi.toml project.

[ruben-arts]

I've read it differently, as for this "default global env" use-case you could already use by adding pixi shell-hook --manifest-path ~/default_env/pixi.toml to your ~/.bashrc file. For other readers, I don't recommend this use-case!

@matthewfeickert's comment in the original discord discussion describes a situation where it isn't a default environment but something developers can set and reuse in other locations. Which I think would be fixed by the workflow @baszalmstra and @synapticarbors are describing.

[oscar6echo]

From what I gather @oscar6echo described in the original post, there are people working in kind of a default environment with different packages they want to edit quickly.

I am afraid I have not been clear enough: I did not mean users work in a default env, similar to a conda base env, but a "work" env they activate and then work in. On the contrary I think it is better to do away with the base env, just like pixi allows.

But yes: Once users have activated their "work" env they may use it in various directories, and they usually pip install -e those corp packages that are mostly likely to need changes on the fly.

To try and clarify this, I have amended the discussion title from "...for global env" to "... for env maybe unrelated to cwd".

[oscar6echo]

@matthewfeickert's comment in the original discord discussion describes a situation where it isn't a default environment but something developers can set and reuse in other locations. Which I think would be fixed by the workflow @baszalmstra and @synapticarbors are describing.

I agree and now cf. issue #1038.

[tdejager]

It can be a business work "setup" (using a different word from env to express a user concept, not the usual conda env everybody knows) where python packages are installed as editable because they occasionally/frequenlty need be modified (if only for extra prints).

Could this use-case be seen as kind of like a: https://www.jetpack.io/devbox/docs/quickstart/ like set-up + a number of editable packages?

[oscar6echo]

I don't know devbox but I think issue #1038 is a good solution.

[oscar6echo]

Assuming issue #1038 is implemented and allows to change directory keeping the same env, I think pixi would benefit from more verbosity and env discoverability.

In brainstorm mode, I would suggest the following commands:

• pixi status with varioug levels of verbosity -v, -vv, -vvv to show all about the current env: location of active pixi.toml, the pixi env variables, whether the env is solved, the caching dir, and maybe suggestion typical commands like pixi shell, pixi list or pixi shell-hook , and maybe the "nearest envs" going up the cwd. And it would contain links to the doc, and a brief summary of pixi approach.
  Similar to git status which you (or is it just me ??) constantly do to get reminded of the situation.

• pixi discover with args: (1) a directory and options (2) optionally a number of levels down. such cmd would show all envs (with their name) up from the directory and optionally down to the number of levels. Ideally there would also be a way to activate (pixi select ?) an env discovered this way by name, maybe storing the latest "discovery" in a pixi global space. Thus on any machine it would be quick to discover all pixi envs and activate them (optionally moving to the dir where they are defined !).

With such commands (or similar) a user whether "IT professional", "business people", "industrial engineer", etc would probably understand pixi workings and candidate envs and then would have a conda-like env selection/activation.

Note that these commands cannot break anything as they essentially provide info (status, discover) or are thin wrappers on existing commands (select). A pixi user only interested in "env in local dir" (à la npm or cargo) can just ignore them. though they may like status.

WDYT ?

[oscar6echo]

I forgot to mention re verbosity that pixi should (if you ask me) allow (optionally) to display the current env name if any like conda/mamba/micromamba do.

[ruben-arts]

Pixi should inject a prompt into your shell, which shell do you use? and do you have a prompt modifier?

Parts of the commands you describe could possibly already be integrated in the existing pixi info already 👍

[oscar6echo]

Pixi should inject a prompt into your shell, which shell do you use? and do you have a prompt modifier?

I use zsh. Yes I have a prompt created by starship. I guess the reason is that pixi is not known by startship at this stage: https://starship.rs/config/

Parts of the commands you describe could possibly already be integrated in the existing pixi info already 👍

I would certainly welcome it: status ~ info.
discover/select (in voc above) would seem better separate maybe.

[ruben-arts]

We support the https://starship.rs/config/#conda currently. Possibly in the future we will be supported by default.

[oscar6echo]

ah yes, wrong manip. I do see the env name in the prompt - and thx for always swift reply 👍