Add ability for many nox tasks to reuse the same session/virtual env

[DDaxler]

How would this feature be useful?
On windows its slow to build a new environment, having to build identical environments seems a waste.

Describe the solution you'd like

import nox

@nox.session(python="3.6")
def all_tests(session):
    session.install('-rrequirements.txt')
    session.install('-rrequirements_test.txt')
    session.install('-rrequirements_docs.txt')
    session.install('-rrequirements_dist.txt')

@nox.session(reuse="all_tests")
def build_docs(session):
    session.run('sphinx-build', 'source', 'build/html')

@nox.session(reuse="all_tests")
def pylint(session):
    session.run("pylint", "code")

@nox.session(reuse="all_tests")
def yapf(session):
    session.run("yapf", "-r", "-d", "code")

reuse is the new "function"

Describe alternatives you've considered
Havent found an easy way to reuse environments.

[stsewd]

I think a way of doing this would be using https://nox.readthedocs.io/en/stable/config.html#nox.sessions.Session.posargs, of course you'll lose the feature of listing sessions (nox -l)

Also, maybe #72 is more useful here?

[stsewd]

Maybe doing this from the command line? Like nox -s lint tests --share-env.

[paunovic]

Are there plans to implement this? I'd be happy to work on it and create a feature PR.

[theacodes]

Go for it! Let us know if you need help.

…

On Fri, Jan 10, 2020, 6:37 PM Marko Paunovic ***@***.***> wrote: Are there plans to implement this? I'd be happy to work on this and create a PR. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#167>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5I43APPEA33IG5BFJNLDQ5EBCDANCNFSM4GYEQ5TA> .

[cs01]

A related approach that would save time around environment creation/reuse is #265.

[theacodes]

I like the solution proposed at #286 (comment)

[Spectre5]

I also like the solution proposed at #286 (comment). Just a quick thought, you can use "notify" to simulate some parts of this, with some caveats:

• Only one session can run on any Nox invocation, as you can't use that shared environment more than once.
• This doesn't attempt to handle the "requires", which would certainly be nice to have at times.

import nox

nox.options.reuse_existing_virtualenvs = True

@nox.session
def shared_venv(session):
    session.install('black')
    session.install('isort')
    session.install('flake8')

    if session.posargs and session.posargs[0] == 'format':
        session.run('black', '.')
        session.run('isort', '.')
    elif session.posargs and session.posargs[0] == 'lint':
        session.run('black', '--check', '.')
        session.run('isort', '--check-only', '.')
        session.run('flake8')

@nox.session
def format(session):
    session.notify('shared_venv', posargs=['format'])

@nox.session
def lint(session):
    session.notify('shared_venv', posargs=['lint'])

This could then be used as below. The first invocation will only create that shared virtual environment since the other two only notify that one, which will be ignored since each session can only run once. The second two only "lint" or only "format" the code. But only one could be done on each invocation of Nox.

$ nox
$ nox -s lint
$ nox -s format

This obviously isn't a very great solution, but just something that came to mind. I'd certainly like to get the proper solution implemented.

[gschaffner]

Hi all! There is an implementation of this over on #631 (comment). It would be great to get some feedback there.

[henryiii]

FYI, this was the syntax I came up with thinking about this before reading the issues:

import nox

@nox.env(python="3.6")
def all_tests(session):
    session.install('-rrequirements.txt')
    session.install('-rrequirements_test.txt')
    session.install('-rrequirements_docs.txt')
    session.install('-rrequirements_dist.txt')

@all_tests.session()
def build_docs(session):
    session.run('sphinx-build', 'source', 'build/html')

@all_tests.session()
def pylint(session):
    session.run("pylint", "code")

@all_tests.session()
def yapf(session):
    session.run("yapf", "-r", "-d", "code")

You could even pass a restricted session in this case that wouldn't allow install in the sessions to reduce potential for order dependent behavior. I assume this is quite a bit of work, just putting down what I though of.

[CAM-Gerlach]

After some consideration, we're in the process of adopting Nox for https://github.com/spyder-ide/spyder-docs ¹ and potentially many of the other, higher-traffic Spyder-IDE org repositories in the future. On Spyder-Docs, we have only a handful of direct dependencies across all tasks (docs, lint, translate, etc) and have adopted a streamlined workflow with them in a single requirements.txt file that all of our various CI services, jobs and local pip/conda could install. Particularly given we want contributions to the docs to be as low-friction as practical, the long delay (on the order of a minute) creating venvs and installing duplicate environments for each of our numerous related sessions was a significant regression from our previous Make-based approach and a blocker to Nox adoption.

@Spectre5 's workaround was very helpful in suggesting a viable path forward, and we've made a number of further improvements and optimizations that avoid most of the (non-inherent) limitations of that approach. In particular, we:

• Simplified the main venv session and kept the logic for each session together by factoring them out to separate functions
• Avoided verbosity and fragility, and simplified adding new sessions by passing the function objects to the venv session directly as a posarg instead requirng of a whole elif block, keeping "install" free of any
• Added support for passing multiple function objects to the venv session, allowing developers to easily pre-define groups of sessions to run sequentially with minimal boilerplate.
• Added passthrough for the existing posargs, so user-specified custom args can still be passed to sessions (commonly needed in our use case)
• Minimized the several-second overhead starting each session down to zero by skipping venv creation by default for all tasks but the install one ²

Here's what that looks like:

import nox

nox.options.sessions = ["run_all"]
nox.options.default_venv_backend = "none"

@nox.session(venv_backend="virtualenv", reuse_venv=True)
def install(session):
    session.install("-r", "requirements.txt")
    if session.posargs:
        for task in session.posargs[0]:
            task(session)

def _docs(session):
    session.run("python", "-m", "sphinx", "doc/", "doc/_build/html", *session.posargs[1:])

@nox.session
def docs(session):
    session.notify("install", posargs=([_docs], *session.posargs))

def _lint(session):
    session.run("pre-commit", "run", *session.posargs[1:])

@nox.session
def lint(session):
    session.notify("install", posargs=([_lint], *session.posargs))

@nox.session
def run_all(session): session.notify("install", posargs=([_docs, _lint]))

You could also even implement a simple run session that can run multiple arbitrary user-specified tasks. Its of course somewhat hacky and not critical for our use case so we didn't go that route, but it could look something like:

@nox.session
def run(session):
    funcs_to_run = [globals()[f"_{name}"] for name in session.posargs]
    session.notify("install", posargs=[funcs_to_run])

It would then be used like so:

$ nox custom -- docs lint

A real implementation would want to register each function in an explicit lookup table instead of just relying on arbitrary globals (and add friendlier error handling, of course).

---

Sidenote: An alternate approach I explored in depth would be to use nested private functions with pre-baked posargs, e.g.

@nox.session(venv_backend="virtualenv", reuse_venv=True)
def install(session):
    session.install("-r", "requirements.txt")
    if session.posargs:
        for task in session.posargs:
            task(session)

@nox.session
def lint(session):
    posargs = session.posargs
    def _docs(session):
        session.run("pre-commit", "run", *posargs)
    session.notify("install", posargs=[_docs])

However, due to the private nested, pre-baked functions it doesn't work with all, the custom run or functions running pre-defined task groups.

Footnotes

1. As a cross-platform, Python-native replacement for Make, in this case—we of course considered Invoke, but with apparently poor project health and minimal maintenance it seems very unwise to rely on ↩

2. Sidenote: Setting nox.options.reuse_venvs = True didn't appear to have any effect per our initial testing (not sure what was going wrong), so we instead set it individually on the one session that actually needs it. ↩