List of disabled plugins is not passed down to runpytest()

[mtelka]

When there are some pytest plugins disabled using the -p no:XXX option the information about disabled plugins is not passed down to the runpytest() call, so the initially disabled plugin appears as enabled there.

Simple test to show the issue:

$ wget https://files.pythonhosted.org/packages/source/p/pytest_metadata/pytest_metadata-3.1.1.tar.gz
$ tar xf pytest_metadata-3.1.1.tar.gz
$ cd pytest_metadata-3.1.1/
$ sed -i -e '/\[testenv\]/a\'$'\n''deps = pytest-checkdocs' tox.ini
$ sed -i -e 's/setenv =/& PYTEST_ADDOPTS="-p no:checkdocs"/' tox.ini
$ tox -e py39
py39: install_deps> python -I -m pip install pytest-checkdocs
.pkg: install_requires> python -I -m pip install 'hatch-vcs>=0.3' 'hatchling>=1.13'
.pkg: _optional_hooks> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
py39: install_package_deps> python -I -m pip install 'pytest>=7.0.0'
py39: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/pytest_metadata-3.1.1/.tox/.tmp/package/1/pytest_metadata-3.1.1.tar.gz
py39: commands[0]> pytest -s -ra --color=yes
============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.1.1, pluggy-1.4.0
cachedir: .tox/py39/.pytest_cache
rootdir: /tmp/pytest_metadata-3.1.1
configfile: tox.ini
testpaths: tests
plugins: metadata-3.1.1
collected 10 items

tests/test_metadata.py ============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.1.1, pluggy-1.4.0
rootdir: /tmp/pytest-of-marcel/pytest-2/test_metadata0
plugins: checkdocs-2.12.0, metadata-3.1.1
collected 1 item

test_metadata.py .                                                       [100%]

============================== 1 passed in 0.02s ===============================

... [snip] ...

Please note the plugins: checkdocs-2.12.0, metadata-3.1.1 line above. I expect to do not see checkdocs-2.12.0 there.

Related issue: #12073

[mtelka]

Maybe the other possible and acceptable approach would be to set PYTEST_DISABLE_PLUGIN_AUTOLOAD and load desired plugins using the -p PLUGIN option. But this does not work either:

$ wget https://files.pythonhosted.org/packages/source/p/pytest_metadata/pytest_metadata-3.1.1.tar.gz
$ tar xf pytest_metadata-3.1.1.tar.gz
$ cd pytest_metadata-3.1.1/
$ sed -i -e '/\[testenv\]/a\'$'\n''deps = pytest-checkdocs' tox.ini
$ sed -i -e '/setenv =/a\'$'\n''  PYTEST_ADDOPTS=-p metadata' tox.ini
$ sed -i -e '/setenv =/a\'$'\n''  PYTEST_DISABLE_PLUGIN_AUTOLOAD=1' tox.ini
$ tox -e py39
py39: install_deps> python -I -m pip install pytest-checkdocs
.pkg: install_requires> python -I -m pip install 'hatch-vcs>=0.3' 'hatchling>=1.13'
.pkg: _optional_hooks> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
py39: install_package_deps> python -I -m pip install 'pytest>=7.0.0'
py39: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/test/pytest_metadata-3.1.1/.tox/.tmp/package/1/pytest_metadata-3.1.1.tar.gz
py39: commands[0]> pytest -s -ra --color=yes
============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.2.2, pluggy-1.5.0
cachedir: .tox/py39/.pytest_cache
rootdir: /tmp/test/pytest_metadata-3.1.1
configfile: tox.ini
testpaths: tests
plugins: metadata-3.1.1
collected 10 items

tests/test_metadata.py ============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.2.2, pluggy-1.5.0
rootdir: /tmp/pytest-of-marcel/pytest-45/test_metadata0
collected 1 item

test_metadata.py E                                                       [100%]

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_pass __________________________
file /tmp/pytest-of-marcel/pytest-45/test_metadata0/test_metadata.py, line 1
  def test_pass(metadata):
E       fixture 'metadata' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory              
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/pytest-of-marcel/pytest-45/test_metadata0/test_metadata.py:1

[...snip...]

In this case the metadata plugin is missing in the test session invoked using runpytest().

[mtelka]

I think I found the way how to achieve what I need: use PYTEST_DISABLE_PLUGIN_AUTOLOAD together with PYTEST_PLUGINS. To make it fully working as expected we just need this small change: #12616.

With #12616 in place I can do this:

$ wget https://files.pythonhosted.org/packages/source/p/pytest_metadata/pytest_metadata-3.1.1.tar.gz
$ tar xf pytest_metadata-3.1.1.tar.gz
$ cd pytest_metadata-3.1.1/
$ sed -i -e '/\[testenv\]/a\'$'\n''deps = pytest-checkdocs' tox.ini
$ sed -i -e '/setenv =/a\'$'\n''  PYTEST_PLUGINS=metadata' tox.ini
$ sed -i -e '/setenv =/a\'$'\n''  PYTEST_DISABLE_PLUGIN_AUTOLOAD=1' tox.ini
$ tox -e py39
.pkg: _optional_hooks> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
.pkg: build_sdist> python /usr/lib/python3.9/vendor-packages/pyproject_api/_backend.py True hatchling.build
py39: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/test/pytest_metadata-3.1.1/.tox/.tmp/package/6/pytest_metadata-3.1.1.tar.gz
py39: commands[0]> pytest -s -ra --color=yes
============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.2.2, pluggy-1.5.0
cachedir: .tox/py39/.pytest_cache
rootdir: /tmp/test/pytest_metadata-3.1.1
configfile: tox.ini
testpaths: tests
plugins: metadata-3.1.1
collected 10 items

tests/test_metadata.py ============================= test session starts ==============================
platform sunos5 -- Python 3.9.19, pytest-8.2.2, pluggy-1.5.0
rootdir: /tmp/pytest-of-marcel/pytest-71/test_metadata0
plugins: metadata-3.1.1
collected 1 item

test_metadata.py .                                                       [100%]

============================== 1 passed in 0.00s ===============================

[...snip...]