pluggy loads disabled pytest plugins

[mtelka]

I do have several cases when I run testing for a pytest plugin (for example pytest-xdist) with other pytest plugin installed and disabled using -p no:XXX set in PYTEST_ADDOPTS environment variable. With such setup it looks like that the disabled plugin gets sometimes loaded for pytest subcalls.

Here is a list of issues with more details:
pytest-dev/pytest-rerunfailures#242
okken/pytest-check#146
bitprophet/pytest-relaxed#31

[RonnyPfannschmidt]

This is a combined bugs of pytest and xdist that pluggy is unrelated to, I'll close this once I link the related issues in xdist and pytest itself

[mtelka]

Here are steps to reproduce the issue (without xdist involved):

$ wget https://files.pythonhosted.org/packages/source/p/pytest_metadata/pytest_metadata-3.1.0.tar.gz
$ tar xf pytest_metadata-3.1.0.tar.gz
$ cd pytest_metadata-3.1.0/
$ sed -i -e '/\[testenv\]/a\'$'\n''deps = pytest-checkdocs' tox.ini
$ tox -e py39 -- -p no:checkdocs

The most important part of the log follows. Please note checkdocs plugin listed in the pytest subcall.

py39: commands[0]> pytest -s -ra --color=yes -p no:checkdocs
============================= test session starts ==============================
platform sunos5 -- Python 3.9.18, pytest-8.0.0, pluggy-1.4.0
cachedir: .tox/py39/.pytest_cache
rootdir: /tmp/pluggy/pytest_metadata-3.1.0
configfile: tox.ini
testpaths: tests
plugins: metadata-3.1.0
collected 10 items

tests/test_metadata.py ============================= test session starts ==============================
platform sunos5 -- Python 3.9.18, pytest-8.0.0, pluggy-1.4.0
rootdir: /tmp/pytest-of-marcel/pytest-4/test_metadata0
plugins: checkdocs-2.10.1, metadata-3.1.0
collected 1 item

test_metadata.py .                                                       [100%]

============================== 1 passed in 0.01s ===============================
.============================= test session starts ==============================
platform sunos5 -- Python 3.9.18, pytest-8.0.0, pluggy-1.4.0
rootdir: /tmp/pytest-of-marcel/pytest-4/test_environment_variables0
collected 1 item

test_environment_variables.py E                                          [100%]

==================================== ERRORS ====================================
_________________________ ERROR at setup of test_pass __________________________
file /tmp/pytest-of-marcel/pytest-4/test_environment_variables0/test_environment_variables.py, line 1
  def test_pass(metadata):
E       fixture 'metadata' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
>       use 'pytest --fixtures [testpath]' for help on them.

/tmp/pytest-of-marcel/pytest-4/test_environment_variables0/test_environment_variables.py:1

[RonnyPfannschmidt]

Pytester doesn't pass over plugin disabled flags from the surrounding pytest, I consider this a different bug

[mtelka]

I believe this is exactly the bug I'm talking about since beginning: pluggy loads disabled pytest plugins. I do not see any other related issue around. Maybe I miss something obvious...

Anyway, if you think this is not a pluggy bug, but pytest, could you please provide some more details? I just found pluggy on the stack calling the disabled plugin, so I reported the problem here once it was obvious that similar reports downstream for pytest plugin projects led to nowhere because they cannot prevent pluggy to load them.

Another way to reproduce the problem (now with PYTEST_ADDOPTS as it was reported initially):

$ wget https://files.pythonhosted.org/packages/source/p/pytest_metadata/pytest_metadata-3.1.0.tar.gz
$ tar xf pytest_metadata-3.1.0.tar.gz
$ cd pytest_metadata-3.1.0/
$ sed -i -e '/\[testenv\]/a\'$'\n''deps = pytest-checkdocs' tox.ini
$ sed -i -e 's/setenv =/& PYTEST_ADDOPTS="-p no:checkdocs"/' tox.ini
$ tox -e py39

[mtelka]

Since it looks like the problem is in pytest I reported the issue there: pytest-dev/pytest#12186