⬆️ Update to Sphinx 7 (#1056)

The principle goal of this commit is to update code/tests to allow for use with sphinx 7 and upgrade to Python 3.8 - 3.11.
To do this a number of other changes are made:

- The development dependency configuratio, in `pyproject.toml`, is modified to use separate [extras](https://python-poetry.org/docs/pyproject/#extras) for tests, benchmarks and docs
  - This allows for the `docs/requirements.txt` to be removed
- The noxfile now directly calls pytest/sphinx, rather than call `make`. This removes a confusing indirection, and allows for [passing arguments to sessions](https://nox.thea.codes/en/stable/config.html#passing-arguments-into-sessions)
- The CI now also directly calls pytest, again removing confusing indirections
  - The tests are split into core and JS tests (JS were previously not actually run)
- The `typing_extensions` backport for python<=3.7 is removed
- The [`sphinxcontrib-jquery`](https://github.com/sphinx-contrib/jquery) dependency is added, which re-adds the jquery JavaScript removed in sphinx>=6
  - This fixes the JS tests for sphinx 7

.github/workflows/benchmark.yaml

@@ -13,20 +13,14 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-      - name: Install/Update  build dependencies
+      - name: Update pip
+        run: python -m pip install --upgrade pip
+      - name: Install dependencies
         run: |
-          python -m pip install --upgrade setuptools
-          python -m pip install --upgrade pip
-      - name: Install Poetry
-        run: |
-          curl -sL https://install.python-poetry.org | python -
-      - name: Update Pip
-        run: poetry run pip install -U pip setuptools
-      - name: Install Nox Dependencies
-        run: |
-          python -m pip install poetry nox nox-poetry pyparsing==3.0.4
+          python -m pip install -e .[test,benchmark,docs]
       - name: Run Benchmark for time
-        run: nox --non-interactive --session benchmark_time -- --full-trace
+        run: pytest --benchmark-json output.json -k _time tests/benchmarks
+
       - name: Download previous benchmark data
         uses: actions/cache@v3
         with:
@@ -53,13 +47,16 @@ jobs:
           gh-pages-branch: "benchmarks"
           benchmark-data-dir-path: "docs/bench"
           alert-comment-cc-users: '@danwos'
-      - name: Run memory benchmark
-        run: nox --non-interactive --session benchmark_memory
+
+      - name: Run Benchmark for memory
+        run: pytest --benchmark-json output.json -k _time tests/benchmarks
+      - name: Create memory flamegraph
+        run: memray flamegraph -o mem_out.html mem_out.bin
       - name: Publish memory flamegraph
         run: |
           git checkout benchmarks
           cp mem_out.html docs/memory.html
           git add docs/memory.html
           git -c "user.name=Sphinx-Needs CI" -c "[email protected]" commit -m "memory.html update"
       - name: Push changes   
-        run: git push origin benchmarks
+        run: git push origin benchmarks

.github/workflows/ci.yaml

@@ -1,70 +1,97 @@
 name: CI
 on: [pull_request]
 jobs:
-  tests:
-    name: "py${{ matrix.python-version }} sp${{ matrix.sphinx-version }} do${{ matrix.docutils-version }} ${{ matrix.os }}"
+
+  lint:
+    name: Lint
+    runs-on: ubuntu-20.04
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python 3.8
+      uses: actions/setup-python@v4
+      with:
+        python-version: '3.8'
+    - uses: pre-commit/[email protected]
+
+  tests-core:
+    name: "Core py${{ matrix.python-version }} sphinx~=${{ matrix.sphinx-version }} ${{ matrix.os }}"
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false  # Set on "false" to get the results of ALL builds
       matrix:
         os: ["ubuntu-latest"]
         # 3.9.8 seems to be broken with type_ast
         # https://www.mail-archive.com/[email protected]/msg1829077.html
-        python-version: ["3.8", "3.10"]
-        sphinx-version: ['5.2.1', '4.5.0']
-        docutils-version: ['0.19', '0.15']
-        exclude:
-          - sphinx-version: "4.5.0"
-            docutils-version: "0.19"
+        python-version: ["3.8", "3.11"]
+        sphinx-version: ["5.0", "6.0", "7.0"]
         include:
-          # Check only newest setups for win server
           - os: "windows-latest"
             python-version: "3.8"
-            sphinx-version: "4.5.0"
-            docutils-version: "0.15"
+            sphinx-version: "5.0"
     steps:
       - uses: actions/checkout@v4
       - name: Set Up Python ${{ matrix.python-version }}
         uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
-      - name: Install Nox Dependencies
-        run: |
-          python -m pip install poetry nox nox-poetry pyparsing==3.0.4
-      - name: Install other dependencies
+      - name: Update pip
+        run: python -m pip install --upgrade pip
+      - name: Install dependencies
         run: |
-          python -m pip install setuptools
-      - name: Final package list
+          python -m pip install sphinx~=${{ matrix.sphinx-version }} -e .[test]
+          python -m pip freeze
+      - name: Run pytest
         run: |
-          pip freeze
-
-      - name: Run Tests on Windows
-        run: nox --non-interactive --session "tests-${{ matrix.python-version }}(sphinx='${{ matrix.sphinx-version }}', docutils='${{ matrix.docutils-version }}')" -- --full-trace
-        if: runner.os == 'Windows'
-
-      - name: Run Tests on Linux
-        run: nox --non-interactive --session "tests-${{ matrix.python-version }}(sphinx='${{ matrix.sphinx-version }}', docutils='${{ matrix.docutils-version }}')" -- --full-trace
-        if: runner.os == 'Linux'
+          python -m pytest -v --ignore=tests/benchmarks -m "not jstest" tests
 
-  lint:
-    name: Lint
-    runs-on: ubuntu-20.04
+  tests-js:
+    name: "JS py${{ matrix.python-version }} sphinx~=${{ matrix.sphinx-version }} ${{ matrix.os }}"
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        include:
+          - os: "ubuntu-latest"
+            python-version: "3.8"
+            sphinx-version: "5.0"
+          - os: "ubuntu-latest"
+            python-version: "3.11"
+            sphinx-version: "7.0"
     steps:
-    - uses: actions/checkout@v4
-    - name: Set up Python 3.8
-      uses: actions/setup-python@v4
-      with:
-        python-version: '3.8'
-    - uses: pre-commit/[email protected]
+      - uses: actions/checkout@v4
+      - name: Use Node.js
+        uses: actions/setup-node@v3
+        with:
+          node-version: 18
+      - name: Install Cypress Test Framework
+        run: npm install cypress
+      - name: Set Up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Update pip
+        run: python -m pip install --upgrade pip
+      - name: Install dependencies
+        run: |
+          python -m pip install sphinx~=${{ matrix.sphinx-version }} -e .[test]
+          python -m pip freeze
+      - name: Run pytest
+        run: |
+          python -m pytest -v --ignore=tests/benchmarks -m "jstest" tests
 
   check:
+
     # This job does nothing and is only used for the branch protection
     # see https://github.com/marketplace/actions/alls-green#why
+
     if: always()
+
     needs:
     - lint
-    - tests
+    - tests-core
+    - tests-js
+
     runs-on: ubuntu-latest
+
     steps:
     - name: Decide whether the needed jobs succeeded or failed
       uses: re-actors/alls-green@release/v1

.github/workflows/docs.yaml

@@ -12,8 +12,11 @@ jobs:
         uses: actions/setup-python@v4
         with:
           python-version: "3.10"
-      - name: Install Nox Dependencies
+      - name: Update pip
+        run: python -m pip install --upgrade pip
+      - name: Install Dependencies
         run: |
-          python -m pip install poetry nox nox-poetry pyparsing==3.0.4
-      - name: Run LinkCheck
-        run: nox --non-interactive --session linkcheck -- --full-trace
+          python -m pip install -e .[docs]
+      - name: Run linkcheck
+        run: sphinx-build -b linkcheck . _build 
+        working-directory: docs

.github/workflows/js_test.yml

@@ -29,12 +29,9 @@ jobs:
         key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
         restore-keys: |
           ${{ runner.os }}-pip-
-    - name: Install Python dependencies
+    - name: Install package
       run: |
-        pip install -r docs/requirements.txt
-    - name: Install Sphinx-Needs from master
-      run: |
-        pip install -e .
+        pip install -e .[docs]
     - name: Install Node dependencies
       run: npm install cypress
     - name: Build Docs

.pre-commit-config.yaml

@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/psf/black
-    rev: 23.7.0
+    rev: 23.10.1
     hooks:
       - id: black
 
@@ -20,14 +20,14 @@ repos:
       - id: isort
 
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.10.1
+    rev: v3.15.0
     hooks:
       - id: pyupgrade
         args:
           - --py36-plus
 
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.5.1
+    rev: v1.6.1
     hooks:
     - id: mypy
       files: sphinx_needs/.*
@@ -39,6 +39,6 @@ repos:
       - types-requests
 
   - repo: https://github.com/python-poetry/poetry
-    rev: 1.5.0
+    rev: 1.7.0
     hooks:
       - id: poetry-check

.readthedocs.yml

@@ -7,7 +7,6 @@ sphinx:
 
 python:
   install:
-    - requirements: docs/requirements.txt
     - method: pip
       path: .
       extra_requirements:

docs/contributing.rst

@@ -26,42 +26,32 @@ Your PR should conform with the following rules:
 Installing Dependencies
 -----------------------
 
-**Sphinx-Needs** requires only
-`Poetry <https://python-poetry.org/>`__ to be installed as a system
-dependency, the rest of the dependencies are 'bootstrapped' and
-installed in an isolated environment by Poetry.
+To develop **Sphinx-Needs**  it can be installed, with development extras, into an existing Python environment using ``pip``:
 
-1. `Install Poetry <https://python-poetry.org/docs/#installation>`__
+.. code-block:: bash
 
-2. Install project dependencies
+   pip install sphinx-needs[test,benchmark,docs]
 
-   .. code-block:: bash
+or using `Poetry <https://python-poetry.org/>`__ to install the dependencies into an isolated environment:
 
-      poetry install
-
-3. `Install Pre-Commit <https://pre-commit.com/>`__
+1. `Install Poetry <https://python-poetry.org/docs/#installation>`__
 
-4. Install the Pre-Commit hooks
+2. Install project dependencies
 
    .. code-block:: bash
 
-      pre-commit install
-
-5. For running tests, install the dependencies of our official documentation:
+      poetry install --all-extras
 
-   .. code-block:: bash
+To run the formatting and linting suite, pre-commit is used:
 
-      pip install -r docs/requirements.txt
+1. `Install Pre-Commit <https://pre-commit.com/>`__
 
+2. Install the Pre-Commit hooks
 
-List make targets
------------------
-
-**Sphinx-Needs** uses ``make`` to invoke most development related actions.
+   .. code-block:: bash
 
-Use ``make list`` to get a list of available targets.
+      pre-commit install
 
-.. program-output:: make --no-print-directory --directory ../ list
 
 Build docs
 ----------
@@ -293,6 +283,14 @@ See the Poetry documentation for a list of commands.
 In order to run custom commands inside the isolated environment, they
 should be prefixed with ``poetry run`` (ie. ``poetry run <command>``).
 
+List make targets
+-----------------
+
+**Sphinx-Needs** uses ``make`` to invoke most development related actions.
+
+Use ``make list`` to get a list of available targets.
+
+.. program-output:: make --no-print-directory --directory ../ list
 
 Publishing a new release
 ------------------------