👌 Improve removal of hidden need nodes (#1013)

This commit improves on the current logic (from #995) and moves it to occur earlier in the build process, directly after the need location has been analysed (i.e. once it is no longer necessary).

docs/configuration.rst

@@ -1792,6 +1792,50 @@ Example:
    The created ``needs.json`` file gets stored in the ``outdir`` of the current builder.
    So if ``html`` is used as builder, the final location is e.g. ``_build/html/needs.json``.
 
+
+.. _needs_build_json_per_id:
+
+needs_build_json_per_id 
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 1.4.0
+    
+Builds list json files for each need. The name of each file is the ``id`` of need.
+This option works like :ref:`needs_build_json`. 
+
+Default: False
+
+Example:
+
+.. code-block:: python
+
+    needs_build_json_per_id = False
+
+.. hint::
+
+    The created single json file per need, located in :ref:`needs_build_json_per_id_path` folder, e.g ``_build/needs_id/abc_432.json``
+ 
+.. _needs_build_json_per_id_path:
+
+needs_build_json_per_id_path 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. versionadded:: 1.4.0
+    
+This option sets the location of the set of ``needs.json`` for every needs-id.
+
+Default value: ``needs_id``
+
+Example:
+
+.. code-block:: python
+
+    needs_build_json_per_id_path = "needs_id"
+
+.. hint::
+
+    The created ``needs_id`` folder gets stored in the ``outdir`` of the current builder. The final location is e.g. ``_build/needs_id``
+
 .. _needs_build_needumls:
 
 needs_build_needumls
@@ -2289,49 +2333,3 @@ If true, need options like status, tags or links are collapsed and shown only af
 Default value: True
 
 Can be overwritten for each single need by setting :ref:`need_collapse`.
-
-.. _needs_build_json_per_id:
-
-needs_build_json_per_id 
-~~~~~~~~~~~~~~~~~~~~~~~
-
-.. versionadded:: 1.4.0
-    
-Builds list json files for each need. The name of each file is the ``id`` of need.
-This option works like :ref:`needs_build_json`. 
-
-Default: False
-
-Example:
-
-.. code-block:: python
-
-    needs_build_json_per_id = False
-
-.. hint::
-
-    The created single json file per need, located in :ref:`needs_build_json_per_id_path` folder, e.g ``_build/needs_id/abc_432.json``
- 
-.. _needs_build_json_per_id_path:
-
-needs_build_json_per_id_path 
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. versionadded:: 1.4.0
-    
-This option sets the location of the set of ``needs.json`` for every needs-id.
-
-Default value: ``needs_id``
-
-Example:
-
-.. code-block:: python
-
-    needs_build_json_per_id_path = "needs_id"
-
-.. hint::
-
-    The created ``needs_id`` folder gets stored in the ``outdir`` of the current builder. The final location is e.g. ``_build/needs_id``
-
-
-

docs/contributing.rst

@@ -203,13 +203,14 @@ The following is an outline of the build events which this extension adds to the
    - Start process timing, if enabled (``prepare_env``)
    - Load external needs (``load_external_needs``)
 
-#. For all removed and changed documents (``env-before-read-docs`` event):
+#. For all removed and changed documents (``env-purge-doc`` event):
 
    - Remove all cached need items that originate from the document (``purge_needs``)
 
 #. For changed documents (``doctree-read`` event, priority 880 of transforms)
 
-   - Determine and add data on containing sections and parents to needs (``add_sections``)
+   - Determine and add data on parent sections and needs(``analyse_need_locations``)
+   - Remove ``Need`` nodes marked as ``hidden`` (``analyse_need_locations``)
 
 #. When building in parallel mode (``env-merge-info`` event), merge ``BuildEnvironment`` data (``merge_data``)
 
@@ -223,7 +224,7 @@ The following is an outline of the build events which this extension adds to the
 
 #. For all changed documents, or their dependants (``doctree-resolved``)
 
-   - Replace all ```Needextract``` nodes with a list of the collected ``Need`` (``process_creator``)
+   - Replace all ``Needextract`` nodes with a list of the collected ``Need`` (``process_creator``)
    - Remove all ``Need`` nodes, if ``needs_include_needs`` is ``True`` (``process_need_nodes``)
    - Call dynamic functions, set as values on the need data items and replace them with their return values (``process_need_nodes -> resolve_dynamic_values``)
    - Replace needs data variant values (``process_need_nodes -> resolve_variants_options``)
@@ -232,15 +233,15 @@ The following is an outline of the build events which this extension adds to the
    - Process constraints, for each ``Need`` node (``process_need_nodes -> process_constraints``)
    - Perform all modifications on need data items, due to ``Needextend`` nodes (``process_need_nodes -> process_needextend``)
    - Format each ``Need`` node to give the desired visual output (``process_need_nodes -> print_need_nodes``)
-   - Process all need specific nodes, replacing them with the desired visual output (``process_creator``)
-   - Remove  ``Need`` nodes marked as ``hidden`` (``remove_hidden_needs``)
+   - Process all other need specific nodes, replacing them with the desired visual output (``process_creator``)
 
 #. At the end of the build (``build-finished`` event)
 
-   - Call all user defined need data checks, a.k.a warnings (``process_warnings``)
-   - Write the ``needs.json`` to the output folder (``build_needs_json``)
-   - Write all required UML files to the output file (``build_needumls_pumls``)
-   - Print process timing, if enabled (``process_timing``)
+   - Call all user defined need data checks, a.k.a `needs_warnings` (``process_warnings``)
+   - Write the ``needs.json`` to the output folder, if `needs_build_json = True` (``build_needs_json``)
+   - Write the ``needs.json`` per ID to the output folder, if `needs_build_json_per_id = True` (``build_needs_id_json``)
+   - Write all UML files to the output folder, if `needs_build_needumls = True` (``build_needumls_pumls``)
+   - Print process timing, if `needs_debug_measurement = True`  (``process_timing``)
 
 .. Include our contributors and maintainers.
 .. include:: ../AUTHORS

sphinx_needs/api/need.py

@@ -337,6 +337,12 @@ def run():
         "external_css": external_css or "external_link",
         "is_modified": False,  # needed by needextend
         "modifications": 0,  # needed by needextend
+        # these are set later in the analyse_need_locations transform
+        "sections": [],
+        "section_name": "",
+        "signature": "",
+        "parent_needs": [],
+        "parent_need": "",
     }
     needs_extra_option_names = list(NEEDS_CONFIG.extra_options)
     _merge_extra_options(needs_info, kwargs, needs_extra_option_names)
@@ -436,8 +442,9 @@ def run():
     node_need.line = needs_info["lineno"]
 
     if needs_info["hide"]:
-        # add node to doctree, so we can later compute the containing section(s)
-        # (for use with section filters)
+        # still add node to doctree,
+        # so we can later compute its relative location in the document
+        # (see analyse_need_locations function)
         node_need["hidden"] = True
         return [node_need]
 

sphinx_needs/data.py

@@ -166,7 +166,7 @@ class NeedsInfoType(NeedsBaseDataType):
     # additional source information
     doctype: str
     """Type of the document where the need is defined, e.g. '.rst'"""
-    # set in add_sections transform
+    # set in analyse_need_locations transform
     sections: list[str]
     section_name: str
     """Simply the first section"""

sphinx_needs/directives/need.py

@@ -285,14 +285,23 @@ def purge_needs(app: Sphinx, env: BuildEnvironment, docname: str) -> None:
             del needs[need_id]
 
 
-def add_sections(app: Sphinx, doctree: nodes.document) -> None:
-    """Add section titles to the needs as additional attributes that can
-    be used in tables and filters"""
+def analyse_need_locations(app: Sphinx, doctree: nodes.document) -> None:
+    """Determine the location of each need in the doctree,
+    relative to its parent section(s) and need(s).
+
+    This data is added to the need's data stored in the Sphinx environment,
+    so that it can be used in tables and filters.
+
+    Once this data is determined, any hidden needs
+    (i.e. ones that should not be rendered in the output)
+    are removed from the doctree.
+    """
     builder = unwrap(app.builder)
     env = unwrap(builder.env)
 
     needs = SphinxNeedsData(env).get_or_create_needs()
 
+    hidden_needs: List[Need] = []
     for need_node in doctree.findall(Need):
         need_id = need_node["refid"]
         need_info = needs[need_id]
@@ -329,6 +338,15 @@ def add_sections(app: Sphinx, doctree: nodes.document) -> None:
             need_info["parent_needs"] = parent_needs
             need_info["parent_need"] = parent_needs[0]
 
+        if need_node.get("hidden"):
+            hidden_needs.append(need_node)
+
+    # now we have gathered all the information we need,
+    # we can remove the hidden needs from the doctree
+    for need_node in hidden_needs:
+        if need_node.parent is not None:
+            need_node.parent.remove(need_node)  # type: ignore[attr-defined]
+
 
 def previous_sibling(node: nodes.Node) -> Optional[nodes.Node]:
     """Return preceding sibling node or ``None``."""
@@ -389,8 +407,6 @@ def process_need_nodes(app: Sphinx, doctree: nodes.document, fromdocname: str) -
     # Used to store needs in the docs, which are needed again later
     found_needs_nodes = []
     for node_need in doctree.findall(Need):
-        if node_need.get("hidden"):
-            continue
         need_id = node_need.attributes["ids"][0]
         found_needs_nodes.append(node_need)
         need_data = needs[need_id]
@@ -544,15 +560,6 @@ def _fix_list_dyn_func(list: List[str]) -> List[str]:
     return new_list
 
 
-def remove_hidden_needs(app: Sphinx, doctree: nodes.document, fromdocname: str) -> None:
-    """Remove hidden needs from the doctree, before it is rendered."""
-    if fromdocname not in SphinxNeedsData(app.env).get_or_create_docs().get("all", []):
-        return
-    for node_need in list(doctree.findall(Need)):
-        if node_need.get("hidden"):
-            node_need.parent.remove(node_need)  # type: ignore
-
-
 #####################
 # Visitor functions #
 #####################

sphinx_needs/needs.py

@@ -29,14 +29,13 @@
 from sphinx_needs.directives.need import (
     Need,
     NeedDirective,
-    add_sections,
+    analyse_need_locations,
     html_depart,
     html_visit,
     latex_depart,
     latex_visit,
     process_need_nodes,
     purge_needs,
-    remove_hidden_needs,
 )
 from sphinx_needs.directives.needbar import Needbar, NeedbarDirective, process_needbar
 from sphinx_needs.directives.needextend import Needextend, NeedextendDirective
@@ -214,15 +213,23 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     # EVENTS
     ########################################################################
     # Make connections to events
-    app.connect("env-purge-doc", purge_needs)
     app.connect("config-inited", load_config)
+    app.connect("config-inited", check_configuration)
+
     app.connect("env-before-read-docs", prepare_env)
     app.connect("env-before-read-docs", load_external_needs)
-    app.connect("config-inited", check_configuration)
-    # app.connect("doctree-resolved", add_sections)
-    app.connect("doctree-read", add_sections)
+
+    app.connect("env-purge-doc", purge_needs)
+
+    app.connect("doctree-read", analyse_need_locations)
+
     app.connect("env-merge-info", merge_data)
 
+    app.connect("env-updated", install_lib_static_files)
+    app.connect("env-updated", install_permalink_file)
+    # This should be called last, so that need-styles can override styles from used libraries
+    app.connect("env-updated", install_styles_static_files)
+
     # There is also the event doctree-read.
     # But it looks like in this event no references are already solved, which
     # makes trouble in our code.
@@ -233,19 +240,12 @@ def setup(app: Sphinx) -> Dict[str, Any]:
     app.connect("doctree-resolved", process_creator(NODE_TYPES_PRIO, "needextract"), priority=100)
     app.connect("doctree-resolved", process_need_nodes)
     app.connect("doctree-resolved", process_creator(NODE_TYPES))
-    app.connect("doctree-resolved", remove_hidden_needs, priority=1000)
 
     app.connect("build-finished", process_warnings)
     app.connect("build-finished", build_needs_json)
+    app.connect("build-finished", build_needs_id_json)
     app.connect("build-finished", build_needumls_pumls)
     app.connect("build-finished", debug.process_timing)
-    app.connect("env-updated", install_lib_static_files)
-    app.connect("env-updated", install_permalink_file)
-
-    #
-    app.connect("build-finished", build_needs_id_json)
-    # This should be called last, so that need-styles can override styles from used libraries
-    app.connect("env-updated", install_styles_static_files)
 
     # Be sure Sphinx-Needs config gets erased before any events or external API calls get executed.
     # So never but this inside an event.