Skip to content

Commit

Permalink
Fix line lengths
Browse files Browse the repository at this point in the history
  • Loading branch information
ubmarco committed Nov 14, 2023
1 parent 6288713 commit 2078aad
Show file tree
Hide file tree
Showing 2 changed files with 36 additions and 28 deletions.
5 changes: 3 additions & 2 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ Features added

.. _`<search>`: https://developer.mozilla.org/en-US/docs/Web/HTML/Element/search

* #10779 and #11448: Parallel execution of post-transformation and write_doc_serialized as an experimental feature.
Speeds up builds featuring expensive post-transforms by a factor of at least 2.
* #10779 and #11448: Parallel execution of post-transformation and
write_doc_serialized as an experimental feature.
Speeds up builds featuring expensive post-transforms by a factor of at least 2.
Patch by Marco Heinemann.

Bugs fixed
Expand Down
59 changes: 33 additions & 26 deletions doc/usage/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -790,17 +790,18 @@ General configuration
.. confval:: enable_parallel_post_transform

Default is ``False``.
This experimental feature flag activates parallel post-transformation during the parallel write phase.
When active, the event :event:`doctree-resolved` as well as the builder function ``write_doc_serialized``
also run in parallel. Parallel post-transformation can greatly improve build time for extensions that do
heave computation in that phase. Depending on machine core count and project size, a build time
reduction by a factor of 2 to 4 and even more was observed.
This experimental feature flag activates parallel post-transformation during
the parallel write phase. When active, the event :event:`doctree-resolved` as
well as the builder function ``write_doc_serialized`` also run in parallel.
Parallel post-transformation can greatly improve build time for extensions that
do heave computation in that phase. Depending on machine core count and project
size, a build time reduction by a factor of 2 to 4 and even more was observed.
The feature flag does nothing in case parallel writing is not used.

*Background*

By default, if parallel writing is active (that is no extensions inhibits it by their
:ref:`metadata <ext-metadata>`) the following logic applies:
By default, if parallel writing is active (that is no extensions inhibits it by
their :ref:`metadata <ext-metadata>`) the following logic applies:

.. code-block:: text
Expand All @@ -810,9 +811,10 @@ General configuration
main process: Builder.write_doc_serialized()
sub process: Builder.write_doc()
This means only the ``write_doc()`` function is executed in parallel. However, the subprocess waits
for the main process preparing the chunk. This is a serious bottleneck that practically inhibits
parallel execution when extensions are used that do CPU intensive calculations during post-transformation.
This means only the ``write_doc()`` function is executed in parallel. However,
the subprocess waits for the main process preparing the chunk. This is a
serious bottleneck that practically inhibits parallel execution when extensions
are used that do CPU intensive calculations during post-transformation.

Activating this feature flag changes the logic as follows:

Expand All @@ -826,27 +828,32 @@ General configuration
sub process: pickle and return certain Builder attributes
main process: merge attributes back to main process builder
This effectively resolves the main process bottleneck as post-transformations run in parallel now.
The expected core logic for doctrees of ``post-transform > write_doc_serialized > write_doc`` is
still intact. The approach can however lead to issues in case extensions write to the environment or the builder
during the post-transformation phase or in ``write_doc_serialized`` and expect that information to be
available after the subprocess has ended. Each subprocess has a completely separated memory space and it is
lost when the process ends. For Sphinx builders and also custom builders, specific attributes can be
returned to the main process. See below note for details.
This effectively resolves the main process bottleneck as post-transformations
run in parallel now. The expected core logic for doctrees of
``post-transform > write_doc_serialized > write_doc`` is still intact. The
approach can however lead to issues in case extensions write to the environment
or the builder during the post-transformation phase or in
``write_doc_serialized`` and expect that information to be available after the
subprocess has ended. Each subprocess has a completely separated memory space
and it is lost when the process ends. For Sphinx builders and also custom
builders, specific attributes can be returned to the main process.
See below note for details.

.. note::
Be sure all active extensions support parallel post-transformation before using this flag.
Be sure all active extensions support parallel post-transformation before
using this flag.

Extensions writing on :py:class:`sphinx.environment.BuildEnvironment` and expecting the
data to be available at a later build stage (e.g. in :event:`build-finished`)
are *not* supported.
Extensions writing on :py:class:`sphinx.environment.BuildEnvironment` and
expecting the data to be available at a later build stage
(e.g. in :event:`build-finished`) are *not* supported.
For the builder object, a mechanism exists to return data to the main process:
The builder class may implement the attribute
:py:attr:`sphinx.builders.Builder.post_transform_merge_attr` to define a list of attributes to be
returned to the main process after parallel post-transformation and writing.
This data is passed to the builder method
:py:meth:`sphinx.builders.Builder.merge_builder_post_transform` to do the actual merging.
In case this is not enough for any of the active extensions, the feature flag cannot be used.
:py:attr:`sphinx.builders.Builder.post_transform_merge_attr` to define a list
of attributes to be returned to the main process after parallel
post-transformation and writing. This data is passed to the builder method
:py:meth:`sphinx.builders.Builder.merge_builder_post_transform` to do the
actual merging. In case this is not enough for any of the active extensions,
the feature flag cannot be used.

.. versionadded:: 7.3

Expand Down

0 comments on commit 2078aad

Please sign in to comment.