From aa64c4c9372b5d0c1a19dfac623dfb44b31f935b Mon Sep 17 00:00:00 2001 From: Jesper Dramsch Date: Wed, 5 Jun 2024 09:31:04 +0000 Subject: [PATCH] chore: pre-commit run --- docs/modules/data_indices.rst | 101 ++++++++++++++++++++------------- docs/modules/layers.rst | 28 ++++----- docs/modules/models.rst | 7 ++- docs/modules/preprocessing.rst | 18 +++--- 4 files changed, 89 insertions(+), 65 deletions(-) diff --git a/docs/modules/data_indices.rst b/docs/modules/data_indices.rst index 59605ee5..8fbff4d0 100644 --- a/docs/modules/data_indices.rst +++ b/docs/modules/data_indices.rst @@ -2,35 +2,44 @@ Data Indices ############## -Data routing with a DataIndex system that knows where data is passed based on config entries. +Data routing with a DataIndex system that knows where data is passed +based on config entries. -This documentation aims to first explain the user side (configs), and then goes into detail for developers. +This documentation aims to first explain the user side (configs), and +then goes into detail for developers. Good news first: You don't need to know the index of data in your Zarr -************* +******************* Usage Information -************* +******************* The routing information can be changed in the config in data. There are currently three types of data with respect to the model: -- Prognostic: Data that is input into and output from the model -- Forcing: Data that informs the model as auxiliary information but isn't output nor fed to the loss -- Diagnostic: Data that is not input into the model, but is produced in the inference. (It's pseudo-diagnostic, as it is fed to the loss during training, so the model is still conditioned on this data.) +- Prognostic: Data that is input into and output from the model -The default in Anemoi Models is that data is prognostic. But data can be switched to forcing or diagnostic by adding the value to the existing config entry: +- Forcing: Data that informs the model as auxiliary information but + isn't output nor fed to the loss -.. code-block:: yaml +- Diagnostic: Data that is not input into the model, but is produced in + the inference. (It's pseudo-diagnostic, as it is fed to the loss + during training, so the model is still conditioned on this data.) - data: - forcing: - - "lsm" - - "sdor" - diagnostic: - - "tp" - - "cp" +The default in Anemoi Models is that data is prognostic. But data can be +switched to forcing or diagnostic by adding the value to the existing +config entry: + +.. code:: yaml + + data: + forcing: + - "lsm" + - "sdor" + diagnostic: + - "tp" + - "cp" .. figure:: ../schemas/data_indices.png :alt: Schematic of IndexCollection with Data Indexing on Data and Model levels. @@ -38,64 +47,76 @@ The default in Anemoi Models is that data is prognostic. But data can be switche The are two Index-levels: -- Data: The data at "Zarr"-level provided by Anemoi-Datasets -- Model: The "squeezed" tensors with irrelevant parts missing. +- Data: The data at "Zarr"-level provided by Anemoi-Datasets +- Model: The "squeezed" tensors with irrelevant parts missing. These are both split into two versions: -- Input: The data going into training / model -- Output: The data produced by training / model +- Input: The data going into training / model +- Output: The data produced by training / model -Each of these four indices has four integer Tensors available to slice into the data: +Each of these four indices has four integer Tensors available to slice +into the data: -- full: The indices of all "relevant" data, i.e. prognostic and forcing for the input data. -- diagnostic: The indices for data that is "output only" and not fed to the model. -- prognostic: The indices for data that is in both the input and output for the training and model. -- forcing: The indices for data that is only input into the model but doesn't exist in the forecast state. +- full: The indices of all "relevant" data, i.e. prognostic and forcing + for the input data. +- diagnostic: The indices for data that is "output only" and not fed to + the model. +- prognostic: The indices for data that is in both the input and output + for the training and model. +- forcing: The indices for data that is only input into the model but + doesn't exist in the forecast state. The data can be accessed via dot notation in such a way that: -.. code-block:: python +.. code:: python data_indices... Examples: +.. code:: python -.. code-block:: python - - data_indices.data.input.full - data_indices.model.output.diagnostic + data_indices.data.input.full + data_indices.model.output.diagnostic -The name_to_index dictionary is available for each of the four indices, which provides a mapping from variable name to index at that level. These are useful for providing users an interface in the config that does not rely on the knowledge of index-locations in the data. Generally, hard-coded locations are to be avoided. +The name_to_index dictionary is available for each of the four indices, +which provides a mapping from variable name to index at that level. +These are useful for providing users an interface in the config that +does not rely on the knowledge of index-locations in the data. +Generally, hard-coded locations are to be avoided. -************* +******************* Index Collections -************* +******************* -The Index Collections module provides a collection of indices, which are used to index and slice the data. +The Index Collections module provides a collection of indices, which are +used to index and slice the data. .. automodule:: anemoi.models.data_indices.collection :members: :no-undoc-members: :show-inheritance: -************* +******* Index -************* +******* -The Index module provides all data indices and methods for indexing and slicing the data. +The Index module provides all data indices and methods for indexing and +slicing the data. .. automodule:: anemoi.models.data_indices.index :members: :no-undoc-members: :show-inheritance: -************* +******** Tensor -************* +******** -This provides the underlying data structure for the indices. It creates a tensor of the appropriate size and shape, and provides methods for indexing and slicing the data. +This provides the underlying data structure for the indices. It creates +a tensor of the appropriate size and shape, and provides methods for +indexing and slicing the data. .. automodule:: anemoi.models.data_indices.tensor :members: diff --git a/docs/modules/layers.rst b/docs/modules/layers.rst index 5bb25cb9..ff21b5ab 100644 --- a/docs/modules/layers.rst +++ b/docs/modules/layers.rst @@ -2,18 +2,18 @@ Layers ######## -******** +********* Mappers -******** +********* .. automodule:: anemoi.models.layers.mapper :members: :no-undoc-members: :show-inheritance: -******** +************ Processors -******** +************ .. automodule:: anemoi.models.layers.processor :members: @@ -38,45 +38,45 @@ :no-undoc-members: :show-inheritance: -******** +******* Graph -******** +******* .. automodule:: anemoi.models.layers.graph :members: :no-undoc-members: :show-inheritance: -******** +****** Conv -******** +****** .. automodule:: anemoi.models.layers.conv :members: :no-undoc-members: :show-inheritance: -******** +*********** Attention -******** +*********** .. automodule:: anemoi.models.layers.attention :members: :no-undoc-members: :show-inheritance: -******** +************************ Multi-Layer Perceptron -******** +************************ .. automodule:: anemoi.models.layers.mlp :members: :no-undoc-members: :show-inheritance: -******** +******* Utils -******** +******* .. automodule:: anemoi.models.layers.utils :members: diff --git a/docs/modules/models.rst b/docs/modules/models.rst index 329791fb..392a9d61 100644 --- a/docs/modules/models.rst +++ b/docs/modules/models.rst @@ -2,11 +2,12 @@ Models ######## -************* +********************************* Encoder Processor Decoder Model -************* +********************************* -The model defines a graph neural network architecture with configurable encoder, processor, and decoder. +The model defines a graph neural network architecture with configurable +encoder, processor, and decoder. .. automodule:: anemoi.models.models.encoder_processor_decoder :members: diff --git a/docs/modules/preprocessing.rst b/docs/modules/preprocessing.rst index eb83d9a4..e68898f3 100644 --- a/docs/modules/preprocessing.rst +++ b/docs/modules/preprocessing.rst @@ -2,30 +2,32 @@ Preprocessing ############### -The preprocessing module is used to pre- and post-process the data. -The module contains the following classes: +The preprocessing module is used to pre- and post-process the data. The +module contains the following classes: .. automodule:: anemoi.models.preprocessing :members: :no-undoc-members: :show-inheritance: -*********** +************ Normalizer -*********** +************ -The normalizer module is used to normalize the data. The module contains the following classes: +The normalizer module is used to normalize the data. The module contains +the following classes: .. automodule:: anemoi.models.preprocessing.normalizer :members: :no-undoc-members: :show-inheritance: -*********** +********* Imputer -*********** +********* -The imputer module is used to impute the data. The module contains the following classes: +The imputer module is used to impute the data. The module contains the +following classes: .. automodule:: anemoi.models.preprocessing.imputer :members: