Skip to content
This repository has been archived by the owner on Dec 20, 2024. It is now read-only.

Commit

Permalink
chore: pre-commit run
Browse files Browse the repository at this point in the history
  • Loading branch information
JesperDramsch committed Jun 5, 2024
1 parent 9484eef commit aa64c4c
Show file tree
Hide file tree
Showing 4 changed files with 89 additions and 65 deletions.
101 changes: 61 additions & 40 deletions docs/modules/data_indices.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,100 +2,121 @@
Data Indices
##############

Data routing with a DataIndex system that knows where data is passed based on config entries.
Data routing with a DataIndex system that knows where data is passed
based on config entries.

This documentation aims to first explain the user side (configs), and then goes into detail for developers.
This documentation aims to first explain the user side (configs), and
then goes into detail for developers.

Good news first: You don't need to know the index of data in your Zarr

*************
*******************
Usage Information
*************
*******************

The routing information can be changed in the config in data.

There are currently three types of data with respect to the model:

- Prognostic: Data that is input into and output from the model
- Forcing: Data that informs the model as auxiliary information but isn't output nor fed to the loss
- Diagnostic: Data that is not input into the model, but is produced in the inference. (It's pseudo-diagnostic, as it is fed to the loss during training, so the model is still conditioned on this data.)
- Prognostic: Data that is input into and output from the model

The default in Anemoi Models is that data is prognostic. But data can be switched to forcing or diagnostic by adding the value to the existing config entry:
- Forcing: Data that informs the model as auxiliary information but
isn't output nor fed to the loss

.. code-block:: yaml
- Diagnostic: Data that is not input into the model, but is produced in
the inference. (It's pseudo-diagnostic, as it is fed to the loss
during training, so the model is still conditioned on this data.)

data:
forcing:
- "lsm"
- "sdor"
diagnostic:
- "tp"
- "cp"
The default in Anemoi Models is that data is prognostic. But data can be
switched to forcing or diagnostic by adding the value to the existing
config entry:

.. code:: yaml
data:
forcing:
- "lsm"
- "sdor"
diagnostic:
- "tp"
- "cp"
.. figure:: ../schemas/data_indices.png
:alt: Schematic of IndexCollection with Data Indexing on Data and Model levels.
:align: center

The are two Index-levels:

- Data: The data at "Zarr"-level provided by Anemoi-Datasets
- Model: The "squeezed" tensors with irrelevant parts missing.
- Data: The data at "Zarr"-level provided by Anemoi-Datasets
- Model: The "squeezed" tensors with irrelevant parts missing.

These are both split into two versions:

- Input: The data going into training / model
- Output: The data produced by training / model
- Input: The data going into training / model
- Output: The data produced by training / model

Each of these four indices has four integer Tensors available to slice into the data:
Each of these four indices has four integer Tensors available to slice
into the data:

- full: The indices of all "relevant" data, i.e. prognostic and forcing for the input data.
- diagnostic: The indices for data that is "output only" and not fed to the model.
- prognostic: The indices for data that is in both the input and output for the training and model.
- forcing: The indices for data that is only input into the model but doesn't exist in the forecast state.
- full: The indices of all "relevant" data, i.e. prognostic and forcing
for the input data.
- diagnostic: The indices for data that is "output only" and not fed to
the model.
- prognostic: The indices for data that is in both the input and output
for the training and model.
- forcing: The indices for data that is only input into the model but
doesn't exist in the forecast state.

The data can be accessed via dot notation in such a way that:

.. code-block:: python
.. code:: python
data_indices.<model/data>.<input/output>.<full/diagnostic/prognostic/forcing>
Examples:

.. code:: python
.. code-block:: python
data_indices.data.input.full
data_indices.model.output.diagnostic
data_indices.data.input.full
data_indices.model.output.diagnostic
The name_to_index dictionary is available for each of the four indices, which provides a mapping from variable name to index at that level. These are useful for providing users an interface in the config that does not rely on the knowledge of index-locations in the data. Generally, hard-coded locations are to be avoided.
The name_to_index dictionary is available for each of the four indices,
which provides a mapping from variable name to index at that level.
These are useful for providing users an interface in the config that
does not rely on the knowledge of index-locations in the data.
Generally, hard-coded locations are to be avoided.

*************
*******************
Index Collections
*************
*******************

The Index Collections module provides a collection of indices, which are used to index and slice the data.
The Index Collections module provides a collection of indices, which are
used to index and slice the data.

.. automodule:: anemoi.models.data_indices.collection
:members:
:no-undoc-members:
:show-inheritance:

*************
*******
Index
*************
*******

The Index module provides all data indices and methods for indexing and slicing the data.
The Index module provides all data indices and methods for indexing and
slicing the data.

.. automodule:: anemoi.models.data_indices.index
:members:
:no-undoc-members:
:show-inheritance:

*************
********
Tensor
*************
********

This provides the underlying data structure for the indices. It creates a tensor of the appropriate size and shape, and provides methods for indexing and slicing the data.
This provides the underlying data structure for the indices. It creates
a tensor of the appropriate size and shape, and provides methods for
indexing and slicing the data.

.. automodule:: anemoi.models.data_indices.tensor
:members:
Expand Down
28 changes: 14 additions & 14 deletions docs/modules/layers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Layers
########

********
*********
Mappers
********
*********

.. automodule:: anemoi.models.layers.mapper
:members:
:no-undoc-members:
:show-inheritance:

********
************
Processors
********
************

.. automodule:: anemoi.models.layers.processor
:members:
Expand All @@ -38,45 +38,45 @@
:no-undoc-members:
:show-inheritance:

********
*******
Graph
********
*******

.. automodule:: anemoi.models.layers.graph
:members:
:no-undoc-members:
:show-inheritance:

********
******
Conv
********
******

.. automodule:: anemoi.models.layers.conv
:members:
:no-undoc-members:
:show-inheritance:

********
***********
Attention
********
***********

.. automodule:: anemoi.models.layers.attention
:members:
:no-undoc-members:
:show-inheritance:

********
************************
Multi-Layer Perceptron
********
************************

.. automodule:: anemoi.models.layers.mlp
:members:
:no-undoc-members:
:show-inheritance:

********
*******
Utils
********
*******

.. automodule:: anemoi.models.layers.utils
:members:
Expand Down
7 changes: 4 additions & 3 deletions docs/modules/models.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,12 @@
Models
########

*************
*********************************
Encoder Processor Decoder Model
*************
*********************************

The model defines a graph neural network architecture with configurable encoder, processor, and decoder.
The model defines a graph neural network architecture with configurable
encoder, processor, and decoder.

.. automodule:: anemoi.models.models.encoder_processor_decoder
:members:
Expand Down
18 changes: 10 additions & 8 deletions docs/modules/preprocessing.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,30 +2,32 @@
Preprocessing
###############

The preprocessing module is used to pre- and post-process the data.
The module contains the following classes:
The preprocessing module is used to pre- and post-process the data. The
module contains the following classes:

.. automodule:: anemoi.models.preprocessing
:members:
:no-undoc-members:
:show-inheritance:

***********
************
Normalizer
***********
************

The normalizer module is used to normalize the data. The module contains the following classes:
The normalizer module is used to normalize the data. The module contains
the following classes:

.. automodule:: anemoi.models.preprocessing.normalizer
:members:
:no-undoc-members:
:show-inheritance:

***********
*********
Imputer
***********
*********

The imputer module is used to impute the data. The module contains the following classes:
The imputer module is used to impute the data. The module contains the
following classes:

.. automodule:: anemoi.models.preprocessing.imputer
:members:
Expand Down

0 comments on commit aa64c4c

Please sign in to comment.