From 3290c88ddc1b68cac55c35b55760b45868f2aa42 Mon Sep 17 00:00:00 2001 From: Giovanni Volpe Date: Thu, 16 May 2024 18:15:31 +0200 Subject: [PATCH] Update docs5.ipynb --- tutorials/developers/docs5.ipynb | 1501 ++++++++++++++++++------------ 1 file changed, 905 insertions(+), 596 deletions(-) diff --git a/tutorials/developers/docs5.ipynb b/tutorials/developers/docs5.ipynb index 3ddcd36b..f2a8d7e7 100644 --- a/tutorials/developers/docs5.ipynb +++ b/tutorials/developers/docs5.ipynb @@ -1,604 +1,913 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Deeplay Developer Tutorial\n", - "\n", - "In the following tutorials, we intend to provide a comprehensive guide to the Deeplay library, focused on helping developers contribute to the project. We will cover the following topics:\n", - "- Project file structure\n", - "- Style guide \n", - " - Naming conventions\n", - " - Code formatting\n", - " - Documentation\n", - " - Testing\n", - "- The base classes and knowing what to subclass\n", - "- The methods and attributes that can be overridden to customize the behavior of the base classes\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Project file structure\n", - "\n", - "### Root level files\n", - "\n", - "The project contains the following files at the root level:\n", - "- `LICENSE.txt`: The license file for the project\n", - "- `.pylintrc`: The configuration file for the pylint tool. It contains the rules for code formatting and style, especially the warning to be ignored.\n", - "- `README.md`: The project's README file\n", - "- `requirements.txt`: The file containing the dependencies for the project\n", - "- `setup.cfg`: The configuration file for the setup tool. It contains the metadata for the project.\n", - "- `setup.py`: The setup file for the project. It contains the instructions for installing the project.\n", - "- `stylestubgen.py`: The script for generating the style stubs for the project. These are type hints for the style system. It creates .pyi files for select classes in the project, and adds overrides to the `style` method to enforce the type hints. It also handles the doc strings for the styles in the same way.\n", - "\n", - "### Root level directories\n", - "\n", - "The project contains the following directories at the root level:\n", - "- `.github`: Contains the GitHub actions workflow files for the project. These run the continuous integration tests for the project.\n", - "- `.vscode`: Contains the Visual Studio Code settings for the project. These settings are used to configure the editor for the project. They include good defaults for the editor, such as the code formatter and the linter.\n", - "- `deeplay`: Contains the source code for the project. This is where the main code for the project is located.\n", - "- `tutorials`: Contains the tutorial files for the project. These are jupyter notebooks that provide a comprehensive guide to the Deeplay library, focused on helping users and developers get started with, and make the most of the library. For now, this script should be run by the developer when a new style is added." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Deeplay directory\n", - "\n", - "The deeplay source code is organized in a hierarchical structure. The main focus is ensuring that files only depend on other files in the same or lower (closer to the root) directories. This is to prevent circular dependencies and make the codebase easier to understand. So, for example in the structure:\n", - "``` bash\n", - "a_folder/\n", - " __init__.py\n", - " a.py\n", - " b_folder/\n", - " __init__.py\n", - " b.py\n", - " ...\n", - " c_folder/\n", - " __init__.py\n", - " c.py\n", - " c_extra.py\n", - " ...\n", - " ...\n", - "```\n", - "\n", - "`a.py` can import `b.py` and `c.py`, but `b.py` and `c.py` cannot import `a.py`. Moreover, `b.py` should not import `c.py` or `c_extra.py`. But `c.py` can import `c_extra.py` and vice versa.\n", - "\n", - "This means that the root level files contain the most general classes and functions, while the lower level files contain more specific classes and functions. This makes it easier to understand the codebase and to find the code you are looking for." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## File scope\n", - "\n", - "In general, each file should export a single class or function. This makes it easier to understand the codebase and to find the code you are looking for. If a file exports multiple classes or functions, they should be related to each other and should be used together. If a file exports multiple unrelated classes or functions, it should be split into multiple files. It is better to organize the codebase such that related objects are in the same folder instead of the same file." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Deeplay root level files\n", - "\n", - "Let's quickly overview the root level files.\n", - "\n", - "#### `module.py`\n", - "\n", - "This file contains the `DeeplayModule` class, which is the base class for all modules in the Deeplay library. It also contains the configuration logic and the selection logic.\n", - "\n", - "#### `meta.py`\n", - "\n", - "This file contains the metaclass that all `DeeplayModule` subclasses should use.\n", - "\n", - "#### `list.py`\n", - "\n", - "This file contains list-like classes (most importantly `LayerList` and `Sequential`), which are used as containers for layers, blocks and components in the Deeplay library.\n", - "\n", - "#### `decorators.py`\n", - "\n", - "This file contains the decorators used in the Deeplay library. These are mainly method decorators that are used to modify the behavior of methods in the library to ensure methods are called at the right point in the lifecycle of the object.\n", - "\n", - "#### `trainer.py`\n", - "\n", - "This file contains the `Trainer` class, which is used to train models in the Deeplay library. It extends the lightning `Trainer` class." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Deeplay subdirectories\n", - "\n", - "The `deeplay` directory contains the following subdirectories:\n", - "\n", - "### `activelearning`\n", - "\n", - "This directory contains the classes and functions related to active learning in the Deeplay library. This includes application wrappers, criterion, and dataset classes.\n", - "\n", - "### `applications`\n", - "\n", - "This directory contains the classes and functions related to applications in the Deeplay library. Applications are classes that contain the training logic for specific tasks, such as classification, regression, segmentation, etc. They handle all the details of training a model for a specific task, except for the model architecture.\n", - "\n", - "Generally, the individual applications will be placed in further subdirectories, such as `classification`, `regression`, `segmentation`, etc. However, this is less strict than the root level file structure.\n", - "\n", - "### `blocks`\n", - "\n", - "This directory contains the classes and functions related to blocks in the Deeplay library. Blocks are the building blocks of models in the Deeplay library. They are used to define the architecture of a model, and can be combined to create complex models. The most important block classes are in the subfolders `conv`, `linear`, `sequence` and in the files `base.py` anb `sequential.py`.\n", - "\n", - "### `callbacks`\n", - "\n", - "Contains deeplay specific callbacks. Mainly the logging of the training history and the custom progress bar.\n", - "\n", - "### `components`\n", - "\n", - "Contains the reusable components of the library. These are generally built as a combination of blocks. They are more flexible than full models, but less flexible than blocks.\n", - "\n", - "### `external`\n", - "\n", - "Contains logic for interacting with external classes and object, such as from `torch`. Most important objects are `Layer` and `Optimizer`.\n", - "\n", - "### `initializers`\n", - "\n", - "Contains the classes for initializing the weights of the models.\n", - "\n", - "### `models`\n", - "\n", - "Contains the models of the library. These are the full models that are used for training and inference. They are built from blocks and components, and are less flexible than both. They generally represent a specific architecture, such as `ResNet`, `UNet`, etc. \n", - "\n", - "### `ops`\n", - "\n", - "Contains individual operations that are used in the blocks and components. These are generally low-level, non-trainable operations, such as `Reshape`, `Cat`, etc. They act like individual layers.\n", - "\n", - "### `tests`\n", - "\n", - "Contains the tests for the library. These are used to ensure that the library is working correctly and to catch any bugs that may arise.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Style guide\n", - "\n", - "The code style should follow the [PEP 8](https://www.python.org/dev/peps/pep-0008/) guidelines. The code should be formatted using\n", - "[black](https://black.readthedocs.io/en/stable/). We are not close to lint-compliance yet, but we are working on it.\n", - "\n", - "Use type hints extensively to make the code more readable and maintainable. The type hints should be as specific as possible.\n", - "For example, if a string can be one of several values, use a `Literal` type hint. Similarly, if a function takes a list of integers,\n", - "the type hint should be `List[int]` instead of `List`. We are currently supporting Python 3.8 and above. Some features of Python 3.9\n", - "are not supported yet, such as the `|` operator for type hints. You can get around this by importing `annotations` from `__future__`.\n", - "\n", - "\n", - "Classes should have their attribute types defined before the `__init__` method. An example is shown below:\n", - "\n", - "```python\n", - "class MyClass:\n", - " \"\"\"A class that does something.\"\"\"\n", - " attribute: int\n", - "\n", - " def __init__(self, num: int):\n", - " self.attribute = num\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Naming conventions\n", - "\n", - "Beyond what is defined in the PEP 8 guidelines, we have the following naming conventions:\n", - "\n", - "- Minimize the use of abbreviations. If an abbreviation is used, it should be well-known and not ambiguous.\n", - "- Use the following names:\n", - " - \"layer\" for a class that represents a single layer in a neural network, typically the learnable part of a block.\n", - " - \"activation\" for a class that represents a non-learnable activation function.\n", - " - \"normalization\" for a class that represents a normalization layer.\n", - " - \"dropout\" for a class that represents a dropout layer.\n", - " - \"pool\" for a class that represents a pooling layer.\n", - " - \"upsample\" for a class that represents an upsampling layer.\n", - " - \"block\" / \"blocks\" for a class that represents a block in a neural network, typically a sequence of layers.\n", - " - \"backbone\" for a class that represents the main part of a neural network, typically a sequence of blocks.\n", - " - \"head\" for a class that represents the final part of a neural network, typically a single layer followed by an optional activation function.\n", - " - \"model\" for a class that represents a full neural network architecture.\n", - " - \"optimizer\" for a class that represents an optimizer.\n", - " - \"loss\" for a class that represents a loss function.\n", - " - \"metric\" for a class that represents a metric.\n", - " \n", - "- If there is a naming conflict within a class, add numbers to the end of the name with an underscore, 0-indexed. For example, `layer_0` and `layer_1`.\n", - " - This is correct: `layer_0`, `layer_1`, `layer_2`.\n", - " - This is incorrect: `layer_1`, `layer_2`, `layer_3`.\n", - " - This is incorrect: `layer`, `layer_1`, `layer_3`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Imports\n", - "\n", - "Use absolute imports for all imports, unless the target is at the same level in the hierarchy inside a `__init__.py`.\n", - "\n", - "```python\n", - "# deeplay/external/optimizers/optimizer.py\n", - "from deeplay.module import DeeplayModule # correct\n", - "from ...module import DeeplayModule # incorrect\n", - "\n", - "from deeplay.external.optimizers.adam import Adam # correct\n", - "from .adam import Adam # also allowed\n", - "```\n", - "\n", - "In the `__init__.py` files, you may use * imports from directories, but not from files. From files, you should import the specific classes or functions you need.\n", - "\n", - "```python\n", - "# deeplay/external/__init__.py\n", - "from .optimizers import * # correct\n", - "from .optimizers.adam import * # incorrect\n", - "from .optimizers.adam import Adam # correct \n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Documentation\n", - "\n", - "Documentation should follow the [NumpyDoc style guide](https://numpydoc.readthedocs.io/en/latest/format.html#style-guide).\n", - "\n", - "TODO: add example here\n", - "\n", - "In general, all non-trivial classes and methods should be documented. The documentation should include a description of the class or method, the parameters, the return value, and any exceptions that can be raised. We sincerely appreciate any effort to improve the documentation, particularly by including examples of how to use the classes and methods." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Testing\n", - "\n", - "All new features should be tested. The tests should cover all possible code paths and should be as comprehensive as possible. The tests should be written using the `unittest` module in Python. The tests should be placed in the `tests` folder. The tests should be run using `unittest`. Not all tests follow our guidelines yet, but we are working on improving them.\n", - "\n", - "In general, we aim to mirror the structure of the `deeplay` package in the `tests` package. For example, `deeplay/external/layer.py` should have a corresponding `tests/external/test_layer.py` file. The name of the file should be the same as the module it tests, but with `test_` prepended. Note that when adding a folder, the `__init__.py` file should be added to the folder to make it a package.\n", - "\n", - "Each test file should contain one `unittest.TestCase` class per class or function being tested. The test methods should be as descriptive as possible. For example, `test_forward` is not a good name for a test method. Instead, use `test_forward_with_valid_input` or `test_forward_with_invalid_input`. The test methods should be as independent as possible, but we value coverage over independence.\n", - "\n", - "It is fine to use multiple subtests using `with self.subTest()` to test multiple inputs or edge cases. This is particularly useful when testing a method that has many possible inputs.\n", - "\n", - "It is fine and preferred to use mocking where appropriate. For example, if a method calls an external API, the API call should be mocked. The `unittest.mock` module is very useful for this purpose.\n", - "\n", - "Keep in mind to try to only test the the class or method itself, not the dependencies. For example, testing an application should focus on the training logic it implements, not the specificis of the architecture it uses." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Testing Applications\n", - "\n", - "Applications can be hard to test because they often require a lot of data and time to train to fully validate. This is not feasible, since we want to retain the unit test speed to within a few minutes. Here are some guidelines to follow when testing applications:\n", - "\n", - "- Test both using build and create methods.\n", - " - Validate that the forward pass works as expected.\n", - " - Validate that the loss and metrics are correctly set.\n", - " - Validate that the optimizer is correctly set.\n", - " - This can be done by manually calling `app.configure_optimizer()` and checking that returned optimizer is correct the expected one.\n", - " - Also, verify that the parameters of the optimizer are correctly set. This can\n", - " be done by doing a forward pass, backward pass, and then checking that the optimizer's parameters have been updated.\n", - " - If there are multiple optimizers with different parameters, ensure that the correct parameters are updated.\n", - "- Test `app.compute_loss` on a small input and verify that the loss is correct.\n", - "- Test `app.training_step` on a small input and verify that the loss is correct. Note that you might need to attach a trainer for this.\n", - "- Test training the application on a single epoch both using `app.fit` and `trainer.fit(app)`.\n", - " - It's a good idea to check that the training history contains the correct keys.\n", - " - Use as small a dataset and model as possible to keep the test fast. Turn off checkpointing and logging to speed up the test.\n", - "- Test that the application can be saved and loaded correctly.\n", - " - Currently, we only guarantee `save_state_dict` and `load_state_dict` to work correctly. A good way to test is to save the state dict, create a new application, load the state dict, and then check that the new application has the same state as the old one.\n", - "- Test any application-specific methods, such as `app.detect`, `app.classify`, etc.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Testing Models\n", - "\n", - "Models are easier to test than applications because they are usually smaller and have fewer dependencies. We do not test if models can be trained.\n", - "\n", - "- Test that the model, created with default arguments, can be created.\n", - "- Test that the model, created with default arguments, has the correct number of parameters.\n", - "- Test that the model, created with default arguments, can be saved and loaded using `save_state_dict` and `load_state_dict`.\n", - "- Test that a previously saved model state dict can be loaded into a new model.\n", - "- test that the model, created with default arguments, has the expected hierarchical structure. This is mostly for forward compatibility.\n", - "- Test the forward pass with a small input and verify that the output is correct in terms of tensor shape.\n", - "- Test that the model can be created with non-default arguments and that the arguments are correctly set.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### TODO: more testing guidelines" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# The base classes and knowing what to subclass\n", - "\n", - "The Deeplay library is built around a few base classes that define the structure of the library. These base classes are designed to be subclassed to create new classes that can be used in the library. The base classes are:\n", - "\n", - "- `DeeplayModule`: The base class for all modules in the Deeplay library. This class defines the configuration logic and the selection logic for the modules.\n", - "- `BaseBlock`: The base class for all blocks in the Deeplay library. This class defines the structure of a block and the methods that should be implemented by subclasses.\n", - "- `Application`: The base class for all applications in the Deeplay library. This class defines the structure of an application and the methods that should be implemented by subclasses.\n", - "\n", - "Components and models do not have base classes, and are instead derived from the `DeeplayModule` class. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "\n", - "The following table provides a sequence of questions to help you decide what type of object you are implementing. Each\n", - "possible answer will have its own style guide in the folder of the same name.\n", - "\n", - "| Question | Answer | Object type |\n", - "| ---------------------------------------------------------------------------------------------------------------- | ------ | ------------- |\n", - "| Does the class represent a task such as classification, without depending heavily on the exact architecture? | Yes | `Application` |\n", - "| Does the class require a non-standard training procedure to make sense (standard is normal supervised learning)? | Yes | `Application` |\n", - "| Does the object represent a specific architecture, such as a ResNet18? | Yes | `Model` |\n", - "| Does the object represent a full architecture, not expected to be used as a part of another architecture? | Yes | `Model` |\n", - "| Does the object represent a generic architecture, such as a ConvolutionalNeuralNetwork? | Yes | `Component` |\n", - "| Is the object a small structural object with a sequential forward pass, such as `LayerActivation`? | Yes | `Block` |\n", - "| Is the object a unit of computation, such as `Conv2d`, `Add` etc.? | Yes | `Operation` |\n", - "\n", - "As a general rule of thumb, for Components, the number of features in each layer should be defineable by the input arguments. For Models, only the input and output features must be defineable by the input arguments.\n", - "\n", - "For models and components, it is recommended to subclass an existing model or component if possible. This will make it easier to implement the required methods and attributes, and will ensure that the new model or component is compatible with the rest of the library." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Deeplay Applications\n", - "\n", - "Applications are broadly defined as classes that represent a task such as classification, without depending heavily on the exact architecture. They are the highest level of abstraction in the Deeplay library. Applications are designed to be easy to use and require minimal configuration to get started. They are also designed to be easily extensible, so that you can add new features without having to modify the existing code.\n", - "\n", - "### What's in an application?\n", - "\n", - "As a general rule of thumb, try to minimize the number of models in an application. Best is if there is a single model, accessed as `app.model`. Some applications require more,\n", - "such as `gan.generator` and `gan.discriminator`. This is fine, but try to keep it to a minimum. A bad example would be for a classifier to include `app.conv_backbone`, `app.conv_to_fc_connector` and `app.fc_head`. This is bad because it limits the flexibility of the application to architectures that fit this exact structure. Instead, the application should have a single model that can be easily replaced with a different model.\n", - "\n", - "#### Training\n", - "\n", - "The primary function of an application is to define how it is trained. This includes the loss function, the optimizer, and the metrics that are used to evaluate the model. Applications also define how the model is trained, including the training loop, the validation loop, and the testing loop. Applications are designed to be easy to use, so that you can get started quickly without having to worry about the details of the training process.\n", - "\n", - "The training step is, broadly, defined as follows:\n", - "\n", - "```python\n", - "x, y = self.train_preprocess(batch)\n", - "y_hat = self(x)\n", - "loss = self.compute_loss(y_hat, y)\n", - "# [...] \n", - "# code for logging\n", - "```\n", - "\n", - "If the training can be defined in this way, then you can implement the `train_preprocess`, `compute_loss`, and `forward` methods to define the training process. If the training process is more complex, then you can override the `training_step` method to define the training process. The default behavior of `train_preprocess` is the identity function, and the default behavior of `compute_loss` is to call `self.loss(y_hat, y)`.\n", - "\n", - "`train_preprocess` is intended to apply any operations that cannot be simply defined as a part of the dataset. For example, in some `self-supervised` models, the target is calculated from the input data. This can be done here. It can also be used to ensure the dtype of the input matches the expected dtype of the model. Most likely, you will not need to override this method.\n", - "\n", - "`compute_loss` is intended to calculate the loss of the model. This can be as simple as calling `self.loss(y_hat, y)`, or it can be more complex. It is more likely that you will need to override this method.\n", - "\n", - "If you need to define a custom training loop, you can override the `training_step` method entirely. This method is called for each batch of data during training. It should return the loss for the batch. Note that if you override the `training_step` method, you will to handle the logging of the loss yourself. This is done by calling `self.log('train_loss', loss, ...)` where `...` is any setting you want to pass to the logger (see `lightning.LightningModule.log` for more information).\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Example of implementing a simple application\n", - "\n", - "We'll be implementing a simple application that trains a model to classify images. This is the minimal amount needed to implement an application.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ + "cells": [ { - "data": { - "text/plain": [ - "ImageClassifier(\n", - " (model): Sequential(\n", - " (0): ConvolutionalEncoder2d(\n", - " (blocks): LayerList(\n", - " (0): Conv2dBlock(\n", - " (layer): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (1): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (2): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (3): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): Identity()\n", - " )\n", - " )\n", - " (postprocess): Identity()\n", - " )\n", - " (1): AdaptiveAvgPool2d(output_size=1)\n", - " (2): MultiLayerPerceptron(\n", - " (blocks): LayerList(\n", - " (0): LinearBlock(\n", - " (layer): Linear(in_features=128, out_features=10, bias=True)\n", - " (activation): Identity()\n", - " )\n", - " )\n", - " )\n", - " )\n", - ")" + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deeplay Developer Tutorial\n", + "\n", + "In the following tutorials, we intend to provide a comprehensive guide to the Deeplay library, focused on helping developers contribute to the project. We will cover the following topics:\n", + "- Project file structure\n", + "- Style guide \n", + " - Naming conventions\n", + " - Code formatting\n", + " - Documentation\n", + " - Testing\n", + "- The base classes and knowing what to subclass\n", + "- The methods and attributes that can be overridden to customize the behavior of the base classes\n" ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import deeplay as dl\n", - "import torch.nn as nn\n", - "\n", - "net = dl.Sequential(\n", - " dl.ConvolutionalEncoder2d(1, [16, 32, 64], 128),\n", - " dl.Layer(nn.AdaptiveAvgPool2d, 1),\n", - " dl.MultiLayerPerceptron(128, [], 10)\n", - ")\n", - "\n", - "class ImageClassifier(dl.Application):\n", - "\n", - " model: nn.Module\n", - "\n", - " def __init__(self, model: nn.Module):\n", - " super().__init__()\n", - " self.model = model\n", - "\n", - " def forward(self, x):\n", - " return self.model(x)\n", - "\n", - "classifier = ImageClassifier(net).create()\n", - "classifier" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Next, we allow the user to set the optimizer. Note that we are using the `create_optimizer_with_params` method to create the optimizer. We also use Adam as the default optimizer. It is better to set the default value of the optimizer to `None` and then set it to Adam in the `__init__` method. This is because the optimizer is a mutable object, and setting it to a default value of `Adam()` will cause all instances of the class to share the same optimizer object." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Project file structure\n", + "\n", + "### Root level files\n", + "\n", + "The project contains the following files at the root level:\n", + "- `LICENSE.txt`: The license file for the project\n", + "- `.pylintrc`: The configuration file for the pylint tool. It contains the rules for code formatting and style, especially the warning to be ignored.\n", + "- `README.md`: The project's README file\n", + "- `requirements.txt`: The file containing the dependencies for the project\n", + "- `setup.cfg`: The configuration file for the setup tool. It contains the metadata for the project.\n", + "- `setup.py`: The setup file for the project. It contains the instructions for installing the project.\n", + "- `stylestubgen.py`: The script for generating the style stubs for the project. These are type hints for the style system. It creates .pyi files for select classes in the project, and adds overrides to the `style` method to enforce the type hints. It also handles the doc strings for the styles in the same way.\n", + "\n", + "### Root level directories\n", + "\n", + "The project contains the following directories at the root level:\n", + "- `.github`: Contains the GitHub actions workflow files for the project. These run the continuous integration tests for the project.\n", + "- `.vscode`: Contains the Visual Studio Code settings for the project. These settings are used to configure the editor for the project. They include good defaults for the editor, such as the code formatter and the linter.\n", + "- `deeplay`: Contains the source code for the project. This is where the main code for the project is located.\n", + "- `tutorials`: Contains the tutorial files for the project. These are jupyter notebooks that provide a comprehensive guide to the Deeplay library, focused on helping users and developers get started with, and make the most of the library. For now, this script should be run by the developer when a new style is added." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Deeplay directory\n", + "\n", + "The deeplay source code is organized in a hierarchical structure. The main focus is ensuring that files only depend on other files in the same or lower (closer to the root) directories. This is to prevent circular dependencies and make the codebase easier to understand. So, for example in the structure:\n", + "``` bash\n", + "a_folder/\n", + " __init__.py\n", + " a.py\n", + " b_folder/\n", + " __init__.py\n", + " b.py\n", + " ...\n", + " c_folder/\n", + " __init__.py\n", + " c.py\n", + " c_extra.py\n", + " ...\n", + " ...\n", + "```\n", + "\n", + "`a.py` can import `b.py` and `c.py`, but `b.py` and `c.py` cannot import `a.py`. Moreover, `b.py` should not import `c.py` or `c_extra.py`. But `c.py` can import `c_extra.py` and vice versa.\n", + "\n", + "This means that the root level files contain the most general classes and functions, while the lower level files contain more specific classes and functions. This makes it easier to understand the codebase and to find the code you are looking for." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## File scope\n", + "\n", + "In general, each file should export a single class or function. This makes it easier to understand the codebase and to find the code you are looking for. If a file exports multiple classes or functions, they should be related to each other and should be used together. If a file exports multiple unrelated classes or functions, it should be split into multiple files. It is better to organize the codebase such that related objects are in the same folder instead of the same file." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deeplay root level files\n", + "\n", + "Let's quickly overview the root level files.\n", + "\n", + "#### `module.py`\n", + "\n", + "This file contains the `DeeplayModule` class, which is the base class for all modules in the Deeplay library. It also contains the configuration logic and the selection logic.\n", + "\n", + "#### `meta.py`\n", + "\n", + "This file contains the metaclass that all `DeeplayModule` subclasses should use.\n", + "\n", + "#### `list.py`\n", + "\n", + "This file contains list-like classes (most importantly `LayerList` and `Sequential`), which are used as containers for layers, blocks and components in the Deeplay library.\n", + "\n", + "#### `decorators.py`\n", + "\n", + "This file contains the decorators used in the Deeplay library. These are mainly method decorators that are used to modify the behavior of methods in the library to ensure methods are called at the right point in the lifecycle of the object.\n", + "\n", + "#### `trainer.py`\n", + "\n", + "This file contains the `Trainer` class, which is used to train models in the Deeplay library. It extends the lightning `Trainer` class." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Deeplay subdirectories\n", + "\n", + "The `deeplay` directory contains the following subdirectories:\n", + "\n", + "### `activelearning`\n", + "\n", + "This directory contains the classes and functions related to active learning in the Deeplay library. This includes application wrappers, criterion, and dataset classes.\n", + "\n", + "### `applications`\n", + "\n", + "This directory contains the classes and functions related to applications in the Deeplay library. Applications are classes that contain the training logic for specific tasks, such as classification, regression, segmentation, etc. They handle all the details of training a model for a specific task, except for the model architecture.\n", + "\n", + "Generally, the individual applications will be placed in further subdirectories, such as `classification`, `regression`, `segmentation`, etc. However, this is less strict than the root level file structure.\n", + "\n", + "### `blocks`\n", + "\n", + "This directory contains the classes and functions related to blocks in the Deeplay library. Blocks are the building blocks of models in the Deeplay library. They are used to define the architecture of a model, and can be combined to create complex models. The most important block classes are in the subfolders `conv`, `linear`, `sequence` and in the files `base.py` anb `sequential.py`.\n", + "\n", + "### `callbacks`\n", + "\n", + "Contains deeplay specific callbacks. Mainly the logging of the training history and the custom progress bar.\n", + "\n", + "### `components`\n", + "\n", + "Contains the reusable components of the library. These are generally built as a combination of blocks. They are more flexible than full models, but less flexible than blocks.\n", + "\n", + "### `external`\n", + "\n", + "Contains logic for interacting with external classes and object, such as from `torch`. Most important objects are `Layer` and `Optimizer`.\n", + "\n", + "### `initializers`\n", + "\n", + "Contains the classes for initializing the weights of the models.\n", + "\n", + "### `models`\n", + "\n", + "Contains the models of the library. These are the full models that are used for training and inference. They are built from blocks and components, and are less flexible than both. They generally represent a specific architecture, such as `ResNet`, `UNet`, etc. \n", + "\n", + "### `ops`\n", + "\n", + "Contains individual operations that are used in the blocks and components. These are generally low-level, non-trainable operations, such as `Reshape`, `Cat`, etc. They act like individual layers.\n", + "\n", + "### `tests`\n", + "\n", + "Contains the tests for the library. These are used to ensure that the library is working correctly and to catch any bugs that may arise.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Style guide\n", + "\n", + "The code style should follow the [PEP 8](https://www.python.org/dev/peps/pep-0008/) guidelines. The code should be formatted using\n", + "[black](https://black.readthedocs.io/en/stable/). We are not close to lint-compliance yet, but we are working on it.\n", + "\n", + "Use type hints extensively to make the code more readable and maintainable. The type hints should be as specific as possible.\n", + "For example, if a string can be one of several values, use a `Literal` type hint. Similarly, if a function takes a list of integers,\n", + "the type hint should be `List[int]` instead of `List`. We are currently supporting Python 3.8 and above. Some features of Python 3.9\n", + "are not supported yet, such as the `|` operator for type hints. You can get around this by importing `annotations` from `__future__`.\n", + "\n", + "\n", + "Classes should have their attribute types defined before the `__init__` method. An example is shown below:\n", + "\n", + "```python\n", + "class MyClass:\n", + " \"\"\"A class that does something.\"\"\"\n", + " attribute: int\n", + "\n", + " def __init__(self, num: int):\n", + " self.attribute = num\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Naming conventions\n", + "\n", + "Beyond what is defined in the PEP 8 guidelines, we have the following naming conventions:\n", + "\n", + "- Minimize the use of abbreviations. If an abbreviation is used, it should be well-known and not ambiguous.\n", + "- Use the following names:\n", + " - \"layer\" for a class that represents a single layer in a neural network, typically the learnable part of a block.\n", + " - \"activation\" for a class that represents a non-learnable activation function.\n", + " - \"normalization\" for a class that represents a normalization layer.\n", + " - \"dropout\" for a class that represents a dropout layer.\n", + " - \"pool\" for a class that represents a pooling layer.\n", + " - \"upsample\" for a class that represents an upsampling layer.\n", + " - \"block\" / \"blocks\" for a class that represents a block in a neural network, typically a sequence of layers.\n", + " - \"backbone\" for a class that represents the main part of a neural network, typically a sequence of blocks.\n", + " - \"head\" for a class that represents the final part of a neural network, typically a single layer followed by an optional activation function.\n", + " - \"model\" for a class that represents a full neural network architecture.\n", + " - \"optimizer\" for a class that represents an optimizer.\n", + " - \"loss\" for a class that represents a loss function.\n", + " - \"metric\" for a class that represents a metric.\n", + " \n", + "- If there is a naming conflict within a class, add numbers to the end of the name with an underscore, 0-indexed. For example, `layer_0` and `layer_1`.\n", + " - This is correct: `layer_0`, `layer_1`, `layer_2`.\n", + " - This is incorrect: `layer_1`, `layer_2`, `layer_3`.\n", + " - This is incorrect: `layer`, `layer_1`, `layer_3`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Imports\n", + "\n", + "Use absolute imports for all imports, unless the target is at the same level in the hierarchy inside a `__init__.py`.\n", + "\n", + "```python\n", + "# deeplay/external/optimizers/optimizer.py\n", + "from deeplay.module import DeeplayModule # correct\n", + "from ...module import DeeplayModule # incorrect\n", + "\n", + "from deeplay.external.optimizers.adam import Adam # correct\n", + "from .adam import Adam # also allowed\n", + "```\n", + "\n", + "In the `__init__.py` files, you may use * imports from directories, but not from files. From files, you should import the specific classes or functions you need.\n", + "\n", + "```python\n", + "# deeplay/external/__init__.py\n", + "from .optimizers import * # correct\n", + "from .optimizers.adam import * # incorrect\n", + "from .optimizers.adam import Adam # correct \n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Documentation\n", + "\n", + "Documentation should follow the [NumpyDoc style guide](https://numpydoc.readthedocs.io/en/latest/format.html#style-guide).\n", + "\n", + "In general, all non-trivial classes and methods should be documented. The documentation should include a description of the class or method, the parameters, the return value, and any exceptions that can be raised. We sincerely appreciate any effort to improve the documentation, particularly by including examples of how to use the classes and methods." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example function or method documentation\n", + "\n", + "Following is an example of how a function or a method should be documented:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from typing import List\n", + "\n", + "def my_function(param1: int, param2: str) -> List[int]:\n", + " \"\"\"This is a short description of the function on one line.\n", + "\n", + " This is a longer description of the function. It should explain what the \n", + " function does and how it works.\n", + "\n", + " Parameters\n", + " ----------\n", + " param1 : int\n", + " This is a description of the first parameter.\n", + " param2 : str\n", + " This is a description of the second parameter.\n", + "\n", + " Returns\n", + " -------\n", + " list of int\n", + " This is a description of the return value.\n", + "\n", + " Examples\n", + " --------\n", + " >>> my_function(1, 'a')\n", + " [1, 2, 3]\n", + "\n", + " Raises\n", + " ------\n", + " ValueError\n", + " This is a description of the exception that can be raised.\n", + " \"\"\"\n", + " ..." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here is a more concrete example of how a function should be documented:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "from typing import Union, Tuple\n", + "\n", + "def predict(\n", + " self, x, *args, batch_size=32, device=None, output_device=None\n", + " ) -> Union[torch.Tensor, Tuple[torch.Tensor, ...]]:\n", + " \"\"\"Predicts the output of the module for the given input.\n", + "\n", + " This method is a wrapper around the `forward` method, which is used to \n", + " predict the output of the module for the given input. It is \n", + " particularly useful for making predictions on large datasets, as it \n", + " allows for the specification of a batch size for processing the input \n", + " data.\n", + "\n", + " Parameters\n", + " ----------\n", + " x : array-like\n", + " The input data for which to predict the output. Should be an \n", + " array-like object with the same length as the input data.\n", + " *args : Any\n", + " Positional arguments for the input data. Should have the same \n", + " length as the input data.\n", + " batch_size : int, optional\n", + " The batch size for processing the input data. Defaults to 32.\n", + " device : str, Device, optional\n", + " The device on which to perform the prediction. If None, the model's\n", + " device is used.\n", + " Defaults to None.\n", + " output_device : str, Device, optional\n", + " The device on which to store the output. If None, the model's \n", + " device is used. Defaults to None.\n", + "\n", + " Returns\n", + " -------\n", + " Tensor or tuple of Tensor\n", + " The output of the module for the given input data.\n", + " Will match the output of the `forward` method. If the output is a single tensor, \n", + " it is returned as is. If the output is a tuple of tensors, it is returned as a tuple.\n", + "\n", + " Examples\n", + " --------\n", + " To predict the output of a module for the given input data:\n", + " \n", + " >>> input_data = torch.randn(100, 3, 32, 32)\n", + " >>> module = ConvolutionalNeuralNetwork(3, [], 1).build()\n", + " >>> module.predict(input_data, batch_size=10)\n", + " tensor([0.1, 0.2, 0.3, ...]) \n", + "\n", + " Raises\n", + " ------\n", + " AssertionError\n", + " If there are multiple input testora that have different lengths.\n", + " \n", + " \"\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example class documentation\n", + "\n", + "Following is an example of how a class should be documented:\n", + "\n", + "Beyond the numpydoc style guide, we have the following sections:\n", + "- Input: This section should describe the input to the forward method. It should include the type of the input, the shape of the input, and any constraints on the input.\n", + "- Output: This section should describe the output of the forward method. It should include the type of the output, the shape of the output, and any constraints on the output.\n", + "- Evaluation: This section should describe the forward path of the class. Typically as code, but can also be in prose if the code is too complex." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "from deeplay import DeeplayModule\n", + "\n", + "class ConvolutionalNeuralNetwork(DeeplayModule):\n", + " \"\"\"Convolutional Neural Network (CNN) component.\n", + "\n", + " This component is a convolutional neural network (CNN) that consists of multiple convolutional blocks.\n", + " Per default, there is no pooling applied between the blocks. The output of the last block is not flattened.\n", + " \n", + " The default structure of the CNN is as follows:\n", + "\n", + " 1. Conv2D(in_channels, hidden_channels[0], kernel_size=3, stride=1, padding=1)\n", + " ReLU\n", + " 2. Conv2D(hidden_channels[0], hidden_channels[1], kernel_size=3, stride=1, padding=1)\n", + " ReLU \n", + " ...\n", + " n. Conv2D(hidden_channels[n-1], out_channels, kernel_size=3, stride=1, padding=1)\n", + " out_activation\n", + "\n", + " Parameters\n", + " ----------\n", + " in_channels: int or None\n", + " Number of input features. If None, the input shape is inferred from the first forward pass\n", + " hidden_channels: list[int]\n", + " Number of hidden units in each layer except the last.\n", + " out_channels: int\n", + " Number of output features in the last layer.\n", + " out_activation: Layer or type[nn.Module], optional\n", + " Specification for the output activation of the last block. (Default: nn.Identity)\n", + " pool: template-like\n", + " Specification for the pooling of the block. Is not applied to the first block. (Default: nn.Identity)\n", + " The pooling will be applied before the layer.\n", + " \n", + " Attributes\n", + " ----------\n", + " in_channels: int\n", + " Number of input features.\n", + " hidden_channels: list[int]\n", + " Number of hidden units in each layer except the last.\n", + " out_channels: int\n", + " Number of output features in the last layer.\n", + " blocks: LayerList\n", + " List of blocks in the CNN.\n", + " input: DeeplayModule\n", + " first block in the CNN\n", + " hidden: DeeplayModule\n", + " all blocks except the last\n", + " output: DeeplayModule\n", + " last block in the CNN\n", + " layer: LayerList\n", + " List of layers in the CNN.\n", + " activation: LayerList\n", + " List of activation functions in the CNN.\n", + " normalization: LayerList\n", + " List of normalization functions in the CNN.\n", + " \n", + " Input\n", + " -----\n", + " x: float32\n", + " (Any, in_channels, Any, Any)\n", + " \n", + " Output\n", + " ------\n", + " float32 : (Any, out_channels, Any, Any)\n", + "\n", + " Evaluation\n", + " ----------\n", + " >>> for block in cnn.blocks:\n", + " >>> x = block(x)\n", + " >>> return x\n", + "\n", + " Examples\n", + " --------\n", + " >>> cnn = ConvolutionalNeuralNetwork(3, [16], 1, out_activation=nn.Sigmoid)\n", + " >>> cnn.strided(2, apply_to_first=True) \\\n", + " .normalized(after_last_layer=False) \\\n", + " .build()\n", + " ConvolutionalNeuralNetwork(\n", + " (blocks): LayerList(\n", + " (0): Conv2dBlock(\n", + " (layer): Conv2d(3, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", + " (activation): ReLU()\n", + " (normalization): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n", + " )\n", + " (1): Conv2dBlock(\n", + " (layer): Conv2d(16, 1, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n", + " (activation): Sigmoid()\n", + " )\n", + " )\n", + "\n", + " \"\"\"\n", + " ..." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Example module documentation\n", + "\n", + "Following is an example of how a module should be documented:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "\"\"\"Docstring for the example.py module.\n", + "\n", + "Modules names should have short, all-lowercase names. The module name may\n", + "have underscores if this improves readability.\n", + "\n", + "Every module should have a docstring at the very top of the file. The\n", + "module's docstring may extend over multiple lines. If your docstring does\n", + "extend over multiple lines, the closing three quotation marks must be on\n", + "a line by itself, preferably preceded by a blank line.\n", + "\n", + "It is a good idea to provide an overview of the module here so that someone \n", + "who reads the module's docstring does not have to read the entire file to\n", + "understand what the module does.\n", + "\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Testing\n", + "\n", + "All new features should be tested. The tests should cover all possible code paths and should be as comprehensive as possible. The tests should be written using the `unittest` module in Python. The tests should be placed in the `tests` folder. The tests should be run using `unittest`. Not all tests follow our guidelines yet, but we are working on improving them.\n", + "\n", + "In general, we aim to mirror the structure of the `deeplay` package in the `tests` package. For example, `deeplay/external/layer.py` should have a corresponding `tests/external/test_layer.py` file. The name of the file should be the same as the module it tests, but with `test_` prepended. Note that when adding a folder, the `__init__.py` file should be added to the folder to make it a package.\n", + "\n", + "Each test file should contain one `unittest.TestCase` class per class or function being tested. The test methods should be as descriptive as possible. For example, `test_forward` is not a good name for a test method. Instead, use `test_forward_with_valid_input` or `test_forward_with_invalid_input`. The test methods should be as independent as possible, but we value coverage over independence.\n", + "\n", + "It is fine to use multiple subtests using `with self.subTest()` to test multiple inputs or edge cases. This is particularly useful when testing a method that has many possible inputs.\n", + "\n", + "It is fine and preferred to use mocking where appropriate. For example, if a method calls an external API, the API call should be mocked. The `unittest.mock` module is very useful for this purpose.\n", + "\n", + "Keep in mind to try to only test the the class or method itself, not the dependencies. For example, testing an application should focus on the training logic it implements, not the specificis of the architecture it uses." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Testing Applications\n", + "\n", + "Applications can be hard to test because they often require a lot of data and time to train to fully validate. This is not feasible, since we want to retain the unit test speed to within a few minutes. Here are some guidelines to follow when testing applications:\n", + "\n", + "- Test both using build and create methods.\n", + " - Validate that the forward pass works as expected.\n", + " - Validate that the loss and metrics are correctly set.\n", + " - Validate that the optimizer is correctly set.\n", + " - This can be done by manually calling `app.configure_optimizer()` and checking that returned optimizer is correct the expected one.\n", + " - Also, verify that the parameters of the optimizer are correctly set. This can\n", + " be done by doing a forward pass, backward pass, and then checking that the optimizer's parameters have been updated.\n", + " - If there are multiple optimizers with different parameters, ensure that the correct parameters are updated.\n", + "- Test `app.compute_loss` on a small input and verify that the loss is correct.\n", + "- Test `app.training_step` on a small input and verify that the loss is correct. Note that you might need to attach a trainer for this.\n", + "- Test training the application on a single epoch both using `app.fit` and `trainer.fit(app)`.\n", + " - It's a good idea to check that the training history contains the correct keys.\n", + " - Use as small a dataset and model as possible to keep the test fast. Turn off checkpointing and logging to speed up the test.\n", + "- Test that the application can be saved and loaded correctly.\n", + " - Currently, we only guarantee `save_state_dict` and `load_state_dict` to work correctly. A good way to test is to save the state dict, create a new application, load the state dict, and then check that the new application has the same state as the old one.\n", + "- Test any application-specific methods, such as `app.detect`, `app.classify`, etc.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Testing Models\n", + "\n", + "Models are easier to test than applications because they are usually smaller and have fewer dependencies. We do not test if models can be trained.\n", + "\n", + "- Test that the model, created with default arguments, can be created.\n", + "- Test that the model, created with default arguments, can be reconstructed (by calling `.__construct__()`)\n", + "- Test that the model, created with default arguments, has the correct number of parameters.\n", + "- Test that the model, created with default arguments, can be saved and loaded using `save_state_dict` and `load_state_dict`.\n", + "- Test that a previously saved model state dict can be loaded into a new model.\n", + "- test that the model, created with default arguments, has the expected hierarchical structure. This is mostly for forward compatibility.\n", + "- Test the forward pass with a small input and verify that the output is correct in terms of tensor shape.\n", + "- Test that the model can be created with non-default arguments and that the arguments are correctly set.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Testing Components\n", + "\n", + "Components are similar to models in terms of testing. They are generally smaller and have fewer dependencies. We do not test if components can be trained.\n", + "\n", + "- Test that the component, created with default arguments, can be created.\n", + "- Test that the component, created with default arguments, can be reconstructed (by calling `.__construct__()`)\n", + "- Test that the component, created with default arguments, has the correct number of parameters.\n", + "- Test that the component, created with default arguments, can be saved and loaded using `save_state_dict` and `load_state_dict`.\n", + "- Test the forward pass with a small input and verify that the output is correct in terms of tensor shape.\n", + "- Test that the component can be created with non-default arguments and that the arguments are correctly set." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Testing Blocks\n", + "\n", + "Blocks are the most complex objects to test, as they are the building blocks of the library. They are generally larger and have more dependencies than models and components. We do not test if blocks can be trained.\n", + "\n", + "- Test that the block, created with default arguments, can be created.\n", + "- Test that the block, created with default arguments, can be reconstructed (by calling `.__construct__()`)\n", + "- Test the various methods of the block (`activated`, `normalized` etc.)\n", + "- Specifically test the `multi` method, ensure that the subblocks have the correct input/output features/channels.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# The base classes and knowing what to subclass\n", + "\n", + "The Deeplay library is built around a few base classes that define the structure of the library. These base classes are designed to be subclassed to create new classes that can be used in the library. The base classes are:\n", + "\n", + "- `DeeplayModule`: The base class for all modules in the Deeplay library. This class defines the configuration logic and the selection logic for the modules.\n", + "- `BaseBlock`: The base class for all blocks in the Deeplay library. This class defines the structure of a block and the methods that should be implemented by subclasses.\n", + "- `Application`: The base class for all applications in the Deeplay library. This class defines the structure of an application and the methods that should be implemented by subclasses.\n", + "\n", + "Components and models do not have base classes, and are instead derived from the `DeeplayModule` class. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "The following table provides a sequence of questions to help you decide what type of object you are implementing. Each\n", + "possible answer will have its own style guide in the folder of the same name.\n", + "\n", + "| Question | Answer | Object type |\n", + "| ---------------------------------------------------------------------------------------------------------------- | ------ | ------------- |\n", + "| Does the class represent a task such as classification, without depending heavily on the exact architecture? | Yes | `Application` |\n", + "| Does the class require a non-standard training procedure to make sense (standard is normal supervised learning)? | Yes | `Application` |\n", + "| Does the object represent a specific architecture, such as a ResNet18? | Yes | `Model` |\n", + "| Does the object represent a full architecture, not expected to be used as a part of another architecture? | Yes | `Model` |\n", + "| Does the object represent a generic architecture, such as a ConvolutionalNeuralNetwork? | Yes | `Component` |\n", + "| Is the object a small structural object with a sequential forward pass, such as `LayerActivation`? | Yes | `Block` |\n", + "| Is the object a unit of computation, such as `Conv2d`, `Add` etc.? | Yes | `Operation` |\n", + "\n", + "As a general rule of thumb, for Components, the number of features in each layer should be defineable by the input arguments. For Models, only the input and output features must be defineable by the input arguments.\n", + "\n", + "For models and components, it is recommended to subclass an existing model or component if possible. This will make it easier to implement the required methods and attributes, and will ensure that the new model or component is compatible with the rest of the library." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deeplay Applications\n", + "\n", + "Applications are broadly defined as classes that represent a task such as classification, without depending heavily on the exact architecture. They are the highest level of abstraction in the Deeplay library. Applications are designed to be easy to use and require minimal configuration to get started. They are also designed to be easily extensible, so that you can add new features without having to modify the existing code.\n", + "\n", + "## What's in an application?\n", + "\n", + "As a general rule of thumb, try to minimize the number of models in an application. Best is if there is a single model, accessed as `app.model`. Some applications require more,\n", + "such as `gan.generator` and `gan.discriminator`. This is fine, but try to keep it to a minimum. A bad example would be for a classifier to include `app.conv_backbone`, `app.conv_to_fc_connector` and `app.fc_head`. This is bad because it limits the flexibility of the application to architectures that fit this exact structure. Instead, the application should have a single model that can be easily replaced with a different model.\n", + "\n", + "### Training\n", + "\n", + "The primary function of an application is to define how it is trained. This includes the loss function, the optimizer, and the metrics that are used to evaluate the model. Applications also define how the model is trained, including the training loop, the validation loop, and the testing loop. Applications are designed to be easy to use, so that you can get started quickly without having to worry about the details of the training process.\n", + "\n", + "The training step is, broadly, defined as follows:\n", + "\n", + "```python\n", + "x, y = self.train_preprocess(batch)\n", + "y_hat = self(x)\n", + "loss = self.compute_loss(y_hat, y)\n", + "# logging\n", + "```\n", + "\n", + "If the training can be defined in this way, then you can implement the `train_preprocess`, `compute_loss`, and `forward` methods to define the training process. If the training process is more complex, then you can override the `training_step` method to define the training process. The default behavior of `train_preprocess` is the identity function, and the default behavior of `compute_loss` is to call `self.loss(y_hat, y)`.\n", + "\n", + "`train_preprocess` is intended to apply any operations that cannot be simply defined as a part of the dataset. For example, in some self-supervised models, the target is calculated from the input data. This can be done here. It can also be used to ensure the dtype of the input matches the expected dtype of the model. Most likely, you will not need to override this method.\n", + "\n", + "`compute_loss` is intended to calculate the loss of the model. This can be as simple as calling `self.loss(y_hat, y)`, or it can be more complex. It is more likely that you will need to override this method.\n", + "\n", + "If you need to define a custom training loop, you can override the `training_step` method entirely. This method is called for each batch of data during training. It should return the loss for the batch. Note that if you override the `training_step` method, you will to handle the logging of the loss yourself. This is done by calling `self.log('train_loss', loss, ...)` where `...` is any setting you want to pass to the logger (see `lightning.LightningModule.log` for more information).\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deeplay Models\n", + "\n", + "Models are broadly defined as classes that represent a specific architecture, such as a ResNet18. Unlike `components`, they are\n", + "generally not as flexible from input arguments, and it should be possible to pass them directly to applications. Models are designed to be\n", + "easy to use and require minimal configuration to get started. They are also designed to be easily extensible, so that you can add new\n", + "features without having to modify the existing code.\n", + "\n", + "## What's in a model?\n", + "\n", + "Generally, a model should define a `__init__` method that takes all the necessary arguments to define the model and a `forward` method that\n", + "defines the forward pass of the model.\n", + "\n", + "Optimally, a model should have an as simple forward pass as possible. A fully sequential forward pass is optimal.\n", + "This is because any hard coded structure in the forward pass limits the flexibility of the model. For example, if the forward pass is defined as\n", + "`self.conv1(x) + self.conv2(x)`, then it is not possible to replace `self.conv1` and `self.conv2` with a single `self.conv` without modifying the\n", + "model.\n", + "\n", + "Moreover, the model architecture should in almost all cases be defined purely out of components and operations. Try to limit direct calls to\n", + "`torch.nn` modules and `blocks`. This is because the `torch.nn` modules are not as flexible as the components and operations in Deeplay. If\n", + "components do not exist for the desired architecture, then it is a good idea to create a new component and add it to the `components` folder.\n", + " \n", + "### Unknown tensor sizes\n", + "\n", + "Tensorflow, and by extension Keras, allows for unknown tensor sizes thanks to the graph structure. This is not possible in PyTorch.\n", + "If you need to support unknown tensor sizes, you can use the `lazy` module. This module allows for unknown tensor sizes by delaying the\n", + "construction of the model until the first forward pass. This is not optimal, so use it sparingly. Examples are `nn.LazyConv2d` and `nn.LazyLinear`.\n", + "\n", + "If a model requires unknown tensor sizes, it is heavily encouraged to define the `validate_after_build` method, which should call the forward\n", + "pass with a small input to validate that the model can be built. This will instantiate the lazy modules directly, allowing for a more\n", + "user-friendly experience.\n" + ] + }, { - "data": { - "text/plain": [ - "ImageClassifier(\n", - " (model): Sequential(\n", - " (0): ConvolutionalEncoder2d(\n", - " (blocks): LayerList(\n", - " (0): Conv2dBlock(\n", - " (layer): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (1): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (2): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): ReLU()\n", - " )\n", - " (3): Conv2dBlock(\n", - " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", - " (layer): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", - " (activation): Identity()\n", - " )\n", - " )\n", - " (postprocess): Identity()\n", - " )\n", - " (1): AdaptiveAvgPool2d(output_size=1)\n", - " (2): MultiLayerPerceptron(\n", - " (blocks): LayerList(\n", - " (0): LinearBlock(\n", - " (layer): Linear(in_features=128, out_features=10, bias=True)\n", - " (activation): Identity()\n", - " )\n", - " )\n", - " )\n", - " )\n", - " (optimizer): Adam[Adam](lr=0.001)\n", - ")" + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ImageClassifier(\n", + " (model): Sequential(\n", + " (0): ConvolutionalEncoder2d(\n", + " (blocks): LayerList(\n", + " (0): Conv2dBlock(\n", + " (layer): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (1): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (2): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (3): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): Identity()\n", + " )\n", + " )\n", + " (postprocess): Identity()\n", + " )\n", + " (1): AdaptiveAvgPool2d(output_size=1)\n", + " (2): MultiLayerPerceptron(\n", + " (blocks): LayerList(\n", + " (0): LinearBlock(\n", + " (layer): Linear(in_features=128, out_features=10, bias=True)\n", + " (activation): Identity()\n", + " )\n", + " )\n", + " )\n", + " )\n", + ")" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import deeplay as dl\n", + "import torch.nn as nn\n", + "\n", + "net = dl.Sequential(\n", + " dl.ConvolutionalEncoder2d(1, [16, 32, 64], 128),\n", + " dl.Layer(nn.AdaptiveAvgPool2d, 1),\n", + " dl.MultiLayerPerceptron(128, [], 10)\n", + ")\n", + "\n", + "class ImageClassifier(dl.Application):\n", + "\n", + " model: nn.Module\n", + "\n", + " def __init__(self, model: nn.Module):\n", + " super().__init__()\n", + " self.model = model\n", + "\n", + " def forward(self, x):\n", + " return self.model(x)\n", + "\n", + "classifier = ImageClassifier(net).create()\n", + "classifier" ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, we allow the user to set the optimizer. Note that we are using the `create_optimizer_with_params` method to create the optimizer. We also use Adam as the default optimizer. It is better to set the default value of the optimizer to `None` and then set it to Adam in the `__init__` method. This is because the optimizer is a mutable object, and setting it to a default value of `Adam()` will cause all instances of the class to share the same optimizer object." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "ImageClassifier(\n", + " (model): Sequential(\n", + " (0): ConvolutionalEncoder2d(\n", + " (blocks): LayerList(\n", + " (0): Conv2dBlock(\n", + " (layer): Conv2d(1, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (1): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (2): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): ReLU()\n", + " )\n", + " (3): Conv2dBlock(\n", + " (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n", + " (layer): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n", + " (activation): Identity()\n", + " )\n", + " )\n", + " (postprocess): Identity()\n", + " )\n", + " (1): AdaptiveAvgPool2d(output_size=1)\n", + " (2): MultiLayerPerceptron(\n", + " (blocks): LayerList(\n", + " (0): LinearBlock(\n", + " (layer): Linear(in_features=128, out_features=10, bias=True)\n", + " (activation): Identity()\n", + " )\n", + " )\n", + " )\n", + " )\n", + " (optimizer): Adam[Adam](lr=0.001)\n", + ")" + ] + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from typing import Optional\n", + "\n", + "class ImageClassifier(dl.Application):\n", + "\n", + " model: nn.Module\n", + " optimizer: dl.Optimizer\n", + "\n", + " def __init__(self, model: nn.Module, optimizer: Optional[dl.Optimizer] = None):\n", + " super().__init__()\n", + " self.model = model\n", + " self.optimizer = optimizer or dl.Adam(lr=0.001)\n", + "\n", + " def forward(self, x):\n", + " return self.model(x)\n", + "\n", + " def configure_optimizers(self):\n", + " return self.create_optimizer_with_params(self.optimizer, self.parameters())\n", + " \n", + "classifier = ImageClassifier(net).create()\n", + "classifier" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.7" } - ], - "source": [ - "from typing import Optional\n", - "\n", - "class ImageClassifier(dl.Application):\n", - "\n", - " model: nn.Module\n", - " optimizer: dl.Optimizer\n", - "\n", - " def __init__(self, model: nn.Module, optimizer: Optional[dl.Optimizer] = None):\n", - " super().__init__()\n", - " self.model = model\n", - " self.optimizer = optimizer or dl.Adam(lr=0.001)\n", - "\n", - " def forward(self, x):\n", - " return self.model(x)\n", - "\n", - " def configure_optimizers(self):\n", - " return self.create_optimizer_with_params(self.optimizer, self.parameters())\n", - " \n", - "classifier = ImageClassifier(net).create()\n", - "classifier" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.10" - } - }, - "nbformat": 4, - "nbformat_minor": 2 + "nbformat": 4, + "nbformat_minor": 2 }