From 8a030d9e50badd8c9564dfe5a3e31e2eef70879f Mon Sep 17 00:00:00 2001 From: Giovanni Volpe Date: Tue, 16 Jul 2024 09:23:47 +0100 Subject: [PATCH] Delete Deeplay Internals.ipynb --- tutorials/developers/Deeplay Internals.ipynb | 372 ------------------- 1 file changed, 372 deletions(-) delete mode 100644 tutorials/developers/Deeplay Internals.ipynb diff --git a/tutorials/developers/Deeplay Internals.ipynb b/tutorials/developers/Deeplay Internals.ipynb deleted file mode 100644 index 4668e72..0000000 --- a/tutorials/developers/Deeplay Internals.ipynb +++ /dev/null @@ -1,372 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Deeplay Internals\n", - "\n", - "This notebook is a deep dive into the internals of the Deeplay library. It is intended for developers who want to understand how the library works and how to extend it.\n", - "\n", - "## DeeplayModule\n", - "\n", - "At the core of deeplay is the `DeeplayModule` class. This class is a subclass of `torch.nn.Module` and is responsible for managing the configurations applied by the user, and how to build the model based on these configurations.\n", - "\n", - "Let's start by understanding the lifecycle of a `DeeplayModule` object. This is managed by the metaclass, `ExtendedConstructorMeta`. This metaclass is responsible for creating the `DeeplayModule` class and managing the configuration of the class. Let's look at the __call__ method of the metaclass.\n", - "\n", - "```python\n", - " def __call__(cls: Type[T], *args, **kwargs) -> T:\n", - " \"\"\"Construct an instance of a class whose metaclass is Meta.\"\"\"\n", - "\n", - " # If the object is being constructed from a checkpoint, we instead\n", - " # load the class from the pickled state and build it using the\n", - " if \"__from_ckpt_application\" in kwargs:\n", - " assert \"__build_args\" in kwargs, \"Missing __build_args in kwargs\"\n", - " assert \"__build_kwargs\" in kwargs, \"Missing __build_kwargs in kwargs\"\n", - "\n", - " _args = kwargs.pop(\"__build_args\")\n", - " _kwargs = kwargs.pop(\"__build_kwargs\")\n", - "\n", - " app = dill.loads(kwargs[\"__from_ckpt_application\"])\n", - " app.build(*_args, **_kwargs)\n", - " return app\n", - "\n", - " # Otherwise, we construct the object as usual\n", - " obj = cls.__new__(cls, *args, **kwargs)\n", - "\n", - " # We store the actual arguments used to construct the object\n", - " object.__setattr__(\n", - " obj,\n", - " \"_actual_init_args\",\n", - " {\n", - " \"args\": args,\n", - " \"kwargs\": kwargs,\n", - " },\n", - " )\n", - " object.__setattr__(obj, \"_config_tape\", [])\n", - " object.__setattr__(obj, \"_is_calling_stateful_method\", False)\n", - "\n", - " # First, we call the __pre_init__ method of the class\n", - " cls.__pre_init__(obj, *args, **kwargs)\n", - "\n", - " # Next, we construct the class. The not_top_level context manager is used to\n", - " # keep track of where in the object hierarchy we currently are.\n", - " with not_top_level(cls, obj):\n", - " obj.__construct__()\n", - " obj.__post_init__()\n", - "\n", - " return obj\n", - "```\n", - "\n", - "### Breaking down the lifecycle\n", - "\n", - "The method is pretty long, so let's break it down into smaller parts.\n", - "\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 1. \n", - "The method first checks if the object is being constructed from a checkpoint. If it is, it loads the object from the checkpoint and returns it.\n", - "```python\n", - "if \"__from_ckpt_application\" in kwargs:\n", - " assert \"__build_args\" in kwargs, \"Missing __build_args in kwargs\"\n", - " assert \"__build_kwargs\" in kwargs, \"Missing __build_kwargs in kwargs\"\n", - "\n", - " _args = kwargs.pop(\"__build_args\")\n", - " _kwargs = kwargs.pop(\"__build_kwargs\")\n", - "\n", - " app = dill.loads(kwargs[\"__from_ckpt_application\"])\n", - " app.build(*_args, **_kwargs)\n", - " return app\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 2. \n", - "Next, it constructs the object as usual. It creates the object using the `__new__` method of the class and sets some internal attributes.\n", - "```python\n", - "obj = cls.__new__(cls, *args, **kwargs)\n", - "object.__setattr__(\n", - " obj,\n", - " \"_actual_init_args\",\n", - " {\n", - " \"args\": args,\n", - " \"kwargs\": kwargs,\n", - " },\n", - ")\n", - "object.__setattr__(obj, \"_config_tape\", [])\n", - "object.__setattr__(obj, \"_is_calling_stateful_method\", False)\n", - "```\n", - "\n", - "These attributes are \n", - "- `_actual_init_args`: The actual arguments used to construct the object. This is used to create new copies of the object.\n", - "- `_config_tape`: A list of configurations applied to the object by the user (more on this later). This is also used to create new copies of the object.\n", - "- `_is_calling_stateful_method`: A flag that is used to check if the object is currently calling a stateful method. This is used to check if something should be added to the _config_tape or not." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 3. \n", - "Next, it calls the `__pre_init__` method of the class. This method is used to perform any pre-initialization steps. For most cases, subclasses do not need to override this method.\n", - "```python\n", - "cls.__pre_init__(obj, *args, **kwargs)\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 4.\n", - "Next, it constructs the object. This is done by calling the `__construct__` method of the object. This method actually calls the `__init__` method of the object and sets up the model. More on the `__construct__` method later.\n", - "Suffice for now to say that this is where deeper initialization of the object happens (i.e. recursively constructing the children of the object).\n", - "\n", - "After constructing the object, it calls the `__post_init__` method of the object. This method is used to perform any post-initialization steps. This does nothing by default.\n", - "\n", - "Note that both the `__pre_init__` and `__post_init__` methods are called within the `not_top_level` context manager. This context manager is used to keep track of where in the object hierarchy we currently are. We'll cover this more later. But, the primary function of this is to help decide the priority of configurations applied to the object. Configurations applied while currently at the top level (as in, called directly by the user) are given higher priority than configurations applied while constructing the object. And the deeper we go, the lower the priority of the configurations.\n", - "\n", - "```python\n", - "with not_top_level(cls, obj):\n", - " obj.__construct__()\n", - " obj.__post_init__()\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### 5.\n", - "Finally, it returns the object." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The main reason this is implemented as a meta class instead of using `__new__` and `__init__` methods is to guarantee we store the exact arguments used to construct the object, not just the arguments passed up through super() calls. This is important for creating new copies of the object. Moreover, the arguments passed to the `__init__` method may not be the same as the arguments passed to the `__new__` method. This is because the configurations applied by the user may change the arguments passed to the `__init__` method between `__pre_init__` and `__construct__` calls." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## The __construct__ method\n", - "\n", - "The construct method is where the actual initialization of the object happens. This is where the `__init__` method of the object is called and the model is set up. The core idea is that the `__construct__` method should restore the state of the object to how it was immediately after the `__pre_init__` method was called, then find the correct arguments to pass to the `__init__` method based on the actual arguments passed to the `__new__` method and the configurations applied by the user.\n", - "\n", - "```python\n", - "def __construct__(self):\n", - " with not_top_level(ExtendedConstructorMeta, self):\n", - " # Reset construction\n", - " self._modules.clear()\n", - " self._user_config.remove_derived_configurations(self.tags)\n", - "\n", - " self.is_constructing = True\n", - "\n", - " args, kwargs = self.get_init_args()\n", - " getattr(self, self._init_method)(*(args + self._args), **kwargs)\n", - "\n", - " self._run_hooks(\"after_init\")\n", - " self.is_constructing = False\n", - " self.__post_init__()\n", - "```\n", - "\n", - "Let's go line by line.\n", - "\n", - "```python\t\n", - "with not_top_level(ExtendedConstructorMeta, self):\n", - "```\n", - "\n", - "This is the same `not_top_level` context manager we saw earlier. This is used to keep track of where in the object hierarchy we currently are.\n", - "\n", - "```python\n", - "self._modules.clear()\n", - "```\n", - "this removes any children of the object. These will only be added during the `__init__` method, so\n", - "they should always be removed.\n", - "\n", - "```python\n", - "self._user_config.remove_derived_configurations(self.tags)\n", - "```\n", - "\n", - "Here we encounter two new terms. `derived` and `tags`. \n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Tags\n", - "Tags are tuples of strings used to identify a module in the hierarchy. These generally correspond to the names of the modules in the hierarchy. For example, (\"block\", \"layer\") would correspond to a module named `block.layer`. A module can have multiple tags if it exists in multiple places. Tags are used to identify the module in the hierarchy and to apply configurations to the module. It's important to refer to modules by their tags instead of them as objects, since the module may be cleared and re-initialized multiple times during the lifecycle of the object. \n", - "\n", - "Tags are always relative to the root module (which we have yet to encounter). The root module is the base of the hierarchy and is the only module that is not a child of any other module. A module may exist in multiple places in the hierarchy, but must always have the same root module. Every `DeeplayModule` object keeps track of the current root." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Derived Configurations\n", - "\n", - "Derived configurations are configurations not explicitly applied by the user. For example,\n", - "if the `__init__` method of a module calls `self.child.configure(\"foo\", 1)`, then the configuration \"foo\" is derived. This is because the user did not explicitly apply the configuration, but it was applied by the module itself. Since the configuration is applied during the `__init__` method, it should be removed before the `__init__` method is called again.\n", - "\n", - "`deeplay` uses the `not_top_level` context manager to decide if a configuration is derived or not. `not_top_level` stores the tags of the currently constructing module in in the `ExtendedConstructorMeta` class. Everytime a configuration is added, it also stores these tags as the `source` of the configuration. \n", - "\n", - "When deciding if a configuration is derived or not, `deeplay` checks if the `source` of the configuration is a parent of the the target of the configuration. If it is, then the configuration is **not** derived. If the source is a child of the target, or the target is the same as the source, then the configuration is derived and should be removed before the `__init__` method is called.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "```python\n", - "self.is_constructing = True\n", - "```\n", - "Next, we set the `is_constructing` flag to `True`. This is used to check if the object is currently being constructed. This is used to prevent certain configurations from being applied while the object is being constructed.\n", - "\n", - "```python\n", - "args, kwargs = self.get_init_args()\n", - "```\n", - "\n", - "This is where the actual arguments to pass to the `__init__` method are determined. This is done by calling the `get_init_args` method. This method is responsible for finding the correct arguments to pass to the `__init__` method based on the actual arguments passed to the `__new__` method and the configurations applied by the user. Each class can override this method to customize how the arguments are determined.\n", - "\n", - "```python\n", - "getattr(self, self._init_method)(*(args + self._args), **kwargs)\n", - "```\n", - "\n", - "Finally, we call the `__init__` method of the object with the correct arguments. The `_init_method` attribute is used to determine the name of the `__init__` method to call. Most of the time, this is just `\"__init__\"`, but it can be overridden by subclasses to call a different method. The reason for this is to make deeplay play nicer with editors. It allows the class to define a dummy `__init__` method that gives the types and names of the arguments, while the actual initialization logic is in a different method. This allows the editor to provide better autocompletion and type checking.\n", - "\n", - "```python\n", - "self._run_hooks(\"after_init\")\n", - "```\n", - "\n", - "After the `__init__` method is called, we run the `after_init` hooks. Hooks are used to run code at specific points in the lifecycle of the object. The `after_init` hook is run after the `__init__` method is called. We'll cover hooks in more detail later.\n", - "\n", - "```python\n", - "self.is_constructing = False\n", - "```\n", - "We set the `is_constructing` flag to `False` to indicate that the object is no longer being constructed.\n", - "\n", - "```python\n", - "self.__post_init__()\n", - "```\n", - "Finally, we call the `__post_init__` method of the object. This method is used to perform any post-initialization steps. This does nothing by default." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## The Config object\n", - "\n", - "For each hierarchy of modules, there is a corresponding `Config` object, which lives on the root module. \n", - "\n", - "It is a dictionary-like object that stores the configurations applied to the modules in the hierarchy. The keys are tags and the name of the configurable (for example `(\"block\", \"layer\", \"foo\")`). The values are lists of `ConfigItem` or `DetachedConfigItem` objects. \n", - "\n", - "`ConfigItem` objects store the `source` of the configuration and the `value` of the configuration. The `source` is the tags of the module that was constructing when the configuration was applied. The `value` is the value of the configuration.\n", - "\n", - "`DetachedConfigItem` objects are in practice very similar to `ConfigItem`s, and should be ephemeral. They are used to store configurations that are applied by an object that is not part of the same hierarchy. As such, the `tags` of the `source` do not make sense. Instead, the `source` is temporarily set to the object itself. This is okay, because all `DetachedConfigItem`s become `ConfigItem`s after the `__construct__` method is called.\n", - "\n", - "Following is an example where a `DetachedConfigItem` is created:\n", - "\n", - "```python\t\n", - "class Module(DeeplayModule):\n", - " def __init__(self):\n", - " child = LinearBlock(10, 10)\n", - "\n", - " # Here, child is not attached to the hierarchy yet, so we don't have tags for it.\n", - " child.configure(\"activation\", nn.ReLU())\n", - "\n", - " # Here, the child is attached. This changes the root_module of `child` and we\n", - " # can now get the tags of the child. The DetachedConfigItem is converted to a ConfigItem.\n", - " self.child = child\n", - "```\n", - "\n", - "No `DetachedConfigItem` should exist after the `__construct__` method is called.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Taking the example of `(\"block\", \"layer\", \"foo\")`, the `Config` object would look something like this:\n", - "\n", - "```python\n", - "{\n", - " (\"block\", \"layer\", \"foo\"): [\n", - " ConfigItem(source=None, value=1),\n", - " ConfigItem(source=(\"block\", \"layer\"), value=2),\n", - " ]\n", - "}\n", - "```\n", - "\n", - "A `None` source means that the configuration was applied by the user. When deciding which item to use as the actual value, the item with the highest priority is used. The priority is determined by the source of the item. The source closest to the root module has the highest priority. If two items have the same source, the item applied later has higher priority. A `None` source has the highest priority." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Hooks\n", - "\n", - "Since modules may be reconstructed at any point, it is important that any state-altering methods are re-run after the module is reconstructed. This is where hooks come in. Hooks are used to run code at specific points in the lifecycle of the object.\n", - "\n", - "To register a method as a hook, you can use the any of the following decorators:\n", - "\n", - "```python\n", - "# does not create a hook, but adds the method to the config tape, which is replayed\n", - "# when model.new() is called\n", - "@stateful \n", - "\n", - "# runs the method after the __init__ method is called (and adds to config tape)\n", - "@after_init\n", - "\n", - "# runs the method before the build method is called (and adds to config tape)\n", - "@before_build\n", - "\n", - "# runs the method after the build method is called (and adds to config tape)\n", - "@after_build\n", - "```\n", - "\n", - "#### The config tape\n", - "\n", - "The config tape is a list of methods that are run when the `new` method is called. This method should create a new, identical but detached object. To do so we first create the object with the same exact input arguments (as stored in the metaclass) and then run the same stateful methods, in the same order, with the same arguments. \n", - "\n", - "One may imagine that one could simply pass the same configuration object the to the new object, but this is far from simple. It is not guaranteed that the configuration object is serializable, and even if it is, it may contain cyclic references and other issues that are hard to resolve." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Checkpointing\n", - "\n", - "Since deeplay modules requires an additional `build` step before the weights are created, the default checkpointing system of `lightning` does not work.\n", - "\n", - "We have solved this by storing the state of the `Application` object immediately before building as a a hyperparameter in the checkpoint. This is then loaded when the model is loaded from the checkpoint, and the `build` method is called with the same arguments as before before the weights are loaded." - ] - } - ], - "metadata": { - "language_info": { - "name": "python" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -}