From 14ee93e2e86593e6c6f0b279ff953bdd1684939d Mon Sep 17 00:00:00 2001 From: Leon Wright Date: Sat, 6 Apr 2024 13:15:22 +0800 Subject: [PATCH] docs: Configuration This adds the configuration documentation, as likely the point of cally isn't super clear without understanding the layered approach to how the configuration is built --- docs/concepts.rst | 36 +----- docs/configuration.rst | 251 +++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 1 + 3 files changed, 253 insertions(+), 35 deletions(-) create mode 100644 docs/configuration.rst diff --git a/docs/concepts.rst b/docs/concepts.rst index a56ce97..1238223 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -69,7 +69,7 @@ Configuration ============= Cally's configuration is built on top of `dynaconf `_ with a custom loader to provide a layered, consitent builder to provide a strong starting point for building your infrastructure -from a very small amount of configuration. +from a very small amount of configuration. Further details can be found in the :ref:`configuration` section. cally.yaml ---------- @@ -87,40 +87,6 @@ follows: stack_vars: example: variable -Central Defaults ----------------- -Often there will be certain constants, that would make sensible defaults, say provider defaults -for `region`, or your state bucket and path name. These can be configured via a `DEFAULTS` key -within your `cally.idp.defaults` file. - -.. code-block:: python - - DEFAULTS = { - 'providers': { 'google': { 'location': 'some-place1' }}, - 'backend': { - 'type': 'GcsBackend', 'path_key': 'prefix', - 'path': 'state-files/{environment}/{name}', - 'config': { 'bucket': 'buckety-mc-bucketface' }, - }, - } - -Which will be available to the consuming services as required - -.. code-block:: shell - - ✗ cally config print-service --environment dev --service pets - BACKEND: - config: - bucket: buckety-mc-bucketface - path: state-files/{environment}/{name} - path_key: prefix - type: GcsBackend - ENVIRONMENT: dev - NAME: pets - PROVIDERS: - google: - location: some-place1 - STACK_TYPE: RandomPets The CDK for Terraform ===================== diff --git a/docs/configuration.rst b/docs/configuration.rst new file mode 100644 index 0000000..f92d317 --- /dev/null +++ b/docs/configuration.rst @@ -0,0 +1,251 @@ +.. _configuration: + +============= +Configuration +============= +Cally's configuration is built on top of `dynaconf `_ with a custom loader to +provide a layered, consitent builder to provide a strong starting point for building your infrastructure +from a very small amount of configuration. + +Cally firmly leans hard on the layered with merged config approach available within Dynaconf. With the +following resolution order, last set, wins: :ref:`config-idp-defaults`, :ref:`config-top-defaults`, :ref:`config-environment` + +At first glance, this may seem quite complex, the layered approach allows for a powerful 'DRY' approach +without a lot of cognitive load. + +cally.yaml +========== +Whislt cally is a tool that is built around configuration, it leans into convention over configuration to +reduce the amount of effort required to get started, improve consistency, and constraining the scope for +bugs, or mis-understood configuration. + +--------- +Top Level +--------- +In this section we'll go into the various top level configuration keys. + +.. _config-top-defaults: + +defaults +-------- +Any second level key configured here will be availble to all services, in all enviroments. + +.. code-block:: yaml + :caption: cally.yaml + + defaults: + providers: + random: + alias: foo + +mixins +------ +Any key can be defined as a set of resuable components, which will be covered in more detail in :ref:`mixins`, but +an example is as below: + +.. code-block:: yaml + :caption: cally.yaml + + mixins: + example-mixin: + providers: + random: + alias: foo + another: + stack_type: ExampleStack + dev: + services: + example-service: + mixins: + - example-mixin + - another + +.. code-block:: shell + + ✗ cally config print-service --environment dev --service example-service + ENVIRONMENT: dev + NAME: example-service + PROVIDERS: + random: + alias: foo + STACK_TYPE: ExampleStack + +.. _config-environment: + +environment +----------- +Any top level key, that does not match `mixins` or `defaults`, will be considered to be an environment that +can be called using the ``--environment`` option or set via the ``CALLY_ENVIRONEMNT`` environment variable. + +Mixins, and defaults can be set at an environment level, and all services are expected to be defined using +the ``services`` key. + +.. code-block:: yaml + :caption: cally.yaml + + dev: + defaults: + providers: + random: + alias: bar + mixins: + example-mixin: + stack_var: + my_key: value + services: + example-service: + mixins: + - example-mixin + stack_type: ExampleStack + +.. code-block:: shell + + ✗ cally config print-service --environment dev --service example-service + ENVIRONMENT: dev + NAME: example-service + PROVIDERS: + random: + alias: bar + STACK_TYPE: ExampleStack + STACK_VAR: + my_key: value + +------------ +Second Level +------------ +All keys in the second level, can be set in **all** levels. From the defaults, to the environment defaults, +mixins, and the service level. + +backend +------- +Terraform requires a backend to store state files. By default this will be a ``LocalBackend``, but it is +quite useful to set your in your idp defaults. The backend key has the following keys + +- ``type`` - this sets the backend to be used ie ``LocalBackend`` +- ``path`` - will be formatted using pythons builtin `str.format `_, and all keys in the ``service`` object are avaialble using the `format syntax `_. +- ``path_key`` - The prefix to the state file varies per backend. For example, ``LocalBackend`` uses ``path``, and ``GcsBackend`` uses ``prefix`` +- ``config`` - A dictionary for the configuration to be suplied to the provider + +.. code-block:: yaml + :caption: cally.yaml + + config: + bucket: my-orgs-state-bucket + path: my/path/to/{environment}/{name} + path_key: prefix + type: GcsBackend + +.. code-block:: json + :caption: cdk.tf.json + + { + "terraform": { + "backend": { + "gcs": { + "bucket": "my-orgs-state-bucket", + "prefix": "my/path/to/dev/example-service" + } + } + } + } + +providers +--------- +The providers key, is where provider configuration defaults will live. Useful for setting things like +a default region. These will be passed to the provider automatically during instantiation. + +.. code-block:: yaml + :caption: cally.yaml + + + +.. _config-idp-defaults: + +IDP Defaults +============ +Often there will be certain constants, that would make sensible defaults, say provider defaults +for ``region``, or your state bucket and path name. These can be configured via a ``DEFAULTS`` key +within your ``cally.idp.defaults`` file. + +.. code-block:: python + :caption: defaults.py + + DEFAULTS = { + 'providers': { 'google': { 'location': 'some-place1' }}, + 'backend': { + 'type': 'GcsBackend', 'path_key': 'prefix', + 'path': 'state-files/{environment}/{name}', + 'config': { 'bucket': 'buckety-mc-bucketface' }, + }, + } + +Which will be available to the consuming services as required + +.. code-block:: shell + + ✗ cally config print-service --environment dev --service pets + BACKEND: + config: + bucket: buckety-mc-bucketface + path: state-files/{environment}/{name} + path_key: prefix + type: GcsBackend + ENVIRONMENT: dev + NAME: pets + PROVIDERS: + google: + location: some-place1 + STACK_TYPE: RandomPets + +.. _mixins: + +Mixins +====== + +Overriding Cally's Loader +========================= +Dynaconf does allow for the loaders to be overriden, there may be further support added to this directly, +however for now, you can at least append your own loader via the `documented methods `_. + +For example, creating a ``loader.py`` in the ``cally.idp`` namespace, you could set an environment variable like +``export LOADERS_FOR_DYNACONF="['cally.idp.loader']"``. With the following contents + +.. code-block:: python + + from dynaconf import LazySettings + + + def cat_all_values(settings: LazySettings) -> None: + for k, v in settings.items(): + if isinstance(v, dict): + cat_all_values(v) + else: + settings[k] = 'meow' + + + def load(obj: LazySettings, *args, **kwargs) -> None: # noqa: ARG001 + cat_all_values(obj) + +Whilst the results are meow amusing, I'm sure a more useful use case could be found for this +functionality. + +.. code-block:: shell + + ✗ cally config print-service --environment test --service test + BACKEND: + config: + bucket: meow + path: meow + path_key: meow + type: meow + BACKEND.PATH: meow + BACKEND.TYPE: meow + ENVIRONMENT: meow + NAME: meow + PROVIDERS: + google: + default_labels: + deployment_tool: meow + git_repo: meow + project: meow + region: meow diff --git a/docs/index.rst b/docs/index.rst index ca6d697..4822029 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -72,6 +72,7 @@ usage patterns. :maxdepth: 2 concepts + configuration Miscellaneous Pages -------------------