Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dot-env-secrets recipe #265

Merged
merged 3 commits into from
Mar 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions configuration/dot-env-secrets/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Dot-env Secrets

Recipe to quickly load your `.env` secrets into
Prefect's [Secret block](https://docs.prefect.io/latest/api-ref/prefect/blocks/system/#prefect.blocks.system.Secret).

## Reasoning

Oftentimes, I find myself adapting existing Python scripts into Prefect.
Storing secret environment variables in a `.env` file is a common solution and
the [dotenv](https://pypi.org/project/python-dotenv/) package can seamlessly load those
variables into the runtime environment.

I needed a tool to load all the variables present in this file into my Prefect workplace
so that my flows no longer need to rely on this file and can just fetch the variable
from Prefect.

## Dependencies

In order to run this script you would
need [python-dot-env](https://pypi.org/project/python-dotenv/).
A *prefect-only* solution is also possible but not (yet) presented in this recipe.
Make sure to be authenticated before running the script.

## Solution

Running this script will:

- Load the variables present in the .env file (specified in the `fp` parameter)
- Convert the variable names to `kebab-case`
- Upload Secret block to your Prefect workspace

## Example

In this example `sample-dot-env` contains a variable `LOAD_DOT_ENV` which is a string.
In the `load_secrets` function we specify the path to the `sample-dot-env` file:

```shell
LOAD_DOT_ENV=it_works
```

```python
if __name__ == "__main__":
env_file = "flows-advanced/dot-env-secrets/sample-dot-env"
load_secrets(env_file)
```

Executing it will print each variable name and its block ID

```shell
load-dot-env ffb9096a-efd3-4fcf-a37e-f79b8549f27f
```

You can now use this Secret block in your Prefect code!

![screenshot](static/block-loaded-example.png)

## More

More could be done to this recipe! For example:

- Provide a solution to 'load' all Prefect's secrets onto a file
- Remove the `dotenv` dependency
- Load all variables concurrently
39 changes: 39 additions & 0 deletions configuration/dot-env-secrets/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
from typing import Any

from dotenv import dotenv_values
from prefect.blocks.system import Secret


def rename_keys(d: dict[str, Any]) -> dict[str, Any]:
"""
Variable names need to be renamed in order to be valid Block names.

Examples:
--------
>>> original = {'TEST_KEY':"test_value", "test-key-valid":"test_value"}
>>> rename_keys(original)
{'test-key': 'test_value', 'test-key-valid': 'test_value'}

"""
n = {}
for k, v in d.items():
n[k.replace("_", "-").casefold()] = v
return n


def load_secrets(fp: str, overwrite: bool = False):
original_values = dotenv_values(fp, verbose=True)

if original_values == {}:
raise IOError("File not found or empty")

d = rename_keys(original_values)
for k, v in d.items():
secret = Secret(value=v)
uuid = secret.save(name=k, overwrite=overwrite)
print(k, uuid)


if __name__ == "__main__":
env_file = "flows-advanced/configuration/dot-env-secrets/sample-dot-env"
load_secrets(env_file)
2 changes: 2 additions & 0 deletions configuration/dot-env-secrets/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
dotenv
prefect
1 change: 1 addition & 0 deletions configuration/dot-env-secrets/sample-dot-env
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
LOAD_DOT_ENV=it_works
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.