-
Notifications
You must be signed in to change notification settings - Fork 29
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Adds dot-env-secrets recipe * Update flows-advanced/dot-env-secrets/README.md Co-authored-by: nate nowack <[email protected]> * Move recipe --------- Co-authored-by: nate nowack <[email protected]>
- Loading branch information
1 parent
f41fd09
commit dc3a601
Showing
5 changed files
with
105 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
# Dot-env Secrets | ||
|
||
Recipe to quickly load your `.env` secrets into | ||
Prefect's [Secret block](https://docs.prefect.io/latest/api-ref/prefect/blocks/system/#prefect.blocks.system.Secret). | ||
|
||
## Reasoning | ||
|
||
Oftentimes, I find myself adapting existing Python scripts into Prefect. | ||
Storing secret environment variables in a `.env` file is a common solution and | ||
the [dotenv](https://pypi.org/project/python-dotenv/) package can seamlessly load those | ||
variables into the runtime environment. | ||
|
||
I needed a tool to load all the variables present in this file into my Prefect workplace | ||
so that my flows no longer need to rely on this file and can just fetch the variable | ||
from Prefect. | ||
|
||
## Dependencies | ||
|
||
In order to run this script you would | ||
need [python-dot-env](https://pypi.org/project/python-dotenv/). | ||
A *prefect-only* solution is also possible but not (yet) presented in this recipe. | ||
Make sure to be authenticated before running the script. | ||
|
||
## Solution | ||
|
||
Running this script will: | ||
|
||
- Load the variables present in the .env file (specified in the `fp` parameter) | ||
- Convert the variable names to `kebab-case` | ||
- Upload Secret block to your Prefect workspace | ||
|
||
## Example | ||
|
||
In this example `sample-dot-env` contains a variable `LOAD_DOT_ENV` which is a string. | ||
In the `load_secrets` function we specify the path to the `sample-dot-env` file: | ||
|
||
```shell | ||
LOAD_DOT_ENV=it_works | ||
``` | ||
|
||
```python | ||
if __name__ == "__main__": | ||
env_file = "flows-advanced/dot-env-secrets/sample-dot-env" | ||
load_secrets(env_file) | ||
``` | ||
|
||
Executing it will print each variable name and its block ID | ||
|
||
```shell | ||
load-dot-env ffb9096a-efd3-4fcf-a37e-f79b8549f27f | ||
``` | ||
|
||
You can now use this Secret block in your Prefect code! | ||
|
||
![screenshot](static/block-loaded-example.png) | ||
|
||
## More | ||
|
||
More could be done to this recipe! For example: | ||
|
||
- Provide a solution to 'load' all Prefect's secrets onto a file | ||
- Remove the `dotenv` dependency | ||
- Load all variables concurrently |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
from typing import Any | ||
|
||
from dotenv import dotenv_values | ||
from prefect.blocks.system import Secret | ||
|
||
|
||
def rename_keys(d: dict[str, Any]) -> dict[str, Any]: | ||
""" | ||
Variable names need to be renamed in order to be valid Block names. | ||
Examples: | ||
-------- | ||
>>> original = {'TEST_KEY':"test_value", "test-key-valid":"test_value"} | ||
>>> rename_keys(original) | ||
{'test-key': 'test_value', 'test-key-valid': 'test_value'} | ||
""" | ||
n = {} | ||
for k, v in d.items(): | ||
n[k.replace("_", "-").casefold()] = v | ||
return n | ||
|
||
|
||
def load_secrets(fp: str, overwrite: bool = False): | ||
original_values = dotenv_values(fp, verbose=True) | ||
|
||
if original_values == {}: | ||
raise IOError("File not found or empty") | ||
|
||
d = rename_keys(original_values) | ||
for k, v in d.items(): | ||
secret = Secret(value=v) | ||
uuid = secret.save(name=k, overwrite=overwrite) | ||
print(k, uuid) | ||
|
||
|
||
if __name__ == "__main__": | ||
env_file = "flows-advanced/configuration/dot-env-secrets/sample-dot-env" | ||
load_secrets(env_file) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
dotenv | ||
prefect |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
LOAD_DOT_ENV=it_works |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.