diff --git a/.licensei.toml b/.licensei.toml index 2dfd636..70118ce 100644 --- a/.licensei.toml +++ b/.licensei.toml @@ -35,7 +35,7 @@ ignored = [ [header] authors = ["Cisco"] -ignorePaths = [".direnv", ".devenv", "vendor"] +ignorePaths = [".direnv", ".devenv", "vendor", "examples"] ignoreFiles = ["zz_generated.*.go"] template = """// Copyright © :YEAR: :AUTHOR: // diff --git a/README.md b/README.md index 45aaa0d..ea4de6b 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,6 @@ **Secret Sync** exposes a generic way to interact with external secret storage systems like [HashiCorp Vault](https://www.vaultproject.io/) and provides a set of API models to interact and orchestrate the synchronization of secrets between them. -This name was chosen in a rush, we are open to naming suggestions 😄 - > [!IMPORTANT] > This is an **early alpha version** and breaking changes are expected. > As such, it is not recommended for usage in production. @@ -27,40 +25,32 @@ This name was chosen in a rush, we are open to naming suggestions 😄 - Advanced templating capabilities for transforming secrets | **Supported store** | **Status** | -|--------------------------------------------------------------------------|--------| -| [HashiCorp Vault](https://www.vaultproject.io) | alpha | -| [Local Provider](#secret-store) | alpha | -| [AWS Secrets Manager](https://aws.amazon.com/secrets-manager) | _planned_ | -| [Google Secrets Manager](https://cloud.google.com/secret-manager) | _planned_ | -| [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) | _planned_ | -| [Kubernetes Secret](https://kubernetes.io/) | _planned_ | +|--------------------------------------------------------------------------|------------| +| [HashiCorp Vault](https://www.vaultproject.io) | alpha | +| [Local Provider](#secret-store) | alpha | +| [AWS Secrets Manager](https://aws.amazon.com/secrets-manager) | _planned_ | +| [Google Secrets Manager](https://cloud.google.com/secret-manager) | _planned_ | +| [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/) | _planned_ | +| [Kubernetes Secret](https://kubernetes.io/) | _planned_ | Check details about upcoming features by visiting the [project issue](https://github.com/bank-vaults/secret-sync/issues) board. ## Goals * Provide safe and simple way to work with secrets -* Common API for secret management regardless of the store backend -* Explicit control over the secret synchronization process +* Expose common API for secret management regardless of the store backend +* Give total control of the secret synchronization process > Consider a situation where Dev teams need access to secrets from different environments. -> Ops teams can provide access to secrets in the form of a sandboxed environment (e.g. new Vault instance) synced only with secrets Devs require; all in GitOps way. - -## Getting Started - -To get familiarized, we will show how you can use these tools to answer two questions: - -- **Ops**: How do I synchronize secrets from one secret storage systems to another? -- **Dev**: How do I consume secrets to bootstrap my application? +> Ops teams can provide access to secrets in the form of an isolated environment (e.g. new Vault instance) synced only with secrets Devs require; all in GitOps way. -To answer the first question, we shall create some database secrets and synchronize them into Vault.
-For the second question, we will use some secrets from Vault to create an access file for an application. +## Getting started -You can find complete examples and instructions in the [EXAMPLE](EXAMPLE.md) file. +To get familiarized, check out the collection of different [examples](examples) using this tool. ## Documentation -Check out the documentation at [API documentation](docs/API.md) page or on [pkg.go.dev](https://pkg.go.dev/mod/github.com/bank-vaults/secret-sync). +Check out the [project documentation](docs) or [pkg.go.dev](https://pkg.go.dev/mod/github.com/bank-vaults/secret-sync). ## Development diff --git a/docs/API.md b/docs/API.md index 6692833..21428fd 100644 --- a/docs/API.md +++ b/docs/API.md @@ -13,8 +13,6 @@ secretsStore: storeName: storeSpec ``` -You can find all the Secret Store specifications in [pkg/apis/v1alpha1/secretstore.go](pkg/apis/v1alpha1/secretstore.go) -
Store Spec: HashiCorp Vault* @@ -68,8 +66,6 @@ sync: - actionSpec ``` -You can find all the Sync Plan specifications in [pkg/apis/v1alpha1/syncjob_types.go](pkg/apis/v1alpha1/syncjob_types.go) -
Action Spec: Synchronize a secret from reference @@ -302,5 +298,5 @@ Note that only YAML configuration files are supported. You can also provide optional params for CRON schedule to periodically sync secrets via `--schedule` flag. All sync actions are indexed in logs based on their order in the sync plan config file. -You can also use [pkg/storesync](pkg/storesync) package to run secret synchronization plan natively from Golang. +You can also use [pkg/storesync](https://pkg.go.dev/github.com/bank-vaults/secret-sync/pkg/storesync) package to run secret synchronization plan natively from Golang. This is how the CLI works as well. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..7f3c633 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,5 @@ +## Documentation + +Here you can find documentation about the way **Secret Sync** works. Refer to: + +- [API](API.md) - Documentation on how to use different secret stores, synchronization policies, and templating rules. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000..3b43368 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,5 @@ +## Usage Examples + +Here you can find various examples on how you can use **Secret Sync**. Examples include: + +- [single-source-advanced](single-source-advanced) - Use a central source of truth to synchronize secrets into different secret stores. Uses templating for fine-grained synchronization. diff --git a/examples/assets/gen-assets.sh b/examples/assets/gen-assets.sh new file mode 100644 index 0000000..50f75fb --- /dev/null +++ b/examples/assets/gen-assets.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash + +# Requirement: https://github.com/terrastruct/d2 + +ASSET_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) + +for d2_file in "$ASSET_DIR"/*.d2 +do + # Get files + png_file=${d2_file%".d2"}.png + + # Generate PNG from diagram + d2 $d2_file $png_file +done diff --git a/examples/assets/native-approach.d2 b/examples/assets/native-approach.d2 new file mode 100644 index 0000000..d7a036c --- /dev/null +++ b/examples/assets/native-approach.d2 @@ -0,0 +1,42 @@ +direction: down + +Vault: { + label: Vault + shape: image + icon: https://www.svgrepo.com/download/448679/vault.svg +} + +AWSSM: { + label: AWS Secrets Manager + shape: image + icon: https://icons.terrastruct.com/aws%2FSecurity%2C%20Identity%2C%20&%20Compliance%2FAWS-Secrets-Manager.svg +} + +GCPSM: { + label: GCP Secret Manager + shape: image + icon: https://www.svgrepo.com/download/375492/secret-manager.svg +} + +AzureKeyVault: { + label: Azure Key Vault + shape: image + icon: https://icons.terrastruct.com/azure%2FSecurity%20Service%20Color%2FKey%20Vaults.svg +} + +Cluster: { + label: Cluster + icon: https://icons.terrastruct.com/azure%2F_Companies%2FKubernetes.svg + + Local: { + Label: Secret + } + Workload: Workload + + Local -> Workload +} + +Cluster.Local <- Vault +Cluster.Local <- AWSSM +Cluster.Local <- GCPSM +Cluster.Local <- AzureKeyVault diff --git a/examples/assets/native-approach.png b/examples/assets/native-approach.png new file mode 100644 index 0000000..8e3e81d Binary files /dev/null and b/examples/assets/native-approach.png differ diff --git a/examples/assets/secret-sync.d2 b/examples/assets/secret-sync.d2 new file mode 100644 index 0000000..469b64d --- /dev/null +++ b/examples/assets/secret-sync.d2 @@ -0,0 +1,137 @@ +direction: down + +Vault: { + label: Vault + shape: image + icon: https://www.svgrepo.com/download/448679/vault.svg +} + +AWSSM: { + label: AWS Secrets Manager + shape: image + icon: https://icons.terrastruct.com/aws%2FSecurity%2C%20Identity%2C%20&%20Compliance%2FAWS-Secrets-Manager.svg + style: { + opacity: 0.4 + } +} + +GCPSM: { + label: GCP Secret Manager + shape: image + icon: https://www.svgrepo.com/download/375492/secret-manager.svg + style: { + opacity: 0.4 + } +} + +AzureKeyVault: { + label: Azure Key Vault + shape: image + icon: https://icons.terrastruct.com/azure%2FSecurity%20Service%20Color%2FKey%20Vaults.svg + style: { + opacity: 0.4 + } +} + +BVSecretSync: { + label: Bank-Vaults Secret Sync + shape: image + icon: https://raw.githubusercontent.com/bank-vaults/bank-vaults.dev/main/assets/icons/logo.svg +} + +Vault -> BVSecretSync +AWSSM -> BVSecretSync { + style: { + opacity: 0.4 + } +} +GCPSM -> BVSecretSync { + style: { + opacity: 0.4 + } +} +AzureKeyVault -> BVSecretSync { + style: { + opacity: 0.4 + } +} + +ClusterA: { + label: Cluster A + icon: https://icons.terrastruct.com/azure%2F_Companies%2FKubernetes.svg + + style: { + opacity: 0.4 + } + + LocalA: Secret { + style: { + opacity: 0.4 + } + } + WorkloadA: Workload { + style: { + opacity: 0.4 + } + } + + LocalA -> WorkloadA { + style: { + opacity: 0.4 + } + } +} + +ClusterB: { + label: Cluster B + icon: https://icons.terrastruct.com/azure%2F_Companies%2FKubernetes.svg + + LocalB: { + label: Local Vault + shape: image + width: 80 + icon: https://www.svgrepo.com/download/448679/vault.svg + } + WorkloadB: Workload + + LocalB -> WorkloadB +} + +LocalC: { + label: Tenant Environment + shape: image + width: 80 + icon: https://www.svgrepo.com/download/375492/secret-manager.svg + style: { + opacity: 0.4 + } +} + +ClusterC: { + label: Cluster C + icon: https://icons.terrastruct.com/azure%2F_Companies%2FKubernetes.svg + + style: { + opacity: 0.4 + } + + WorkloadC: Workload { + style: { + opacity: 0.4 + } + } +} + +LocalC -> ClusterC.WorkloadC + +BVSecretSync -> ClusterB.LocalB: Sync +BVSecretSync -> ClusterA.LocalA: Sync { + style: { + opacity: 0.4 + } +} +BVSecretSync -> LocalC: Sync { + style: { + opacity: 0.4 + } +} diff --git a/examples/assets/secret-sync.png b/examples/assets/secret-sync.png new file mode 100644 index 0000000..ea12f62 Binary files /dev/null and b/examples/assets/secret-sync.png differ diff --git a/EXAMPLE.md b/examples/single-source-advanced/README.md similarity index 72% rename from EXAMPLE.md rename to examples/single-source-advanced/README.md index 6f458cb..84efced 100644 --- a/EXAMPLE.md +++ b/examples/single-source-advanced/README.md @@ -1,22 +1,23 @@ ## Getting Started -To get familiarized, we will show how you can use these tools to answer two questions: +To get familiarized, we will show how you can use Secret Sync tool to cover two scenarios: -- **Ops**: How do I synchronize secrets from one secret storage systems to another? -- **Dev**: How do I consume secrets to bootstrap my application? +- **Case 1**: Synchronize secrets from one secret store to another. + - We will use our local file store as a source of truth to synchronize some database secrets into Vault. +- **Case 2**: Consume secrets to bootstrap an application. + - We will use Vault instance to fetch database secrets to our local store in a form of a configuration file for an application. -To answer the first question, we shall create some database secrets and synchronize them into Vault.
-For the second question, we will use some secrets from Vault to create an access file for an application. +*Note:* The same logic applies to any other combination of secret stores. -### 1. Prepare environment +### Step 1: Prepare environment You will need the following tools to continue: -- Docker - Git - Makefile - Golang `>= 1.21` To set up the environment, you can build from source: + ```bash git clone https://github.com/bank-vaults/secret-sync.git /tmp/secret-sync cd /tmp/secret-sync @@ -25,16 +26,17 @@ alias secret-sync="/tmp/secret-sync/build/secret-sync" ``` Alternatively, you can also use only Docker: + ```bash -alias secret-sync="docker run --rm -v /tmp:/tmp ghcr.io/bank-vaults/secret-sync:latest secret-sync" +alias secret-sync="docker run --rm -v /tmp:/tmp ghcr.io/bank-vaults/secret-sync:v0.1.0 secret-sync" ``` -### 2. Define secret stores +### Step 2: Define secret stores -Documentation and examples on how to use different secret stores can be found in chapter [Secret Store](README.md#secret-store). +#### Local store -#### 2.1. Local store Create a directory and a config file to use as the _local secret store_. + ```bash # Create local store directory mkdir -p /tmp/example/local-store @@ -47,8 +49,10 @@ secretsStore: EOF ``` -#### 2.2. Vault store +#### Vault store + Deploy Vault and create config file to use as the _Vault secret store_. + ```bash # Deploy a Vault instance docker compose -f dev/vault/docker-compose.yml up -d @@ -64,12 +68,11 @@ secretsStore: EOF ``` -### 3. Define sync plans -Documentation and examples on how to create a more extensive sync plan can be found in chapter [Sync Plan](README.md#sync-plan). +### Step 3: Define sync plans + +#### Database secrets -#### 3.1. Database secrets -Define a sync plan for `db-host`, `db-user`, `db-pass` secrets. -These secrets will be synced from our local to Vault secret store. +Define a sync plan for `db-host`, `db-user`, `db-pass` secrets. These secrets will be synced from our local to Vault secret store. ```bash cat < /tmp/example/db-secrets-sync.yml @@ -81,10 +84,9 @@ sync: EOF ``` -#### 3.1. Application access secret -Define a sync plan for app-specific secret `app-access-config` created from various other secrets (e.g. database). -This secret will be synced from Vault to our local secret store (as a file). -It can also be synced against the same store to refresh the secret. +#### Application access secret + +Define a sync plan for app-specific secret `app-access-config` created from various other secrets (e.g. database). This secret will be synced from Vault to our local secret store (as a file). It can also be synced against the same store to refresh the secret. ```bash cat < /tmp/example/app-access-config-sync.yml @@ -109,20 +111,21 @@ sync: EOF ``` -### 4. Create database secrets +### Step 4: Create database secrets Create database access secrets in our local secret store. + ```bash echo -n "very-secret-hostname" > /tmp/example/local-store/db-host echo -n "very-secret-username" > /tmp/example/local-store/db-user echo -n "very-secret-password" > /tmp/example/local-store/db-pass ``` -### 5. Perform sync +### Step 5: Perform sync -Secret synchronization is performed using the [CLI](README.md#running-the-synchronization) by executing the sync plan between source and target secret stores. +Secret synchronization is performed using the CLI by executing the sync plan between source and target secret stores. -#### 5.1. Database secrets +#### Database secrets To synchronize database secrets from our local to Vault secret store, run: @@ -141,7 +144,7 @@ If successful, your output should contain something like: You can also navigate to the local Vault instance and verify these secrets. -#### 5.2. Application access secret +#### Application access secret To synchronize application access secret from Vault to our local secret store, run: @@ -157,7 +160,7 @@ cat /tmp/example/local-store/app-access-config ``` -### 6. Cleanup +### Step 6: Cleanup ```bash # Destroy Vault instance