Proposal Secert Manager sub-task: Obtain startup-time sensitive information from the vault

[Sn0rt]

Description

this proposal base on #9915, Adjust the field's design.

this part of #8319

Description

As we know the APISIX support 3 mode for starting: Traditional, Decoupled, Standalone. this proposal mainly for the Traditional mode and Decoupled mode .

Under this 2 mode the APISIX requires 2 core configurations when starting up

1. Configuration parameters of admin API
2. Configuration parameters of etcd connection

The original configuration method

The admin key and etcd password are configured in clear text, and the certificate is the path

deployment:
  admin:
    admin_key:
      - name: admin
        key: edd1c9f034335f136f87ad84b625c8f1
        role: admin
      - name: viewer
        key: 4054f7cf07e344346cd3f287985e76a2
        role: viewer
    admin_api_mtls:
      admin_ssl_cert: /path/to/cert
      admin_ssl_cert_key: /path/to/key
      admin_ssl_ca_cert: /path/to/cert
  etcd:
    password: 5tHkHhYkjr6cQY
    tls:
      cert: /path/to/cert
      key: /path/to/key

Among them, in the plugin_attr of config-default, there is no special need to be stored in vault for the time being. If there is, add it later.

Add configuration items
An example is as follows:

deployment:
    secret_manager:
        vault:
            uri: https://vault.vault:8200
            prefix: "kv/apisix"
            token: "${{VAULT_TOKEN}}"

The above configuration file is equivalent to the following environment variable configuration.

export APISIX_DEPLOYMENT_SM_VAULT_URI=https://vault.vault:8200
export APISIX_DEPLOYMENT_SM_VAULT_PREFIX="kv/apisix"
export APISIX_DEPLOYMENT_SM_VAULT_TOKEN=${{VAULT_TOKEN}}

When the configuration file and environment variables are configured at the same time, the environment variables have higher priority

Scope definition of sensitive data required for startup

deployment section

The specific configuration items are as follows

• deployment.admin.admin_key[].key
• deployment.admin.admin_api_mtls.admin_ssl_cert
• deployment.admin.admin_api_mtls.admin_ssl_cert_key
• deployment.admin.admin_api_mtls.admin_ssl_ca_cert
• deployment.etcd.password
• deployment.etcd.tls.cert
• deployment.etcd.tls.key

Some parameters were lost in the previous proposal. For completeness, we need these parameters.
But it doesn't have to be implemented in one PR. I think

The above-mentioned sensitive information needs to be designed and implemented according to priority, and stored in the vault.

Example of connecting to secret manager's vault backend through configuration file

deployment:
  admin:
    admin_key:
      - name: admin
        key: "$secret://vault/apisix_config/admin_key"
        role: admin
      - name: viewer
        key: "$secret://vault/apisix_config/viewer_key"
        role: viewer
    admin_api_mtls:
      admin_ssl_cert_content: "$secret://vault/apisix_config/admin_ssl_cert"
      admin_ssl_cert_key_content: "$secret://vault/apisix_config/admin_ssl_cert_key"
      admin_ssl_ca_cert_content: "$secret://vault/apisix_config/admin_ssl_ca_cert"
  etcd:
    password: "$secret://vault/apisix_config/etcd_password"
    tls:
      cert_content: "$secret://vault/apisix_config/etcd_cert"
      key_content: "$secret://vault/apisix_config/etcd_cert_key"

APISIX section

APISIX itself also provides encryption of core data. APISIX uses symmetric encryption to encrypt some sensitive data and store it in ETCD.

more info : https://github.com/apache/apisix/pull/8487/files#diff-ffcc5f951f0adfe5baeabfcd28555c3c148d87a58d475c6e2239344426795ebf

There is a scenario: when the user expects to use symmetric encryption to encrypt and store sensitive data in etcd instead of storing it in the vault backend of secert manager.
But this encrypted key is expected to be stored in the secert manager's backend vault.

anyway. the APISIX section is with a lower priority.

• apisix.ssl.key_encrypt_salt[]
• apisix.data_encryption.keyring[]

apisix:
...
  ssl:
...
    key_encrypt_salt: 
      - "$secret://vault/apisix_config/apisix_ssl_key_encrypt_salt0"
  data_encryption:                # Encrypt fields specified in `encrypt_fields` in plugin schema.
...
    keyring: 
      - "$secret://vault/apisix_config/apisix_data_encryption_keyring0"
...

discovery section

The discovery field in the configuration file is mainly used for service discovery and is not the core of APISIX. this is not a core field, so it has a lower priority.

• apisix.discovery.nacos.host

discovery:                      # Service Discovery
...
 nacos:                         # Nacos
   host:                        # Nacos address(es)
     - "$secret://vault/apisix_config/discovery_nacos_host0"

If any vault blank is obtained in the vault instance, the startup fails and an obvious error is given.

APISIX Standalone

If you use standalone mode to start APISIX, you cannot start the admin API. The configuration backend is also based on yaml.

There will be no sensitive information at startup.

[Sn0rt]

we have a PR for the orginal version #9930 cc @rodman10 thx you contribute

and @monkeyDluffy6017 @leslie-tsang pls help me to review this desgin.

[soulbird]

Can the secret manager configuration here be used in the authentication plugin?

[Sn0rt]

authentication plugin?

From a design perspective, the configuration file discussed in the current proposal or the vault specified by the environment variable cannot be used to configure and store sensitive information of the plugin. It is only used to store sensitive information at startup.

The main reason is that the vault referenced in the authentication plugin has a multi-instance concept. For example, vault1 and vault2 correspond to different vault backends. This in turn requires the configuration file to support the connection of multiple vault instances.

[ssignik]

This function will be useful in production, so does it has any progress? @Sn0rt

[Sn0rt]

@ssignik Sorry, I haven't had the time to work on this feature lately.