Here we present a continuous delivery method for a service or services running in a Kubernetes cluster. It is expected that one or more of these services has Ethereum contracts developed alongside the service. It is expected that there is a relationship between service and contract to a degree that we must ensure both service and contract are deployed continuously. The entirety of the solution is targeted at development and test environments. The requirements for utilizing the outlined approach in production will not be covered here. Our one and only service at this time is the keep-client. The RFC will focus on the keep-client as a result, however the solution is applicable outside the keep-client. This RFC requires a basic understanding of Kubernetes objects.
There are three parts to deployments with keep-clients:
Service Code and Service Configuration:
Deploying manually is a pain in the ass, even at a few client deployments (5) the manual process is untenable. It’s possible to manage this manually at such a low number of clients though you run a non-trivial risk of human error and deployment step repetition as as result. This has an adverse effect on both the engineering experience and time efficiency.
When scaling the solution beyond 5 clients manual configuration becomes untenable regardless of due-diligence.
Contract Code:
The keep-client service leans on a collection of Ethereum contracts that are in development alongside the service. When contracts are changed they have to be deployed to the Ethereum network that the keep-clients are connecting to.
This arrangement creates a dependency between the service configuration and contract code on a couple levels:
-
keep-client configuration files require contract addresses. These addresses change when contracts are updated. This means we need to be aware of contract changes on the keep-client configuration side and make sure keep-client configuration files get updated contract addresses.
-
We can assume keep-client code will be updated to utilize the latest contract functionality. We can assume that this may or may not break keep-client behavior if the service is deployed against old contracts. This can potentially happen the other way around as well, where an updated contract is deployed and removes/changes functionality an old keep-client can’t deal with. Therefore we must ensure both keep-clients and Keep contracts get deployed together, contracts before clients due to client startup behavior.
Bootstrap Peer
# This is a TOML configuration file for DKG, P2P networking and connection to Ethereum
# Provider Initialization Example
[ethereum]
URL = "ws://eth-host:ws-port"
URLRPC = "http://eth-host:rpc-port"
[ethereum.account]
Address = "0x6299496199d99941193fdd2d717ef585f431ea05"
KeyFile = "path/to/in-use/keyfile"
[ethereum.ContractAddresses]
# Hex-encoded address of KeepRandomBeaconOperator contract
KeepRandomBeaconOperator = "0x6299496449d11141193fdd2d717ef585f431er02"
# TokenStaking
Staking = "0x6297776199d99942293fdd2d717ef585f431er03"
# Bootstrap node creating a new network.
[LibP2P]
Seed = seed-number-for-id-generation
Port = keep-client-portStandard Peer
# This is a TOML configuration file for DKG, P2P networking and connection to Ethereum
# Provider Initialization Example
[ethereum]
URL = "ws://eth-host:ws-port"
URLRPC = "http://eth-host:rpc-port"
[ethereum.account]
Address = "0x6299496199d99941193fdd2d717ef585f431ea05"
KeyFile = "path/to/in-use/keyfile"
[ethereum.ContractAddresses]
# Hex-encoded address of KeepRandomBeaconOperator contract
KeepRandomBeaconOperator = "0x6299496449d11141193fdd2d717ef585f431er02"
# Hex-encoded address of TokenStaking contract
Staking = "0x6297776199d99942293fdd2d717ef585f431er03"
[LibP2P]
Peers = ["comma separated list of bootstrap peers"]
Port = keep-client-portWe currently have a single cloud environment (keep-dev) that is in a mixed state of
continuous delivery and manual configuration. Current functionality will be
outlined as the "three parts" described in Background.
keep-client Service Deployment:
keep-client service deployment is done automatically whenever code is merged to master. This is facilitated by a lightweight tool called Keel that runs in the Kubernetes cluster. This is a pull based deployment mechanism where Keel is subscribed to our container registry and listens for publish events. On image publish Keel checks Kubernetes deployments for annotations and matches labels set there for what should be deployed. On match Keel pulls the image and does a deployment.
-
Keel Infrastructure: Keel Terraform Module
keep-dev -
Keel Implementation: keep-dev Terraform main.tf
keep-client Configuration:
Our first goal was to separate service configuration from the service image. This exists in the form of each deployed keep-client having its own Kubernetes ConfigMap. This allows for the use of a "standard" keep-client service image across all deployed keep-clients. There are two pieces of data that make up a keep-client ConfigMap:
-
keep-client-config.toml
-
Ethereum keyfile
Configuration of the data that populates the ConfigMaps and ConfigMaps themselves are updated manually.
-
Maintenance log and Kubernetes CRUD commands: kube-setup
-
keep-client config files: keep-client.toml
Ethereum Contract Deployment:
Ethereum contracts and subsequent steps are managed against keep-dev manually.
This is done from a local machine on the keep-dev VPN using Truffle.
It’s worth noting that when contracts are deployed we need to do the following:
-
Ensure Ethereum accounts are unlocked.
-
Stake Ethereum accounts with KEEP tokens.
-
Update the keep-client configuration files and ConfigMaps with new contract address. (see previous section)
-
re-deploy keep-clients.
Sample Commands:
# migrate contracts
truffle migrate --reset --network keep_dev
# unlock ETH accounts
KEEP_ACCOUNT_PASSWORD=eth-account-passphrase \
truffle exec ./unlock-eth-accounts.js --network keep_dev
# stake ETH accounts
truffle exec ./delegate-tokens.js --network keep_devTo bring parts Ethereum Contract Deployment and keep-client Configuration
into automated configuration such that they can be continuously deployed with
the already automated keep-client service deployment.
To automatically provision the keep-dev environment on master merge with all
appropriate configurations and app code without human intervention.
To reiterate: The implementation will aim to automate
Ethereum Contract Deployment and keep-client Configuration.
keep-client service deployment is already automated via Keel.
Either a new workflow or new jobs to existing workflow will be added to the
keep-core circle config. Before image publish on master merge Circle
will run this workflow/job to trigger a script that will:
-
migrate all contracts
-
Copy compiled contract JSON to Circle and store in a Workspace for persistence in the
InitContainerimage. -
Here we must implement an access point for Circle into the private Kubernetes cluster. We can do this with the
gcp_push_deployTerraform module.
On each keep-client Kubernetes deployment we’ll run an InitContainer that does
the following:
-
create ETH account
-
unlock ETH account
-
stake ETH account
-
write contract addresses, ETH host/port, bootstrap peer addresses to config template
-
write configuration file and ETH account keyfile to a persistent volume
-
mount configuration file / keyfile persistent volume to keep-client deployment
The InitContainer will operate exclusively on the ETH account and assigned to the
keep-client being deployed. If there are 100 keep-client Kubernetes
deployments, there would be 100 InitContainer instances unlocking and staking
for each of the assigned ETH accounts. This is infinitely scalable (system
resources aside) and reduces unlock + stake time to a ceiling of the time it
takes to operate on a single account.
The InitContainer will have a copy of each Keep contracts compiled JSON. The
files will be used to fetch contract ABI’s for account staking and contract
addresses for configuration file setup. The updated configuration values will be
fed into a complete keep-client configuration template and stored in a Kubernetes
persistent volume. This volume will be mounted to the keep-client deployment
where the config can be passed via command line argument on service start.
For now we’re going to bake a custom image for the InitContainer with the script
for doing ETH account creation, unlock and stake, and keep-client configuration.
This will be checked into the keep-core/infrastructure/kube directory.
We provide configuration values via environment variables at two points in this process: Circle contexts and Kube deployment configuration files.
We have environment and client properties. Where some N configuration values are of context/properties environment and some are of context/properties client. Environment properties
An example:
Using the environment/client context to organize configurations we can draw a line at where config values get populated.
ETH_HOSTNAME is a property of the environment, where KEEP_CLIENT_ETH_ACCOUNT
is a property of a client (because we’ve assigned it so).
Environment context property ETH_HOSTNAME gets configured at the Circle context
level and baked into the InitCointainer and KEEP_CLIENT_ETH_ACCOUNT
gets configured on the Kube deployment.
-
It requires Kubernetes
-
It requires Keel
-
It require CircleCI
-
It requires Truffle
-
All contracts are migrated, can’t be selective
-
No rollback mechanism if things go sideways
-
No order to which type of keep-client gets deployed first (bootstrap vs standard)
-
Various Github Issues discussing one or more of the 3 deployment parts outlined here: