cosign-ecs-verify

A prototype analog of a Kubernetes admission controller for Amazon Elastic Container Service (ECS) which:

• Terminates any running images without a valid cosign signature.
• Sends alerts to an SNS topic (which can forward to your email).
• Supports configurable keys to use for signature validation.

NOTE: This is Proof of Concept code and not yet production-ready. In the event of a bug or misconfiguration, it can prevent any ECS tasks from running. This system intercepts tasks as they start (not before) and may not fully prevent unsigned containers.

If you'd like to learn more or give feedback, please file an issue or send an email to [email protected].

How it works

This system comprises a Lambda function (4) which listens to EventBridge events (3) triggered on every ECS task run (1). The function checks that the task's container image has a valid cosign signature in ECR (2) with a specific public key, provided directly or stored in KMS (5). If the check fails, the function terminates the task and sends a notification to an SNS topic (7) which you can subscribe to via email.

In order:

1. Start an ECS task in the cluster
2. The task definition has the container image stored in ECR
3. EventBridge sends a notification to Lambda
4. Cluster and Task definition is sent to function
5. KMS key that has signed an image
6. Lambda function evaluates if container image is signed w/ KMS
7. If not signed with specified key it does two things
   1. Stop task definition
   2. SNS notification email to alert that the service/task has been stopped

Quickstart

Requirements and preliminaries.

You will need the following tools installed:

• make (e.g., GNU make)
• Terraform: for local testing
• AWS CLI and AWS SAM CLI: for deploying
• cosign: for generating keys
• docker: if you need to make images

You should configure the AWS CLI for your project and account.

Deploy

To deploy, run:

make sam_deploy

It will check against the key in cosign.pub (see detailed instructions for how to change this).

Test it

To test, we need an ECS cluster on which to run our signed/unsigned tasks. The terraform subdirectory contains a Terraform template for such a cluster, with corresponding task definitions. First, initialize (to download the AWS provider for Terraform), then deploy:

make tf_init
make tf_apply  # run `make tf_plan` to see the plan first

We can then run our tasks (these will run two public "hello world" Alpine images with and without a signature):

make run_unsigned_task
make run_signed_task

Check that it worked. You should see the unsigned task in the STOPPED tasks and the signed task in the RUNNING tasks:

make task_status

Cleanup

To clean up, run:

make stop_tasks
make tf_destroy
make sam_delete

Detailed instructions

cosign-ecs-verify uses a SAM template (template.yml) to create:

• A Lambda function (source in cosign-ecs-function/) which:
  • Runs on every ECS task state change, triggered by EventBridge.
  • Gets the key for signature verification.
  • For each container in the event:
    • Verifies the container image, terminating the task and sending a notification if it is invalid.
• An Amazon SNS topic: if the function stops an unsigned container image, it will send a message to this topic.
  • You can configure email notifications for this topic to be alerted whenever an unverified image is stopped.
  • Messages sent to the topic are encrypted using a key in KMS.

To configure, run sam deploy with either the KeyArn set to a KMS key to use, or KeyPem set to a full public key in PEM format. In the provided Makefile, we hardcode the key in cosign.pub.

We provide a test ECS cluster configuration (in terraform/), containing;

• An ECS cluster:
  • Configured to log to CloudWatch (which will trigger the Lambda via EventBridge).
  • Associated task definitions signed and unsigned, which (by default) run images.
• KMS keys and IAM permissions for the above.

If you would like to use your own images (for example, from a previous post on the Chainguard blog), export $IMAGE_URL_SIGNED and $IMAGE_URL_UNSIGNED before running make tf_apply. We give instructions below for making your own to test.

Key Pair for cosign

We need a key against which to verify image signatures. If you have an existing keypair for cosign in AWS KMS, set it:

export KEY_ALIAS=my-key

Otherwise, we can make one:

export KEY_ALIAS=my-key
export AWS_SDK_LOAD_CONFIG=true
make key_gen

Signed and unsigned images

To see cosign-ecs-verify in action, you need an example of a signed and unsigned image. Here, we'll build two simple images, push them to Amazon ECR, and sign only one.

First, login to ECR with Docker. We recommend using a credential helper for docker, but we also provide a make target make ecr_auth that will authenticate you to the default registry.

Then, we can create a repository for the signed/unsigned images.

REPO_URL=$(aws ecr create-repository \
    --repository-name $REPO_NAME \
    --query repository.repositoryUri \
    --output text)

Finally, we can build and push two simple images (see Dockerfile):

# Export these so we can make ECR task definitions for running them.
export IMAGE_URL_SIGNED=$REPO_URL:signed
export IMAGE_URL_UNSIGNED=$REPO_URL:unsigned
# Make 2 example images and push both.
# The --build-arg is to make sure the images have different digests.
docker build . --build-arg signed=true --tag $IMAGE_URL_SIGNED
docker build . --build-arg signed=false --tag $IMAGE_URL_UNSIGNED
docker push $IMAGE_URL_SIGNED
docker push $IMAGE_URL_UNSIGNED

And sign only one of them (make sure you have a key pair):

cosign sign --key awskms:///alias/$KEY_ALIAS $IMAGE_URL_SIGNED

Now, you can proceed with the Terraform instructions.

Local Development and Testing

We require Go 1.17 for development.

You can also use the SAM local feature to run in a simulated Lambda environment:

make sam_local
make sam_local_debug

License

Apache License 2.0