tf-prefect2-ecs-agent

Prefect 2 Agent on ECS Fargate

This recipe demonstrates how to deploy a Prefect 2 agent onto ECS Fargate using Terraform. It is intended to be used as a Terraform module as described in Usage below. It assumes you have Terraform installed, and was tested with Terraform v1.2.7.

Note that flows will run inside the agent ECS task, as opposed to becoming their own ECS tasks.

Usage

To start with you will need your Prefect account ID, workspace ID, and API key. You will also need to pick one or more subnets that Fargate will launch into, as well as give your deployment a name.

In order to avoid accidentally committing your API key, consider structuring your project as follows,

.
├── main.tf
├── terraform.tfvars
└── variables.tf

// variables.tf
variable prefect_api_key {}

// terraform.tfvars
// Don't panic! This isn't a real API key
prefect_api_key = "pnu_bcf655365883614d468990896264f6a30372"

// main.tf

provider "aws" {
  region = "us-east-1"
}

// Don't panic! These values are just random uuid.uuid4()s
module "prefect_ecs_agent" {
  source = "github.com/PrefectHQ/prefect-recipes//devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent"

  agent_subnets        = [
    "subnet-014aa5f348034e45b",
    "subnet-df23ae9eab1f49af9"
  ]
  name                 = "dev"
  prefect_account_id   = "6e02a1db-07de-4760-a15d-60d8fe0b04e1"
  prefect_api_key      = var.prefect_api_key
  prefect_workspace_id = "54cdfc71-9f13-41ba-9492-e1cf24eed185"
  vpc_id               = "vpc-acfc2092275244ca8"
}

Assuming the file structure above, you can run terraform init followed by terraform apply to create the resources. Check out the Inputs section below for more options.

Reference

The terraform docs below can be generated with the following command:

terraform-docs markdown table . --output-file README.md

Requirements

Name	Version
aws	~> 4.0

Providers

Name	Version
aws	4.27.0

Modules

No modules.

Resources

Name	Type
aws_cloudwatch_log_group.prefect_agent_log_group	resource
aws_ecs_cluster.prefect_agent_cluster	resource
aws_ecs_cluster_capacity_providers.prefect_agent_cluster_capacity_providers	resource
aws_ecs_service.prefect_agent_service	resource
aws_ecs_task_definition.prefect_agent_task_definition	resource
aws_iam_role.prefect_agent_execution_role	resource
aws_iam_role.prefect_agent_task_role	resource
aws_secretsmanager_secret.prefect_api_key	resource
aws_secretsmanager_secret_version.prefect_api_key_version	resource
aws_security_group.prefect_agent	resource
aws_security_group_rule.https_outbound	resource
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
agent_cpu	CPU units to allocate to the agent	number	1024	no
agent_desired_count	Number of agents to run	number	1	no
agent_extra_pip_packages	Packages to install on the agent assuming image is based on prefecthq/prefect	string	"prefect-aws s3fs"	no
agent_image	Container image for the agent. This could be the name of an image in a public repo or an ECR ARN	string	"prefecthq/prefect:2-python3.10"	no
agent_log_retention_in_days	Number of days to retain agent logs for	number	30	no
agent_memory	Memory units to allocate to the agent	number	2048	no
agent_queue_name	Prefect queue that the agent should listen to	string	"default"	no
agent_subnets	Subnets to place the agent in	list(string)	n/a	yes
agent_task_role_arn	Optional task role ARN to pass to the agent. If not defined, a task role will be created	string	null	no
name	Unique name for this agent deployment	string	n/a	yes
prefect_account_id	Prefect cloud account ID	string	n/a	yes
prefect_api_key	Prefect cloud API key	string	n/a	yes
prefect_workspace_id	Prefect cloud workspace ID	string	n/a	yes
vpc_id	VPC ID in which to create all resources	string	n/a	yes

Outputs

Name	Description
prefect_agent_cluster_name	n/a
prefect_agent_execution_role_arn	n/a
prefect_agent_security_group	n/a
prefect_agent_service_id	n/a
prefect_agent_task_role_arn	n/a