Skip to content

Latest commit

 

History

History

tf-prefect2-ecs-agent

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

Prefect 2 Agent on ECS Fargate

This recipe demonstrates how to deploy a Prefect 2 agent onto ECS Fargate using Terraform. It is intended to be used as a Terraform module as described in Usage below. It assumes you have Terraform installed, and was tested with Terraform v1.2.7.

Note that flows will run inside the agent ECS task, as opposed to becoming their own ECS tasks.

Usage

image

To start with you will need your Prefect account ID, workspace ID, and API key. You will also need to pick one or more subnets that Fargate will launch into, as well as give your deployment a name.

In order to avoid accidentally committing your API key, consider structuring your project as follows,

.
├── main.tf
├── terraform.tfvars
└── variables.tf
// variables.tf
variable prefect_api_key {}
// terraform.tfvars
// Don't panic! This isn't a real API key
prefect_api_key = "pnu_bcf655365883614d468990896264f6a30372"
// main.tf

provider "aws" {
  region = "us-east-1"
}

// Don't panic! These values are just random uuid.uuid4()s
module "prefect_ecs_agent" {
  source = "github.com/PrefectHQ/prefect-recipes//devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent"

  agent_subnets        = [
    "subnet-014aa5f348034e45b",
    "subnet-df23ae9eab1f49af9"
  ]
  name                 = "dev"
  prefect_account_id   = "6e02a1db-07de-4760-a15d-60d8fe0b04e1"
  prefect_api_key      = var.prefect_api_key
  prefect_workspace_id = "54cdfc71-9f13-41ba-9492-e1cf24eed185"
  vpc_id               = "vpc-acfc2092275244ca8"
}

Assuming the file structure above, you can run terraform init followed by terraform apply to create the resources. Check out the Inputs section below for more options.

Reference

The terraform docs below can be generated with the following command:

terraform-docs markdown table . --output-file README.md

Requirements

Name Version
aws ~> 4.0

Providers

Name Version
aws 4.27.0

Modules

No modules.

Resources

Name Type
aws_cloudwatch_log_group.prefect_agent_log_group resource
aws_ecs_cluster.prefect_agent_cluster resource
aws_ecs_cluster_capacity_providers.prefect_agent_cluster_capacity_providers resource
aws_ecs_service.prefect_agent_service resource
aws_ecs_task_definition.prefect_agent_task_definition resource
aws_iam_role.prefect_agent_execution_role resource
aws_iam_role.prefect_agent_task_role resource
aws_secretsmanager_secret.prefect_api_key resource
aws_secretsmanager_secret_version.prefect_api_key_version resource
aws_security_group.prefect_agent resource
aws_security_group_rule.https_outbound resource
aws_region.current data source

Inputs

Name Description Type Default Required
agent_cpu CPU units to allocate to the agent number 1024 no
agent_desired_count Number of agents to run number 1 no
agent_extra_pip_packages Packages to install on the agent assuming image is based on prefecthq/prefect string "prefect-aws s3fs" no
agent_image Container image for the agent. This could be the name of an image in a public repo or an ECR ARN string "prefecthq/prefect:2-python3.10" no
agent_log_retention_in_days Number of days to retain agent logs for number 30 no
agent_memory Memory units to allocate to the agent number 2048 no
agent_queue_name Prefect queue that the agent should listen to string "default" no
agent_subnets Subnets to place the agent in list(string) n/a yes
agent_task_role_arn Optional task role ARN to pass to the agent. If not defined, a task role will be created string null no
name Unique name for this agent deployment string n/a yes
prefect_account_id Prefect cloud account ID string n/a yes
prefect_api_key Prefect cloud API key string n/a yes
prefect_workspace_id Prefect cloud workspace ID string n/a yes
vpc_id VPC ID in which to create all resources string n/a yes

Outputs

Name Description
prefect_agent_cluster_name n/a
prefect_agent_execution_role_arn n/a
prefect_agent_security_group n/a
prefect_agent_service_id n/a
prefect_agent_task_role_arn n/a