Skip to content

Latest commit

 

History

History

prefect2-mig

README.md

Prefect Agent on a GCP Managed Instance Group

Purpose

This recipe will walk you through the process to deploy a Prefect Agent using a GCP Managed Instance Group

Prerequisites

  1. Privileges to create service accounts & instances in GCP
  2. Terraform CLI Locally
  3. A Public Subnet with allowed ingress and egress Firewall rules to deploy the instances to or a Private Subnet connected to the internet via a NAT Gateway

Steps

  1. Set your default GCP Project to the Project in which the instance will be deployed to via gcloud auth application-default login
  2. Create a VPC and Subnet that have external internet access
  3. Create a Work Queue in Prefect Cloud that the agent will be associated with
  4. Run terraform apply from your local machine
    1. Pass in requested variables
  5. Wait for the Work Queue to become Healthy in the Prefect Cloud UI (Typically within 5 minutes)
  6. You should now be able to run Deployments against your new Prefect Agent
    1. Note that the VM has only Docker and Prefect installed by default. Other possible python modules may need to be added by updating the prefect-agent.sh.tpl file to include the installation of other python modules.

Requirements

Name Version
terraform ~> 1
google 4.44.1

Providers

Name Version
google 4.44.1

Modules

Name Source Version
instance_group terraform-google-modules/vm/google//modules/mig 7.9.0
instance_template terraform-google-modules/vm/google//modules/instance_template 7.9.0

Resources

Name Type
google_project_iam_binding.prefect_agent_instance_group resource
google_service_account.prefect_agent resource

Inputs

Name Description Type Default Required
prefect_account_id prefect cloud account ID string n/a yes
prefect_api_key prefect cloud api key string n/a yes
prefect_workspace_id prefect cloud workspace ID string n/a yes
project_id google cloud project ID string n/a yes
region region to deploy the resources to string n/a yes
subnet subnet to deploy the managed instance group string n/a yes
work_queue prefect cloud work queue name string n/a yes
disk_size Size of the Prefect Agent VM disk string "20" no
disk_type Disk type to be used by the Prefect Agent VM string "pd-standard" no
machine_type GCP Machine type to be used for the Prefect Agent VM string "n2d-highcpu-2" no
name_prefix Prefix for the instance name string "prefect-agent" no
num_vm Number of deployed VMs in the managed instance group number 1 no
preemptible prefect cloud account ID bool false no

Outputs

No outputs.