Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add variable for ephemeral_storage in metadata-service aws_ecs_task_definition #86

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Add variable for ephemeral_storage in metadata-service aws_ecs_task_definition #86

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2aa9146
add variable for ECS task ephemeral_storage
jkuypers93 Feb 12, 2024
2a0df44
fix typo + format
jkuypers93 Feb 12, 2024
ba59eae
add README definitions
jkuypers93 Feb 12, 2024
5de55df
set ephemeral storage as block + correct readme
jkuypers93 Feb 14, 2024
56be04a
fix docs format
jkuypers93 Feb 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,7 @@ resource "local_file" "metaflow_config" {
| <a name="input_metadata_service_container_image"></a> [metadata\_service\_container\_image](#input\_metadata\_service\_container\_image) | Container image for metadata service | `string` | `""` | no |
| <a name="input_metadata_service_enable_api_basic_auth"></a> [metadata\_service\_enable\_api\_basic\_auth](#input\_metadata\_service\_enable\_api\_basic\_auth) | Enable basic auth for API Gateway? (requires key export) | `bool` | `true` | no |
| <a name="input_metadata_service_enable_api_gateway"></a> [metadata\_service\_enable\_api\_gateway](#input\_metadata\_service\_enable\_api\_gateway) | Enable API Gateway for public metadata service endpoint | `bool` | `true` | no |
| <a name="input_metadata_service_ephemeral_storage"></a> [metadata\_service\_ephemeral\_storage](#input\_metadata\_service\_ephemeral\_storage) | The Gb amount of disk storage to set for the ECS task [21-200] | `number` | `21` | no |
| <a name="input_resource_prefix"></a> [resource\_prefix](#input\_resource\_prefix) | string prefix for all resources | `string` | `"metaflow"` | no |
| <a name="input_resource_suffix"></a> [resource\_suffix](#input\_resource\_suffix) | string suffix for all resources | `string` | `""` | no |
| <a name="input_subnet1_id"></a> [subnet1\_id](#input\_subnet1\_id) | First subnet used for availability zone redundancy | `string` | n/a | yes |
Expand Down
37 changes: 19 additions & 18 deletions main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,24 +23,25 @@ module "metaflow-metadata-service" {
resource_prefix = local.resource_prefix
resource_suffix = local.resource_suffix

access_list_cidr_blocks = var.access_list_cidr_blocks
database_name = module.metaflow-datastore.database_name
database_password = module.metaflow-datastore.database_password
database_username = module.metaflow-datastore.database_username
db_migrate_lambda_zip_file = var.db_migrate_lambda_zip_file
datastore_s3_bucket_kms_key_arn = module.metaflow-datastore.datastore_s3_bucket_kms_key_arn
enable_api_basic_auth = var.metadata_service_enable_api_basic_auth
enable_api_gateway = var.metadata_service_enable_api_gateway
fargate_execution_role_arn = module.metaflow-computation.ecs_execution_role_arn
iam_partition = var.iam_partition
metadata_service_container_image = local.metadata_service_container_image
metaflow_vpc_id = var.vpc_id
rds_master_instance_endpoint = module.metaflow-datastore.rds_master_instance_endpoint
s3_bucket_arn = module.metaflow-datastore.s3_bucket_arn
subnet1_id = var.subnet1_id
subnet2_id = var.subnet2_id
vpc_cidr_blocks = var.vpc_cidr_blocks
with_public_ip = var.with_public_ip
access_list_cidr_blocks = var.access_list_cidr_blocks
database_name = module.metaflow-datastore.database_name
database_password = module.metaflow-datastore.database_password
database_username = module.metaflow-datastore.database_username
db_migrate_lambda_zip_file = var.db_migrate_lambda_zip_file
datastore_s3_bucket_kms_key_arn = module.metaflow-datastore.datastore_s3_bucket_kms_key_arn
enable_api_basic_auth = var.metadata_service_enable_api_basic_auth
enable_api_gateway = var.metadata_service_enable_api_gateway
fargate_execution_role_arn = module.metaflow-computation.ecs_execution_role_arn
iam_partition = var.iam_partition
metadata_service_container_image = local.metadata_service_container_image
metadata_service_ephemeral_storage = var.metadata_service_ephemeral_storage
metaflow_vpc_id = var.vpc_id
rds_master_instance_endpoint = module.metaflow-datastore.rds_master_instance_endpoint
s3_bucket_arn = module.metaflow-datastore.s3_bucket_arn
subnet1_id = var.subnet1_id
subnet2_id = var.subnet2_id
vpc_cidr_blocks = var.vpc_cidr_blocks
with_public_ip = var.with_public_ip

standard_tags = var.tags
}
Expand Down
3 changes: 2 additions & 1 deletion modules/metadata-service/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ If the `access_list_cidr_blocks` variable is set, only traffic originating from
| <a name="input_metadata_service_container_image"></a> [metadata\_service\_container\_image](#input\_metadata\_service\_container\_image) | Container image for metadata service | `string` | `""` | no |
| <a name="input_metadata_service_cpu"></a> [metadata\_service\_cpu](#input\_metadata\_service\_cpu) | ECS task CPU unit for metadata service | `number` | `512` | no |
| <a name="input_metadata_service_memory"></a> [metadata\_service\_memory](#input\_metadata\_service\_memory) | ECS task memory in MiB for metadata service | `number` | `1024` | no |
| <a name="input_metadata_service_ephemeral_storage"></a> [metadata\_service\_ephemeral\_storage](#input\_metadata\_service\_ephemeral\_storage) | Disk space for disk ECS task [21-200] | `number` | `21` | no |
| <a name="input_metaflow_vpc_id"></a> [metaflow\_vpc\_id](#input\_metaflow\_vpc\_id) | ID of the Metaflow VPC this SageMaker notebook instance is to be deployed in | `string` | n/a | yes |
| <a name="input_rds_master_instance_endpoint"></a> [rds\_master\_instance\_endpoint](#input\_rds\_master\_instance\_endpoint) | The database connection endpoint in address:port format | `string` | n/a | yes |
| <a name="input_resource_prefix"></a> [resource\_prefix](#input\_resource\_prefix) | Prefix given to all AWS resources to differentiate between applications | `string` | n/a | yes |
Expand All @@ -52,4 +53,4 @@ If the `access_list_cidr_blocks` variable is set, only traffic originating from
| <a name="output_metadata_svc_ecs_task_role_arn"></a> [metadata\_svc\_ecs\_task\_role\_arn](#output\_metadata\_svc\_ecs\_task\_role\_arn) | This role is passed to AWS ECS' task definition as the `task_role`. This allows the running of the Metaflow Metadata Service to have the proper permissions to speak to other AWS resources. |
| <a name="output_migration_function_arn"></a> [migration\_function\_arn](#output\_migration\_function\_arn) | ARN of DB Migration Function |
| <a name="output_network_load_balancer_dns_name"></a> [network\_load\_balancer\_dns\_name](#output\_network\_load\_balancer\_dns\_name) | The DNS addressable name for the Network Load Balancer that accepts requests and forwards them to our Fargate MetaData service instance(s) |
<!-- END_TF_DOCS -->
<!-- END_TF_DOCS -->
3 changes: 3 additions & 0 deletions modules/metadata-service/ecs.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ EOF
execution_role_arn = var.fargate_execution_role_arn
cpu = var.metadata_service_cpu
memory = var.metadata_service_memory
ephemeral_storage {
size_in_gib = var.metadata_service_ephemeral_storage
}

tags = merge(
var.standard_tags,
Expand Down
6 changes: 6 additions & 0 deletions modules/metadata-service/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,12 @@ variable "metadata_service_memory" {
description = "ECS task memory in MiB for metadata service"
}

variable "metadata_service_ephemeral_storage" {
type = number
default = 21
description = "The Gb amount of disk storage to set for the ECS task [21-200]"
}

variable "metaflow_vpc_id" {
type = string
description = "ID of the Metaflow VPC this SageMaker notebook instance is to be deployed in"
Expand Down
6 changes: 6 additions & 0 deletions variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,12 @@ variable "metadata_service_container_image" {
description = "Container image for metadata service"
}

variable "metadata_service_ephemeral_storage" {
type = number
default = 21
description = "The Gb amount of disk storage to set for the ECS task [21-200]"
}

variable "metadata_service_enable_api_basic_auth" {
type = bool
default = true
Expand Down