[Bug]: aws_db_instance - configured value for vpc_security_group_ids is incorrectly changed during TF plan

[ktham]

Terraform Core Version

1.7.5

AWS Provider Version

5.84.0

Affected Resource(s)

• aws_db_instance
• aws_rds_cluster

Note: My examples are for aws_db_instance, but the same problem also applies to aws_rds_cluster

Expected Behavior

We should expect to see Terraform plan output that looks like this when a new security group is added to vpc_security_group_ids

      ~ vpc_security_group_ids                = [
          + "sg-46c08c0f",
          + (known after apply),
        ]

Actual Behavior

However, the aws_db_instance resource is doing something unexpected, we instead see this

      ~ vpc_security_group_ids                = [
          - "sg-46c08c0f",
        ] -> (known after apply)

👉 👉 sg-46c08c0f should be retained, but in the plan renderer, what we're seeing is that the vpc_security_group_ids list as a whole is being replaced with an unknown value.

This is very concerning behavior because it gives the impression that the AWS provider might potentially be revoking security group IDs that we expect to be retained.

Terraform Configuration Files

Terraform configuration (step 1)

resource "aws_security_group" "example" {
  # ...
}

resource "aws_db_instance" "example" {
  vpc_security_group_ids = [
    aws_security_group.example.id,
  ]
  # ...
}

Terraform configuration (step 2)

resource "aws_security_group" "example" {
  # ...
}


# Add this:
resource "aws_security_group" "additional" {
  # ...
}

resource "aws_db_instance" "example" {
  vpc_security_group_ids = [
    aws_security_group.example.id,

    # Add this:
    aws_security_group.additional.id,
  ]
  # ...
}

Steps to Reproduce

Run terraform apply for step 1 Terraform configuration, then run terraform apply for step 2 configuration from above.

References

This is not a problem with Terraform Core, but a problem with the provider itself.

Please See:

• Terraform plans should show set members that will be retained post-apply terraform#36221
• OpenTofu plans should show set members that will be retained post-apply opentofu/opentofu#2296

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @ktham 👋 Thank you for taking the time to raise this! In this case, the provider is behaving as I would expect. When you add the additional aws_security_group, its ID cannot be known until it's created. Since you've referenced that ID to aws_db_instance.example.vpc_security_group_ids, the full set cannot be known until apply time, and thus shows in the plan as (known after apply). While list types are ordered would show individual values like you expected, set types are unordered, and as a result, appear as wholly known or unknown like this. The vpc_security_group_ids needs to be a set, as the order they're sent to/returned from AWS in is inconsequential, and so isn't guaranteed. The set/list behavior is controlled by Terraform Core, and not something the AWS Provider has any control over.

If that is unacceptable in your situation, you can get around this by refactoring your configuration to separate the creation of the security groups and DB instance. Doing so would mean that by the time the change to the DB is being planned, the ID of the new security group is known and will be shown in the plan.

Since this is a behavior that the AWS provider is unable to control, and can be solved by refactoring the configuration, I'm going to close this issue. If you run into any other unexpected behavior, please do let us know!

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

[ktham]

Hi @justinretzolk , I believe you have prematurely closed the issue. It is true that the full set cannot be known until apply time but I believe you have the incorrect understanding of expected Terraform behavior. Please see the linked issues.

[ktham]

cc @jbardin and @apparentlymart

[ktham]

Posting here for better clarity - @jbardin mentioned in hashicorp/terraform#36221 that Terraform Core currently does what I am asking for above

resource "terraform_data" "a" {
}

resource "terraform_data" "b" {
  input = toset([
    "known value",
    terraform_data.a.id,
  ])
}

Results in the plan:

  # terraform_data.b will be created
  + resource "terraform_data" "b" {
      + id     = (known after apply)
      + input  = [
          + "known value",
          + (known after apply),
        ]
      + output = (known after apply)
    }

Since you are providing the configuration for the vpc_security_group_ids attribute, it seems the provider is incorrectly changing the configured value during the plan, but as a legacy provider it can get away with the behavior for compatibility reasons (there should be a corresponding warning in the logs about Provider "aws" produced an invalid plan for aws_db_instance...).

It would be up to the provider to fix the resource implementation, which involves upgrading the SDK to use the newer Terraform provider framework.

However, the aws_db_instance resource does not do this and it seems like the provider is likely producing an invalid plan and is able to get away with this incorrect behavior due to the legacy plugin SDK. If this is true, then I think @justinretzolk , this GH issue should remain open and needs to be closed with a proper fix.

So in other words, Terraform is expected to be able to render the full detail of a partially-unknown set if the provider is indeed providing one, and it seems like the AWS provider isn't currently doing that here for aws_db_instance and aws_rds_cluster