Removing default tags fails to untag resources on v6

[danielrbradley]

The final step (5) in the test TestAccDefaultTagsGo is currently failing in CI but passes locally.

We're disabling the test temporarily to unblock the release but should diagnose the failure as a matter of urgency post-release.

Step 5 is asserting that when all default tags are removed from the provider, then they're also removed from all resources. This fails and the previous default tags remain on the resources.

[danielrbradley]

Example of original failure: https://github.com/pulumi/pulumi-aws/actions/runs/5602049680/job/15176971731#step:23:115
Example of failure after refactoring tests: https://github.com/pulumi/pulumi-aws/actions/runs/5643381135/job/15286148417#step:24:286

[mikhailshilkov]

@t0yv0 I assigned to you since you were looking at it yesterday - please let me know if you don't have time to continue

[t0yv0]

I can reproduce this issue locally on the latest v6 branch.

I have been digging into this issue.

The issue appears to be an incorrect resource Diff when moving from a state that defines some provider-default tags to a state where the state defines no provider-default tags.

Adding or removing some but not all default tags, or changing tags works as expected.

[t0yv0]

When this feature does work, we get a non-empty InstanceDiff; the fix for default tags looked at how we used to suppress non-empty diffs for computed attributes and we stopped doing that for XComputedInput to make an exception for tags_all. However in this case that's not happening at all, we even don't get to this call site:

• on v6 we use simpleDiffViaPlanState
• as inputs to this we have tags_all in prior state set to {"thwomp": "pow"} but nil in rawConfig
• when we compute objchange.ProposedNew(priorState, rawConfig) it copies {"thwomp": "pow"} to planned state; this seems bad
• res.SimpleDiff returns empty diff because it's comparing {"thwomp": "pow"} to {"thwomp": "pow"}
• therefore forceNew detection fails to find anything
• therefore provider responds with Diff: None
• therefore bucket does not get updated, Update does not execute

[t0yv0]

I'm removing flaky/test labels AFAIK this is a legit product bug.

[t0yv0]

On second thought, when this does work, like adding a new tag, we also copy {"thwomp": "pow"} to the inputs of res.SimpleDiff however res.SimpleDiff responds with a correct diff. It figures out there's a new tag.. So somehow SimpleDiff engages TF hooks here. I need to dig deeper.

[iwahbe]

res.SimpleDiff eventually calls res.CustomizeDiff which is SetTagsDiff. SetTagsDiff is responsible for utilizing the provider's state to determine what the diff should be. Unfortunately, it looks like SetTagsDiff only works when there is a non-empty tag set. I don't think the special TF hook is what we are missing.

[t0yv0]

One laborious, but possible avenue of investigation is this. Does this example work when executed via TF CLI in TF world? If it does work, does that end up invoking SetTagsDiff, or how does it ensure the end result?

[mikhailshilkov]

Before we look at TF directly - Do I remember correctly that this test works in 5.x of our provider? If so, do we know what changed?

[iwahbe]

@t0yv0 The example does work in the TF world. I confirmed that before investing too much time in fixing the bug on our end.

@mikhailshilkov hashicorp/terraform-provider-aws#30793 changed how SetTagsDiff works. We can revert, but their PR fixed other bugs so I'm hesitant to back it out.

[iwahbe]

I've been experimenting with running terraform-provider-aws directly against TF. I've been using this program:

provider "aws" {
 region = "us-east-1"
-  default_tags { tags = { "foo": "bar" } }
+  # default_tags { tags = { "foo": "bar" } }
}
resource "aws_s3_bucket" "example" {}

This calls into SetTagsDiff as expected, but follows a different execution path then expected. diff.HasChange("tags") returns true, so the diff is recorded here.

This is different to how pulumi-aws runs through SetTagsDiff with an equivalent program. For us diff.HasChange("tags") returns false. This is sufficient to cause the bug we are observing.

Investigating further, we don't appear to set tags at all:

Running this program against v5 or v6 of pulumi-aws:

name: secret-random-yaml
runtime: yaml
resources:
  p:
    type: pulumi:providers:aws
    properties:
      defaultTags:
        tags:
          foo: buz
      region: us-west-2
    defaultProvider: true
  b:
    type: aws:s3:BucketV2
outputs:
  tags: ${b.tags}

We get an empty tags output on both versions:

𝛌 pulumi stack output tags
{}

[t0yv0]

Excellent sleuthing. This is very surprising and violates my mental model of what's happening. I thought that in the listed TF program "tags" is in fact not changing. I assume "tags" refers to the tags attribute of "example" bucket, not the provider. Do you have a test harness around to reproduce this result? Tracing the contents of diff *schema.ResourceDiff and/or the inputs into the diff computation can help understand how it's doing it.

[iwahbe]

More digging makes it even less cut and dry. Given the above program, the initial terraform apply gives this diff (tags_all but no tags):

  # aws_s3_bucket.example will be created
  + resource "aws_s3_bucket" "example" {
      + acceleration_status         = (known after apply)
      + acl                         = (known after apply)
      + arn                         = (known after apply)
      + bucket                      = (known after apply)
      + bucket_domain_name          = (known after apply)
      + bucket_prefix               = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + object_lock_enabled         = (known after apply)
      + policy                      = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags_all                    = {
          + "foo" = "bar"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)
    }

However, running terraform plan directly after that (on the same program) does create a diff on tags:

𝛌 terraform plan 
aws_s3_bucket.example: Refreshing state... [id=terraform-20230728140713668600000001]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # aws_s3_bucket.example has changed
  ~ resource "aws_s3_bucket" "example" {
        id                          = "terraform-20230728140713668600000001"
      + tags                        = {}
        # (11 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Changes to Outputs:
  + tags = {}

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.

Commenting out default_tags { tags = { "foo": "bar" } } and running terraform plan we see this diff (note the out of band tags addition):

𝛌 terraform plan 
aws_s3_bucket.example: Refreshing state... [id=terraform-20230728140713668600000001]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply" which may have affected this plan:

  # aws_s3_bucket.example has changed
  ~ resource "aws_s3_bucket" "example" {
        id                          = "terraform-20230728140713668600000001"
      + tags                        = {
          + "foo" = "bar"
        }
        # (11 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan
may include actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_s3_bucket.example will be updated in-place
  ~ resource "aws_s3_bucket" "example" {
        id                          = "terraform-20230728140713668600000001"
      ~ tags                        = {
          - "foo" = "bar" -> null
        }
      ~ tags_all                    = {
          - "foo" = "bar"
        } -> (known after apply)
        # (10 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Changes to Outputs:
  + tags = {}

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.

[iwahbe]

Another interesting note, where terraform plan results in an out of bound tags change:

  # aws_s3_bucket.example has changed
  ~ resource "aws_s3_bucket" "example" {
        id                          = "terraform-20230728140713668600000001"
      + tags                        = {
          + "foo" = "bar"
        }
        # (11 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

And then leads to this diff:

  # aws_s3_bucket.example will be updated in-place
  ~ resource "aws_s3_bucket" "example" {
        id                          = "terraform-20230728140713668600000001"
      ~ tags                        = {
          - "foo" = "bar" -> null
        }
      ~ tags_all                    = {
          - "foo" = "bar"
        } -> (known after apply)
        # (10 unchanged attributes hidden)

        # (3 unchanged blocks hidden)
    }

If we run terraform plan -refresh=false, we see this instead:

𝛌 terraform plan -refresh=false 

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

[iwahbe]

Running pulumi up --refresh=true does not duplicate terraforms behavior with regards to noticing a tags diff and then rectifying it.

[iwahbe]

Closed by #2655