Annotations ending in kubernetes.io not being ignored

[mersive-raypitmon]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.14.6
Kubernetes provider version: v2.25.2
Kubernetes version: v1.26.6-gke.1700

Note: I just ran similar terraform against Kubernetes version: v1.25.10-gke.2700 and I don't see the issue, so is this a k8s problem and not a provider issue?

Affected Resource(s)

kubernetes_stateful_set

Steps to Reproduce

terraform plan

Expected Behavior

My STS resource should not show any changes.

Actual Behavior

The output from the plan command states that it will null out a server-set annotation ending in kubernetes.io. The provider documentation states this:

! Note
By default, the provider ignores any annotations whose key names end with kubernetes.io.
...

It is not ignoring that annotation, as here is my output from terraform plan:

  # kubernetes_stateful_set.pgwatch2_sts will be updated in-place
  ~ resource "kubernetes_stateful_set" "my_sts" {
        id               = "default/mysts"
        # (1 unchanged attribute hidden)


      ~ spec {
            # (4 unchanged attributes hidden)


          ~ template {
              ~ metadata {
                  ~ annotations = {
                      - "kubectl.kubernetes.io/restartedAt" = "2023-06-14T14:48:46-06:00" -> null
                    }
                    # (2 unchanged attributes hidden)
                }

                # (1 unchanged block hidden)
            }

            # (2 unchanged blocks hidden)
        }
        # (1 unchanged block hidden)
    }

I've ran terraform plan on this bit of code several times over the past year, and I haven't noticed this until now. Also note that I am not manually setting any annotations in spec.template.metadata.annotations

I checked the actual YAML for the resource running in the GKE console and it does have the annotation:

spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: myapp
  serviceName: myservice
  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/restartedAt: "2023-06-14T14:48:46-06:00"
      creationTimestamp: null
      labels:
        app: myapp

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jrhouston]

Thanks for opening this @mersive-raypitmon. This looks like a regression. I tried the following steps:

1. use terraform apply to apply a config with a stateful_set
2. use kubectl rollout restart to restart the statefulset
3. run terraform plan again

On v2.52.2 this results in a diff showing the restartedAt label being deleted. On v2.24.0 there is no diff – so this is a bug.

[suhaylsoprano]

Upgraded to v2.26.0 today and this issue seems to have regressed again. Any applies will attempt to remove the restartedAt annotations and cause the pods to restart.

Have fallen back to v2.24.0 in the meantime.

[suhaylsoprano]

Upgraded to v2.26.0 today and this issue seems to have regressed again. Any applies will attempt to remove the restartedAt annotations and cause the pods to restart.

Have fallen back to v2.24.0 in the meantime.

Missed out the documentation, this is resolved. Reference for the next person: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#examples-1

[jundersand-chwy]

Upgraded to v2.26.0 today and this issue seems to have regressed again. Any applies will attempt to remove the restartedAt annotations and cause the pods to restart.
Have fallen back to v2.24.0 in the meantime.

Missed out the documentation, this is resolved. Reference for the next person: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#examples-1

@BBBmau @arybolovlev I see that there was clarification added to docs regarding this issue, but I am unable to get kubectl.kubernetes.io/restartedAt properly ignored after v2.24.0 even with the documented ignore_changes lifecycle example as shown below. We have had to pin our provider to v2.24.0 in the meantime to prevent unwanted restarts.

Terraform version: v0.13.7
Kubernetes provider version: v2.27.0
Kubernetes version: v1.25.16-eks-77b1e4e

resource "kubernetes_deployment" "my_deployment" {
  lifecycle {
    ignore_changes = [
      spec[0].template[0].metadata[0].annotations["kubectl.kubernetes.io/restartedAt"]
    ]
  }
}

Still results in the following plan output following a manual kubectl rollout restart of my_deployment.

  # module.my-eks.kubernetes_deployment.my_deployment will be updated in-place

   ~ resource "kubernetes_deployment" "my_deployment" {

      . . .

       ~ spec {

        . . .

           ~ template {

               ~ metadata {

                   ~ annotations = {

                       - "kubectl.kubernetes.io/restartedAt" = "2024-03-15T14:06:31-05:00" -> null

                     }

                     . . .

Should this issue be reopened, or should I create a new one? Thanks.

[mersive-raypitmon]

@jundersand-chwy - I just ran into this, and when I added the lifecycle block the change went away, so the lifecycle block is working for me, at least on GKE.
My versions:

$ terraform -version
Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/google v2.20.3
+ provider registry.terraform.io/hashicorp/kubernetes v2.27.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.0
+ provider registry.terraform.io/terraform-providers/ignition v1.2.1

The lifecycle block I just added:

  lifecycle {
    ignore_changes = [
      spec[0].template[0].metadata[0].annotations["kubectl.kubernetes.io/restartedAt"],
    ]
  }

Terraform plan before adding the lifecycle:

  # kubernetes_deployment.server will be updated in-place
  ~ resource "kubernetes_deployment" "server" {
        id               = "ns1/server"
        # (1 unchanged attribute hidden)


      ~ spec {
            # (5 unchanged attributes hidden)



          ~ template {
              ~ metadata {
                  ~ annotations = {
                      - "kubectl.kubernetes.io/restartedAt" = "2024-03-13T08:56:54-06:00" -> null
                    }
                    # (2 unchanged attributes hidden)
                }

                # (1 unchanged block hidden)
            }
            # (2 unchanged blocks hidden)
        }
        # (1 unchanged block hidden)
    }

(After adding the lifecycle block, the resource didn't show in my plan)

[jundersand-chwy]

@jundersand-chwy - I just ran into this, and when I added the lifecycle block the change went away, so the lifecycle block is working for me, at least on GKE. My versions:

$ terraform -version
Terraform v0.14.6
+ provider registry.terraform.io/hashicorp/google v2.20.3
+ provider registry.terraform.io/hashicorp/kubernetes v2.27.0
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.0
+ provider registry.terraform.io/terraform-providers/ignition v1.2.1

Thanks, I confirmed locally that upgrading to Terraform v0.14.6 in our project results in the expected behavior. So, anyone stuck on Terraform v0.13.7 may need to pin provider kubernetes v2.24.0 or migrate to a newer TF version.

[jackpordi]

I am currently using Terraform CDK, and ignoreAnnotations isn't working at the provider level.

import { KubernetesProvider } from "@cdktf/provider-kubernetes/lib/provider";

void new KubernetesProvider(this, "k8s", {
  host: this.cluster.eksCluster.endpoint,
  clusterCaCertificate: Fn.base64decode(this.cluster.eksCluster.certificateAuthority.get(0).data),
  token: this.cluster.auth.token,
  ignoreAnnotations: [
    "kubectl.kubernetes.io/restartedAt",
  ],
});

I have also tried the "kubectl\\.kubernetes\\.io\\/restartedAt" to no avail.

I am on Terraform 1.8.6, cdktf version 0.20.7, @cdktf/provider-kubernetes version 11.5.0 which bundles prebuilt bindings for hashicorp/kubernetes provider version 2.30.0

Doing it on a per-deployment basis with the life cycle ignore does work, but it obviously is a bit less ideal given the duplication.