Operation cannot be fulfilled on deployments- the object has been modified; please apply your changes to the latest version and try again

[kjq]

I am really perplexed as to what is causing this issue and flooding our logs each time a knative service is invoked. These are "fresh" first-time deployments applied with kubectl apply and not revisions.

failed to update deployment "function-XXX-50706878-2-00001-deployment": Operation cannot be fulfilled on deployments.apps "function-XXX-50706878-2-00001-deployment": the object has been modified; please apply your changes to the latest version and try again

It does not seem to harm anything but happens whenever the knative service is invoked.

I dont know what could be modifying it.

Running alongside the Knative Operator we have Tekton Operator, Strimzi Kafka Operator, etc.

I see this error in the controller logs:

{
    "severity": "ERROR",
    "timestamp": "2024-08-04T17:32:44.517466244Z",
    "logger": "controller",
    "caller": "revision/reconciler.go:303",
    "message": "Returned an error",
    "commit": "cc2085d",
    "knative.dev/pod": "controller-6dd9b8448-r4gvk",
    "knative.dev/controller": "knative.dev.serving.pkg.reconciler.revision.Reconciler",
    "knative.dev/kind": "serving.knative.dev.Revision",
    "knative.dev/traceid": "0b979477-7d41-4efa-bb3b-4e3e02fc1bf6",
    "knative.dev/key": "acme/function-XXX-50706878-2-00001",
    "targetMethod": "ReconcileKind",
    "error": "failed to update deployment \"function-XXX-50706878-2-00001-deployment\": Operation cannot be fulfilled on deployments.apps \"function-XXX-50706878-2-00001-deployment\": the object has been modified; please apply your changes to the latest version and try again",
    "stacktrace": "knative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision.(*reconcilerImpl).Reconcile\n\tknative.dev/serving/pkg/client/injection/reconciler/serving/v1/revision/reconciler.go:303\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/[email protected]/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/[email protected]/controller/controller.go:491"
}

The events I see, if this helps any, are:

acme               2m47s       Normal    Scheduled               pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Successfully assigned acme/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8 to ip-10-0-23-124.us-east-2.compute.internal
acme               2m48s       Normal    SuccessfulCreate        replicaset/function-XXX-50706878-2-00001-deployment-9d49cd465   Created pod: function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8
acme               2m48s       Normal    ScalingReplicaSet       deployment/function-XXX-50706878-2-00001-deployment             Scaled up replica set function-XXX-50706878-2-00001-deployment-9d49cd465 to 1 from 0
acme               87s         Warning   InternalError           revision/function-XXX-50706878-2-00001                          failed to update deployment "function-XXX-50706878-2-00001-deployment": Operation cannot be fulfilled on deployments.apps "function-XXX-50706878-2-00001-deployment": the object has been modified; please apply your changes to the latest version and try again
acme               2m47s       Normal    Pulling                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Pulling image "index.docker.io/logicdrop/clients@sha256:f25434f2f33ee4e3b9bfa9983beabbb00f1d2a2db93ce6f58182edfb16be2af1"
acme               2m39s       Normal    Created                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Created container user-container
acme               2m39s       Normal    Pulled                  pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Successfully pulled image "index.docker.io/logicdrop/clients@sha256:f25434f2f33ee4e3b9bfa9983beabbb00f1d2a2db93ce6f58182edfb16be2af1" in 8.305s (8.305s including waiting). Image size: 219639857 bytes.
acme               2m38s       Normal    Pulling                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Pulling image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:55e7d89003b3fb97ddfaf9d65326a7ccc69b437773c3eccab43cece771bc1eda"
acme               2m38s       Normal    Started                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Started container user-container
acme               2m37s       Normal    Created                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Created container queue-proxy
acme               2m37s       Normal    Pulled                  pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Successfully pulled image "gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:55e7d89003b3fb97ddfaf9d65326a7ccc69b437773c3eccab43cece771bc1eda" in 1.733s (1.733s including waiting). Image size: 10809573 bytes.
acme               2m36s       Normal    Started                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Started container queue-proxy
acme               2m32s       Warning   Unhealthy               pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Readiness probe failed: Get "http://10.0.27.136:8012/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
acme               87s         Normal    ScalingReplicaSet       deployment/function-XXX-50706878-2-00001-deployment             Scaled down replica set function-XXX-50706878-2-00001-deployment-9d49cd465 to 0 from 1
acme               87s         Normal    SuccessfulDelete        replicaset/function-XXX-50706878-2-00001-deployment-9d49cd465   Deleted pod: function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8
acme               87s         Normal    Killing                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Stopping container queue-proxy
acme               87s         Normal    Killing                 pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Stopping container user-container
acme               57s         Warning   Unhealthy               pod/function-XXX-50706878-2-00001-deployment-9d49cd465-j8ct8    Readiness probe failed: HTTP probe failed with statuscode: 503

This is pretty much an out-of-the-box install...

References:

• KPA and cluster autoscaler compatibility #14939 (comment)