Strimzi is not honoring the reason for restart events during the certificate renewal #10743

saeedya · 2024-10-21T12:01:09Z

saeedya
Oct 21, 2024

Hi
When the new cert renewal is happening , we are expecting the events as below:
CaCertRenewed CaCertRenewed CA certificates have been renewed, and the pod is restarted to run with the updated certificates.
Restart reasons
But we are getting "Pod has old revision".
@scholzj
Thanks
Saeed

scholzj · 2024-10-21T12:17:09Z

scholzj
Oct 21, 2024
Maintainer

I'm not sure this is unexpected. To make sure that the rolling update happens even when the in-memory state of the operator is lost, the certificate versions are tracked through annotations and when they change, that creates the Pod has old revision restart reason. So this does not seem necessarily wrong - at leats not without seeing the exact logs and understanding the actual scenario.

8 replies

saeedya Oct 23, 2024
Author

I have all those annotation, but when the client ca cert renewal happens still the annotation "strimzi.io/clients-ca-cert-generation: 0" and won't change. Pods are getting restarted but in the events i am getting "PodHasOldRevision" message not "CaCertRenewed".

scholzj Oct 23, 2024
Maintainer

Again, you would need to explain the exact scenarios, configs, and logs. You are asking for a very detailed explanations to a very generic questions.

saeedya Oct 23, 2024
Author

apiVersion: v1
items:
- apiVersion: kafka.strimzi.io/v1beta2
  kind: Kafka
  metadata:
    annotations:
      jmx-filter-configmap-name: eric-data-message-bus-kf-metrics-config
      mydomain.io/app: msg1
      mydomain.io/instance: msg1-instance
    creationTimestamp: "2024-10-18T15:14:53Z"
    generation: 16
    labels:
      application: msg1
      eric-data-distributed-coordinator-ed-access: "true"
      eric-sec-key-management-access: "true"
      eric-sec-sip-tls-access: "true"
      instance: msg1-instance
    name: eric-data-message-bus-kf-test
    namespace: saeed-op
    ownerReferences:
    - apiVersion: mbkf.data.ericsson.com/v1
      blockOwnerDeletion: false
      controller: false
      kind: MessageBus
      name: eric-data-message-bus-kf-test
      uid: bf8ba719-16ce-4704-8d13-81bfd477fd17
    resourceVersion: "2115903355"
    uid: 9899c51d-b8ec-401f-82d1-d7a202bf4526
  spec:
    entityOperator:
      template:
        pod:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/kind: Kafka
                      strimzi.io/name: eric-data-message-bus-kf-test-kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
          tolerations:
          - effect: NoSchedule
            key: key1
            operator: Equal
            value: value1
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                strimzi.io/kind: Kafka
                strimzi.io/name: eric-data-message-bus-kf-test-entity-operator
            maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
        userOperatorContainer:
          env:
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_ENTITY_OPERATOR_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
      userOperator:
        additionalVolumes:
        - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
          name: eric-sec-sip-tls-trusted-root-cert
          targetContainers:
          - user-operator
          volumeSource:
            secret:
              items:
              - key: ca.crt
                path: ca.crt
              secretName: eric-sec-sip-tls-trusted-root-cert
            type: secret
        logging:
          loggers:
            appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
              "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
              TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
              "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
              "user-operator", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file":
              "%file", "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
            appender.console.layout.type: PatternLayout
            appender.console.name: consoleLogger
            appender.console.target: SYSTEM_OUT
            appender.console.type: Console
            rootLogger.additivity: "false"
            rootLogger.appenderRef.console.ref: consoleLogger
          type: inline
        resources:
          limits:
            cpu: "2"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 512Mi
    kafka:
      additionalVolumes:
      - mountPath: /opt/kafka/certificates/tls-siptls-9093-certs/eric-data-message-bus-kf-test-client-ca-secret
        name: siptls-9093-eric-data-message-bus-kf-test-client-ca-secret-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: clientcacertbundle.pem
              path: ca.crt
            secretName: eric-data-message-bus-kf-test-client-ca-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
        name: eric-sec-sip-tls-trusted-root-cert
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca.crt
              path: ca.crt
            secretName: eric-sec-sip-tls-trusted-root-cert
          type: secret
      - mountPath: /opt/kafka/certificates/metrics-server-secret
        name: metrics-server-cert
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: clicert.pem
              path: clicert.pem
            - key: cliprivkey.pem
              path: cliprivkey.pem
            secretName: eric-data-message-bus-kf-metrics-server-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-pm-server-ca
        name: eric-pm-server-ca
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: client-cacertbundle.pem
              path: ca.crt
            secretName: eric-pm-server-ca
          type: secret
      - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca-0
        name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-0-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca-bundle.crt
              path: ca-bundle.crt
            secretName: eric-bss-msg-kafka-external-kafka-client-ca-0
          type: secret
      - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca
        name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-ca-certs
        targetContainers:
        - kafka
        volumeSource:
          secret:
            items:
            - key: ca-bundle.crt
              path: ca-bundle.crt
            secretName: eric-bss-msg-kafka-external-kafka-client-ca
          type: secret
      authorization:
        superUsers:
        - CN=eric-adp-mbkf-superuser
        - CN=eric-bss-msg-kafka
        type: simple
      config:
        default.replication.factor: 1
        min.insync.replicas: 1
      listeners:
      - authentication:
          type: tls
        configuration:
          brokerCertChainAndKey:
            certificate: srvcert.crt
            key: srvprivkey.key
            secretName: eric-data-message-bus-kf-test-server-cert-operator-secret
          createBootstrapService: true
        name: siptls
        port: 9093
        tls: true
        type: internal
      - authentication:
          type: tls
        configuration:
          bootstrap: {}
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: eric-bss-msg-kafka-external-kafka-server-cert
          brokers:
          - advertisedHost: eric-data-message-bus-kf-test-kafka-0.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 0
          - advertisedHost: eric-data-message-bus-kf-test-kafka-1.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 1
          - advertisedHost: eric-data-message-bus-kf-test-kafka-2.hahn157.rnd.gic.ericsson.se
            advertisedPort: 443
            broker: 2
          createBootstrapService: true
        name: externaltls
        port: 9094
        tls: true
        type: internal
      logging:
        loggers:
          appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
            "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
            TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
            "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
            "kafka", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
            "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
          appender.console.layout.type: PatternLayout
          appender.console.name: consoleLogger
          appender.console.target: SYSTEM_OUT
          appender.console.type: Console
          logger.kafka.additivity: "false"
          logger.kafka.appenderRef.console.ref: consoleLogger
          logger.kafka.name: kafka
          logger.kafka.server.DynamicConfigManager.additivity: "false"
          logger.kafka.server.DynamicConfigManager.appenderRef.BufferedSocketAppender.ref: streamLogger
          logger.kafka.server.DynamicConfigManager.appenderRef.console.ref: consoleLogger
          rootLogger.additivity: "false"
          rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
      rack:
        topologyKey: topology.kubernetes.io/zone
      replicas: 3
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 512Mi
      storage:
        class: network-block
        size: 15Gi
        type: persistent-claim
      template:
        bootstrapService:
          ipFamilyPolicy: PreferDualStack
        brokersService:
          ipFamilyPolicy: PreferDualStack
        kafkaContainer:
          env:
          - name: METRICS_COMPONENT
            value: kafka
          - name: METRICS_EXPORTER_ENABLED
            value: "true"
          - name: DATADIR
            value: /var/lib/kafka/data
          - name: PVCNAME
            value: data-eric-data-message-bus-kf-test-kafka
          - name: METRIC_PREFIX
            value: mbkf
          - name: METRIC_PORT
            value: "9087"
          - name: LOG_LEVEL
            value: info
          - name: ENABLE_TLS_METRIC
            value: "true"
          - name: SERVER_CERT_DIR
            value: /opt/kafka/certificates/metrics-server-secret
          - name: SERVER_CERT_NAME
            value: clicert.pem
          - name: SERVER_CERT_KEY
            value: cliprivkey.pem
          - name: SCRAPE_CA_CERT_DIR
            value: /opt/kafka/certificates/eric-pm-server-ca
          - name: SCRAPE_CA_CERT_NAME
            value: ca.crt
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_KAFKA_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
        pod:
          affinity:
            podAntiAffinity:
              preferredDuringSchedulingIgnoredDuringExecution:
              - podAffinityTerm:
                  labelSelector:
                    matchLabels:
                      strimzi.io/kind: Kafka
                      strimzi.io/name: eric-data-message-bus-kf-test-kafka
                  topologyKey: kubernetes.io/hostname
                weight: 100
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              authorization-enabled: "true"
              external-listener-ca-thumbprint: d7566f6f,d7566f6f
              kafka-listeners: '{listeners-names=siptls,externaltls}'
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
          tolerations:
          - effect: NoSchedule
            key: key1
            operator: Equal
            value: value1
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                strimzi.io/kind: Kafka
                strimzi.io/name: eric-data-message-bus-kf-test-kafka
            maxSkew: 1
            topologyKey: zone
            whenUnsatisfiable: ScheduleAnyway
      version: 3.5.1
    zookeeper:
      additionalVolumes:
      - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
        name: eric-sec-sip-tls-trusted-root-cert
        targetContainers:
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: ca.crt
              path: ca.crt
            secretName: eric-sec-sip-tls-trusted-root-cert
          type: secret
      - mountPath: /opt/kafka/certificates/metrics-server-secret
        name: metrics-server-cert
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: clicert.pem
              path: clicert.pem
            - key: cliprivkey.pem
              path: cliprivkey.pem
            secretName: eric-data-message-bus-kf-metrics-server-secret
          type: secret
      - mountPath: /opt/kafka/certificates/eric-pm-server-ca
        name: eric-pm-server-ca
        targetContainers:
        - kafka
        - zookeeper
        volumeSource:
          secret:
            items:
            - key: client-cacertbundle.pem
              path: ca.crt
            secretName: eric-pm-server-ca
          type: secret
      logging:
        loggers:
          appender.console.filter.1.onMatch: DENY
          appender.console.filter.1.onMismatch: NEUTRAL
          appender.console.filter.1.regex: .*running stat.*
          appender.console.filter.1.type: RegexFilter
          appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
            "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
            TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
            "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
            "zookeeper", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
            "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
          appender.console.layout.type: PatternLayout
          appender.console.name: consoleLogger
          appender.console.target: SYSTEM_OUT
          appender.console.type: Console
          rootLogger.additivity: "false"
          rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
      replicas: 1
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 512Mi
      storage:
        class: network-block
        size: 15Gi
        type: persistent-claim
      template:
        clientService:
          ipFamilyPolicy: PreferDualStack
        nodesService:
          ipFamilyPolicy: PreferDualStack
        pod:
          affinity: {}
          imagePullSecrets:
          - name: armdocker
          metadata:
            annotations:
              mydomain.io/app: msg1
              mydomain.io/instance: msg1-instance
            labels:
              application: msg1
              eric-data-distributed-coordinator-ed-access: "true"
              eric-sec-key-management-access: "true"
              eric-sec-sip-tls-access: "true"
              instance: msg1-instance
          securityContext:
            fsGroup: 10000
            runAsNonRoot: true
        zookeeperContainer:
          env:
          - name: METRICS_COMPONENT
            value: zookeeper
          - name: METRICS_EXPORTER_ENABLED
            value: "true"
          - name: DATADIR
            value: /var/lib/zookeeper
          - name: PVCNAME
            value: data-eric-data-message-bus-kf-test-zookeeper
          - name: METRIC_PREFIX
            value: mbkf
          - name: METRIC_PORT
            value: "9087"
          - name: LOG_LEVEL
            value: info
          - name: ENABLE_TLS_METRIC
            value: "true"
          - name: SERVER_CERT_DIR
            value: /opt/kafka/certificates/metrics-server-secret
          - name: SERVER_CERT_NAME
            value: clicert.pem
          - name: SERVER_CERT_KEY
            value: cliprivkey.pem
          - name: SCRAPE_CA_CERT_DIR
            value: /opt/kafka/certificates/eric-pm-server-ca
          - name: SCRAPE_CA_CERT_NAME
            value: ca.crt
          - name: METRICS_JMX_ENABLED
            value: "true"
          - name: METRICS_JMX_PORT
            value: "9404"
          - name: LOG4J_LOG_LEVEL
            value: INFO
          - name: STRIMZI_CUSTOM_ZOOKEEPER_LABELS
            value: 'app.kubernetes.io/version: 3.5.0-ha4ca65e3'
          - name: HELMCHART_VERSION
            value: 3.5.0-ha4ca65e3
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
              - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsNonRoot: true
  status:
    clusterId: CAVSR2sNS8ynLfltpYt_fA
    conditions:
    - lastTransitionTime: "2024-10-23T13:47:54.216102875Z"
      status: "True"
      type: Ready
    listeners:
    - addresses:
      - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc
        port: 9093
      bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc:9093
      certificates:
      - |-
        -----BEGIN CERTIFICATE-----
        MIIDozCCA0mgAwIBAgIUENmCIXSvAnUzPCgRJDbiCq9qvAUwCgYIKoZIzj0EAwIw
        NDEyMDAGA1UEAxMpZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBJbnRlcm1lZGlh
        dGUgQ0EwHhcNMjQxMDE4MTUxMTEyWhcNMjUwMzE5MTcxMTQyWjAoMSYwJAYDVQQD
        Ex1lcmljLWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdDBZMBMGByqGSM49AgEGCCqG
        SM49AwEHA0IABA6FIkcr5QVtfymuIF2MgEf+ahD6azBC/8D0tSFsFWtFUsKBQvxx
        gM25J5xytrZcSSVcw0jfTb2BpbS3nG4tICajggJDMIICPzAdBgNVHQ4EFgQUlfhe
        dLK/oSbv+cwvtoE7DnhxrVQwHwYDVR0jBBgwFoAUN4UPHUOoOFR5cBuRR4l71gAY
        tR8wXgYIKwYBBQUHAQEEUjBQME4GCCsGAQUFBzAChkJodHRwczovL2VyaWMtc2Vj
        LWtleS1tYW5hZ2VtZW50OjgyMDAvdjEvZXJpYy1zZWMtc2lwLXRscy1zdWItY2Ev
        Y2EwggF2BgNVHREEggFtMIIBaYIdZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRl
        c3SCJmVyaWMtZGF0YS1tZXNzYWdlLWJ1cy1rZi10ZXN0LnNhZWVkLW9wgiplcmlj
        LWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdC5zYWVlZC1vcC5zdmOCOGVyaWMtZGF0
        YS1tZXNzYWdlLWJ1cy1rZi10ZXN0LnNhZWVkLW9wLnN2Yy5jbHVzdGVyLmxvY2Fs
        gi1lcmljLWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdC1rYWZrYS1ib290c3RyYXCC
        OiouZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fma2EtYnJva2Vycy5z
        YWVlZC1vcC5zdmOCNiouZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fm
        a2EtYnJva2Vycy5zYWVlZC1vcIIXY2VydGlmaWVkLXNjcmFwZS10YXJnZXQwDgYD
        VR0PAQH/BAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAoGCCqGSM49BAMCA0gA
        MEUCIQCpgCc5Q2KVCC4rFXX//BHUm/+NjaUfHzqnaVqWpN9mUQIgFzKLIoCzNImk
        8R+lN6sdzqqR4DVUF05WuU62YcvsRMs=
        -----END CERTIFICATE-----
        -----BEGIN CERTIFICATE-----
        MIICKzCCAdCgAwIBAgIUa5RyaG1VGoi9jYkBRYG/8UfoRgMwCgYIKoZIzj0EAwIw
        LDEqMCgGA1UEAxMhZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBSb290IENBMCAX
        DTI0MTAxODE0MDcxNVoYDzIxMjQwOTI0MTQwNzQ1WjA0MTIwMAYDVQQDEyllcmlj
        LXNlYy1zaXAtdGxzIEludGVybmFsIEludGVybWVkaWF0ZSBDQTBZMBMGByqGSM49
        AgEGCCqGSM49AwEHA0IABBnmmnDQvL7SlUP5TcrFtKNjK8jw7W+meHlJ0YivDoRP
        +NJ1iOJclcV3lAP60cRfs2UP713a17o0d4yJixAJiVyjgcUwgcIwDgYDVR0PAQH/
        BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFDeFDx1DqDhUeXAbkUeJ
        e9YAGLUfMB8GA1UdIwQYMBaAFPp0Mq4fIIOyw4ocI11gBVLENt5KMF8GCCsGAQUF
        BwEBBFMwUTBPBggrBgEFBQcwAoZDaHR0cHM6Ly9lcmljLXNlYy1rZXktbWFuYWdl
        bWVudDo4MjAwL3YxL2VyaWMtc2VjLXNpcC10bHMtcm9vdC1jYS9jYTAKBggqhkjO
        PQQDAgNJADBGAiEA8uqrhtyOyodbmVhfLiJ+MiZA9b4WKdFiD0MYPxNaa7ICIQD1
        YaduAL6qq9KrAR4R6gzrr0tMfTid4omSl613MNZKWA==
        -----END CERTIFICATE-----
      name: siptls
      type: siptls
    - addresses:
      - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc
        port: 9094
      bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed-op.svc:9094
      certificates:
      - |
        -----BEGIN CERTIFICATE-----
        MIIEGTCCAgGgAwIBAgICA+kwDQYJKoZIhvcNAQELBQAwEDEOMAwGA1UEAxMFU1JW
        Q0EwHhcNMjQxMDE4MTQ0MzI1WhcNMjUxMDI4MTU0MzI1WjAmMSQwIgYDVQQDDBsq
        LmthZmthLWV4dC50ZXN0ZXhhbXBsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IB
        DwAwggEKAoIBAQCpaPSubgm69wYiDyUsWXjSOiHNikiAukwOoO5hzAiDHhDV1kmL
        tWuAUXHPu1Ko8g4Vo8xsYE1NVFT35AanoTqzDaIwOz25tJ5sxy3IwyGhZSQlensL
        MyDrjM5+6iuFxRHGVFecn3Kw+YFZgovWqJ84jVS0CIwgMc90br0obVmR+4xmm8RF
        kT6qroNJlTq3/rUeLuby13Rgwsfgyyuvf60HYXorUcjgClLf3CkVl12HF+5/Sw3Z
        Dh0ANRPq1jDIjQIxn12nkTlt1Ua9aeYOONGCIvGKjqfUNsZwGyvnaXCsb8E4qy1a
        bflbgDLexRkB2ZKAtuHtTFBA2qy5awv3JX7xAgMBAAGjZzBlMA4GA1UdDwEB/wQE
        AwIF4DATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4EFgQUkmBeHnzQRtdk2evX
        D35/6qRGfdQwHwYDVR0jBBgwFoAUa+qzLNG5SOLi27PQTRllkrlHJHYwDQYJKoZI
        hvcNAQELBQADggIBAGtgAvTz1KuAnDYNHbrbpbwkjIygQZb50lVbjIRmcOA3ntq8
        zhE3cPU2OBQdED7OSMzEncqsHp0/1K9OcqdEaR9FI3rAPsCJP2id/Pnwbj1jLf9F
        EYVdgOGjUbfIvSpLt7TQiWon9s+2ZI2K4h2cggtQ2FH00LbT+EKMQYXBB8hSuQqc
        Yu/CXouSC1Wq2p88F+CL7bLXs7oy7eCWSnYo9SuzXM5mGrFuq0Aqrc4tCEH8My6d
        EGE8coisWDOGG0mUR3JJNWfY7j4aVtt+QQLHwpP4OP4eLMjYBpkQ3PbZHlNgi6QX
        PefFM7BxYbyfgJpOiKaWHJi8skG02YpHWFv7T0X1xKw/OzGdOyRZFYHTBzYCcuzW
        jWTpmujURmicCrwmPivkex7YZTCsy9QQxbwX2ZoCgTiM7gt72eYvBLzbGljC8HK6
        un4sK46qo+qt0R0hjiCEyLcvJG2aD7huTGXaCALDwp/BbVEd7KbWM+Y232+9xDzp
        6xY5I86E4JuxTc9M2jMrNn4kFL5pUP6FlNPVZLxNLpyh0UV9hh7NIJVU/uUPHxON
        cgL9uGOHZ9wGxTgK6yD/Vi12XvM6kjVJGOd/X75RmBcBTQXHKggrCv648WWArNZ+
        WfyAuy8YnXtJHmAjdVCML7maE5YuChy5VZsB5blbuGFF7cbG3shZnrPwxLiO
        -----END CERTIFICATE-----
      name: externaltls
      type: externaltls
    observedGeneration: 16
kind: List
metadata:
  resourceVersion: ""

This is the config and i am going to delete "eric-bss-msg-kafka-external-kafka-client-ca" secret and re-create it again, that will cause the pods to restart but when i check the events i am expecting "CaCertRenewed" but it is still "PodHasOldRevision"

scholzj Oct 23, 2024
Maintainer

This is the config and i am going to delete "eric-bss-msg-kafka-external-kafka-client-ca" secret and re-create it again, that will cause the pods to restart but when i check the events i am expecting "CaCertRenewed" but it is still "PodHasOldRevision"

Wait, why would you do that?

That is not a CA renewal
And you are also not using external CA but Strimzi generated CAs based on the configuration you shared

If you want to use your own CA or trigger a renewal, you should follow the docs. Just by going in and deleting some secret you are just doing some random disaster recovery testing.

saeedya Oct 25, 2024
Author

This is the new logs:
0s Normal Issuing certificate/ca-certificate Renewing certificate as renewal was scheduled at 2024-10-25 12:37:15 +0000 UTC
0s Normal Reused certificate/ca-certificate Reusing private key stored in existing Secret resource "eric-bss-msg-kafka-external-kafka-client-ca"
0s Normal Requested certificate/ca-certificate Created new CertificateRequest resource "ca-certificate-4"
0s Normal WaitingForApproval certificaterequest/ca-certificate-4 Not signing CertificateRequest until it is Approved
0s Normal WaitingForApproval certificaterequest/ca-certificate-4 Not signing CertificateRequest until it is Approved
0s Normal WaitingForApproval certificaterequest/ca-certificate-4 Not signing CertificateRequest until it is Approved
0s Normal WaitingForApproval certificaterequest/ca-certificate-4 Not signing CertificateRequest until it is Approved
0s Normal WaitingForApproval certificaterequest/ca-certificate-4 Not signing CertificateRequest until it is Approved
0s Normal cert-manager.io certificaterequest/ca-certificate-4 Certificate request has been approved by cert-manager.io
0s Normal CertificateIssued certificaterequest/ca-certificate-4 Certificate fetched from issuer successfully
0s Normal Issuing certificate/ca-certificate The certificate has been successfully issued
0s Normal KeyPairVerified issuer/ca-issuer Signing CA verified

0s Normal Killing pod/eric-data-message-bus-kf-test-kafka-1 Stopping container kafka
0s Normal PodHasOldRevision pod/eric-data-message-bus-kf-test-kafka-1 Pod has old revision
0s Normal Scheduled pod/eric-data-message-bus-kf-test-kafka-1 Successfully assigned saeed-op/eric-data-message-bus-kf-test-kafka-1 to node-10-120-217-200

0s Normal Created pod/eric-data-message-bus-kf-test-kafka-1 Created container kafka-init
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-1 Started container kafka-init

0s Normal Created pod/eric-data-message-bus-kf-test-kafka-1 Created container kafka
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-1 Started container kafka
61m Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
61m Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
61m Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
61m Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
61m Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
61m Normal cert-manager.io certificaterequest/ca-certificate-3 Certificate request has been approved by cert-manager.io
61m Normal CertificateIssued certificaterequest/ca-certificate-3 Certificate fetched from issuer successfully
61m Normal Issuing certificate/ca-certificate Renewing certificate as renewal was scheduled at 2024-10-25 11:37:15 +0000 UTC
61m Normal Requested certificate/ca-certificate Created new CertificateRequest resource "ca-certificate-3"
0s Normal Killing pod/eric-data-message-bus-kf-test-kafka-2 Stopping container kafka
0s Normal PodHasOldRevision pod/eric-data-message-bus-kf-test-kafka-2 Pod has old revision
0s Normal Scheduled pod/eric-data-message-bus-kf-test-kafka-2 Successfully assigned saeed-op/eric-data-message-bus-kf-test-kafka-2 to node-10-120-217-199

0s Normal Created pod/eric-data-message-bus-kf-test-kafka-2 Created container kafka-init
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-2 Started container kafka-init

0s Normal Created pod/eric-data-message-bus-kf-test-kafka-2 Created container kafka
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-2 Started container kafka

0s Normal Killing pod/eric-data-message-bus-kf-test-kafka-0 Stopping container kafka
0s Normal PodHasOldRevision pod/eric-data-message-bus-kf-test-kafka-0 Pod has old revision
0s Normal Scheduled pod/eric-data-message-bus-kf-test-kafka-0 Successfully assigned saeed-op/eric-data-message-bus-kf-test-kafka-0 to node-10-120-217-198
0s Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf-be-operator:3.5.0-h3392d34f" already present on machine
0s Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka-init
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka-init
0s Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf:3.5.0-h3392d34f" already present on machine
0s Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka
0s Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka

Here you can see the certificate rebewal happend, and the kafka pods are going to restart, but the message is "PodHasOldRevision" not "CaCertRenewed".

saeedya · 2024-11-08T12:22:11Z

saeedya
Nov 8, 2024
Author

Hi

Kafka version: 3.5.1
Strimzi Version: 0.36.1

Issuer config:

`apiVersion: v1
items:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cert-manager.io/v1","kind":"Issuer","metadata":{"annotations":{},"name":"selfsigned-issuer","namespace":"saeed"},"spec":{"selfSigned":{}}}
creationTimestamp: "2024-11-08T11:06:59Z"
generation: 1
name: selfsigned-issuer
namespace: saeed
resourceVersion: "2138749465"
uid: d2a18a65-56e4-4d35-af5d-4ca165c86096
spec:
selfSigned: {}
status:
conditions:
- lastTransitionTime: "2024-11-08T11:06:59Z"
  observedGeneration: 1
  reason: IsReady
  status: "True"
  type: Ready
  kind: List
  metadata:
  resourceVersion: ""`

Certificate config:

`apiVersion: v1
items:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{},"name":"ca-certificate","namespace":"saeed"},"spec":{"commonName":"my-ca","duration":"2h","isCA":true,"issuerRef":{"name":"selfsigned-issuer"},"privateKey":{"algorithm":"RSA","size":2048},"renewBefore":"1h","secretName":"eric-bss-msg-kafka-external-kafka-client-ca"}}
creationTimestamp: "2024-11-08T11:07:11Z"
generation: 1
name: ca-certificate
namespace: saeed
resourceVersion: "2138750269"
uid: 9af43624-4b49-4004-b6a9-eda7920319f4
spec:
commonName: my-ca
duration: 2h
isCA: true
issuerRef:
name: selfsigned-issuer
privateKey:
algorithm: RSA
size: 2048
renewBefore: 1h
secretName: eric-bss-msg-kafka-external-kafka-client-ca
status:
conditions:
- lastTransitionTime: "2024-11-08T11:07:51Z"
  message: Certificate is up to date and has not expired
  observedGeneration: 1
  reason: Ready
  status: "True"
  type: Ready
  notAfter: "2024-11-08T13:07:51Z"
  notBefore: "2024-11-08T11:07:51Z"
  renewalTime: "2024-11-08T12:07:51Z"
  revision: 2
  kind: List
  metadata:
  resourceVersion: ""
  kafka config: =====================apiVersion: v1
  items:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
annotations:
jmx-filter-configmap-name: eric-data-message-bus-kf-metrics-config
mydomain.io/app: msg1
mydomain.io/instance: msg1-instance
creationTimestamp: "2024-11-08T10:56:18Z"
generation: 3
labels:
application: msg1
eric-data-distributed-coordinator-ed-access: "true"
eric-sec-key-management-access: "true"
eric-sec-sip-tls-access: "true"
instance: msg1-instance
name: eric-data-message-bus-kf-test
namespace: saeed
ownerReferences:
- apiVersion: mbkf.data.ericsson.com/v1
  blockOwnerDeletion: false
  controller: false
  kind: MessageBus
  name: eric-data-message-bus-kf-test
  uid: 8386f6d8-ee5a-4423-a143-3fc65f8410b5
  resourceVersion: "2138752755"
  uid: 2e8d8703-9170-42b0-b407-73ffdbd63999
  spec:
  entityOperator:
  template:
  pod:
  affinity:
  podAntiAffinity:
  preferredDuringSchedulingIgnoredDuringExecution:
  - podAffinityTerm:
  labelSelector:
  matchLabels:
  strimzi.io/kind: Kafka
  strimzi.io/name: eric-data-message-bus-kf-test-kafka
  topologyKey: kubernetes.io/hostname
  weight: 100
  imagePullSecrets:
  - name: armdocker
  metadata:
  annotations:
  mydomain.io/app: msg1
  mydomain.io/instance: msg1-instance
  labels:
  application: msg1
  eric-data-distributed-coordinator-ed-access: "true"
  eric-sec-key-management-access: "true"
  eric-sec-sip-tls-access: "true"
  instance: msg1-instance
  securityContext:
  fsGroup: 10000
  runAsNonRoot: true
  tolerations:
  - effect: NoSchedule
  key: key1
  operator: Equal
  value: value1
  topologySpreadConstraints:
  - labelSelector:
  matchLabels:
  strimzi.io/kind: Kafka
  strimzi.io/name: eric-data-message-bus-kf-test-entity-operator
  maxSkew: 1
  topologyKey: zone
  whenUnsatisfiable: ScheduleAnyway
  userOperatorContainer:
  env:
  - name: METRICS_JMX_ENABLED
  value: "true"
  - name: METRICS_JMX_PORT
  value: "9404"
  - name: LOG4J_LOG_LEVEL
  value: INFO
  - name: STRIMZI_CUSTOM_ENTITY_OPERATOR_LABELS
  value: 'app.kubernetes.io/version: 3.5.0-h1d4818e6'
  - name: HELMCHART_VERSION
  value: 3.5.0-h1d4818e6
  securityContext:
  allowPrivilegeEscalation: false
  capabilities:
  drop:
  - ALL
  privileged: false
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  userOperator:
  additionalVolumes:
  - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
    name: eric-sec-sip-tls-trusted-root-cert
    targetContainers:
    - user-operator
      volumeSource:
      secret:
      items:
      - key: ca.crt
        path: ca.crt
        secretName: eric-sec-sip-tls-trusted-root-cert
        type: secret
        logging:
        loggers:
        appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
        "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
        TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
        "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
        "user-operator", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file":
        "%file", "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
        appender.console.layout.type: PatternLayout
        appender.console.name: consoleLogger
        appender.console.target: SYSTEM_OUT
        appender.console.type: Console
        rootLogger.additivity: "false"
        rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
        resources:
        limits:
        cpu: "2"
        memory: 1Gi
        requests:
        cpu: "1"
        memory: 512Mi
        kafka:
        additionalVolumes:
  - mountPath: /opt/kafka/certificates/tls-siptls-9093-certs/eric-data-message-bus-kf-test-client-ca-secret
    name: siptls-9093-eric-data-message-bus-kf-test-client-ca-secret-ca-certs
    targetContainers:
    - kafka
      volumeSource:
      secret:
      items:
      - key: clientcacertbundle.pem
        path: ca.crt
        secretName: eric-data-message-bus-kf-test-client-ca-secret
        type: secret
  - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
    name: eric-sec-sip-tls-trusted-root-cert
    targetContainers:
    - kafka
      volumeSource:
      secret:
      items:
      - key: ca.crt
        path: ca.crt
        secretName: eric-sec-sip-tls-trusted-root-cert
        type: secret
  - mountPath: /opt/kafka/certificates/metrics-server-secret
    name: metrics-server-cert
    targetContainers:
    - kafka
    - zookeeper
      volumeSource:
      secret:
      items:
      - key: clicert.pem
        path: clicert.pem
      - key: cliprivkey.pem
        path: cliprivkey.pem
        secretName: eric-data-message-bus-kf-metrics-server-secret
        type: secret
  - mountPath: /opt/kafka/certificates/eric-pm-server-ca
    name: eric-pm-server-ca
    targetContainers:
    - kafka
    - zookeeper
      volumeSource:
      secret:
      items:
      - key: client-cacertbundle.pem
        path: ca.crt
        secretName: eric-pm-server-ca
        type: secret
  - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca-0
    name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-0-ca-certs
    targetContainers:
    - kafka
      volumeSource:
      secret:
      items:
      - key: ca.crt
        path: ca.crt
        secretName: eric-bss-msg-kafka-external-kafka-client-ca-0
        type: secret
  - mountPath: /opt/kafka/certificates/tls-externaltls-9094-certs/eric-bss-msg-kafka-external-kafka-client-ca
    name: externaltls-9094-eric-bss-msg-kafka-external-kafka-client-ca-ca-certs
    targetContainers:
    - kafka
      volumeSource:
      secret:
      items:
      - key: ca.crt
        path: ca.crt
        secretName: eric-bss-msg-kafka-external-kafka-client-ca
        type: secret
        authorization:
        superUsers:
    - CN=eric-adp-mbkf-superuser
    - CN=eric-bss-msg-kafka
      type: simple
      config:
      default.replication.factor: 1
      min.insync.replicas: 1
      listeners:
  - authentication:
    type: tls
    configuration:
    brokerCertChainAndKey:
    certificate: srvcert.crt
    key: srvprivkey.key
    secretName: eric-data-message-bus-kf-test-server-cert-operator-secret
    createBootstrapService: true
    name: siptls
    port: 9093
    tls: true
    type: internal
  - authentication:
    type: tls
    configuration:
    bootstrap: {}
    brokerCertChainAndKey:
    certificate: tls.crt
    key: tls.key
    secretName: eric-bss-msg-kafka-external-kafka-server-cert
    brokers:
    - advertisedHost: eric-data-message-bus-kf-test-kafka-0.hahn157.rnd.gic.ericsson.se
      advertisedPort: 443
      broker: 0
      createBootstrapService: true
      name: externaltls
      port: 9094
      tls: true
      type: internal
      logging:
      loggers:
      appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
      "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
      TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
      "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
      "kafka", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
      "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
      appender.console.layout.type: PatternLayout
      appender.console.name: consoleLogger
      appender.console.target: SYSTEM_OUT
      appender.console.type: Console
      logger.kafka.additivity: "false"
      logger.kafka.appenderRef.console.ref: consoleLogger
      logger.kafka.name: kafka
      logger.kafka.server.DynamicConfigManager.additivity: "false"
      logger.kafka.server.DynamicConfigManager.appenderRef.BufferedSocketAppender.ref: streamLogger
      logger.kafka.server.DynamicConfigManager.appenderRef.console.ref: consoleLogger
      rootLogger.additivity: "false"
      rootLogger.appenderRef.console.ref: consoleLogger
      type: inline
      rack:
      topologyKey: topology.kubernetes.io/zone
      replicas: 1
      resources:
      limits:
      cpu: "2"
      memory: 1Gi
      requests:
      cpu: "1"
      memory: 512Mi
      storage:
      class: network-block
      size: 15Gi
      type: persistent-claim
      template:
      bootstrapService:
      ipFamilyPolicy: PreferDualStack
      brokersService:
      ipFamilyPolicy: PreferDualStack
      kafkaContainer:
      env:
    - name: METRICS_COMPONENT
      value: kafka
    - name: METRICS_EXPORTER_ENABLED
      value: "true"
    - name: DATADIR
      value: /var/lib/kafka/data
    - name: PVCNAME
      value: data-eric-data-message-bus-kf-test-kafka
    - name: METRIC_PREFIX
      value: mbkf
    - name: METRIC_PORT
      value: "9087"
    - name: LOG_LEVEL
      value: info
    - name: ENABLE_TLS_METRIC
      value: "true"
    - name: SERVER_CERT_DIR
      value: /opt/kafka/certificates/metrics-server-secret
    - name: SERVER_CERT_NAME
      value: clicert.pem
    - name: SERVER_CERT_KEY
      value: cliprivkey.pem
    - name: SCRAPE_CA_CERT_DIR
      value: /opt/kafka/certificates/eric-pm-server-ca
    - name: SCRAPE_CA_CERT_NAME
      value: ca.crt
    - name: METRICS_JMX_ENABLED
      value: "true"
    - name: METRICS_JMX_PORT
      value: "9404"
    - name: LOG4J_LOG_LEVEL
      value: INFO
    - name: STRIMZI_CUSTOM_KAFKA_LABELS
      value: 'app.kubernetes.io/version: 3.5.0-h1d4818e6'
    - name: HELMCHART_VERSION
      value: 3.5.0-h1d4818e6
      securityContext:
      allowPrivilegeEscalation: false
      capabilities:
      drop:
      - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        pod:
        affinity:
        podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
      - podAffinityTerm:
        labelSelector:
        matchLabels:
        strimzi.io/kind: Kafka
        strimzi.io/name: eric-data-message-bus-kf-test-kafka
        topologyKey: kubernetes.io/hostname
        weight: 100
        imagePullSecrets:
    - name: armdocker
      metadata:
      annotations:
      authorization-enabled: "true"
      external-listener-ca-thumbprint: 8ec02c59,8ec02c59
      kafka-listeners: '{listeners-names=siptls,externaltls}'
      mydomain.io/app: msg1
      mydomain.io/instance: msg1-instance
      labels:
      application: msg1
      eric-data-distributed-coordinator-ed-access: "true"
      eric-sec-key-management-access: "true"
      eric-sec-sip-tls-access: "true"
      instance: msg1-instance
      securityContext:
      fsGroup: 10000
      runAsNonRoot: true
      tolerations:
    - effect: NoSchedule
      key: key1
      operator: Equal
      value: value1
      topologySpreadConstraints:
    - labelSelector:
      matchLabels:
      strimzi.io/kind: Kafka
      strimzi.io/name: eric-data-message-bus-kf-test-kafka
      maxSkew: 1
      topologyKey: zone
      whenUnsatisfiable: ScheduleAnyway
      version: 3.5.1
      zookeeper:
      additionalVolumes:
  - mountPath: /opt/kafka/certificates/eric-sec-sip-tls-trusted-root-cert
    name: eric-sec-sip-tls-trusted-root-cert
    targetContainers:
    - zookeeper
      volumeSource:
      secret:
      items:
      - key: ca.crt
        path: ca.crt
        secretName: eric-sec-sip-tls-trusted-root-cert
        type: secret
  - mountPath: /opt/kafka/certificates/metrics-server-secret
    name: metrics-server-cert
    targetContainers:
    - kafka
    - zookeeper
      volumeSource:
      secret:
      items:
      - key: clicert.pem
        path: clicert.pem
      - key: cliprivkey.pem
        path: cliprivkey.pem
        secretName: eric-data-message-bus-kf-metrics-server-secret
        type: secret
  - mountPath: /opt/kafka/certificates/eric-pm-server-ca
    name: eric-pm-server-ca
    targetContainers:
    - kafka
    - zookeeper
      volumeSource:
      secret:
      items:
      - key: client-cacertbundle.pem
        path: ca.crt
        secretName: eric-pm-server-ca
        type: secret
        logging:
        loggers:
        appender.console.filter.1.onMatch: DENY
        appender.console.filter.1.onMismatch: NEUTRAL
        appender.console.filter.1.regex: .running stat.
        appender.console.filter.1.type: RegexFilter
        appender.console.layout.pattern: '{"version": "1.2.0", "timestamp": "%d{yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZ}",
        "severity": "%p{WARN=warning, DEBUG=debug, ERROR=error, FATAL=critical,
        TRACE=debug, INFO=info}", "service_id": "eric-data-messagebus-kf", "message":
        "%enc{%m}{JSON}%enc{%ex{separator(|)}}{JSON}", "metadata": {"container_name":
        "zookeeper", "pod_name": "$${env:HOSTNAME}"}, "extra_data": {"file": "%file",
        "line": "%line", "mbkf_cluster_name": "eric-data-message-bus-kf-test"}}%n'
        appender.console.layout.type: PatternLayout
        appender.console.name: consoleLogger
        appender.console.target: SYSTEM_OUT
        appender.console.type: Console
        rootLogger.additivity: "false"
        rootLogger.appenderRef.console.ref: consoleLogger
        type: inline
        replicas: 1
        resources:
        limits:
        cpu: "2"
        memory: 1Gi
        requests:
        cpu: "1"
        memory: 512Mi
        storage:
        class: network-block
        size: 15Gi
        type: persistent-claim
        template:
        clientService:
        ipFamilyPolicy: PreferDualStack
        nodesService:
        ipFamilyPolicy: PreferDualStack
        pod:
        affinity: {}
        imagePullSecrets:
      - name: armdocker
        metadata:
        annotations:
        mydomain.io/app: msg1
        mydomain.io/instance: msg1-instance
        labels:
        application: msg1
        eric-data-distributed-coordinator-ed-access: "true"
        eric-sec-key-management-access: "true"
        eric-sec-sip-tls-access: "true"
        instance: msg1-instance
        securityContext:
        fsGroup: 10000
        runAsNonRoot: true
        zookeeperContainer:
        env:
      - name: METRICS_COMPONENT
        value: zookeeper
      - name: METRICS_EXPORTER_ENABLED
        value: "true"
      - name: DATADIR
        value: /var/lib/zookeeper
      - name: PVCNAME
        value: data-eric-data-message-bus-kf-test-zookeeper
      - name: METRIC_PREFIX
        value: mbkf
      - name: METRIC_PORT
        value: "9087"
      - name: LOG_LEVEL
        value: info
      - name: ENABLE_TLS_METRIC
        value: "true"
      - name: SERVER_CERT_DIR
        value: /opt/kafka/certificates/metrics-server-secret
      - name: SERVER_CERT_NAME
        value: clicert.pem
      - name: SERVER_CERT_KEY
        value: cliprivkey.pem
      - name: SCRAPE_CA_CERT_DIR
        value: /opt/kafka/certificates/eric-pm-server-ca
      - name: SCRAPE_CA_CERT_NAME
        value: ca.crt
      - name: METRICS_JMX_ENABLED
        value: "true"
      - name: METRICS_JMX_PORT
        value: "9404"
      - name: LOG4J_LOG_LEVEL
        value: INFO
      - name: STRIMZI_CUSTOM_ZOOKEEPER_LABELS
        value: 'app.kubernetes.io/version: 3.5.0-h1d4818e6'
      - name: HELMCHART_VERSION
        value: 3.5.0-h1d4818e6
        securityContext:
        allowPrivilegeEscalation: false
        capabilities:
        drop:
        
        ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        status:
        clusterId: ojClkvCySHiOACEjsdr87w
        conditions:
- lastTransitionTime: "2024-11-08T11:10:26.895079266Z"
  status: "True"
  type: Ready
  listeners:
- addresses:
  - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed.svc
    port: 9093
    bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed.svc:9093
    certificates:
  - |-
    -----BEGIN CERTIFICATE-----
    MIIDkzCCAzqgAwIBAgIUTtV4VEcB7FMMAJKPGIoPxhiLkTkwCgYIKoZIzj0EAwIw
    NDEyMDAGA1UEAxMpZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBJbnRlcm1lZGlh
    dGUgQ0EwHhcNMjQxMTA4MTA1NTM3WhcNMjUwNDA5MTI1NjA3WjAoMSYwJAYDVQQD
    Ex1lcmljLWRhdGEtbWVzc2FnZS1idXMta2YtdGVzdDBZMBMGByqGSM49AgEGCCqG
    SM49AwEHA0IABLNC+BGa6ck8WjGDCXF7xJ449oNS2kqjJeJJdLtVhTjFHjzBDulU
    tePST52PtmEvbQz/Oxj0Kxign98RCgEHsm2jggI0MIICMDAdBgNVHQ4EFgQUEIqC
    zQUG5jYmnSw1lalhliflwxwwHwYDVR0jBBgwFoAUdkxmYADZ1qor73VFsk13Cmiq
    l6QwXgYIKwYBBQUHAQEEUjBQME4GCCsGAQUFBzAChkJodHRwczovL2VyaWMtc2Vj
    LWtleS1tYW5hZ2VtZW50OjgyMDAvdjEvZXJpYy1zZWMtc2lwLXRscy1zdWItY2Ev
    Y2EwggFnBgNVHREEggFeMIIBWoIdZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRl
    c3SCI2VyaWMtZGF0YS1tZXNzYWdlLWJ1cy1rZi10ZXN0LnNhZWVkgidlcmljLWRh
    dGEtbWVzc2FnZS1idXMta2YtdGVzdC5zYWVlZC5zdmOCNWVyaWMtZGF0YS1tZXNz
    YWdlLWJ1cy1rZi10ZXN0LnNhZWVkLnN2Yy5jbHVzdGVyLmxvY2Fsgi1lcmljLWRh
    dGEtbWVzc2FnZS1idXMta2YtdGVzdC1rYWZrYS1ib290c3RyYXCCNyouZXJpYy1k
    YXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fma2EtYnJva2Vycy5zYWVlZC5zdmOC
    MyouZXJpYy1kYXRhLW1lc3NhZ2UtYnVzLWtmLXRlc3Qta2Fma2EtYnJva2Vycy5z
    YWVlZIIXY2VydGlmaWVkLXNjcmFwZS10YXJnZXQwDgYDVR0PAQH/BAQDAgeAMBMG
    A1UdJQQMMAoGCCsGAQUFBwMBMAoGCCqGSM49BAMCA0cAMEQCIFfzDcMtAjJQ75vR
    cZTfNPL3nX2jatLPfMS0ru5Q0Uf7AiBrCM7bgxn9X8Pbo8meXH8IteMkbYo/E/RA
    rQ6F3ta3eQ==
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    MIICKzCCAdCgAwIBAgIUa6w2gxJhWP/dxIFPiFr0EcwzXXgwCgYIKoZIzj0EAwIw
    LDEqMCgGA1UEAxMhZXJpYy1zZWMtc2lwLXRscyBJbnRlcm5hbCBSb290IENBMCAX
    DTI0MTEwNzE0NTAyOVoYDzIxMjQxMDE0MTQ1MDU5WjA0MTIwMAYDVQQDEyllcmlj
    LXNlYy1zaXAtdGxzIEludGVybmFsIEludGVybWVkaWF0ZSBDQTBZMBMGByqGSM49
    AgEGCCqGSM49AwEHA0IABG3nEMkznsgWVE3WW1ykQscmXWhs97HQobfhnro9H6Er
    i43Cs47UvV3HT8LMeZXMfN2uDbCwkA7bVswQitI256qjgcUwgcIwDgYDVR0PAQH/
    BAQDAgEGMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFHZMZmAA2daqK+91RbJN
    dwpoqpekMB8GA1UdIwQYMBaAFBelyMp5/zX/Yh9yfqBHKqpFVlDUMF8GCCsGAQUF
    BwEBBFMwUTBPBggrBgEFBQcwAoZDaHR0cHM6Ly9lcmljLXNlYy1rZXktbWFuYWdl
    bWVudDo4MjAwL3YxL2VyaWMtc2VjLXNpcC10bHMtcm9vdC1jYS9jYTAKBggqhkjO
    PQQDAgNJADBGAiEA70VWFlc5NjxP7pUG/ipVuJXHLZhBzMB/dEQSKUZHpPQCIQCU
    8ObRmyo/hRjPbeaQKU5AUqyArit37Xrd+pcyL7hCtA==
    -----END CERTIFICATE-----
    name: siptls
    type: siptls
- addresses:
  - host: eric-data-message-bus-kf-test-kafka-bootstrap.saeed.svc
    port: 9094
    bootstrapServers: eric-data-message-bus-kf-test-kafka-bootstrap.saeed.svc:9094
    certificates:
  - |
    -----BEGIN CERTIFICATE-----
    MIIECzCCAfMCFHE2YXgXQjGJMJYS5xgwiJkg1r2cMA0GCSqGSIb3DQEBCwUAMEIx
    CzAJBgNVBAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl
    ZmF1bHQgQ29tcGFueSBMdGQwHhcNMjQxMTA4MTA1MDEyWhcNMjUxMTA4MTA1MDEy
    WjBCMQswCQYDVQQGEwJYWDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQK
    DBNEZWZhdWx0IENvbXBhbnkgTHRkMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
    CgKCAQEAyj1Hv0FsPqf2zPTMoolSS9GxR5vHKcq1hqTIrdbaodN7dMjF7Rkqq3aO
    nk4vlzEHVoL06w6PYEpLnBxfIJPc9d97ICaJ2Ae6C9SMOOeCgkZ2rMYlLuKSdjB+
    mKL9noivf/Idh6jziwuU2STftgH/ATZmfxgi73AzEwCdHGRadengR20KJbTV2Xfm
    YgXb6Hf3B/mCcdIMmwhzy6oIJl1pqRLR/ATPksfhBtkNJZE9wC2cm2JtWEknSQ14
    i8PpoErkHzKJ9+Ylr2AlhLusghGdJbWVkQNkIzMU7KKIyJR1WMtzvclVVO+lItiC
    vzQRBqhT5U88bKRdafkwiiKI6bnkkQIDAQABMA0GCSqGSIb3DQEBCwUAA4ICAQA6
    kKdIKqlGwuMdOJTg3E0v3lXQlT60encFYg80VxTb2ZivianSSXKbfi+BdwUXYx/a
    nY12P3FDih+THXZfg5MPi5XjdtFckUh0lvui6jNsaoPWwO0uIxd6evxvMM9okZnK
    HQgKpSV+U8RSZErIrBWPkCEayuTlo/VzrwGzZRIXQxDUV53wwYrulJMAVK0wRP78
    1IrRQkFku73ILP2kPv2h/vu+QGM5JyyqR//vzc2irBa9864QzCVJFXYp15NGjkIf
    mNqD01lAzpGwc+frGxryvV29wIfRGRNMi3c3HcgCKSJNnDzf0dfdrfBlROiG0AKF
    5YJf5GNgLdgQVoowwn56ElKkkjZipoq/CdDvSs1LnfH2bl6nNQXKWmhm6ilb9yQI
    Ov9s7BB+BhEQj2bqv20vwI15v0PwrX8pLHyIKfXu+iuFzhp6VkexpGYpkaGrB/q2
    1B6FEoAI5lDdaNE95/Vp05oU9dLL15g6a0CGtDLjHvEzmaWiriywXhIDSdRONWam
    GioZFxBUiMizHtye6SuOJGtdvhjyRgHQmzmX6jgbFc9T13tujlqM0Yuq3EQvRiP/
    b7CvN8ZTr8a7GtazYQxBogIJIu/paoIjCH08juNsEq00Tbv9FNYpffOyUv6vZ0MG
    1s/tp9VRiGptILC8xCCPV5nAXczOcnd2pBV9UfwLUg==
    -----END CERTIFICATE-----
    name: externaltls
    type: externaltls
    observedGeneration: 3
    kind: List
    metadata:
    resourceVersion: ""`

=================
what we expect?
When the renewal time is reached, we expect to see 'CaCertRenewed,' but instead we are seeing 'PodHasOldRevision.'

And certificate is autorenewed. This is the "kubectl get events log"

0s 0s 0s 0s 0s 0s 0s 0s 0s 0s 0s 61m 61m 61m 61m 61m 61m 61m 60m 60m 60m 60m 60m 60m 60m 61m 61m 61m 60m 60m 60m 61m 60m 60m 60m 60m 60m 60m 60m 0s 0s 0s 0s 0s 0s 0s 0s 0s 61m 61m 0s 0s 0s Normal Issuing certificate/ca-certificate Renewing certificate as renewal was scheduled at 2024-11-08 12:07:51 +0000 UTC
Normal Reused certificate/ca-certificate Reusing private key stored in existing Secret resource "eric-bss-msg-kafka-external-kafka-client-ca"
Normal Requested certificate/ca-certificate Created new CertificateRequest resource "ca-certificate-3"
Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-3 Not signing CertificateRequest until it is Approved
Normal cert-manager.io certificaterequest/ca-certificate-3 Certificate request has been approved by cert-manager.io
Normal CertificateIssued certificaterequest/ca-certificate-3 Certificate fetched from issuer successfully
Normal Issuing certificate/ca-certificate The certificate has been successfully issued
Normal WaitingForApproval certificaterequest/ca-certificate-1 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-1 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-1 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-1 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-1 Not signing CertificateRequest until it is Approved
Normal cert-manager.io certificaterequest/ca-certificate-1 Certificate request has been approved by cert-manager.io
Normal CertificateIssued certificaterequest/ca-certificate-1 Certificate fetched from issuer successfully
Normal WaitingForApproval certificaterequest/ca-certificate-2 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-2 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-2 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-2 Not signing CertificateRequest until it is Approved
Normal WaitingForApproval certificaterequest/ca-certificate-2 Not signing CertificateRequest until it is Approved
Normal cert-manager.io certificaterequest/ca-certificate-2 Certificate request has been approved by cert-manager.io
Normal CertificateIssued certificaterequest/ca-certificate-2 Certificate fetched from issuer successfully
Normal Issuing certificate/ca-certificate Issuing certificate as Secret does not contain a private key
Normal Generated certificate/ca-certificate Stored new private key in temporary Secret resource "ca-certificate-kwzhw"
Normal Requested certificate/ca-certificate Created new CertificateRequest resource "ca-certificate-1"
Normal Issuing certificate/ca-certificate Issuing certificate as Secret does not exist
Normal Generated certificate/ca-certificate Stored new private key in temporary Secret resource "ca-certificate-hgrgx"
Normal Requested certificate/ca-certificate Created new CertificateRequest resource "ca-certificate-2"
Warning FailedMount pod/eric-data-message-bus-kf-test-kafka-0 MountVolume.SetUp failed for volume "externaltls-9094-eric-bss-msg-kafka-external-kafka-cli-581e6bd2" : secret "eric-bss-msg-kafka-external-kafka-client-ca-0" not found
Normal Killing pod/eric-data-message-bus-kf-test-kafka-0 Stopping container kafka
Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf-be-operator:3.5.0-h1d4818e6" already present on machine
Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka-init
Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka-init
Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf:3.5.0-h1d4818e6" already present on machine
Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka
Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka
Normal Killing pod/eric-data-message-bus-kf-test-kafka-0 Stopping container kafka
Warning Unhealthy pod/eric-data-message-bus-kf-test-kafka-0 Readiness probe failed: Get "http://192.168.151.248:9000/health/readiness": dial tcp 192.168.151.248:9000: connect: connection refused
Normal NoPods poddisruptionbudget/eric-data-message-bus-kf-test-kafka No matching pods found
Normal NoPods poddisruptionbudget/eric-data-message-bus-kf-test-kafka No matching pods found
Normal PodHasOldRevision pod/eric-data-message-bus-kf-test-kafka-0 Pod has old revision
Normal Scheduled pod/eric-data-message-bus-kf-test-kafka-0 Successfully assigned saeed/eric-data-message-bus-kf-test-kafka-0 to node-10-120-217-200
Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf-be-operator:3.5.0-h1d4818e6" already present on machine
Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka-init
Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka-init
Normal Scheduled pod/eric-data-message-bus-kf-test-kafka-0 Successfully assigned saeed/eric-data-message-bus-kf-test-kafka-0 to node-10-120-217-200
Normal PodHasOldRevision pod/eric-data-message-bus-kf-test-kafka-0 Pod has old revision
Normal Pulled pod/eric-data-message-bus-kf-test-kafka-0 Container image "armdocker.rnd.ericsson.se/proj-adp-message-bus-kf-internal/eric-data-message-bus-kf:3.5.0-h1d4818e6" already present on machine
Normal Created pod/eric-data-message-bus-kf-test-kafka-0 Created container kafka
Normal Started pod/eric-data-message-bus-kf-test-kafka-0 Started container kafka

Thanks

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

Strimzi is not honoring the reason for restart events during the certificate renewal #10743

{{title}}

Replies: 2 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

Strimzi is not honoring the reason for restart events during the certificate renewal #10743

saeedya Oct 21, 2024

Replies: 2 comments · 8 replies

scholzj Oct 21, 2024 Maintainer

saeedya Oct 23, 2024 Author

scholzj Oct 23, 2024 Maintainer

saeedya Oct 23, 2024 Author

scholzj Oct 23, 2024 Maintainer

saeedya Oct 25, 2024 Author

saeedya Nov 8, 2024 Author

Issuer config:

Certificate config:

saeedya
Oct 21, 2024

Replies: 2 comments 8 replies

scholzj
Oct 21, 2024
Maintainer

saeedya Oct 23, 2024
Author

scholzj Oct 23, 2024
Maintainer

saeedya Oct 23, 2024
Author

scholzj Oct 23, 2024
Maintainer

saeedya Oct 25, 2024
Author

saeedya
Nov 8, 2024
Author