Updating the cluster-operator from 1.2.0 to 1.3.0 did not result in a rolling STS update, as mentioned in the release notes

[gerhard]

Part of rabbitmq/tgir#20 we bumped the cluster-operator from 1.2.0 to 1.3.0 and were expecting to see STS updating, as per the release notes, but this did not happen.

We (+@Gsantomaggio) have two questions:

1. How can we tell which cluster-operator updates will result in rolling updates?
2. Should we update the 1.3.0 release notes to clarify that a rolling update may not happen?

cc @mkuratczyk

More context

cluster-operator logs

2020-12-28T11:38:52.135Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":12345"}
2020-12-28T11:38:52.137Z	INFO	setup	started controller
2020-12-28T11:38:52.137Z	INFO	setup	starting manager
I1228 11:38:52.137597       1 leaderelection.go:242] attempting to acquire leader lease  rabbitmq-system/rabbitmq-cluster-operator-leader-election...
2020-12-28T11:38:52.138Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I1228 11:39:09.745631       1 leaderelection.go:252] successfully acquired lease rabbitmq-system/rabbitmq-cluster-operator-leader-election
2020-12-28T11:39:09.745Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"rabbitmq-system","name":"rabbitmq-cluster-operator-leader-election","uid":"0772424c-9f73-4f2b-8084-0ec0217010fa","apiVersion":"v1","resourceVersion":"11057"}, "reason": "LeaderElection", "message": "rabbitmq-cluster-operator-6b78c48b86-s7rxr_3d9c8373-dbb9-4c07-99a1-fd2fbe56784e became leader"}
2020-12-28T11:39:09.746Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:09.846Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:09.847Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:09.847Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:09.847Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:09.948Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:10.048Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:10.149Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "source": "kind source: /, Kind="}
2020-12-28T11:39:10.249Z	INFO	controller	Starting Controller	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster"}
2020-12-28T11:39:10.250Z	INFO	controller	Starting workers	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "worker count": 1}
2020-12-28T11:39:10.353Z	INFO	rabbitmqcluster-controller	Start reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bugs", "spec": "{\"replicas\":1,\"image\":\"rabbitmq:3.8.9-management\",\"service\":{\"type\":\"LoadBalancer\"},\"persistence\":{\"storage\":\"10Gi\"},\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"1\",\"memory\":\"2Gi\"}},\"rabbitmq\":{},\"tls\":{},\"override\":{},\"terminationGracePeriodSeconds\":604800}"}
2020-12-28T11:39:10.368Z	INFO	rabbitmqcluster-controller	updated resource rabbitmq-bugs-server of Type *v1.StatefulSet
2020-12-28T11:39:10.368Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"RabbitmqCluster","namespace":"default","name":"rabbitmq-bugs","uid":"1c2e739d-c948-4798-8882-302a5b414d60","apiVersion":"rabbitmq.com/v1beta1","resourceVersion":"3585"}, "reason": "SuccessfulUpdate", "message": "updated resource rabbitmq-bugs-server of Type *v1.StatefulSet"}
2020-12-28T11:39:10.382Z	INFO	rabbitmqcluster-controller	Finished reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bugs"}
2020-12-28T11:39:10.382Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "name": "rabbitmq-bugs", "namespace": "default"}
2020-12-28T11:39:10.383Z	INFO	rabbitmqcluster-controller	Start reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bunny", "spec": "{\"replicas\":1,\"image\":\"rabbitmq:3.8.9-management\",\"service\":{\"type\":\"LoadBalancer\"},\"persistence\":{\"storage\":\"10Gi\"},\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"1\",\"memory\":\"2Gi\"}},\"rabbitmq\":{},\"tls\":{},\"override\":{},\"terminationGracePeriodSeconds\":604800}"}
2020-12-28T11:39:10.462Z	INFO	rabbitmqcluster-controller	updated resource rabbitmq-bunny-server of Type *v1.StatefulSet
2020-12-28T11:39:10.462Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"RabbitmqCluster","namespace":"default","name":"rabbitmq-bunny","uid":"a856bf33-c54e-4de2-991f-f9ea6d5ceabf","apiVersion":"rabbitmq.com/v1beta1","resourceVersion":"3909"}, "reason": "SuccessfulUpdate", "message": "updated resource rabbitmq-bunny-server of Type *v1.StatefulSet"}
2020-12-28T11:39:10.474Z	INFO	rabbitmqcluster-controller	Finished reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bunny"}
2020-12-28T11:39:10.474Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "name": "rabbitmq-bunny", "namespace": "default"}
2020-12-28T11:39:10.474Z	INFO	rabbitmqcluster-controller	Start reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bugs", "spec": "{\"replicas\":1,\"image\":\"rabbitmq:3.8.9-management\",\"service\":{\"type\":\"LoadBalancer\"},\"persistence\":{\"storage\":\"10Gi\"},\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"1\",\"memory\":\"2Gi\"}},\"rabbitmq\":{},\"tls\":{},\"override\":{},\"terminationGracePeriodSeconds\":604800}"}
2020-12-28T11:39:10.485Z	INFO	rabbitmqcluster-controller	updated resource rabbitmq-bugs-server of Type *v1.StatefulSet
2020-12-28T11:39:10.486Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"RabbitmqCluster","namespace":"default","name":"rabbitmq-bugs","uid":"1c2e739d-c948-4798-8882-302a5b414d60","apiVersion":"rabbitmq.com/v1beta1","resourceVersion":"3585"}, "reason": "SuccessfulUpdate", "message": "updated resource rabbitmq-bugs-server of Type *v1.StatefulSet"}
2020-12-28T11:39:10.495Z	INFO	rabbitmqcluster-controller	Finished reconciling RabbitmqCluster	{"namespace": "default", "name": "rabbitmq-bugs"}
2020-12-28T11:39:10.495Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "rabbitmq.com", "reconcilerKind": "RabbitmqCluster", "controller": "rabbitmqcluster", "name": "rabbitmq-bugs", "namespace": "default"}

events

LAST SEEN   TYPE      REASON                   OBJECT                                                      MESSAGE
13m         Normal    SuccessfulUpdate         rabbitmqcluster/rabbitmq-bugs                               updated resource rabbitmq-bugs-server of Type *v1.StatefulSet
13m         Normal    SuccessfulUpdate         rabbitmqcluster/rabbitmq-bunny                              updated resource rabbitmq-bunny-server of Type *v1.StatefulSet

rabbitmq-bugs describe that shows no pod updates

Name:         rabbitmq-bugs
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  rabbitmq.com/v1beta1
Kind:         RabbitmqCluster
Metadata:
  Creation Timestamp:  2020-12-28T10:51:27Z
  Finalizers:
    deletion.finalizers.rabbitmqclusters.rabbitmq.com
  Generation:  2
  Managed Fields:
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:image:
        f:persistence:
          .:
          f:storage:
        f:replicas:
        f:resources:
          .:
          f:limits:
            .:
            f:memory:
          f:requests:
            .:
            f:memory:
        f:service:
          .:
          f:type:
        f:terminationGracePeriodSeconds:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2020-12-28T10:51:27Z
    API Version:  rabbitmq.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
      f:spec:
        f:override:
        f:rabbitmq:
        f:resources:
          f:limits:
            f:cpu:
          f:requests:
            f:cpu:
        f:tls:
      f:status:
        .:
        f:conditions:
        f:defaultUser:
          .:
          f:secretReference:
            .:
            f:keys:
              .:
              f:password:
              f:username:
            f:name:
            f:namespace:
          f:serviceReference:
            .:
            f:name:
            f:namespace:
    Manager:         manager
    Operation:       Update
    Time:            2020-12-28T10:52:10Z
  Resource Version:  3585
  Self Link:         /apis/rabbitmq.com/v1beta1/namespaces/default/rabbitmqclusters/rabbitmq-bugs
  UID:               1c2e739d-c948-4798-8882-302a5b414d60
Spec:
  Image:  rabbitmq:3.8.9-management
  Override:
  Persistence:
    Storage:  10Gi
  Rabbitmq:
  Replicas:  1
  Resources:
    Limits:
      Cpu:     2
      Memory:  2Gi
    Requests:
      Cpu:     1
      Memory:  2Gi
  Service:
    Type:                            LoadBalancer
  Termination Grace Period Seconds:  604800
  Tls:
Status:
  Conditions:
    Last Transition Time:  2020-12-28T10:52:10Z
    Reason:                AllPodsAreReady
    Status:                True
    Type:                  AllReplicasReady
    Last Transition Time:  2020-12-28T10:52:10Z
    Reason:                AtLeastOneEndpointAvailable
    Status:                True
    Type:                  ClusterAvailable
    Last Transition Time:  2020-12-28T10:51:27Z
    Reason:                NoWarnings
    Status:                True
    Type:                  NoWarnings
    Last Transition Time:  2020-12-28T10:51:27Z
    Message:               Created or Updated all child resources
    Reason:                Success
    Status:                True
    Type:                  ReconcileSuccess
  Default User:
    Secret Reference:
      Keys:
        Password:  password
        Username:  username
      Name:        rabbitmq-bugs-default-user
      Namespace:   default
    Service Reference:
      Name:       rabbitmq-bugs
      Namespace:  default
Events:
  Type    Reason            Age                From                        Message
  ----    ------            ----               ----                        -------
  Normal  SuccessfulUpdate  17m (x2 over 17m)  rabbitmqcluster-controller  updated resource rabbitmq-bugs-server of Type *v1.StatefulSet

[mkuratczyk]

If I'm not mistaken, only TLS-enabled clusters will be restarted (since we mount TLS secrets differently in 1.3.0). I don't think there is any simple way to check that on the user side - we need to be more explicit in the release note based on our understanding of what we change (eg. in this case we know we change the way TLS secrets are defined in the StatefulSet and therefore we know this will only affect TLS-enabled clusters).

[gerhard]

That sounds reasonable. Will test this and update the release notes accordingly. Thanks!