how long should client wait on dead connection for message draining

[lnthai2002]

Hello, from this article https://artemiscloud.io/docs/tutorials/scaleup_and_scaledown/#step-5---scale-down-with-message-draining I understand that the operator will move messages to other broker when a broker is down (scaled down or crash?) i assume this draining process would take some time and thus the client (both consumer and producer) must wait until all messages has been moved over to the new broker. How long should i set clientFailureCheckPeriod on the client connection to wait for dead connection before aborting the current connection and move on to the next one?

[brusdev]

@lnthai2002 the messages are migrated only if the brokers are scaled down (decreasing the DeploymentPlan.Size field), can you share the connection url that you are using to connect to the broker cluster?

[lnthai2002]

Here is my connection:
spring.artemis.broker-url=(tcp://localhost:61610,tcp://localhost:61611,tcp://localhost:61612)?clientFailureCheckPeriod=30000&sslEnabled=false

I'm using localhost because my consumer and provider are running on my desktop, so i port-forwarded request from 61610, 61611, 61612 to the 3 kubernetes services at 61617:

kubectl port-forward service/activemq-artemis-broker-all-protocols-acceptor-0-svc 61610:61617 -n nle
kubectl port-forward service/activemq-artemis-broker-all-protocols-acceptor-1-svc 61611:61617 -n nle
kubectl port-forward service/activemq-artemis-broker-all-protocols-acceptor-2-svc 61612:61617 -n nle

The 3 brokers are deployed on kuberntes:

activemq-artemis-broker-ss-0                 1/1     Running   0          13m
activemq-artemis-broker-ss-1                 1/1     Running   0          13m
activemq-artemis-broker-ss-2                 1/1     Running   0          13m

and their kubernetes services:

activemq-artemis-broker-ping-svc                       ClusterIP   None            <none>        8888/TCP             13m
activemq-artemis-broker-hdls-svc                       ClusterIP   None            <none>        8161/TCP,61616/TCP   13m
activemq-artemis-broker-all-protocols-acceptor-0-svc   ClusterIP   10.43.221.182   <none>        61617/TCP            13m
activemq-artemis-broker-all-protocols-acceptor-1-svc   ClusterIP   10.43.32.131    <none>        61617/TCP            13m
activemq-artemis-broker-all-protocols-acceptor-2-svc   ClusterIP   10.43.75.219    <none>        61617/TCP            13m
activemq-artemis-broker-wconsj-0-svc                   ClusterIP   10.43.202.17    <none>        8161/TCP,8162/TCP    13m
activemq-artemis-broker-wconsj-1-svc                   ClusterIP   10.43.107.206   <none>        8161/TCP,8162/TCP    13m
activemq-artemis-broker-wconsj-2-svc                   ClusterIP   10.43.82.138    <none>        8161/TCP,8162/TCP    13m

Here is my deployment:

apiVersion: broker.amq.io/v1beta1
kind: ActiveMQArtemis
metadata:
  name: activemq-artemis-broker
spec:
  acceptors:
    - name: "all-protocols-acceptor"
      bindToAllInterfaces: true
      port: 61617
  addressSettings:
    applyRule: merge_all
    addressSetting:
    - match: '#'
      deadLetterAddress: DLQ
      expiryAddress: ExpiryQueue
      redistributionDelay: 60000
      redeliveryDelay: 0
      maxSizeBytes: "-1"
      messageCounterHistoryDayLimit: 10
      addressFullPolicy: PAGE
      autoCreateQueues: true
      autoCreateAddresses: true
      autoCreateJmsQueues: true
      autoCreateJmsTopics: true
      autoDeleteQueues: false
      autoDeleteCreatedQueues: false
      autoDeleteAddresses: false
      defaultConsumerWindowSize: 0
    - match: 'JOB_QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'CRON_JOB_QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'QUEUE.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'TOPIC.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
      autoDeleteQueuesDelay: 300000
      autoDeleteQueuesMessageCount: 0
      autoDeleteAddresses: true
    - match: 'TOPIC.internal.#'
      autoDeleteQueues: true
      autoDeleteCreatedQueues: true
      autoDeleteQueuesDelay: 10000
      autoDeleteQueuesMessageCount: -1
      autoDeleteAddresses: true
    - match: 'TOPIC.retro.#'
      retroactiveMessageCount: 1
      autoDeleteQueues: true
      autoDeleteQueuesDelay: 0
      autoDeleteQueuesMessageCount: -1
    - match: '$.artemis.internal.#'
      autoDeleteQueues: false
      autoDeleteAddresses: true
  console:
    expose: true
    sslEnabled: false
  deploymentPlan:
    size: 3
    requireLogin: true
    messageMigration: true
    persistenceEnabled: true
    podSecurityContext:
      fsGroup: 0
    storage:
      size: {{ .Values.persistence.size }}
      storageClassName: {{ .Values.persistence.storageClass }}
    resources:
      requests:
        cpu: {{ .Values.resources.requests.cpu }}
        memory: {{ .Values.resources.requests.memory }}
  adminUser: {{ .Values.artemisUser }}
  adminPassword: {{ .Values.artemisPassword }}

The reason i am trying to use clientFailureCheckPeriod on the connection string is because i though if there is problem with 1 broker , the client will wait 30 sec before switching to a new broker and when the artemis cluster sees that no consumer is connected to a queue, it will move messages to the other broker that the consumer is now connected to. However, when i stop port forwarding for one of the broker where my consumer is connected to for a few sec and start the port forwarding again, my consumer still could not connect to any broker. Listing queues in all 3 broker i see the same queue that my consumer should use is being created in all 3 brokers.

[gaohoward]

when you stop port-forwarding on one broker the broker pod is still alive, so no scaledown will happen. (you will have to change your deploymentPlan.size from 3 to 2 for it).

[lnthai2002]

Thanks you are right, I misunderstood that message migration also happen when client reconnect to different pod due to any error. However, this brings up the question as why a network error on one broker (simulated by restart port forward) causes client to forever lost connection to broker cluster. I got a feeling that on the client side i need to do more than just listing all broker URLs in the spring.artemis.broker-url=(tcp://localhost:61610,tcp://localhost:61611,tcp://localhost:61612)?clientFailureCheckPeriod=30000&sslEnabled=false. To correctly use the cluster of broker, do i need to setup client discovery?

[gaohoward]

No you don't need. I think your client need to add a ExceptioListener to catch the connection falures and re-try connect. By default the client will try the endpoints in your url until it connects one of them.

[brusdev]

ARTEMIS-4251 will support automatic failover to other live servers. It will be included in v1.0.12.