RabbitMQ acknowledges multiple publishes in a single TCP packet

[shaxiaozz]

Describe the bug

I'm using a java spring-boot-starter-amqp client for Basic.Publish and I'm in production to push messages to MQ and I need MQ to return acknowledgements. But it often occurs occasionally in my production environment:

Caused by: java.util.concurrent.TimeoutException: null
	at com.rabbitmq.client.impl.ChannelN.waitForConfirms(ChannelN.java:229)
	at org.springframework.amqp.rabbit.connection.PublisherCallbackChannelImpl.waitForConfirms(PublisherCallbackChannelImpl.java:665)
	at sun.reflect.GeneratedMethodAccessor845.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1184)
	at com.sun.proxy.$Proxy313.waitForConfirms(Unknown Source)
	at org.springframework.amqp.rabbit.core.RabbitTemplate.waitForConfirms(RabbitTemplate.java:2241)
	... 35 common frames omitted

My RabbitMQ-Server version is 3.7.8 Erlang 21.1

Here are the packets I grabbed from the client to communicate with RabbitMQ:

I found that when there is a waitForConfirms timeout, instead of RabbitMQ replying to the Basic.ACK of the corresponding delivery_tag on this channel, one packet replies to almost 20 different or duplicate delivery_tag Basic.ACKs, and then after 3s (the client waitForConfirms timeout=3s) the client voluntarily closed this connection.

Please pay attention to the place of the red frame, client 10.82.48.237 Basic.Publish production message, and after a packet, RabbitMQ server is the same packet to return more than one Basic.ACK, and after 3s, the client 10.82.49.237 initiates the Channel.Close, thus triggering the waitForConfirms timeout

I don't know why RabbitMQ responds to the Basic.ACK in this way, ideally RabbitMQ should return the Basic.ACK with delivery_tag=4. Instead of a packet containing multiple Basic.ACKs, I'd like to see it with delivery_tag=1 on top of the packet. Delivery_tag=1, delivery_tag=2, delivery_tag=3 that way operation.

I don't know what the problem is, you guys.

Reproduction steps

When the client has finished producing messages, and is waiting for RabbitMQ to return ACKs, and RabbitMQ contains multiple ACKs in the same TCP packet level, I will have a waitForConfirms timeout.

Expected behavior

I don't know why RabbitMQ responds to the Basic.ACK in this way, ideally RabbitMQ should return the Basic.ACK with delivery_tag=4. Instead of a packet containing multiple Basic.ACKs, I'd like to see it with delivery_tag=1 on top of the packet. Delivery_tag=1, delivery_tag=2, delivery_tag=3 that way operation.

Additional context

No response

[michaelklishin]

I assume that by "did not send a single packet" you really mean "multiple acknowledgements were sent in the same TCP packet" because "not send a single packet" means that none acknowledgements were sent and this is not the case.

@shaxiaozz RabbitMQ does not really control how the data it sends is organized into TCP packets. More importantly, acknowledging multiple deliveries at once is a perfectly fine thing to do. AMQP 0-9-1's basic.ack frame allows both clients and RabbitMQ to acknowledge multiple deliveries (or published messages) at once.

I don't see why it would "ideally not do that". It makes publisher confirms handling somewhat more complex if you use streaming confirms but some users would say multi-acks is the right thing to do under load.

[shaxiaozz]

@michaelklishin @lukebakken
The publisher code is as follows(Publisher ConfirmType is CORRELATED):

rabbitTemplate.invoke(template -> {
template.convertAndSend(exchangeName, routingKey, jsonMessage);
template.waitForConfirms(3000);
return true;
});

[lukebakken]

Please carefully read what I wrote:

Please provide the code you are using to publish messages and receive confirmations. I should be able to clone it and run it, so provide instructions if it is not obvious. I don't have time to guess at how to reproduce this.

I can't run the code you provided.

The fact is, I don't have time to look at this. If there is a bug, it is in your code. If something isn't working they way you expect, then compare the behavior with a well-known tool, like PerfTest. We know RabbitMQ and our official client libraries work correctly with regard to delivery tags - perhaps our documentation needs to be updated to reflect that they may not be exactly sequential.

[shaxiaozz]

@lukebakken
I apologize for the delay~

I'll take some time to pull out the publisher and consumer code and clearly describe how to reproduce the issue

Thanks again for your reply~

[michaelklishin]

@shaxiaozz like I said above, many protocols, including TCP, deal with confirmations (whatever exactly is being confirmed in a particular protocol does not matter) arriving out of order. We cover three different approaches to working with publisher confirms in the docs.

[shaxiaozz]

@michaelklishin Thank you very much for your response, I will take another hard look at this phenomenon with my team. Thanks for the link.

[lukebakken]

@shaxiaozz - here is what I did to observe a basic set of AMQP publishes with confirmations in my local environment:

• Start RabbitMQ 3.12.10 / Erlang 26.1.2
• Start Wireshark - listen to localhost:5672
• Run PerfTest with the following arguments

  --rate 10 --confirm 1 --pmessages 10 --producers 1 --consumers 0
  

• Here is the capture: localhost-rabbitmq-3.12-perftest-capture.zip

You will note that the delivery tags are sequential.

Please take the time to run the same PerfTest command that I ran in your environment.

I just noticed this in your original message:

My RabbitMQ-Server version is 3.7.8 Erlang 21.1

RabbitMQ 3.7.8 is OVER FIVE YEARS OLD, and is unsupported. I will not spend any more time on this issue until you update to the latest version of RabbitMQ.

Do not take the time to provide code if you are unable to upgrade RabbitMQ.

[shaxiaozz]

@lukebakken Okay, copy that.Thanks.