Client connection closed after crc validation error

[lukaszsamson]

Describe the bug

Version 3.12.2
erlang 25.0.4
os ubuntu 20.04

Occasionally an application using https://github.com/rabbitmq/rabbitmq-dotnet-client subscribing to a stream on a 3 node cluster via AMQP 0.9.1 subscription will get a channel close

Channel shutdown AMQP close-reason, initiated by Peer, code=541, text='INTERNAL_ERROR', classId=0, methodId=0

A server log indicates an erlang process crash with crc validation failure at chunk id... and supervisor terminating the child due to max restart intensity.

The cluster is sufficiently provisioned and in the moment of crash was running without other problems.

Possibly related to https://groups.google.com/g/rabbitmq-users/c/29yEH-2F3hc/m/BQtGQstqCAAJ

Reproduction steps

No idea. It happens from time to time. The client restarts and reconnects with no persisting problem

Expected behavior

No crash

Additional context

server log:
rabbitmq-server-8972-server-log.txt

[michaelklishin]

CRC checks are performed only in one place and they can indicate genuine on disk file corruption. RabbitMQ cannot avoid
them entirely since there are at two or three layers lower on the file write path. The entire idea is to detect issues with on disk storage, whatever the root cause may be.

There was an incorrectly performed check in Ra that affected quorum queues on Erlang 26. It was addressed a few months ago. I do not remember seeing this in the context of streams that use a different library, Osiris.

Can you share the segment files of the stream in question? There is no other way for us to perform root cause analysis.

[lukebakken]

cc @kjnilsson

@lukaszsamson just an FYI, if you have more than a few lines of text to add to a GitHub issue or discussion it's better to attach a file with the content. I have edited the original post to do so.

I'm not exactly sure where the segment files are kept for this particular stream, but it should be a subdirectory of this:

/var/lib/rabbitmq/mnesia/rabbit@MyService-RabbitMQv2-2/stream

[lukaszsamson]

@michaelklishin I couldn't find stream file with the exact chunk id but this one had a closest modification date
https://drive.google.com/file/d/1oSC9NUGrg454yKSj3irliW7qdmj9P3SM/view?usp=drive_link

[lukaszsamson]

I noticed this PR by @kjnilsson rabbitmq/osiris#137 that may be related to this issue. It got included in osiris v1.6.4 which made it to rabbitmq 3.12.5. I'll post an update after some testing.

[michaelklishin]

RabbitMQ 3.12 is out of community support. We will not investigate any reports for versions older than 3.13.6.

[lukaszsamson]

I’m aware of that