Compatibility rabbitmq with glusterfs StorageClass

[lretrain91]

Hello,
First i already post the question on rabbitmq-user but we get no solution, then i post here.
here the related ticket : https://groups.google.com/g/rabbitmq-users/c/cwCzOYBmx2g

We are facing issue by using rabbitmq with glusterfs StorageClass in k8s env.
We are deploying 3 rabbitmq pod.

First we thought that it's could be a race/locking issue but can't be because, yes we use glusterfs so we might think that the data is shared between the 3 pod but in our case we are declaring a pvc for each pod.

We get the following error :
2020-11-05 08:51:13.892 [error] <0.338.0> Mnesia('rabbit@ornery-clownfish-crmq-1'): ** ERROR ** (core dumped to file: "/var/lib/rabbitmq/MnesiaCore.rabbit@ornery-clownfish-crmq-1_1604_566273_884697") ** FATAL ** {error,{"Cannot open dets table",schema,[{file,"/var/lib/rabbitmq/mnesia/rabbit@ornery-clownfish-crmq-1/schema.DAT"},{keypos,2},{repair,true},{type,set}],{not_a_dets_file,"/var/lib/rabbitmq/mnesia/rabbit@ornery-clownfish-crmq-1/schema.DAT"}}} 2020-11-05 08:51:23.887 [error] <0.338.0> Mnesia('rabbit@ornery-clownfish-crmq-1'): ** ERROR ** mnesia_event got unexpected event: {'EXIT',<0.342.0>,killed} 2020-11-05 08:51:23.888 [info] <0.338.0> [{initial_call,{gen_event,init_it,['Argument__1','Argument__2','Argument__3','Argument__4','Argument__5','Argument__6']}},{pid,<0.338.0>},{registered_name,mnesia_event},{error_info,{exit,killed,[{gen_event,terminate_server,4,[{file,"gen_event.erl"},{line,397}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[mnesia_sup,<0.336.0>]},{message_queue_len,0},{messages,[]},{links,[]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,28},{reductions,1387}], [] 2020-11-05 08:51:23.888 [error] <0.338.0> CRASH REPORT Process mnesia_event with 0 neighbours exited with reason: killed in gen_event:terminate_server/4 line 397 2020-11-05 08:51:23.891 [info] <0.335.0> [{initial_call,{application_master,init,['Argument__1','Argument__2','Argument__3','Argument__4']}},{pid,<0.335.0>},{registered_name,[]},{error_info,{exit,killed,[{application_master,terminate,2,[{file,"application_master.erl"},{line,232}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}},{ancestors,[<0.334.0>]},{message_queue_len,0},{messages,[]},{links,[<0.44.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,987},{stack_size,28},{reductions,87875}], [] 2020-11-05 08:51:23.891 [error] <0.335.0> CRASH REPORT Process <0.335.0> with 0 neighbours exited with reason: killed in application_master:terminate/2 line 232 2020-11-05 08:51:23.892 [info] <0.44.0> Application mnesia exited with reason: killed 2020-11-05 08:51:23.892 [info] <0.44.0> Application mnesia exited with reason: killed

it seems that when using glusterfs schema.DAT is corrupted
Then in our case because our file is corrupted when crmq-1 try to access this file to update it's own it fail then restart.

and after restarting we get :

dets: file "/var/lib/rabbitmq/mnesia/rabbit@ornery-clownfish-crmq-2/schema.DAT" not properly closed, repairing ...

Do you know if there is already limitation in glusterfs use, i was not able to find anything related to that.

If you need more info please ask.

Thank's a lot
Léo Retrain.

[michaelklishin]

Neither RabbitMQ nor its underlying runtime or libraries have any dependencies on a specific filesystem.

Cannot open dets table

means that the node failed to open a DETS table. Those tables are backed by pretty basic files, nothing sophisticated. You can run the node under strace to see what specific file system operations are involved.

As far as using distributed filesystems for RabbitMQ node data goes, we recommend against it. These latency of such file systems can fluctuate greatly, and it is very important for real world throughput and latency of RabbitMQ. One thing that you absolutely cannot do with RabbitMQ nodes is to share their data directories (meaning multiple nodes use a single directory for their data).