Different values of "File descriptors" in WebUI and "rabbitmqctl status" output and new clients not being able to establish connection. #11738
Unanswered
sarlashin-8x8
asked this question in
Other
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We have a RabbitMQ 3.13 cluster of 3 nodes running on Oracle Linux 9 (systemd) (we also saw a similar behaviour with 3.7.x on CentOS 7.x (systemd) ).
Several times we had a problem when new clients were not able to establish connection to the cluster while the clients that had been already connected didn't have any problems.
Also
rabbitmqctl status
on 2 nodes out of 3 got stuck.We didn't see any error messages or warnings in RabbitMQ logs either.
We did an investigation and found out that the value for
File descriptors
in Web UI was around 200-600 and matched the output ofls /proc/${pid}/fd |wc -l
, but the value reported byrabbitmqctl status
(where it was available) was way bigger - very close to the configured limit -32768
.So we increased the limit of open files for all RabbitMQ processes via
prlimit
to3276800
on all nodes of the cluster and then set a new value forfile_handle_cache
viarabbitmqctl eval 'file_handle_cache:set_limit(3276800).'
. Right after that all the new clients that were not able to establish the connection to the cluster managed to connect successfully. Andrabbitmqctl status
started to work too.This is the current output of
rabbitmqctl status
(after we increasedMax open files
limit )And WebUI at the moment is showing
641
.We also noticed that Prometheus metrics report same value as Web UI , and we cannot find the value reported by
rabbitmqctl status
anywhere else.So can anyone please explain if this is a normal behaviour?
And also should we monitor the value for
.file_descriptors.total_used
fromrabbitmqctl status
and adjustMax open files
limit according to it?Thanks!
Beta Was this translation helpful? Give feedback.
All reactions