Different values of "File descriptors" in WebUI and "rabbitmqctl status" output and new clients not being able to establish connection.

[sarlashin-8x8]

We have a RabbitMQ 3.13 cluster of 3 nodes running on Oracle Linux 9 (systemd) (we also saw a similar behaviour with 3.7.x on CentOS 7.x (systemd) ).

Several times we had a problem when new clients were not able to establish connection to the cluster while the clients that had been already connected didn't have any problems.

Also rabbitmqctl status on 2 nodes out of 3 got stuck.

We didn't see any error messages or warnings in RabbitMQ logs either.

We did an investigation and found out that the value for File descriptors in Web UI was around 200-600 and matched the output of ls /proc/${pid}/fd |wc -l, but the value reported by rabbitmqctl status (where it was available) was way bigger - very close to the configured limit - 32768.

So we increased the limit of open files for all RabbitMQ processes via prlimit to 3276800 on all nodes of the cluster and then set a new value for file_handle_cache via rabbitmqctl eval 'file_handle_cache:set_limit(3276800).'. Right after that all the new clients that were not able to establish the connection to the cluster managed to connect successfully. And rabbitmqctl status started to work too.

This is the current output of rabbitmqctl status (after we increased Max open files limit )

# rabbitmqctl status --formatter json | jq '.file_descriptors'
{
  "total_used": 33471,
  "total_limit": 3276800,
  "sockets_used": 567,
  "sockets_limit": 2949118
}

And WebUI at the moment is showing 641.

We also noticed that Prometheus metrics report same value as Web UI , and we cannot find the value reported by rabbitmqctl status anywhere else.

So can anyone please explain if this is a normal behaviour?
And also should we monitor the value for .file_descriptors.total_used from rabbitmqctl status and adjust Max open files limit according to it?

Thanks!