Different behavior around internode TLS on Erlang 26

[daleksic-godaddy]

Describe the bug

When Erlang 26 is used, specified config file with-ssl_dist_optfile option is not applied, therefore inter communication node on port 25672 is not configured serve TLS certificate.

Same configuration works when rabbitmq 3.12 & erlang 25 is used, but it fails in combinations rabbitmq 3.12 / erlang 26 and rabbitmq 3.13 / erlang 26.

Also, rabbitmqctl and rabbitmq-diagnostics tools fails to connect on affected versions.

Reproduction steps

PoC with steps to reproduce can be found at https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/tree/old-docker-poc?tab=readme-ov-file

Expected behavior

• inter_node_tls.config to be applied when RabbitMQ is is started with -pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config additional flags.
• Inter communication endpoint on 25672 port should serve SSL certificate

Additional context

No response

[michaelklishin]

@daleksic-godaddy RabbitMQ does not implement TLS and is not responsible for handling of -proto_dist and friends. The runtime does/is.

Port 5672 is not supposed to be used for TLS.

The only user-facing TLS-related change that RabbitMQ had to adapt to is the new default at least one peer verification setting. RabbitMQ has retained the old default where it can control the value (this is not the case with TLS for inter-node communication), most of the changes I recall in this area have shipped in 3.13.1.

[michaelklishin]

Highly relevant erlang/otp#7497.

[lukebakken]

I've checked and, at first glance, @daleksic-godaddy's configuration file does not have that issue - https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/blob/main/docker/config/inter_node_tls.config

[lukebakken]

Hello, thanks for using RabbitMQ and for providing such a comprehensive report. I wish every RabbitMQ user were as thorough.

I have the following project to form RabbitMQ clusters, and this branch will set one up using TLS -

https://github.com/lukebakken/docker-rabbitmq-cluster/tree/tls

I'm making sure it still works as intended, but you should be able to compare what I do with what you do to figure out the difference.

[lukebakken]

Hmm, something isn't quite right with your project, because running rabbitmqctl status using the rabbitmq312-erlang25 service isn't working, either. I'm investigating.

[lukebakken]

@daleksic-godaddy I have spent probably more time today than I should trying to figure out why your project doesn't work out-of-the-box in my docker environment. I had to make the following changes so that the make test_erlang25_rmq312 target works once the rabbitmq312-erlang25 service is running:

daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc#1

If I docker compose exec into the rabbitmq312-erlang25 container, and run rabbitmqctl status, it ONLY works if I disable peer verification on the inter-node TLS client:

https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/pull/1/files#diff-c2d90e5845596a73ed9aff5c445e389d3233d90e887474a8dcbaf4ece784825eR18

I can't explain this behavior.

One major difference between my working project and yours is that I use the official RabbitMQ docker image.

I don't have the time to continue investigating this issue as I have to focus support efforts on RabbitMQ users with support contracts. I will return to this as my time and interest allow. Thanks again for the comprehensive report.

For what it's worth, this is what I see when I clone your project as-is and run make:
rabbitmq-server-11074-make.log

2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> BOOT FAILED
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> ===========
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> Error during startup: {error,failed_to_prepare_configuration}
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0>
Error during startup: {error,failed_to_prepare_configuration}

2024-04-24 17:25:20.491289+00:00 [error] <0.131.0> ssl_options.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.512177+00:00 [error] <0.131.0> ssl_options.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514094+00:00 [error] <0.131.0> management.ssl.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514195+00:00 [error] <0.131.0> management.ssl.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514466+00:00 [error] <0.131.0> Error preparing configuration in phase validation:
2024-04-24 17:25:20.514500+00:00 [error] <0.131.0>   - ssl_options.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514561+00:00 [error] <0.131.0>   - ssl_options.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514620+00:00 [error] <0.131.0>   - management.ssl.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514673+00:00 [error] <0.131.0>   - management.ssl.keyfile invalid, file does not exist or cannot be read by the node

I don't see how it could work in your environment. Please review my PR.

[daleksic-godaddy]

@lukebakken Thanks for in-depth analysis.

Ok, this is related to how docker mounts files inside container (owned by root) - my mistake 😅. I overlooked it, as I was able to replicate non-serving cert issue when checking with openssl command. Our actual setup is not using containers, we are on Ubuntu 22.04 VMs. This was my quick attempt to replicate issue using lightweight PoC environment instead full blown VMs. That's why I didn't used official rabbimtq docker images, instead I've used custom one which mimics our ansible provisioning.

I will check once more configuration and examples that you provided, and properly replicate issue if still persists.

[daleksic-godaddy]

@lukebakken I was able to fully replicate issue using Vagrant local environment, and also find the cause 🎉 .

Here is the new PoC enviroment https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/tree/60f91070f6d8bc6b0341e632b73361cf16dc8648

In a nutshell, difference is due to how ssl options are handled in Erlang 26. It seems that excess options are not ignored anymore in erlang26, therefore:

[
  {server, [
    {cacertfile, "/etc/rabbitmq/tls/rabbitmq_server_wildcard.ca"},
    {certfile,   "/etc/rabbitmq/tls/rabbitmq_server_wildcard.crt"},
    {keyfile,    "/etc/rabbitmq/tls/rabbitmq_server_wildcard.key"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true},
    {customize_hostname_check, [
      {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]},
  {client, [
    {cacertfile, "/etc/rabbitmq/tls/rabbitmq_client_wildcard.ca"},
    {certfile,   "/etc/rabbitmq/tls/rabbitmq_client_wildcard.crt"},
    {keyfile,    "/etc/rabbitmq/tls/rabbitmq_client_wildcard.key"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {customize_hostname_check, [
      {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]}
].

This config will work fine with Erlang 25, but it would fail on Erlang 26 as customize_hostname_check is only valid for client section (clue 01 & clue 02).

This then leads that server section is not loaded at all, therefore ssl certificate is not served on 25672 (internode communication) port (as seen in the README with client_variant: "wildcard_cert_both_customize" set in the Vagrant PoC. Didn't have time to deeply analyze Erlang code, this is based on my observation and testing.

Concerning part is that this is breaking change for folks that configured theirs setup with customize_hostname_check in both section, and want to upgrade to Erlang 26 & newer RabbitMQ versions. Also, a found few examples like this one #10398 (comment) that contains example where customize_hostname_check is specified in both places. I've also found similar one where {fail_if_no_peer_cert, true} was also included in both client and server section and it was failing to to same reason as here, but can't find link at the moment.

I would strongly suggest to add/update documentation to include:

1. note about new behavior in Erlang 26 that may lead to breaking setups
2. note about ensuring that all configuration options are only put in appropriate sections / remove excess

Also, rabbitmq/rabbitmq-website#1791 to be resolved, would be highly appreciated, as I was also affected same issue where customize_hostname_check is not metioned as required for wildcard certs in the docs.

[lukebakken]

Thanks for following up.

Yep, this caught me as well last year when I started using Erlang 26.

Please see the following:

• Distributed Erlang over TLS does not work correctly using OTP 26 erlang/otp#7497
• Distributed Erlang over TLS does not work correctly using OTP 26 erlang/otp#7497 (comment)
• Remove invalid TLS settings rabbitmq-website#1687

Our docs have been updated for a while to reflect fail_if_no_peer_cert, but not customize_hostname_check, because it is very rare for people to use wildcard certs for clusters.

If someone from the community would like to update the docs, a pull request would be greatly appreciated.

[michaelklishin]

And here's where the website source code lives. Note that in this case contributors should make sure to update both main and the versioned_docs 3.13.x versions.