TLS server alert "Handshake Failure malformed_handshake_data"

[kotireddy478]

Describe the bug

Hi,

I'm trying to install rabbitmq 4.0.2/4.0.1 on Red Hat 8 Linux but it's giving below error:

TLS server: In state hello at tls_dtls_connection.erl:696 generated SERVER ALERT: Fatal - Handshake Failure
malformed_handshake_data

Could you please help? Please let me know if you need any additional info.

Reproduction steps

1. download rabbitmq-server and erlang off the office repositories, direct download
2. install both the packages using dnf localinstall
3. generate ssl, configure, restart rabbitmq-server. The restart giving the above error.

Expected behavior

UI should work

Additional context

No response

[mkuratczyk]

Are you using Erlang 27.1.1? If so, this is a known bug in Erlang. You can downgrade to 27.1 to avoid it for now.
Keep in mind that Erlang 27 is not officially supported by RabbitMQ:
https://www.rabbitmq.com/docs/which-erlang

[kotireddy478]

@mkuratczyk Thanks for the reply. No - I've installed Erlang 26.2.5.3

[mkuratczyk]

Yeah, same bug. Seems like already fixed in 26.2.5.4:
erlang/otp@1c6172c

[michaelklishin]

The issue is described in a bit more detail in erlang/otp#8908

[kotireddy478]

@mkuratczyk Thanks again, I have downloaded and installed erlang version 26.2.5.4 but still giving the same error. Do I need to clear any cache or anything else please?

[michaelklishin]

@kotireddy478 there is no cache to clear. We cannot really help you with a single line problem definition.

Start with putting together an executable example that reproduces this behavior with tls-gen [1] and [2].

1. https://github.com/rabbitmq/tls-gen
2. https://www.rabbitmq.com/docs/troubleshooting-ssl

[michaelklishin]

This should always have been a discussion.

[michaelklishin]

generate ssl, configure, restart rabbitmq-server.

is not a set of reproduction steps. There are dozens of potential differences in how a certificate can be generated.

Please let me know if you need any additional info.

It on you to provide evidence of a TLS implementation issue, not on us.

In any case, we explicitly mention that without clear evidence of a TLS implementation bug (in Erlang/OTP, since RabbitMQ does not implement TLS), this topic is explicitly not covered for non-paying, non-contributing users. See Troubleshooting TLS for a methodology we use and recommend, and tls-gen for a way to quickly generate certificates (certificate chains) with adequate x.509 fields and extensions to be used with RabbitMQ.

[michaelklishin]

Reviewing recent Erlang/OTP release notes suggests that a fix for erlang/otp#8908, while known and verified by us, hasn't been included into the latest round of patches.

We'll have to wait for 26.2.5.5, 27.1.1 and 25.3.2.16 (if we get any more 25.3.x releases at all).

That said, we don't have enough information to conclude that this is erlang/otp#8908.

[kotireddy478]

Hi @michaelklishin @mkuratczyk, thanks for all your input and apologies for not providing enough info earlier. I have downgraded rabbitmq to 3.13.7 and erlang to 26.0 and not seeing those errors and all working as expected. I have created a server and client authentication ssl cert using The Network Device Enrolment Service (NDES) and deployed and configured same as https://weblogs.asp.net/jeffreyabecker/Using-SSL-client-certificates-for-authentication-with-RabbitMQ. Here are my sample configs:

rabbit.conf:

auth_backends.1   = ldap
auth_backends.1   = internal
management.http_log_dir = /var/log/rabbitmq
log.dir = /var/log/rabbitmq
log.file = rabbit.log
log.file.level = info
log.exchange = true
log.exchange.level = info
log.exchange.formatter = json

advanced.config:
[

    {ssl,  [
                {versions, ['tlsv1.2']}
    ]},
    {rabbit, [
     {ssl_listeners, [5671]},
     {ssl_options, 
                 [{cacertfile, "/etc/pki/tls/certs/MyCA.pem"},
                    {certfile,   "/etc/pki/tls/certs/server.pem"},
                    {keyfile,    "/etc/pki/tls/private/server.key"},
                    {password,  "t0p$3kRe7"},
                    {versions, ['tlsv1.2']},
                    {ciphers,
                                 [{ecdhe_ecdsa,aes_256_gcm,aead,sha384},
                                  {ecdhe_rsa,aes_256_gcm,aead,sha384},
                                  {ecdh_ecdsa,aes_256_gcm,aead,sha384},
                                  {dhe_rsa,aes_256_gcm,aead,sha384},
                                  {dhe_dss,aes_256_gcm,aead,sha384}
                                 ]},
                    {honor_cipher_order, true},
                    {honor_ecc_order, true},
                    {verify,     verify_none},
                    {fail_if_no_peer_cert, false}]}
                 ]} 
    ]},
     {rabbit_management, [
     {listeners,
           [{port, 15671},
            {ssl, true},
            {ssl_opts, 
                 [{cacertfile, "/etc/pki/tls/certs/MyCA.pem"},
                    {certfile,   "/etc/pki/tls/certs/server.pem"},
                    {keyfile,    "/etc/pki/tls/private/server.key"},
                    {password,  "t0p$3kRe7"},
                    {versions, ['tlsv1.2']},
                    {verify, verify_none},
                    {fail_if_no_peer_cert, false},
                    {client_renegotiate, false},
                    {secure_renegotiate, true}
                 ]}
           ]} 
    ]},
    {rabbitmq_auth_backend_ldap, [
          {servers, ["ldap.eng.megacorp.local", "192.168.0.100"]},
          {port,    6389},
          {log, true},
          {connection_pool_size, 64},
          {idle_timeout, 120000},
          {group_lookup_base, "[group-dn, e.g. OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]"},
          {vhost_access_query, 
            { in_group, "[admin-group-dn, e.g. CN=RabbitMQAdmins,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]" }
          },
          {vhost_access_query, 
            { in_group, "[user-group-dn, e.g. CN=RabbitMQUsers,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]" }
          },
          {resource_access_query,
            { for, [
              {permission, configure, 
                  {in_group, "[admin-group-dn, e.g. CN=RabbitMQAdmins,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]"}},
              {permission, write, 
                  {in_group, "[admin-group-dn, e.g. CN=RabbitMQAdmins,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]"}},
              {permission, read, 
                  {in_group, "[user-group-dn, e.g. CN=RabbitMQUsers,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]"}}
            ]}
          },
          {tag_queries, [
            {administrator, { in_group, "[admin-group-dn, e.g. CN=RabbitMQAdmins,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]" }},
            {monitoring,    { in_group, "[user-group-dn, e.g. CN=RabbitMQUsers,OU=SecurityGroups,OU=UserAccounts,DC=gopivotal,DC=com]", "uniqueMember" }},
            {management,    { constant, true }}
          ]}
        ]}
].

Please let me know if you need any additional info from me.

[michaelklishin]

@kotireddy478 we won't be troubleshooting TLS for you, this is very clearly explained in the Community Support policy.

If you have a way to reproduce this behavior without LDAP, advanced.config (everything in the advanced.config above can be set up via rabbitmq.conf) and using certificates generated using one of the tls-gen profiles, we can try to reproduce and report what we find to the Erlang/OTP team.

[michaelklishin]

https://weblogs.asp.net/jeffreyabecker/Using-SSL-client-certificates-for-authentication-with-RabbitMQ is a blog post from 2014.

A number of things have changed:

• tls-gen has long been a standalone project that allows you to avoid using openssl manually in many if not almost all cases (for RabbitMQ client needs, that is)
• rabbitmq.conf supports all key TLS-related settings
• TLS implementations have progressed significantly, changed their defaults, and so on

RabbitMQ does not implement TLS, Erlang/OTP does, so running 4.0.x (the only community-supported series at the moment) on Erlang 26.1.x or even 26.0.x will work exactly the same way when it comes to TLS.