oauth2 connection authenticated only once per resource server

[Martin-4Spaces]

Describe the bug

I can successfully connect to the broker with oauth2, but only once for each resource server. How can this be?

Error message after first connection: "Authentication using an OAuth 2/JWT token failed: {error,no_key}"

RabbitMQ is running local with this Dockerfile

FROM rabbitmq:3.13-alpine

RUN rabbitmq-plugins enable rabbitmq_management
RUN rabbitmq-plugins enable rabbitmq_web_stomp
RUN rabbitmq-plugins enable rabbitmq_auth_backend_oauth2

rabbitmq.conf

default_user = guest
default_pass = guest

listeners.tcp.default = 5672

management.tcp.port = 15672
management.load_definitions = /etc/rabbitmq/definitions.json
management.oauth_enabled = true
management.oauth_client_id = rabbitmq-management
management.oauth_scopes = rabbitmq.tag:administrator
management.oauth_provider_url = http://localhost:8963/api/oauth

log.file.level = debug

auth_backends.1 = rabbit_auth_backend_oauth2
auth_oauth2.jwks_url = https://app/api/oauth/.well-known/openid-configuration/jwks
auth_oauth2.resource_servers.1.id = rabbitmq
auth_oauth2.resource_servers.2.id = rabbitmq1
auth_oauth2.resource_servers.3.id = webclient
auth_oauth2.resource_servers.4.id = rabbitmq-management
auth_oauth2.default_key = id1
auth_oauth2.https.verify = verify_none
auth_oauth2.scope_prefix = rabbitmq.

definitions.conf

{
  "rabbit_version": "3.12.12",
  "rabbitmq_version": "3.12.12",
  "product_name": "RabbitMQ",
  "product_version": "3.12.12",
  "users": [],
  "vhosts": [
    {
      "name": "realm1"
    }
  ],
  "permissions": [  ],
  "topic_permissions": [],
  "parameters": [],
  "policies": [],
  "queues": [],
  "exchanges": [],
  "bindings": []
}

I appreciate any help.

Reproduction steps

Reproduced with management ui
https://github.com/user-attachments/assets/dd771821-de3a-41a9-8ab2-e7062d02b036

I have to restart the rabbitmq server to successfully authenticate again.

Reproduced with PhpAmqpLib as producer client

1. Client requests an access token from OAuth 2.0 provider with resource_server_id/aud "rabbitmq"
2. Client connects with PHP code

$this->connection = new AMQPStreamConnection(
    getenv('RABBITMQ_HOST'),
    getenv('RABBITMQ_PORT'),
    'backend',  // username, ignored by rabbitmq
    $authApi->getAccessToken(),
    'realm1', // vhost
    false, // insist
    PLAIN' // login method
);

3. RabbitMQ Log

2024-08-07 09:37:31 2024-08-07 07:37:31.962904+00:00 [info] <0.764.0> accepting AMQP connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672)
2024-08-07 09:37:32 2024-08-07 07:37:32.269797+00:00 [info] <0.764.0> connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672): user '' authenticated and granted access to vhost 'realm1'
2024-08-07 09:37:32 2024-08-07 07:37:32.305577+00:00 [info] <0.764.0> closing AMQP connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672, vhost: 'realm1', user: '')

4. Client requests a new access token from OAuth 2.0 provider with resource_server_id/aud "rabbitmq"

eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzUxMiJ9.eyJpZCI6ImI5ZDg0OWE2OGU2NTkzNWZhNjA5ZDkxYTBmMDVlODlmYzkwMTM3NmQiLCJqdGkiOiJiOWQ4NDlhNjhlNjU5MzVmYTYwOWQ5MWEwZjA1ZTg5ZmM5MDEzNzZkIiwiaXNzIjoiaHR0cDpcL1wvbG9jYWxob3N0Ojg5NjNcL2FwaVwvb2F1dGgiLCJhdWQiOiJyYWJiaXRtcSIsInN1YiI6IiIsImV4cCI6MTcyMzAxNzE2NywiaWF0IjoxNzIzMDE2MjY3LCJ0b2tlbl90eXBlIjoiYmVhcmVyIiwic2NvcGUiOiJvcGVuaWQgcmFiYml0bXEucmVhZDpyZWFsbTFcLypcLyogcmFiYml0bXEud3JpdGU6cmVhbG0xXC8qXC8qIHJhYmJpdG1xLmNvbmZpZ3VyZTpyZWFsbTFcLypcLyoifQ.dDG7eTALu9qOfDrS96YGwbSMOu2Ir7nXfhGdaRSIhQhvltgSFgKlzShohsi77KmrKa4hX9wewuZirp8vCiwgXk-EdhNia8_uuU8AqIhEZTOU-4OzG1QQeNzIxkO_w026QbOZw9ImKq9I8yNbim4SK6ihOW1P3Uz0_YwFqGSmu5oN0N_qclLE9X6m5u7wqHPXF_uNRe4IKM9SnxO-Ado9lJugiocfo06omLQvEZuBHJC-RbsTnUQ5isVzuYxV7n_TuaecAfkCilISERee5j-86JF7COi6xvkEoWL6udKaWhoajmDCiMDSekM2g-HbE-2zGZH9PSFAaT7CzJ9ZttX6vc1OscNOcPfp2GFsLZWXhcT5XtkB8jOvYgQIXHtwpy2Q_LlHUDhnp8f8qEHoOBInjrLP3xTw07UxYSRgM_hXv0ZQ0GzzJV2cuN1aWchTvNqU5J2S0EVGu2g6cIv6bLcwUesPSeiE7ltUi_1Bbkvfg5Bv_Ie6YTKpqmWF2b5qzjeQgSSEy9_OaPtzb62lvKnDjkeATqDBd8BcVvvfDdhG_SX4culPqAlTCXbmkhAhUaADSM97ivuuF_OPsgOPElGLhTxsoHmVKY6ExxVH4m2Q1MN04F-lgT8oUilCUZdR4KXm2EI31X74AbUFWSENRHE7VauR7mB45pkQXLdD9DCNH4g

5. Client connects with PHP code

$this->connection = new AMQPStreamConnection(
    getenv('RABBITMQ_HOST'),
    getenv('RABBITMQ_PORT'),
    'backend',  // username, ignored by rabbitmq
    $authApi->getAccessToken(),
    'realm1', // vhost
    false, // insist
    PLAIN' // login method
);

6. RabbitMQ Log

2024-08-07 09:37:47 2024-08-07 07:37:47.282890+00:00 [info] <0.793.0> accepting AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672)
2024-08-07 09:37:47 2024-08-07 07:37:47.290199+00:00 [error] <0.793.0> Error on AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672, state: starting):
2024-08-07 09:37:47 2024-08-07 07:37:47.290199+00:00 [error] <0.793.0> PLAIN login refused: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:37:47 2024-08-07 07:37:47.290671+00:00 [info] <0.793.0> closing AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672)

7. Client requests a new access token from OAuth 2.0 provider with resource_server_id/aud "rabbitmq1"

eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzUxMiJ9.eyJpZCI6ImNlMzg4NjI1MWZhNWQ0ZmY0ZGE5NzdiNmZmOGYxNTExYmNhNWFkNWYiLCJqdGkiOiJjZTM4ODYyNTFmYTVkNGZmNGRhOTc3YjZmZjhmMTUxMWJjYTVhZDVmIiwiaXNzIjoiaHR0cDpcL1wvbG9jYWxob3N0Ojg5NjNcL2FwaVwvb2F1dGgiLCJhdWQiOiJyYWJiaXRtcTEiLCJzdWIiOiIiLCJleHAiOjE3MjMwMTc3MTksImlhdCI6MTcyMzAxNjgxOSwidG9rZW5fdHlwZSI6ImJlYXJlciIsInNjb3BlIjoib3BlbmlkIHJhYmJpdG1xLnJlYWQ6cmVhbG0xXC8qXC8qIHJhYmJpdG1xLndyaXRlOnJlYWxtMVwvKlwvKiByYWJiaXRtcS5jb25maWd1cmU6cmVhbG0xXC8qXC8qIn0.jMbMcJrtGjTf-OxbDFYq-ZKP6W9nTcg9seECCE_TjOdk4dUxMCKz9aOqQ3sQHkG20jpOm4DKzayV4a_WPXlJOJHuJOww_QmPFTWniYft7wb2KySjdWlSMmDlAK8ZpAkHyghQPcPPAqjPwaL0OD8jq6luLjXitm_wd9-AWvBLha6LFReda1B8RBnLRUqdebZMhgDxoWcOT1ZK4g6E_ALJrqv3TRrUK_vvpk4svhAou4c2DC7jQH4wI7cXrUF9uqCS39rQIcEXDTtPJgm4J24VQy_UDGNYtsaoWDHxsUf4l72X8f-LSWjauX9O0ehYFOuPKnsX2l1aNkSO63cApH7l1w205kLa-or2nN2bUWl1iJZRzbib_GCnOTHt6hpHFtwcNs9QLx3Vj45gWTiyfYp6-ks8fosGDudCphd0g0rg-btKGp3M8CKKFQIFXWc2PB3w5qX03aMxrv6Jkjz10CFS1GPatA06FNCgziH4KNQ7SiUhwnN6nbZrn2Fz3PpK40_WwPYPXr8ahlVLueebPXlLceDpr5FtUlPhZsAkHd1nHBjzRYcVDv0ssMpGbWIvga_9Va1vJEW-JqKR2Vou9dd3Z4Im6bjtlE3epg04K3GUs-xEmwDGXNR4q2agvIIXnV-j4s30lyythZtNaYJ-xskORUkEudu_eVUDuM2iZVpFf2g

8. Client connects with PHP code - twice
9. RabbitMQ Log

2024-08-07 09:46:55 2024-08-07 07:46:55.301265+00:00 [info] <0.821.0> connection <0.821.0> (192.168.128.2:40536 -> 192.168.128.4:5672): user '' authenticated and granted access to vhost 'realm1'
2024-08-07 09:46:55 2024-08-07 07:46:55.336525+00:00 [info] <0.821.0> closing AMQP connection <0.821.0> (192.168.128.2:40536 -> 192.168.128.4:5672, vhost: 'realm1', user: '')
2024-08-07 09:46:59 2024-08-07 07:46:59.667902+00:00 [info] <0.843.0> accepting AMQP connection <0.843.0> (192.168.128.2:40546 -> 192.168.128.4:5672)
2024-08-07 09:46:59 2024-08-07 07:46:59.675688+00:00 [error] <0.843.0> Error on AMQP connection <0.843.0> (192.168.128.2:40546 -> 192.168.128.4:5672, state: starting):
2024-08-07 09:46:59 2024-08-07 07:46:59.675688+00:00 [error] <0.843.0> PLAIN login refused: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:46:59 2024-08-07 07:46:59.676187+00:00 [info] <0.843.0> closing AMQP connection <0.843.0> (192.168.128.2:40546 -> 192.168.128.4:5672)

Complete log

-08-07 09:33:55 2024-08-07 07:33:55.603368+00:00 [info] <0.555.0> Ready to start client connection listeners
2024-08-07 09:33:55 2024-08-07 07:33:55.607877+00:00 [info] <0.722.0> started TCP listener on [::]:5672
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0> Server startup complete; 8 plugins started.
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_prometheus
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_federation
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_auth_backend_oauth2
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_web_stomp
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_stomp
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_management
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_management_agent
2024-08-07 09:33:55 2024-08-07 07:33:55.839024+00:00 [info] <0.555.0>  * rabbitmq_web_dispatch
2024-08-07 09:33:55  completed with 8 plugins.
2024-08-07 09:33:56 2024-08-07 07:33:56.001430+00:00 [info] <0.9.0> Time to start RabbitMQ: 20185 ms
2024-08-07 09:34:21 2024-08-07 07:34:21.029835+00:00 [warning] <0.740.0> HTTP access denied: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:34:21 2024-08-07 07:34:21.051960+00:00 [warning] <0.742.0> HTTP access denied: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:34:21 2024-08-07 07:34:21.064897+00:00 [warning] <0.743.0> HTTP access denied: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:37:31 2024-08-07 07:37:31.962904+00:00 [info] <0.764.0> accepting AMQP connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672)
2024-08-07 09:37:32 2024-08-07 07:37:32.269797+00:00 [info] <0.764.0> connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672): user '' authenticated and granted access to vhost 'realm1'
2024-08-07 09:37:32 2024-08-07 07:37:32.305577+00:00 [info] <0.764.0> closing AMQP connection <0.764.0> (192.168.128.2:37632 -> 192.168.128.4:5672, vhost: 'realm1', user: '')
2024-08-07 09:37:47 2024-08-07 07:37:47.282890+00:00 [info] <0.793.0> accepting AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672)
2024-08-07 09:37:47 2024-08-07 07:37:47.290199+00:00 [error] <0.793.0> Error on AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672, state: starting):
2024-08-07 09:37:47 2024-08-07 07:37:47.290199+00:00 [error] <0.793.0> PLAIN login refused: Authentication using an OAuth 2/JWT token failed: {error,no_key}
2024-08-07 09:37:47 2024-08-07 07:37:47.290671+00:00 [info] <0.793.0> closing AMQP connection <0.793.0> (192.168.128.2:33176 -> 192.168.128.4:5672)

Expected behavior

I excepted oauth2 to accept every connection in attached examples.

Additional context

Content of jwks_url endpoint:

{
    "keys": [
        {
            "kty": "RSA",
            "use": "sig",
            "kid": "id1",
            "e": "AQAB",
            "n": "vtS-Evrj0ZDvk2Fa3N-KWOGyf9JxJLX2IcuzfJSTElLacA-ZUuts6isrXTnXqymwQ83cFn8J3XbPbm5Fz8_uT15iyx08v2_KxiVZDk0zV8uIEGS9LyR3_MsMkqCyvV15xlMdR2LIvjChuJ5ATqMggVoKlH1g8cl__RNHjEjshl1Lfsi537VN72n9jaox06jRSWAvP1yI3WCWsMM1keW9Lwi3IRaPMAprjGlLIpl0IAgzfcd0vmWrygiN-Caj5YKdQYqkgqnWx4Ziz4mFvQYM2nY0zHTLsdmrOdOiUUWlLq4wit97U35CpZgAatBBhmr9f3IMVYE3BJU4QWIG5Hgqs6AykTLdUeBHPECVhXw53QSwMLTwozOq5LI6SkS1zNcEnmd3L3KioKwk_24cDTUmeq-swtxUvNsT7mr59zhODSPTn5jbYav9o-_DY5V9n4xbb3Ko0ni16XBLTVujlv-pVlFc_7YDKN82JXXpCbKtaawcKXHulhfv5FcB5v9OsZHhRQgW4OrmJQNzaXZe_F_QG747mhGO-5oWxO6cpXlClFNq3Yw3AkwlQEUVM2gzhNzs_aP__54QPlWqHDkMPsjS1T4ubuqaHvp5QLQxRESCOISe079VM9Xzq1iMDTz4P14wJa3lxNfZThxkOMU2U0Xvy5t4iaEoKYDUGNoJhyqPS1E",
            "alg": "RS256"
        }
    ]
}

[lukebakken]

Please take the time to reproduce this issue using the latest version of RabbitMQ (3.13.6). We do not provide free community support for any version other than the latest.

https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md#who-is-eligible-for-community-support

[michaelklishin]

In fact, we do not provide any OAuth 2 troubleshooting support at all to non-paying users who are not regular contributors.

But there are three extensive documentation guides:

• OAuth 2
• OAuth 2 examples that cover five IDPs and explain why one more is unsupported
• OAuth 2 troubleshooting

Plus enabling debug logging will reveal a lot of decisions made by the authN and authZ backends.

[michaelklishin]

There is only one place in the code where this specific error is returned: when no key for the resource server exists and no default key is configured.

Why that might be, you will have to narrow down on your own using the doc guides above (and source code if necessary).