redpanda: Client auth failure: user 'kubernetes-controller' wrong password

[chrisseto]

It seems the chart (and/or operator) can get itself into a state where it generates the following log lines:

WARN  2024-11-12 11:32:11,242 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:11,253 [shard 6:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,121 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,127 [shard 6:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,136 [shard 0:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password
WARN  2024-11-12 11:32:13,140 [shard 5:admi] request_auth - request_auth.cc:150 - Client auth failure: user 'kubernetes-controller' wrong password

This has been discovered in a cloud cluster (5.9.8) and has been reported by a self hosted user in the community slack in both 5.9.8 and 5.9.7.

In both cases, SASL was enabled, admin_api_require_auth was NOT enabled, and users were being specified via a secret created out of band of the chart itself rather than the inline sasl.users option.

JIRA Link: K8S-415