Added ability to skip authorization for describe requests #56
Conversation
…thorizer.skip.auth.for.describe.requests` property.
|
We use opa-authorizer in a production environment where OPA itself is hosted on separate servers. A request to OPA, including all network overhead, can take approximately 1-5 ms. In Kafka, we have around 3k topics and roughly 3-4k consumer instances, primarily using librdkafka. Some of the consumers utilize regular expression subscriptions, for which librdkafka retrieves metadata for all existing topics and periodically updates it. All of this generates a significant stream of "describe" requests to OPA, accounting for more than 99% of all requests, and consumes a substantial number of network threads in Kafka. Yes, we are aware that this is not an ideal pattern for Kafka, but unfortunately, it is a necessity for us. |
|
It seems to me like this can be pretty dangerous to just go and add flags to selectively disable some authorization checks. Why are those requests not cached? |
|
Yes, we understand that this can be potentially dangerous in some cases... Caches are being used, and we currently have a cache expiration set to 20 minutes, which results in a significant number of requests hitting the cache. However, there is still a substantial volume of requests when the cache is invalidated. A good solution would be to invalidate the cache when rules are changed, as described in #6 |
|
That seems a bit weird. I'm not really sure why would you see 99% of requests for Describe if the cache is capturing them. Kafka normally authorizes every fetch or produce. So they should normally happen at least with the same frequency. So how come you have 99% of describe requests? |
|
Here's what we see in our monitoring Every client generates "describe" requests based on the number of topics every 20 minutes. |
|
I share @scholzj concerns around having this option in config. Even if we'd add it though, we'd need another option for the next exception someone wants, and another one... and so on. While I've not looked into the details of #42 (i.e. I don't know if it's feasible / possible) that seems like a more generally useful pattern for this, IMHO. Systems that chain authorization decisions are quite flexible, and not too uncommon (see e.g. Kubernetes). While Kafka doesn't natively provide this, or at least it didn't last time I checked, I think this would be possible to do in a plugin — if not this one then perhaps a standalone plugin that did nothing but wire up other authorizers in the order defined by the plugin configuration. I'm afraid I don't have the bandwidth to work on this myself, but if someone would want to pick that up, I'd be happy to help review it. |
|
Alright, in that case, I'll close this PR. We'll address this issue in a different way. Thank you. |
No description provided.