dekaf: Add metrics to track fetch request counts by topic and partition #1730
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
In order to debug what's happening with infrequent consumer lags, we want to keep track of how many read requests Dekaf is getting, broken down by collection and partition. This tracks that, as well as breaking down by read request type:
state => read_pending
means the read request came in after a previous request had already fetched the same offset for the topic and partition, and that request is pending.state => collection_not_found
andstate => partition_not_found
are self-evident, though I believe a normal Kafka consumer would have figured out that these topics don't exist during the metadata discovery phasestate => new_data_preview_read
indicates a fetch request that we flagged as being for a data preview. These requests have special handling and poison the entire connection to only be usable for other data preview readsstate => new_regular_read
indicates a fetch request for a new topic/partition/offset combination.Ex:
This change is