Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation on DLT strategies with batch mode listeners #3554

Open
StephenFlavin opened this issue Oct 10, 2024 · 0 comments
Open

Documentation on DLT strategies with batch mode listeners #3554

StephenFlavin opened this issue Oct 10, 2024 · 0 comments

Comments

@StephenFlavin
Copy link

StephenFlavin commented Oct 10, 2024

Expected Behavior

there is docs

Current Behavior

there is no docs (on this page at least)

Context

I'm investigating DLT and retry flows with spring-kafka however, I stumbled on the difficulties of using it in tandem with a batch listener.
I can see historically things have changed around this area which has left me a little confused.

Originally I had just assumed that underneath the listener, when a batch processing exception is thrown that it would either produce the whole batch to the -retry topic or the -dlt or it would do something like binary search on the batch to find bad messages and pausing the consumer until complete.
I then discovered BatchListenerFailedException where you need to find the bad messages yourself during processing and throw this exception but I'm still not clear on what will happen here without delving into the code, is the bad message published to the dlt? do we just commit the offset up to that message and continually retry? how can this be integrated with @DltHandler?

Ultimately, if the answers to the questions I have is, "it's not supported" then that's fine although a little disappointing, but it would be good to update the documentation found here.

I also think the documentation could be better around the handling of deserialization exceptions vs processing exceptions e.g. if the error happens during processing you need to have a serializer set up in a KafkaTemplate that's able to serialize the message OR with batch I currently don't understand what the ErrorHandlingDeserializer will do with my batch mode consumer.

As for batch consuming as a concept, it's far more efficient in my scenarios where I'm consuming from a topic then performing database operations etc.

This issue describes my pain somewhat but there's not been much activity on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant