High performance consumer examples?

[Matt-Esch]

Forgive me if this has been asked before, I've only recently come to learn about seastar and some of the systems that have been built on it such as redpanda and scylladb. I would really like to experiment with building a kafka consumer based on seastar. Are there any examples of this? I think redpanda for performance is an obvious choice but unless there is a client with the same order of magnitude for throughput it does somewhat detract from the benefits when looking at the overall performance of systems built on top of redpanda. Is there any plan to maintain a high performance client?

[emaxerrno]

hi @Matt-Esch atm we have no plans of building a client. what you point out is true: a better client will give better performance.

for better or worse, librdkafka has been good enough for people. But I hear you and i think it would be useful indeed to build a high perf client.

what people have in practice/prod deployments is often is tens, hundreds and many times thousands of clients. So in practice this has not been an issue for people.

I am agreeing it can definitely improve, but we won't have a chance to work on it this year. potentially in the future tho.