[Task]: Add tests to Kafka SDF and fix known and discovered issues

[johnjcasey]

What needs to happen?

Kafka IO SDF currently has an issue where it is not restarting properly: #21730

In addition, there was a separate issue where it would not commit back to Kafka correctly.

As such, it appears that Kafka SDF has not been sufficiently tested, and the scope of this task is to test it more robustly, and fix both the linked issue, and any other issues discovered.

Issue Priority

Priority: 1

Issue Component

Component: io-java-kafka

[johnjcasey]

@Abacn this issue can be used to track the kafka testing improvements

[johnjcasey]

#22715 also

[pabloem]

how should we triage this? should we assign it to @johnjcasey or @Abacn ?

[Abacn]

Apparently Kafka performance test is failing again after #24879 in. This is because we removed shuffle=appliance there and then read from shuffle becomes faster. There are some reports that runner v2 is putting more pressure on kafka, and resulting in higher error rate. Now kafka server is overwhelmed with write requests and making write pipeline fail.

This reveals real issue of the code.

[johnjcasey]

Closing as out of date. SDF Kafka has been significantly reworked, and is now in a much better state