WriteBatch Method in BatchOutput Interface Does Not Return Errors Anymore

[mohitmarwal]

Previously, when using the BatchOutput interface in Benthos, the WriteBatch method used to return an error if delivery was not possible. This was crucial for handling error scenarios and responding appropriately, especially in HTTP server contexts where the client expects meaningful error responses.

However, with the latest update, it seems that the WriteBatch method does not return errors anymore. Instead, when an error occurs, the client receives a request timeout response, which is misleading and does not provide useful information about the actual error.

Expected Behavior:

The WriteBatch method in the BatchOutput interface should return an error if delivery is not possible, as it did before. This error should be propagated back to the caller so that appropriate error handling can be performed, such as returning meaningful HTTP responses with error details.

Steps to Reproduce:

Use the BatchOutput interface in Benthos.
Implement a custom output that implements this interface.
Invoke the WriteBatch method with a scenario where delivery is not possible (e.g., invalid message format, connection failure).
Observe that instead of receiving an error, the client gets a request timeout response.

Impact:

This change affects users who rely on the BatchOutput interface for writing batches of messages, especially in scenarios where error handling and response generation are critical, such as HTTP servers.

Proposed Solution:

Restore the previous behavior of the WriteBatch method to return errors when delivery is not possible. This ensures consistency and enables users to handle error scenarios appropriately.

[mohitmarwal]

Earlier for example we were getting message error in the input - httpServer: . But now this error is not returned back to http server instead we get request time out with error printed in console in loop.
func (o *BatchOutput) WriteBatch(ctx context.Context, batch types.MessageBatch) error {
return fmt.Errorf("message error: %w", err)

}

[mihaitodor]

Hey @mohitmarwal, would you mind sharing a sample config and code to reproduce this issue? I'm struggling to piece together a way to reproduce the issue you're seeing.

[FerroEduardo]

A sample config can be found here: #2549 (comment)

[mihaitodor]

OK, I think the explanation provided by Ash here should give some insight.