Degraded performance for proxying requests responding with Server Sent Events

[noyoshi]

Report

When the connection that the HTTP add on is proxying is handling server sent events, we get weird behavior where the events are all buffered and returned in several large chunks, rather than a continuous stream.

Expected Behavior

The events come back just as they would without the proxy sitting between the client and the server - as a continuous stream.

Actual Behavior

The events come back in several large chunks, all at once, rather than as a stream.

Steps to Reproduce the Problem

We are doing the following:

The server is a model running with this: https://github.com/predibase/lorax

The server has an endpoint called /generate_stream which returns a stream of events using SSE

If you put the keda proxy in front of a pod running a lorax model, and try calling the /generate_stream endpoint, you get the described behavior.

Logs from KEDA HTTP operator

example

HTTP Add-on Version

0.5.0

Kubernetes Version

1.28

Platform

Other

Anything else?

No response

[JorTurFer]

Hello,
Could you confirm that you are using the version 0.5.0? I'm checking the diff between 0.5.0 and 0.7.0 and there are several improvements related with the routing and the streaming:

• HTTPSO-based Routing Table #669
• feat: flush writes to http response on every chunk #743 (I think that this will be the fix for you)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.