Nats: Make MaxAckPending and MaxAckWaitTime configurable

[kobergj]

When putting heavy load to nats receivers the nats service will possibly redeliver events and therefore duplicate them. This is because we do not configure it correctly.

When pulling events from the queue the expected behaviour is the following:
The event is pulled from the queue and delivered into a channel. When consumed from this channel the event will be acknowledged on the nats server. On high load this can cause problems. Sine MaxAckWaitTime is 30s by default, the nats server will redeliver the event after 30s. Leading to two identical events waiting to be picked up from the channel. Since MaxAckPending is set to 1000 by default, this can lead to up to 1000 identical events waiting for being processed.

We already introduced worker groups to be able to handle multiple events at the same time. We now need to make MaxAckWaitTime and MaxAckPending configurable, so they can configured individually.

Acceptance Criteria:

• update nats-js go-micro package to expose MaxAckWaitTime and MaxAckPending
• introduce envvars to make them configurable
• set sane defaults

[wkloucek]

should we already add those settings to our oCIS chart with NATS deployment example? https://github.com/owncloud/ocis-charts/blob/main/deployments/ocis-nats/streams/ocis.yaml exposes all the stream settings already.

Our Kubernetes deployments already could benefit from this a lot!?

[kobergj]

Our Kubernetes deployments already could benefit from this a lot!?

Yes it should. If possible please set them.

[wkloucek]

@kobergj do you have proposals for MaxAckWaitTime and MaxAckPending that we could set on all / a subset of streams?

Just to interweave our ticket clusters a little bit:

#8949 would maybe take out some pressure on high message count situations!? It also would allow more fine granular redeliver settings. Eg. the postprocessing messages could need a very high MaxAckWaitTime if we had a long running postprocessing step while SSE related events need to be dropped anyways if they are not delivered within < 10 seconds or so!?

[micbar]

g. the postprocessing messages could need a very high MaxAckWaitTime if we had a long running postprocessing step while SSE related events need to be dropped anyways if they are not delivered within < 10 seconds or so!?

That sounds reasonable. The requirements for the different use cases could be fulfilled better in separate streams.