Add ADR 005: Publishing API debouncing #230
Conversation
| Upon receiving a message, a worker would: | ||
| - use the [`redlock-rb` gem][redlock-gem] to acquire a lock of `lock:<content_id>` with a sensible | ||
| timeout | ||
| - if the lock currently exists, [wait for it to be available again](#addendum-how-to-wait-for-lock) | ||
| - check `latest:<content_id>` to see if a newer message has recently been successfully processed | ||
| - if so, do nothing and go straight to releasing the lock | ||
| - process the message (update or delete the document in Discovery Engine) | ||
| - if processing fails, handle as usual (requeue or discard) | ||
| - otherwise use `SET latest:<content_id> <version>` to update the latest processed version | ||
| - release the lock |
There was a problem hiding this comment.
Is there still a scenario here where we could exceed Discovery Engine's rate
limit if the messages come in quickly in the correct order? Say:
| Time(ms) | Payload Version | Content ID |
|---|---|---|
| 0 | 0 | cafebabe |
| 10 | 1 | cafebabe |
| 20 | 2 | cafebabe |
| 30 | 3 | cafebabe |
| 40 | 4 | cafebabe |
| 50 | 5 | cafebabe |
If each of thos messages gets processed quickly enough, we'll hit the rate limit.
Do we want to put some kind of deliberate delay in to handle that case?
For example, if we made the locks timeout after 1 second, and then just didn't
release them explicitly, that would throttle messages to one message per
second.
If we do release the locks explicitly, then this won't necessarily slow down
the processing of sequential messages by much.
There was a problem hiding this comment.
Good shout - with the caveat that during peak times latency for an upsert API call to Vertex is about 1s anyway, and only the parallel requests are a problem. But an enforced delay of sorts would probably still help us during off-peak times when it's 0.1s.
No description provided.