Investigate 'adaptive' mode for retries and improve concurrency of message sends from multiple services

[terrazoon]

Right now when we do large sends we are hitting some ThrottlingExceptions with a "max retries exceeded: 4" message. This is not our celery retries, this is built into AWS. This occurs both for publishing messages and for calling FilterLogEvents when we are getting the delivery receipts.

Apparently there is a new, and currently experimental, 'adaptive' mode for AWS retries which takes into account service limits and hopefully reduces the number of AWS retries:

https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html

Investigate switching over to this adaptive mode when it becomes less experimental.

Some additional guidance is also available here:

• https://aws.amazon.com/blogs/messaging-and-targeting/how-to-utilise-amazon-pinpoint-to-retry-unsuccessful-sms-delivery/
• https://aws.amazon.com/blogs/messaging-and-targeting/queueing-amazon-pinpoint-api-calls-to-distribute-sms-spikes/

Ultimately, we also want to make sure that a large batch from one partner doesn't become a blocker for all other work to go through the system from other partners.

[terrazoon]

So adaptive mode is beta-ish, and we don't really have problems with retries themselves as we currently have them. The problem was the 'incomplete jobs' task was flagging jobs still running after 30 minutes as incomplete and trying to reprocess, which we have already fixed.

Regarding improving concurrency we've done all that and had a successful load test with 5 jobs running 2000 messages each simultaneously. So closing as done.