Refactor daemon loop

[AndyRae]

Is this the right issue type?

• Yes, I'm planning work for this project team.

Summary

The main daemon while loop could do with moving out to testable functions and cleaning up.

Acceptance Criteria

• Code is clean, modular, and testable
• Tests are written for the logic
• Graceful degradation for retries

Tasks

• Extract code to functions
• Write unit tests

Confirm creation

• This issue is ready

[elementechemlyn]

Should the main loop catch network exceptions? A temporary network problem will cause the loop to end. This possibly isn't a problem if/when the container is set to restart on failure but should the code be as robust as we can make it without relying on that?

[AndyRae]

Should the main loop catch network exceptions? A temporary network problem will cause the loop to end. This possibly isn't a problem if/when the container is set to restart on failure but should the code be as robust as we can make it without relying on that?

Yes is my view. We currently rely on container restarts to manage the service, it feels completely unnecessary and fragile, and just adds complexity to observability / debugging.

The best outcome is graceful degradation - catch the exception, log it, and wait twice as long each time to retry (with a maximum time of 1 minute (which can be over-ride in config..)).
If a request succeeds - then the retry is reset.

I had something like this in my mind

def fetch_task(client: TaskApiClient, polling_endpoint: str) -> response:
    attempt = 0
    while True:
        try:
            response = client.get(endpoint=polling_endpoint)
            response.raise_for_status()
            return response
        except requests.RequestException as e:
            wait_time = min(2 ** attempt, MAX_RETRY_BACKOFF)
            logger.warning("Network error fetching task: %s. Retrying in %s ...", e, wait_time)
            time.sleep(wait_time)
            attempt += 1