Prefect tasks not responding to timeout_seconds

[tjgalvin]

This is related to a separate issue described in #187

I have tried to uses prefects timeout_seconds functionality exposed as part of the task decorator. My hope was that if the bane error was tripped, prefect would kill and relaunch. The specifics:

# BANE sometimes gets cauht in some stalled staTE
@task(timeout_seconds=60 * 45, retries=3)
def task_run_bane_and_aegean(
    image: Union[WSCleanCommand, LinmosCommand], aegean_container: Path
) -> AegeanOutputs:

My understanding of this is that after 45 minutes the task will be cancelled, moved to the failed state, and then rescheduled. However it seems like this is not being respected. Will need to mock up something to test with. Potential issues are around:

• dask task runner
• slurm queue