Pipeline runner improvements

[ljgray]

This is a pretty involved PR, so it may take some time to review.

Features

This PR addresses most of the issues in #181 and #152. In particular:

• The pipeline runner uses a priority system to choose the next task to run, based on
  • Number of times a task can run and its net consumption per iteration
  • Base priority as set by the user
  • Pipeline config order
• A task which is a pure producer (i.e. has no input keys) will never be available, meaning that these will only run when nothing else is able to do anything
• A task will not be able to run next if setup did not run successfully
• Separate the selection of task to run next from the main run loop, which makes it easier to implement different task selection methods
• Include a legacy option which will mimic the current pipeline behaviour
• Add two extra task config features:
  • limit_outputs: restrict the number of times a task can produce output in the next state. Once this limit is reached, the task will immediately advance its state.
  • base_priority: priority modifier which is added to the calculated priority. Can be negative.

Two other features are added which are helpful for memory analysis:

1. A function to find the total memory footprint of an object, including any objects it references.
2. A monitoring function in PSUtilProfiler which checks memory usage every 0.5 seconds in an attempt to log peak memory used by a task

Refactoring

• Additional refactoring of parts of pipeline.Manager class related to initial task loading and pipeline creation. This does not change any behaviour, but slightly improves the distribution of responsibility between methods
• Update to use f-strings

Remaining:

• Input broadcasting. This is tricky as the task would likely have to know how many total inputs to expect for each queue (i.e. how many total combinations of inputs), so it would have to wait for all inputs to accumulate instead of clearing them as soon as possible

Note: this re-implements the multi-output functionality of #263, but that has some extra tests

[ljgray]

I think this is ready for a review and some thoughts on the priority implementation. I've been testing the daily pipeline config with good results, but I'm still trying to think of edge cases where this might fail/behave worse than the old implementation. Input is welcome.

The memory tools seem to be working as expected.

[ketiltrout]

I don't see any real show-stoppers here. I think the new priority system is well designed: it's both simple (to explain/implement) but flexible enough to provide users the necessary tools to adjust things when necessary.

Some extra documentation on how it works may help.

You're right that there may be some edge cases that come up in production, but I don't see anything so obvious just looking at the PR.

The legacy ordering at least means the previous behaviour can be returned to until things are fixed, if everything goes pear-shaped. And I don't think we're locking ourselves into anything that's going to get us into trouble and we couldn't fix if needed.

I have left some comments which I think are mostly minor in scale.

[ketiltrout]

This looks reasonable. It's difficult, with static analysis, to anticipate what sort of corner cases (if any) we might get ourselves into with this change, but I think this is implemented well enough that's it's a good place to start.

I found a couple of typos: