Test suite clean up

[JDBetteridge]

Description

This PR started as an experiment to "cheaply" speed up the test suite by calling mpiexec wrapping pytest, rather than forking a subprocess which calls mpiexec (which is also problematic for other reasons).

This PR now carries around multiple test suite fixes that should be merged back to master and includes fixes including:

• Adding comm arguments to function calls that need them.
• Freeing comms that are created.
• Disabling a test that pollutes the tape.
• "Fixing" ensemble parallel tests by using the simple partitioner (just in tests, Ensemble needs a proper fix!)
• This work had to be rebased on JDBetteridge/update caching #3730 and uses PyOP2 #724 and FInAT #134 due to the deadlocks that they call.

We need to consider what aspects of this experiment we want to incorporate back into master.

Some timings for the actual speed-up (the original intention):

Results

(Real only)

Master

This week's scheduled execution:

Total (inc install): 50m 45s

This branch

With fixed caches, mpispawn, fixed FInAT hashes and pytest-split based on a timed execution.
NB: We tweak vertexonly/test_poisson_inverse_conductivity.py to only do 3 iterations (see diff)

Serial: 17m51s
2: 2m59s
3: 6m43s
4: 45s
6: 19s
7: 48s
8: 12s
Total (inc install): 46m 6s

Important, this branch only runs a maximum of 12 ranks/threads!

[connorjward]

This is cool, but isn't it a bad idea to effectively remove test coverage? If CI doesn't run all the tests no one will.

I can see this being useful in the context of a bigger change where we run the test suite with a number of Firedrake configurations and only one of them would run these slow tests.

[github-actions]

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake complex	8067 ran	6423 passed	1644 skipped	0 failed

[github-actions]

	Tests	Passed ✅	Skipped ⏭️	Failed ❌
Firedrake real	8042 ran	7224 passed	818 skipped	0 failed

[connorjward]

Generally very happy with this.

[connorjward]

Before we merge this we should probably put this into the firedrakeproject organisation, or pin to a version or something?