Self-test script rewrite

[Luc45]

Background & Motivation

The motivation for this PR came after I wasted a couple of hours while doing the sync with 9.6.0-beta.1. One of the self-tests didn't start, and I only found out about this 2h later. Debugging why it failed was much harder than it should be, because the self-test script would just dispatch 20 tests in parallel, keep them all alive with --wait and --json, where each would poll their own test status individually.

This PR greatly simplifies this setup, we dispatch the tests async and collect their test run IDs, then we poll their status ourselves with one request to the newly introduced get-multiple endpoint, which takes a comma-separated list of test run IDs.

This made it possible to also improve the output, and added extensive logging to all requests to last-self-test.log, that gets reset with each self test.

What Changed: Before vs. After

Before

• Multiple Parallel Processes: We would start many processes, each:

  1. Dispatching its QIT test.
  2. Polling individually for status until completion.
  3. Reporting success or failure.

• Drawbacks:

  • Hard to coordinate and debug: each process produced its own logs and status updates.
  • Risk of wasted time: if one test fails to run, it would be very hard to know why.
  • More overhead managing N distinct polling loops concurrently.

After

1. Asynchronous Dispatch

   • All tests are dispatched quickly (no blocking).
   • We gather every test_run_id upfront.

2. Unified Polling Loop

   • Rather than multiple pollers, a single loop calls qit get-multiple for all test_run_ids in one go.
   • This repeats every X seconds until each test reports a final status.

3. Snapshot Verification

   • As soon as a test finishes, its JSON result is stored and checked against the PHPUnit snapshot.
   • Any mismatch shows up right away (or updates, if in update mode).

Implementation Details

• QitRunner.php:
  • Dispatches all tests asynchronously and records their IDs.
  • Repeatedly calls qit get-multiple until all have a final result.
• QITLiveOutput.php:
  • Adjusted to show a unified table of test statuses during each poll iteration.
• PhpUnitRunner.php:
  • Receives completed test data and checks it against snapshots.
• Documentation:
  • Updated to explain the single polling flow (instead of many parallel pollers).

Testing instructions

• Run the self-tests as usual, see how it goes

[zhongruige]

I'll dig into this a bit more tomorrow and run the tests but so far loving these changes!

[Luc45]

No worries! Ignore if you see some activity in GH, I'm just taking the opportunity to run some self-tests and scheduling the PC to auto-shutdown in a couple of hours.

[zhongruige]

Went through the self tests (php QITSelfTests.php) and it batched and incremented as it was running:

However, I did hit a timeout and it eventually exited:

Wanted to just double check on the above behavior.

[Luc45]

Wanted to just double check on the above behavior.

Nice catch, I've limitted the get-multiple endpoint to 20 tests per request on the backend, and updated the logic here to:

• Stop requesting tests that are complete
• Batch polling in chunks of 20 tests, instead of trying to fetch them all at once

I was able to run a mass test with all tests at once, so it seems to have solved it.

[zhongruige]

Re-ran this again and just finished (ran this twice just to see) and got this result both times:

Would we expect these all to pass or is this expected as we have some no-op tests we might expect to fail?

[Luc45]

It should pass on the 9.6.0-beta.1 sync branch

[zhongruige]

Weird I got the same results on that branch @Luc45:

All snapshots have been verified.

Some snapshots still failed. Final outcome: ❌

For more details, see last-self-test.log.
➜  managed_tests git:(24-12/sync-960) 

Not sure, could it be due to the PHP version I'm using?

[zhongruige]

Perfect that fixed it, thanks @Luc45!