Blocking metrics updates on test regressions

[foolip]

As discussed in web-platform-tests/results-analysis#143, any new non-OK harness status (for testharness.js tests) will prevent the Interop 2023 scores from updating until that test is added to the list of ignored tests statuses.

This mechanism was inherited from Interop 2022. During 2022, there were no regressions, so we never noticed this.

Unfortunately, web-platform-tests/wpt#38258 broke the score updating for almost a week before @gsnedders alerted me to the fact in web-platform-tests/results-analysis#143 and I just fixed it.

I don't think we have the right tradeoff here, we shouldn't block Interop 2023 updates because a regression in a single test, and we shouldn't require the interop team to triage and fix such regressions.

I think we should:

• Land the change in Make the same logging change for 2022 script results-analysis#142 and do the same for Interop 2023 scoring
• In the scoring script, log changes to test scores to a text file, one per focus area. That can be used both to identify both improvements to celebrate and regressions to investigate.
• Investigate fixing "Failed to fetch test runs" on prod with many (38) labels wpt.fyi#3149 so that we can query for all harness errors in the latest test results.

[gsnedders]

The thing that's obvious here is how neither web-platform-tests/wpt#38203 nor web-platform-tests/wpt#38258 failed CI on WPT (see web-platform-tests/wpt#10877).

[foolip]

@gsnedders I didn't confirm it, but assumed based on the timing (~4 hours apart) that both PRs passed the CI separately and were then both merged.

[foolip]

This is blocked on getting review of web-platform-tests/results-analysis#150.

[foolip]

Closing this as I won't be driving any changes here. I've closed web-platform-tests/results-analysis#150 but note that web-platform-tests/results-analysis#173 remains open. @jgraham can you file a new issue if you want to move forward with some change?