Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script for computing plain pass/total scores over time #156

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

foolip
Copy link
Member

@foolip foolip commented Feb 21, 2023

No description provided.

@foolip
Copy link
Member Author

foolip commented Feb 21, 2023

This is what I was playing with when I noticed we've passed 50,000 tests, and took a look at the growth over time:
https://mastodon.nu/@foolip/109879573138978166
https://mastodon.nu/@foolip/109897803628119305

@foolip
Copy link
Member Author

foolip commented Feb 21, 2023

A few observations after tinkering with this:

  • The big picture is visually very similar regardless of scoring method. To me that's an argument for the simplest possible "binary" approach, which takes out the nuances of harness status and subtests.
  • It seems important to fix the "there are periods of time, mostly mid-late 2018, where we ran both Safari 11.1 and 12.1, and the results are massively different" TODO and that might take out some of the noise. It would be better to have a clean series of runs per browser, skipping missing results per browser, at least in graphs like these that don't "join" results between browsers.
  • It would be good to make use of the manifest as the source of truth for which tests exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant