Run tests in puppeteer

[samreid]

As suggested in phetsims/binder#8 (comment)

It crossed my mind that parallel puppeteer may be significantly better than sequential aqua--maybe at some point we should redo aqua to use puppeteer?

My first motivation is to make tests run in parallel. Puppeteer may be able to do that with client code that parallelizes, or we may benefit from something like: https://github.com/thomasdondorf/puppeteer-cluster

The end goal is to have something that can run easily/automatically and run a large amount of tests quickly, with a uniform text report. Cross platform, something we may switch Bayes CT over to at some point.

[samreid]

I emailed @kathy-phet and @ariel-phet

Kathy & Ariel,

May I please have a little time to investigate a strategy for speeding up our tests by parallelizing? Context is #52

Best Regards,
Sam

[ariel-phet]

@samreid what is your estimate for an initial investigation?

[samreid]

I am guessing around 2 hours to set up initial harness to get unit tests running in puppeteer using async/await. I'll be learning how to set up Promises and run them in parallel and would benefit from a mentor (<1 hour), if someone has time to help on that step.

If that step goes well, the next step would be to add fuzz tests.

[ariel-phet]

@samreid take 4 hours to begin and lets see where you get. Automated testing is certainly quite beneficial to the project, so worthy of some investment in improving efficiency.

Check with @chrisklus or @jbphet regarding Promises.

[samreid]

Notes from discussion with @jonathanolson:

bayes CT: all tests take around 4 hours.
it runs 9 at a time, so takes around 4/9 hour = 27 minutes to run all tests for one snapshot.

We considered send set of shas to bayes, then it can run tests on those.

Focus on problems that could develop PhET developer workflow. Bayes CT can catch lower-frequency issues (like building scenery).
(0) grunt lint-everything
(1) We should create intelligent subsets of tests that we can run anyways.
- fuzzing normal brand
- fuzzing phet-io brand
- fuzzing accessibility with fuzzBoard
- phet-io wrappers
(2) Make it easy to run them locally and in parallel
(3) Fuzz testing time should be from when sim starts, not from when loading begins
(4) This should run from command line, not by opening a browser, so we can do everything from one starting point. No need to rely on client-server.
(5) Focus on cross-platform, so this code can run on every developer machine and also on Bayes

Major disadvantage of this approach, it only tests chromium. Does not test mobile safari, ie11, firefox, etc. Or is there a way to reuse the code above, but make it work with multiple browsers?

But puppeteer could still be good to get started for local testing.

It would be good if it can run the most important tests first. Focus on finding an issue fast. Interleave the tests, in case there is a phet-io problem--don't save all the phet-io tests until the end.

[samreid]

A heads up to @zepumph that some SimTests are moving to aqua/js/local/test.js

[samreid]

After the preceding commits we have:

1. A pruned list of unit tests (omit ones that take too long)
2. One randomly selected fuzz test
3. A script that subdivides the unit tests across multiple processes
4. Timing so that unit testing and one fuzz complete in about the same time as lint-everything

I've set this up to automatically run on every commit. Putting issue on hold while I use this in practice for a while.

[zepumph]

Could you please provide basic steps to run this. I tried the following and nothing worked:

1. cd aqua; ./scripts/auto-tests-local.sh --> Error: Cannot find module 'puppeteer'
2. cd aqua/scripts; ./auto-tests-local.sh --> Error: Cannot find module 'C:\Users\Michael\PhET\git\aqua\scripts\js\local\test.js'
3. cd friction; ../aqua/scripts/auto-tests-local.sh --> Error: Cannot find module 'C:\Users\Michael\PhET\git\friction\js\local\test.js'`
4. cd friction; grunt tests
5. cd aqua; grunt tests

[samreid]

In the preceding commit, I added puppeteer to aqua package.json, it will need npm install. Then your step (1) is worth a try, but I haven't tested it on windows at all.

To run all unit tests, you can use:

node js/local/test.js 123 1 0 UNIT

To run a fuzz test (randomly picks one sim at the moment), you can use:

node js/local/test.js 123 1 0 FUZZ

Sorry for the wonky syntax--working on something amenable to parallelization.

[samreid]

Not planning more development on this at the moment, but we should revisit it when we spend more time on automated testing. Self-unassigning for now.

[samreid]

I removed the auto-test-local script since we have been focusing on precommit hooks.