Single target testing

[ppenzin]

Enable building and testing via a single target (check) - require chakracore
library to be built before the test run.

For #6547

[ppenzin]

The idea is that we can invoke one target, which would be check for debug builds, and default for release builds, this would keep CI build steps for all builds mostly identical.

[ppenzin]

Three Linux builds are working, trying to add OSX. I am going to make sure all basic varieties work, then will try to replicate some of the testing old CI used to do. I am not going to wait for this - will check tomorrow.

With this much variety we are getting close to the number of jobs Azure allows for free - after 10 jobs are launched the rest would have to wait their turn.

[rhuanjl]

The macOS debug build is failing all the tests but rather than saying it's failed it's carrying on failing over and over.

Two problems to solve:

1. Why is it failing everything (some failures look like missing ICU but there are others which that doesn't explain)?
2. Why is it carrying on when it's failed? Is there anything we can change/set to make it stop when it fails?

EDIT: the test failures are this issue: #5876 it's a long standing problem with PAL that surfaces in macOS shared debug builds - the old CI dodged it by using a static build.

[ppenzin]

The test runner tries to run until the end. I will set up a timeout - I was hoping Azure would kill it, but it did not.

EDIT: the test failures are this issue: #5876 it's a long standing problem with PAL that surfaces in macOS shared debug builds - the old CI dodged it by using a static build.

Thank you for finding it! Converting the build to static should be a relatively easy fix.

[rhuanjl]

The test runner tries to run until the end. I will set up a timeout - I was hoping Azure would kill it, but it did not.

EDIT: the test failures are this issue: #5876 it's a long standing problem with PAL that surfaces in macOS shared debug builds - the old CI dodged it by using a static build.

Thank you for finding it! Converting the build to static should be a relatively easy fix.

I wonder if we could/should add a "terminate on failure" flag to runtests.py or even make that the default behaviour with a flag to cancel it if you want to find multiple failures offline. I could have a look at how to do that.

[ppenzin]

I wonder if we could/should add a "terminate on failure" flag to runtests.py or even make that the default behaviour with a flag to cancel it if you want to find multiple failures offline. I could have a look at how to do that.

Maybe - I think it is useful to see all failures, though we can fail on the first one if it is too expensive to keep running. I wondering why the 'nix testing is slower - maybe we can improve the script somehow.

[ppenzin]

This does not seem to help, not sure what I am doing wrong.

[ppenzin]

Turns out I had brackets around an environment variable name :)