ci/transformers: add baseline checks for test cases

[dvrogozh]

Baseline is set in a python script. Script looks into the results and categorizes test cases as:

• New failures - if detected workflow is marked as failed
• New passes - if detected workflow is marked as failed (since baseline requires an update)
• Skipped flakies - if detected workflow is marked as failed (since baseline requires an update)
• Known failures - these are failing tests knowing to fail per baseline, workflow is marked as passed (if nothing from above detected)

Test cases from above categories are printed into summary.

[chuanqi129]

LGTM

[chuanqi129]

BTW, what does the "None" mean here? Do we need to root cause and fix those test cases?

[dvrogozh]

Yes, these are all test cases which fail and which need to be root caused and fixed. We require that no more tests should fail on top of this list - in such a case workload should fail.

None here is in a place for the dictionary to pass additional information about failing tests. In the simplest case, we don't pass anything, i.e. we pass None, since test name in this list by itself signifies that test fails. However, in some cases we need to mark that test is flaky. That's where this placeholder comes handy:

failing_cases = {
    'tests.models.detr.test_image_processing_detr.DetrImageProcessingTest': {
        'test_fast_is_faster_than_slow': { 'flaky': True },
    ...

I use this already for few flaky tests in the middle of the list. See lines 48-57.

Actually we can further expand this idea if needed and pass for example links to the known bugs or PRs associated with the failing case and further print this in the result table.