Test results show wrong test class

[stephan-herrmann]

I was searching for recent test results for this test:
org.eclipse.jdt.core.tests.compiler/org.eclipse.jdt.core.tests.compiler.regression.ScannerTest.test034()
But in none of the recent test results like I20240519-1800 I could find any trace of the test. In fact searching for .ScannerTest shows exactly 5 matches while 86 matches should be expected.

By contrast, github-triggered builds like https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Github/job/master/222/testReport/org.eclipse.jdt.core.tests.compiler.regression/ScannerTest/ show the expected number of tests (here multiplied with 5 for 5 compliance levels).

Going back all the way to 4.29 it shows 18 tests, still far from complete.

[stephan-herrmann]

@iloveeclipse has an initial result in this comment

[stephan-herrmann]

If this is broken at all, it probably broken at the test report or creation level (org.eclipse.jdt.core.tests.junit.extension.TestCase.buildTestsList(Class, int, long) ?), I see that there are tests from ScannerTest like testBug531716_001_since_13 but they are marked as org.eclipse.jdt.core.tests.compiler.regression.TestAll.

OMG, you're right, there are 196 results for "test034", only 2 of which are assigned to a specific test class.

[stephan-herrmann]

At the end of org.eclipse.jdt.core.tests.compiler.regression.TestAll.suite() the structure looks like this:

• all = TestSuite ("org.eclipse.jdt.core.tests.compiler.regression.TestAll")
  • fTests[3] = RegressionTestSetup
    • nested fTests 158 elements, e.g.:
      • fTests[13] = TestSuite("org.eclipse.jdt.core.tests.compiler.regression.ScannerTest")
        • fTests has 86 elements as expected.

This runtime structure looks OK and I'm not aware of anything that might have changed here in years.

How does the I-build create & collect test results?

[jukzi]

The file is not incomplete.
jenkins reads the result from an .xml which is generated when the tests run
https://ci.eclipse.org/releng/job/AutomatedTests/job/ep432I-unit-cen64-gtk3-java17/lastSuccessfulBuild/artifact/workarea/I20240520-1800/eclipse-testing/results/xml/org.eclipse.jdt.core.tests.compiler_ep432I-unit-cen64-gtk3-java17_linux.gtk.x86_64_17.xml

<testcase classname="org.eclipse.jdt.core.tests.compiler.parser.TestAll" name="test034 - 16" time="0.0" />

I never figured out how exactly the values inside the .xml are generated, but both Testnames and time are reproducible wrong. The testname probably is wrongly setup in the junit test/suite itself.