twister: harness: Fix several issues at Console harness #63902

golowanow · 2023-10-13T06:31:01Z

Twister Console Harness several fixes and small improvements:

Fix unordered pattern matching to treat the ztest as failed when not all of the expected patterns were found in the console output, but the ztest application itself reports 'PROJECT EXECUTION SUCCESSFUL'.
Extends twister: harness: Fix Console pattern matching for ztest #63621
Unify debug logging on pattern match for ordered and unordered patterns.
Check for non-empty pattern list configured for Console: If Console Harness has no regexp pattens configured, then ConfigurationError is raised, handled, and the failure of the test case correctly accounted in the summary results.
Implement a workaround for Console harness to compose TestCase identifier correctly when Ztest suite with a single TestCase is executed (generally, Ztest suite should use Ztest harness).
Without the fix each TestCase result at Console is duplicated (and written into twister.json) with short 'identifier' set to TestSuite id only, without TestCase suffix added. The resulting entry with the full TestCase id is also stored, but its values are set empty to defaults.

scripts/pylib/twister/twisterlib/harness.py

gchwier

Looks good, one minor comment

scripts/pylib/twister/twisterlib/harness.py

golowanow · 2023-10-23T08:02:16Z

@nashif, @galak, @gopiotr, @hakehuang, @PerMac - up to your attention

nashif · 2023-10-23T12:43:15Z

Implement a workaround for Console harness to compose TestCase identifier correctly when only one TestCase is recognized as executed.
Without the fix each TestCase result at Console was duplicated (and written into twister.json) with short 'identifier' set to TestSuite id only, without TestCase suffix added.
The resulting entry with the full TestCase id was also stored, but without its values set being empty with defaults.

can you point to an example where this happens?

golowanow · 2023-10-23T13:15:49Z

Implement a workaround for Console harness to compose TestCase identifier correctly when only one TestCase is recognized as executed.
Without the fix each TestCase result at Console was duplicated (and written into twister.json) with short 'identifier' set to TestSuite id only, without TestCase suffix added.
The resulting entry with the full TestCase id was also stored, but without its values set being empty with defaults.

can you point to an example where this happens?

current main branch (92045a02) with ./scripts/twister -vv -j 1 -p qemu_x86 -T tests/subsys/debug/coredump_backends

{
    "environment":{
        "os":"posix",
        "zephyr_version":"zephyr-v3.5.0-324-g92045a021ed3",
        "toolchain":"zephyr",
        "commit_date":"2023-10-23T14:32:39+02:00",
        "run_date":"2023-10-23T13:07:05+00:00"
    },
    "testsuites":[
        {
            "name":"tests/subsys/debug/coredump_backends/debug.coredump.backends.logging",
            "arch":"x86",
            "platform":"qemu_x86",
            "path":"tests/subsys/debug/coredump_backends",
            "run_id":"021ad5ff8f59fbf1f913df452da28c22",
            "runnable":true,
            "retries":0,
            "status":"passed",
            "execution_time":"2.15",
            "testcases":[
                {
                    "identifier":"debug.coredump.backends.logging.coredump_backend",
                    "execution_time":"0.00",
                    "status":null
                },
                {
                    "identifier":"debug.coredump.backends.logging",
                    "execution_time":"0.00",
                    "status":"passed"
                }
            ]
        },

with this fix:

{
    "environment":{
        "os":"posix",
        "zephyr_version":"v3.5.0-rc2-203-g9edae344d00d",
        "toolchain":"zephyr",
        "commit_date":"2023-10-17T08:17:24+02:00",
        "run_date":"2023-10-23T13:12:07+00:00"
    },
    "testsuites":[
        {
            "name":"tests/subsys/debug/coredump_backends/debug.coredump.backends.logging",
            "arch":"x86",
            "platform":"qemu_x86",
            "path":"tests/subsys/debug/coredump_backends",
            "run_id":"533f73cf18c87813a3aaf17ec40faef3",
            "runnable":true,
            "retries":0,
            "status":"passed",
            "execution_time":"2.13",
            "testcases":[
                {
                    "identifier":"debug.coredump.backends.logging.coredump_backend",
                    "execution_time":"2.13",
                    "status":"passed"
                }
            ]
        },

nashif · 2023-10-23T13:36:54Z

current main branch (92045a02) with ./scripts/twister -vv -j 1 -p qemu_x86 -T tests/subsys/debug/coredump_backends

ok. While your workaround might deal with the issue, the problem here is that the test is wrong. It can't be both a ztest and a console test. We either use ztest and rely on passing tests reporting based on ztest output or we rely on the console output, having both is wrong.

nashif · 2023-10-23T13:28:38Z

scripts/pylib/twister/twisterlib/harness.py

@@ -162,6 +163,14 @@ class Console(Harness):

    def configure(self, instance):
        super(Console, self).configure(instance)
+        if self.regex is None or len(self.regex) == 0:
+            self.state = "failed"


if the definition of the testcase is wrong, this should be treated as an error. So, instead of doing this here and calling this a fail, error already during the parsing of the test (in config_parser.py)

i wonder if this can be dealt with on the schema level, i.e.. disallowing empty sequences.

i wonder if this can be dealt with on the schema level, i.e.. disallowing empty sequences.

pykwalify has allowempty but for `map' type only
https://pykwalify.readthedocs.io/en/unstable/validation-rules.html#allowempty

if the definition of the testcase is wrong, this should be treated as an error.

changed to 'error' status for instances

So, instead of doing this here and calling this a fail, error already during the parsing of the test (in config_parser.py)

it seems not easy to avoid these checks here with the current twister harness design: testsute-schema.yaml defines common harness_config for all types of harnesses simultaneously and parser checks for its presence with allowed (not required) properties. Later harness configure() needs to check what it got from the test instance in regard of the specific harness class.

Fix the Twister Console harness unordered pattern matching to treat the ztest as failed when not all of the expected patterns were found in the console output, but the ztest application itself reports 'PROJECT EXECUTION SUCCESSFUL'. Unify debug logging on pattern match for ordered and unordered patterns. Signed-off-by: Dmitrii Golovanov <[email protected]>

If Console Harness 'harness_config' properties have no 'regex' patterns or no correct 'type' set, then ConfigurationError exception is raised, handled, and the test instance error is accounted in the summary results. Signed-off-by: Dmitrii Golovanov <[email protected]>

golowanow · 2023-11-08T12:17:06Z

current main branch (92045a02) with ./scripts/twister -vv -j 1 -p qemu_x86 -T tests/subsys/debug/coredump_backends

ok. While your workaround might deal with the issue, the problem here is that the test is wrong. It can't be both a ztest and a console test. We either use ztest and rely on passing tests reporting based on ztest output or we rely on the console output, having both is wrong.

@nashif , that particular testcase wrong implementation (coredump_backends.logging) is now addressed by #64937

hakehuang · 2023-11-08T15:38:10Z

scripts/pylib/twister/twisterlib/harness.py

+        Console Harness id has only TestSuite id without TestCase name suffix.
+        TODO: Only the first TestCase name is taken if available.
+        '''
+        if self.instance is not None and len(self.instance.testcases) == 1:


in python shall we just use below?

Suggested change

if self.instance is not None and len(self.instance.testcases) == 1:

if self.instance and self.instance.testcases:

this workaround is applicable with a single Ztest testcase only.
I've simplified this if statement; also better description is added for this method as well as for the commit message.

hakehuang · 2023-11-08T15:50:09Z

scripts/pylib/twister/twisterlib/harness.py

+        else:
+            self.state = "failed"
+            tc = self.instance.set_case_status_by_name(
+                self.id,


why not use self.get_testcase_name() for this?

good catch, I've forgot it here.

Implement a workaround for Console harness to compose TestCase identifier correctly when a Ztest suite with a single testcase uses this harness type. Normally, a Ztest suite should use the Ztest Twister harness. Without this workaround each Ztest TestCase result on Console is duplicated (and written into twister.json) with its 'identifier' attribute set to TestSuite id only, no TestCase suffix added; the resulting entry with the full TestCase id is also stored, but its values are empty with defaults. Signed-off-by: Dmitrii Golovanov <[email protected]>

golowanow · 2023-11-14T11:55:45Z

@nashif, @galak, @gchwier, @gopiotr, @hakehuang, @PerMac - up to your attention

golowanow marked this pull request as ready for review October 13, 2023 07:27

golowanow requested a review from nashif as a code owner October 13, 2023 07:27

zephyrbot added the area: Twister Twister label Oct 13, 2023

zephyrbot requested review from galak, gchwier, gopiotr, hakehuang and PerMac October 13, 2023 07:28

zephyrbot assigned nashif Oct 13, 2023

golowanow added the bug The issue is a bug, or the PR is fixing a bug label Oct 13, 2023

gchwier reviewed Oct 13, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/harness.py Outdated Show resolved Hide resolved

scripts/pylib/twister/twisterlib/harness.py Show resolved Hide resolved

golowanow force-pushed the twister_fix_console_patterns_unordered branch from 73984f3 to 3dd7717 Compare October 13, 2023 19:20

golowanow changed the title ~~twister: harness: Fix Console unordered pattern matching for ztest~~ twister: harness: Fix several issues at Console harness Oct 13, 2023

golowanow requested a review from gchwier October 13, 2023 19:29

gchwier previously approved these changes Oct 16, 2023

View reviewed changes

scripts/pylib/twister/twisterlib/harness.py Outdated Show resolved Hide resolved

golowanow dismissed gchwier’s stale review via 9edae34 October 17, 2023 06:18

golowanow force-pushed the twister_fix_console_patterns_unordered branch from 3dd7717 to 9edae34 Compare October 17, 2023 06:18

golowanow requested a review from gchwier October 17, 2023 06:19

gchwier previously approved these changes Oct 17, 2023

View reviewed changes

PerMac previously approved these changes Oct 23, 2023

View reviewed changes

nashif requested changes Oct 23, 2023

View reviewed changes

golowanow dismissed stale reviews from PerMac and gchwier via c670eed October 31, 2023 11:37

golowanow force-pushed the twister_fix_console_patterns_unordered branch from 9edae34 to c670eed Compare October 31, 2023 11:37

golowanow requested review from nashif and gchwier October 31, 2023 12:34

golowanow requested a review from PerMac October 31, 2023 12:34

golowanow force-pushed the twister_fix_console_patterns_unordered branch from c670eed to 37f57b6 Compare October 31, 2023 12:55

golowanow added 2 commits November 2, 2023 12:18

golowanow force-pushed the twister_fix_console_patterns_unordered branch from 37f57b6 to 52c39d6 Compare November 2, 2023 11:18

gchwier previously approved these changes Nov 3, 2023

View reviewed changes

hakehuang reviewed Nov 8, 2023

View reviewed changes

golowanow dismissed gchwier’s stale review via 2ea008e November 8, 2023 19:06

golowanow force-pushed the twister_fix_console_patterns_unordered branch from 52c39d6 to 2ea008e Compare November 8, 2023 19:06

golowanow requested review from hakehuang and gchwier November 8, 2023 19:14

gchwier approved these changes Nov 14, 2023

View reviewed changes

nashif approved these changes Nov 14, 2023

View reviewed changes

nashif merged commit 7d0d3f8 into zephyrproject-rtos:main Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

twister: harness: Fix several issues at Console harness #63902

twister: harness: Fix several issues at Console harness #63902

golowanow commented Oct 13, 2023 •

edited

Loading

gchwier left a comment

golowanow commented Oct 23, 2023

nashif commented Oct 23, 2023

golowanow commented Oct 23, 2023

nashif commented Oct 23, 2023

nashif Oct 23, 2023

nashif Oct 23, 2023

golowanow Oct 31, 2023

golowanow Oct 31, 2023 •

edited

Loading

golowanow commented Nov 8, 2023 •

edited

Loading

hakehuang Nov 8, 2023

golowanow Nov 8, 2023

hakehuang Nov 8, 2023

golowanow Nov 8, 2023

golowanow commented Nov 14, 2023

	if self.instance is not None and len(self.instance.testcases) == 1:
	if self.instance and self.instance.testcases:

twister: harness: Fix several issues at Console harness #63902

twister: harness: Fix several issues at Console harness #63902

Conversation

golowanow commented Oct 13, 2023 • edited Loading

gchwier left a comment

Choose a reason for hiding this comment

golowanow commented Oct 23, 2023

nashif commented Oct 23, 2023

golowanow commented Oct 23, 2023

nashif commented Oct 23, 2023

nashif Oct 23, 2023

Choose a reason for hiding this comment

nashif Oct 23, 2023

Choose a reason for hiding this comment

golowanow Oct 31, 2023

Choose a reason for hiding this comment

golowanow Oct 31, 2023 • edited Loading

Choose a reason for hiding this comment

golowanow commented Nov 8, 2023 • edited Loading

hakehuang Nov 8, 2023

Choose a reason for hiding this comment

golowanow Nov 8, 2023

Choose a reason for hiding this comment

hakehuang Nov 8, 2023

Choose a reason for hiding this comment

golowanow Nov 8, 2023

Choose a reason for hiding this comment

golowanow commented Nov 14, 2023

golowanow commented Oct 13, 2023 •

edited

Loading

golowanow Oct 31, 2023 •

edited

Loading

golowanow commented Nov 8, 2023 •

edited

Loading