CI: Add Coverage Analysis #68624

gbarkadiusz · 2024-02-06T11:58:05Z

Extend codecov workflow to include coverage analysis.
Coverage_analysis.py generates coverage report in both (json and xlsx) formats.
The generated report contains information about coverage per component, covered and uncovered functions.
Report.json

{
    "components": [
        {
            "name": "acpi",
            "sub_components": [ 
                {
                    "name": "basic",
                    "test_suites": [
                        {
                            "name": "tests/lib/acpi/acpi.basic",
                            "path": "tests/lib/acpi",
                            "platforms": [
                                "qemu_x86_64"
                            ],
                            "runnable": true,
                            "status": "sim_only",
                            "test_cases": [
                                {
                                    "name": "acpi.basic.irq_routing_table"
                                },
                                ...
                            ]
                        },
                        ...
                    ]
                },
                ...
            ]
        },
        ...
    ]
}

Moreover, this script parses Testplan.json generated by twister ( twister -E testplan.json ) to collect the data regarding all testsuites.
Report.xlsx

Components	TestSuites	Runnable	Build only	Simulators only	Hardware only	Coverage [%]	Total Functions	Uncovered Functions	Comment
linker	2	2	0	2	0	100.00	2	0
zdsp	3	3	0	3	0	100.00	60	0
message_bus	7	7	0	7	0	99.70	34	0
utilities	62	62	0	0	62	92.03	246	9
printk	3	3	0	3	0	91.76	399	21
kernel	164	150	14	157	7	90.94	826	36
pm	13	12	1	12	1	87.81	80	3
rtio	4	4	0	4	0	87.56	52	0
settings	11	9	2	9	2	86.34	80	0
libraries	191	187	4	191	0	84.76	755	64
lib	1	1	0	0	1	84.76	755	64
storage	7	6	1	6	1	84.70	27	2
input	12	12	0	12	0	82.18	48	3
portability	32	29	3	30	2	81.81	380	15
filesystem	21	20	1	20	1	79.35	367	23
logging	106	106	0	106	0	79.20	248	31
modem	8	6	2	6	2	78.20	128	12
gpio	1	1	0	1	0	77.84	115	28
tracing	2	2	0	2	0	74.82	153	43
gnss	1	1	0	1	0	74.70	31	2
arch	60	49	11	57	3	73.63	268	62
debug	8	8	0	8	0	70.67	64	8
mgmt	22	18	4	18	4	67.52	175	35
canbus	5	5	0	5	0	66.46	225	50
cpp	14	8	6	14	0	60.00	7	4
net	216	206	10	215	1	57.95	5553	1653
llext	4	4	0	4	0	57.85	31	10
shell	19	5	14	19	0	48.21	276	108
dfu	4	4	0	4	0	47.29	17	6
boards	20	2	18	3	17	45.65	622	297
flash	1	0	1	0	1	43.95	45	19
usb	6	6	0	6	0	42.44	498	203
crypto	8	8	0	8	0	40.00	23	7
bluetooth	276	160	116	102	174	39.64	4844	2646
drivers	422	87	335	160	262	34.48	3778	1918
sample	643	114	529	324	319	32.03	18	11
ipc	2	2	0	2	0	25.26	33	26
sensors	8	1	7	8	0	17.81	1936	1377
acpi	1	1	0	1	0	11.57	33	27
buildsystem	14	6	8	14	0	0.00	0	0
application_development	5	3	2	2	3	0.00	0	0
bootloader	6	0	6	0	6	0.00	0	0
boot	3	1	2	1	2	0.00	0	0
kconfig	2	2	0	2	0	0.00	0	0
init	1	0	1	1	0	0.00	0	0	Missed in maintainers.yml file.
benchmark	24	18	6	24	0	0.00	0	0
sip_svc	1	0	1	0	1	0.00	0	0
testing	30	28	2	25	5	0.00	0	0	Missed in maintainers.yml file.
mem_mgmt	2	2	0	2	0	0.00	0	0	Missed in maintainers.yml file.
openthread	3	3	0	3	0	0.00	0	0
modbus	2	0	2	1	1	0.00	0	0
edac	6	0	6	0	6	0.00	0	0
bindesc	7	7	0	7	0	0.00	0	0
emul	1	1	0	1	0	0.00	0	0	Missed in maintainers.yml file.
samples	2	0	2	0	2	0.00	0	0

nashif · 2024-02-06T14:04:06Z

Please fix the CI issues

gbarkadiusz · 2024-02-06T14:44:43Z

Please fix the CI issues

Fixed.

nashif · 2024-02-12T21:02:31Z

@gbarkadiusz why is this still in draft?

nashif

see https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/7891137291/workflow for a workflow file that works

nashif · 2024-02-13T15:12:16Z

.github/workflows/codecov.yaml

+        with:
+          name: Coverage Reports
+          path: |
+            coverage/reports/${{ steps.run_date.outputs.run_date_short }}.json


prefix file name with coverage-report

scripts/ci/coverage/coverage_analysis.py

nashif · 2024-02-13T15:20:34Z

.github/workflows/codecov.yaml

+      - name: Generate Coverage Report
+        if: always()
+        run: |
+          pip install xlsxwriter, ijson, json, yaml


remove the comma, just leave xlsxwrite and ijson.

Why do you need ijson and json and why cant this all be accomplished with json module alone?

not blocking, I would consider using openpyxl for excel in the future, it is much more powerful.

nashif · 2024-02-13T15:35:31Z

.github/workflows/codecov.yaml

+          pip install xlsxwriter, ijson, json, yaml
+          cd ./coverage/reports
+          ../scripts/twister -E testplan.json
+          python3 ../scripts/ci/coverage/coverage_analysis.py -t testplan.json -m ../MAINTAINERS.yml -c merged.json \


this is wrong, use something like that instead:

- cd ./coverage/reports - ../scripts/twister -E testplan.json - python3 ../scripts/ci/coverage/coverage_analysis.py -t testplan.json -m ../MAINTAINERS.yml -c merged.json \ - -o ${{ steps.run_date.outputs.run_date_short }} -f all + ./scripts/twister -E testplan.json + python3 ./scripts/ci/coverage/coverage_analysis.py -t testplan.json -m MAINTAINERS.yml -c coverage/reports/merged.json \ + -o coverage-report-${{ steps.run_date.outputs.run_date_short }} -f all + cp coverage-report-* coverage/reports/ +

nashif · 2024-02-13T18:13:14Z

scripts/ci/coverage/coverage_analysis.py

+        for item in self.maintainers_file:
+            _found_flag = False
+            try:
+                tests = self.maintainers_file[item]['tests']


that will fail if an area does not have tests....
You should look into using the maintainer file parser, see scripts/get_maintainer.py then
from get_maintainer import Maintainers

Regarding this. I tried to use get_maintainer, but there is no functionality. Search the area according to the name of the tests.
AFAIK, this script provides the following functionalities:
Get area according to paths. get_maintainer path PATH_TO_FILE
Get a list of files regarding the area. get_maintainer list AREA
While creating the report, I'm based on the component's name instead of area. The component's name came from test_case name.
That's why, I used that workaround.

nashif · 2024-02-14T12:35:47Z

.github/workflows/codecov.yaml

+        if: always()
+        run: |
+          pip install xlsxwriter ijson
+          ./scripts/twister -E testplan.json


This will not work, please look at the workflow link I provided. running twister here requires a different runner, you do not have the dependencies needed for twister in the default runner. see https://github.com/zephyrproject-rtos/zephyr-testing/actions/runs/7891137291/workflow

You're right. I needed to take a look at this part of workflow again.

nashif

thw workflow will not work, you really need to take my changes from here: zephyrproject-rtos/zephyr-testing@c6f555a

I can also push those changes into this PR if you like

nashif · 2024-02-26T14:10:30Z

.github/workflows/codecov.yaml

@@ -90,6 +90,7 @@ jobs:
          export ZEPHYR_TOOLCHAIN_VARIANT=zephyr
          mkdir -p coverage/reports
          pip3 install gcovr==6.0
+          ./scripts/twister -E testplan.json


this will not work, we will generate this for each platform, so we have later to deal with the same file name. Just pin it tot he platform name and later just pick one to avoid issues. see my workflow on zephyr-testing

gbarkadiusz · 2024-02-26T15:06:35Z

thw workflow will not work, you really need to take my changes from here: zephyrproject-rtos/zephyr-testing@c6f555a

I can also push those changes into this PR if you like

If my changes are not enough to work properly this workflow. Please push those changes into this PR. I'd like to learn what I did wrong.

Extend codecov workflow to include coverage analysis. Signed-off-by: Arkadiusz Cholewinski <[email protected]> Signed-off-by: Anas Nashif <[email protected]>

gbarkadiusz added the area: Continuous Integration label Feb 6, 2024

gbarkadiusz force-pushed the coverage_analysis branch 4 times, most recently from b29bd6b to 88154fa Compare February 6, 2024 12:38

nashif self-assigned this Feb 6, 2024

gbarkadiusz force-pushed the coverage_analysis branch from 88154fa to ba07f4b Compare February 6, 2024 13:21

gbarkadiusz force-pushed the coverage_analysis branch 2 times, most recently from 3ef37a9 to d462d57 Compare February 6, 2024 14:16

PerMac self-requested a review February 6, 2024 15:56

gbarkadiusz marked this pull request as ready for review February 13, 2024 07:15

zephyrbot requested review from fabiobaltieri, kartben, nashif and stephanosio February 13, 2024 07:15

nashif requested changes Feb 13, 2024

View reviewed changes

gbarkadiusz force-pushed the coverage_analysis branch from d462d57 to 1a128c9 Compare February 14, 2024 12:16

nashif reviewed Feb 14, 2024

View reviewed changes

gbarkadiusz force-pushed the coverage_analysis branch 2 times, most recently from 771b0ee to c7f9001 Compare February 14, 2024 12:59

gbarkadiusz requested a review from nashif February 23, 2024 09:50

nashif requested changes Feb 26, 2024

View reviewed changes

gbarkadiusz force-pushed the coverage_analysis branch from c7f9001 to dbcc207 Compare February 26, 2024 14:39

nashif force-pushed the coverage_analysis branch from dbcc207 to ff83321 Compare February 28, 2024 15:42

nashif previously approved these changes Feb 28, 2024

View reviewed changes

ci: coverage: add coverage analysis reports

50854b8

Extend codecov workflow to include coverage analysis. Signed-off-by: Arkadiusz Cholewinski <[email protected]> Signed-off-by: Anas Nashif <[email protected]>

nashif dismissed their stale review via 50854b8 February 28, 2024 20:20

nashif force-pushed the coverage_analysis branch from ff83321 to 50854b8 Compare February 28, 2024 20:20

nashif approved these changes Mar 5, 2024

View reviewed changes

stephanosio approved these changes Mar 5, 2024

View reviewed changes

nashif merged commit 0e2a25b into zephyrproject-rtos:main Mar 5, 2024
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: Add Coverage Analysis #68624

CI: Add Coverage Analysis #68624

gbarkadiusz commented Feb 6, 2024 •

edited

Loading

nashif commented Feb 6, 2024

gbarkadiusz commented Feb 6, 2024

nashif commented Feb 12, 2024

nashif left a comment •

edited

Loading

nashif Feb 13, 2024

nashif Feb 13, 2024

nashif Feb 13, 2024

nashif Feb 13, 2024

gbarkadiusz Feb 14, 2024

nashif Feb 14, 2024

gbarkadiusz Feb 14, 2024

nashif left a comment

nashif Feb 26, 2024

gbarkadiusz commented Feb 26, 2024

CI: Add Coverage Analysis #68624

CI: Add Coverage Analysis #68624

Conversation

gbarkadiusz commented Feb 6, 2024 • edited Loading

nashif commented Feb 6, 2024

gbarkadiusz commented Feb 6, 2024

nashif commented Feb 12, 2024

nashif left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nashif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gbarkadiusz commented Feb 26, 2024

gbarkadiusz commented Feb 6, 2024 •

edited

Loading

nashif left a comment •

edited

Loading