Skip to content

Mbed CHIP functional testing

Jump to bottom
ATmobica edited this page May 25, 2021 · 9 revisions

Brief

This page describes the CHIP project's functional testing approach for Mbed applications. It allows you to better understand the use of tools and frameworks for device validation.

Functional testing approach

According to the CHIP project design idea:

"The ability to run tests on actual and emulated hardware is paramount in embedded projects. CHIP is no exception. We want on-device testing to be a first class goal of CHIP architecture. On-device testing requirements apply both to Continuous Integration testing for main CHIP software stack development and to eventual CHIP product certification."

However, functional testing requires a host machine to run and control the testing program. Connection and data exchange between the host machine and the device under test is necessary. Various communication interfaces and protocols are used for this purpose (i.e serial port, network communication via Ethernet, or WiFi).

The common functional test scenario involves the following steps:

  • Get binaries image of testing application (build directly or download CI artifacts)
  • Flash binary image to the device under test
  • Configuration functional tests
  • Run testing
  • Collecting and processing the results

CHIP Mbed functional Testing happens at system level on real hardware. The host machine running the tests should be connected to the board under test via serial port. The tests send a succession of commands to the connected boards or trigger external tools action that changes the device state and asserts the results reported by the hardware.

For example, here is a test validating that WiFi connection works:

Preparation: The device is connected to the host via serial port and executes the shell example application.

The test:

  1. Checking if the device is connected
  2. Sending the command connecting with network credentials
  3. Check device response
  4. Check the device IP address

The Mbed CHIP functional tests are coded in Python and use the pyTest framework. This also allows for easy integration with external CHIP tools such as device-controller. The functional test setup is designed for Linux OS (it is conditioned by the requirements of the CHIP project). The preferred runtime environment is Raspberry Pi 4 with Ubuntu 20.04 LTS.

Setup

The Mbed CHIP functional testing requires a setup testing environment.

The first step is to clone the CHIP project repository.

In your user workspace call:

mkdir -p CHIP
git clone --branch test-functional-tests --recurse-submodules https://github.com/ARMmbed/connectedhomeip.git CHIP

Then setup Linux environment by calling a specific script:

bash CHIP/scripts/setup/setup_ubuntu_20_04_lts.sh

The last step is to run a script to setup a functional test environment:

bash CHIP/scripts/setup/setup_functional_test_env.sh

After these steps, the functional test environment is created with:

  • FunctionalTests directory - test workspace with python virtual environment (FUNCTIONAL_TESTS_DIR environment variable)
  • Running WiFi access point (AP_SSID, AP_PASSWORD and AP_GATEWAY environment variables)
  • Running TCP and UDP echo servers (with AP_GATEWAY IP address on port 7)
  • Ready-to-use CHIP tools (CHIP_TOOLS_DIR environment variable)

There is also a special script for the easy Mbed board mounting. It adds a static entry in fstab with a new connected device.

Then you can just need to call sudo mount -a to mount the device.

To add the new MBed device you should:

  1. Run script bash CHIP/src/test_driver/mbed-functional/setup_new_device.sh
  2. Plug the device
  3. Run sudo mount -a

That's all. The new device is ready for testing.

Prepare binaries

Binary image preparation is an independent process from functional testing. Pytest scripts need the build application images.

There are two ways to get a binary file:

  • building an application from sources (on the host machine or testing platform directly)
  • download artifacts from CI (Github actions in our case)

The building option will be useful to run tests "manually" in our testing environment but it requires a proper build environment.

The downloading artifacts will be good to connect testing with the CI process.

Flash binaries to device

The board flashing step uses the mbed-flasher tool. It is integrated with functional tests as a PyTest fixture. During the creation device, the allocation process checks if we want to flash a board and execute the mbed-flasher function.

You should use the --binaries argument to flash the board. The format is platform:binary_path.

For example:

pytest --binaries=CY8CPROTO_062_4343W::binaries/CY8CPROTO_062_4343W/chip-mbed-shell-example simple_test.py

Tests configuration

Mbed functional tests can be configured by PyTest command line arguments. Every test call may contain a specific test configuration. The list of supported parameters:

[Common]

  • platforms - list of platforms that can be used to run the tests. Platforms are separated by a comma
  • binaries - platform and associated binary in the form platform:binary. Multiple values are separated by a comma
  • serial_inter_byte_delay - time in second between two bytes sent on the serial line (accepts floats), default=None
  • serial_baudrate - baudrate of the serial port used, default=115200

[Test specific]

  • network - WiFi network credentials to which we want to connect device. Format network_ssid:network_password
  • echo_server - address and port of echo server. Format server_ip:server_port
  • chip_tools_dir - path to CHIP tools directory

Test run

To run Mbed functional tests execute the pytest command with the arguments mentioned above. For example:

pytest --platforms=CY8CPROTO_062_4343W --network=$AP_SSID:$AP_PASSWORD ...

The Mbed functional cases are divided into separate directory depends on the testing CHIP application:

  1. shell - CHIP MBed shell example for testing base functionalities such as date, network, socket, BLE, servers
  2. lock-app - CHIP Mbed lock-app for testing WiFi provisioning and execute ZCL command by CHIP device

For more details on how to run tests using PyTest see: PyTest doc

Several Pytest markers have been added to run a specific set of tests:

  • smoketest - base communication and running device tests
  • networktest - networking tests specific
  • bletest - BLE tests specific

Tests results

Adding -rAv arguments to Pytest cause that short tests summary is on the output.

For example:

pytest -rAv simple_test.py

Output:

=================================================================================================================== short test summary info ====================================================================================================================
PASSED CHIP/src/test_driver/mbed-functional/shell/test_base64.py::test_base64_encode_decode
PASSED CHIP/src/test_driver/mbed-functional/shell/test_btp.py::test_btp_device_visiable
PASSED CHIP/src/test_driver/mbed-functional/shell/test_btp.py::test_btp_adv_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_date.py::test_date_set_date
PASSED CHIP/src/test_driver/mbed-functional/shell/test_device.py::test_device_status_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_device.py::test_device_sta_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_device.py::test_device_internet_connection_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_misc.py::test_echo_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_misc.py::test_log_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_misc.py::test_rand_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_network.py::test_network_interface_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_network.py::test_network_interface_address_check
PASSED CHIP/src/test_driver/mbed-functional/shell/test_socket.py::test_socket_client_echo
FAILED CHIP/src/test_driver/mbed-functional/shell/test_device.py::test_device_connection_check - AssertionError: assert 'true' == 'false'
FAILED CHIP/src/test_driver/mbed-functional/shell/test_server.py::test_server_udp - FileNotFoundError: [Errno 2] No such file or directory: './chip-echo-requester'
FAILED CHIP/src/test_driver/mbed-functional/shell/test_socket.py::test_socket_example_check - AssertionError: DUT: Didn't find Done in 10 s
===================================================================================================== 3 failed, 13 passed, 7 warnings in 209.71s (0:03:29) =====================================================================================================

There is also an option to save test results to HTML file. Adding --html=<file_name.html> arguments to Pytest cause that all test results are saved to HTML file and can be open in your browser.

For example:

pytest --html=result.html simple_test.py
Clone this wiki locally