Testers design

Introduction

Kyua provides support to run test programs that follow different interfaces. At the moment, this includes the atf and the plain interfaces, respectively described in the kyua-atf-interface(7) and kyua-plain-interface(7) manual pages.

The support to run a particular test interface is known as a tester and all testers are currently implemented in the same process as the rest of Kyua. This makes the implementation as a whole more complex and error-prone than it could be and poses some major challenges for the addition of parallel test case execution. The text below explain this in more detail.

This page proposes an alternate design in which all the logic to deal with a particular test interface is placed in a separate binary whose sole purpose is to run individual test cases in a safe manner. A side-effect of this proposal is that testers become dynamically pluggable into Kyua, and that Kyua loses all internal knowledge of different test interfaces by delegating this to separate, self-contained entities.

Problems to solve

Let's start by describing the problems of the current implementation. The description of each try not to make any implications as to how a possible solution looks like.

Parallel execution of test cases

Test programs are untrusted entities: they are programs designed to cause failures on other code and, when these failures arise, they can produce all kinds of side-effects either in the calling process or in the operating system as a whole. Kyua, which is a runtime engine for test cases, mitigates these as much as possible.

Test case isolation provides a mechanism by which test cases are insulated from the test program that contains them and also insulates the test cases from global OS resources that could otherwise make their behavior non-deterministic. More in detail, the current actions are currently taken by the test case isolation code:

• Run the test case in a separate process so that it cannot harm the rest of the test cases in the same test program.

• Set up a process group for the test case and its descendants so that the group as a whole can be terminated when the test case completes or its deadline expires.

• Run the test case in a clean temporary directory so that it can create files at will. The temporary directory is automatically cleaned on termination.

  • Clean environment variables like HOME, TZ, LANG, etc. by setting them to deterministic values.

  • Set the umask to a known value.

  • Bump process limits to allow the creation of a core file for debugging purposes.

Implementing test case isolation involves dealing with many different OS-specific interfaces: processes, signals and timers to name a few. Programming these portably is often a process-wide operation; some of these resources may not be programmable at a thread level at all, or this operation may not be portable.

Kyua needs to implement parallel execution of test cases to take advantage of multiprocessor systems and to drastically reduce the run times of large test suites. The "obvious" approach to this would be to add threading to Kyua, but this would be non-trivial due to what has been mentioned previously: programming OS features is often a process-wide operation. Additionally, using threads may not be the wisest choice as this increases programming and debugging complexity: it would be nice if we could come up with a design in which threads were not necessary.

Code complexity

Kyua's codebase includes an abstraction layer in the utils libraries to expose all native system resources in a portable and clean manner to C++ code. In particular, these libraries make extensive use of the RAII programming pattern to ensure that no resources are leaked in the face of exceptions, which are extensively used for error reporting in Kyua. Unfortunately, this C++ abstraction has a cost in terms of code complexity and total build times. To make the integration of Kyua less problematic into the various BSD systems, reducing the amount of C++ code is worthwhile as long as the new implementation is simpler and maintainable.

Modularity

As mentioned earlier, Kyua currently supports two different test interfaces: atf and plain. There is a plan to add support for additional interfaces (e.g. googletest). However, the support for all interfaces is complex as different parts of Kyua (the run-time engine, the database, the internal representations) must have knowledge of all these possible types.

It would be nice if there was a clearly-defined and abstract interface to deal with all tests regardless of their implementation. This would simplify the code in Kyua significantly, and thus indirectly also address the goal of reducing build times.

Overview of the solution

All the problems described above can be addressed by moving all the tricky code to interact with a test program into a separate binary:

• Implementing parallel execution of test cases becomes simple because Kyua need not program OS resources any more: all it has to do is execute the different testers (one per test case) and collect their results. Additionally, because this operation can be done asynchronously with multiple subprocesses running at once, there is no need to introduce threads in the execution.

• If the test case isolation code lives in a separate binary, many simplifications can be done. Because such binary would only be responsible for running an individual test case, and because such binary does not need to interact with the user, it can be easily implemented in pure C code. This can reduces the amount of C++ code in Kyua significantly, particularly by getting rid of a lot of abstraction layers that become unused.

• If the independent testers are made to have a common interface, Kyua does not need to maintain any knowledge of the various test interfaces: testers can be plugged dynamically to support new test programs, and Kyua can use them without modifications.

File system layout

Testers are binaries installed in libexecdir/kyua/ and have names of the form <INTERFACE>_tester. Whenever a Kyuafile defines a test program of type T, Kyua will check if a tester named T_tester exists and use that to run the test program. Kyua does not need to know anything else about the way the tester is written nor the test programs it interacts with.

All tester interfaces must support the exact same command-line interface so that Kyua can talk to them without worrying about the specifics of a test program. In the rest of this document, we will consider the ATF tester for all examples (as the ATF interface it is the most complex one to be supported), but the explanations should be applicable to the plain tester or any other tester that we may implement in the future.

Command-line interface

Every invocation of a tester results in a corresponding invocation of a test program. Therefore, there is always a need to specify the restrictions for the execution of the test program, such as a deadline for its successful completion, and is why these options have to be global.

The syntax is:

*_tester [generic options] command [command options] [command arguments]

The generic options are:

• -t seconds: The timeout for the test program, in seconds.

• -w workdir: Template of the work directory to use. If this includes a XXXXXX substring, a new temporary directory is created using mkdtemp(3) using the given template; otherwise, the provided directory is used, which must exist beforehand.

The exit codes of the program depends on the particular command that is invoked.

Use case: list tests

The list command generates a list of the test cases in a test program and dumps it to stdout in a machine-parseable format. The syntax is:

*_tester [generic options] list test_program

And the format of the output is:

test_case_name1
prop1 value1
...
propN valueN

test_case_name2

test_case_name3
prop1 value1
...
propN valueN

Note that test cases are separated from others by a blank line, and that the list of properties may be empty.

The test case listing should not require a work directory. Kyua will pass an existing, read-only directory to the list command so that, if the test case attempts to create a temporary file during a listing (an invalid operation), it fails early.

Use case: run a test

The run command executes a test case and cleans it up in an atomic manner. The syntax is:

*_tester [generic options] run [-u uid] [-v var=name] test_program test_case results_file

The flags are as follows:

• -u uid: UID of the unprivileged user to drop the privileges to when running the test case, if any and if running as root.

• -v var=name: Sets the value of a configuration variable. These are passed verbatim to the test case, although this depends on the specific tester.

The results file has the following format, and is machine-parseable:

result_type[: result_reason]

The possible combinations of the type and the reason are, along with the exit code of the run command:

Result type	Has reason	Exit code	Description
passed	no	0	Test passed.
skipped	yes	0	Test not applicable for whatever condition.
failed	yes	1	Test failed for any reason.
xfail	yes	0	Test detected an expected failure (e.g. known bug)

The exit code of the runner is as detailed in the previous table. The result file is always created independently of the exit code, as the exit code is not enough to determine all the necessary status information.

Enforcement of requirements

The tester may implement the enforcement of generic requirements before executing a test case. For example, if a test case indicates that it needs to run on an amd64 host and the host is not amd64, the tester should return a skipped result without even executing the test case.

This is a feature required by the ATF interface only. The split of the list and run commands in the tester makes it hard to implement this in an efficient manner: Kyua gets the list of properties of a test case as the first step for its own consumption, but when it executes a test through a tester, the tester does not have access to these properties without reexecuting the test program in list mode.

A possibility to improve this, if it proves to be a problem, is to make Kyua pass the collected test case metadata to the run command in a special way so that the run command does not need to reinvoke the test program. Another alternative could be to change the test programs themselves to report their metadata during a regular execution through an out-of-band channel.

Use case: debug a test

The purpose of the debug command is to provide a mechanism to run all the parts of a test case (body, cleanup) individually so that, for example, a debugger can be attached to them.

debug [-u uid] [-v var=name] test_program test_case part

The flags are as follows:

• -u uid: Same as in the run command.

• -v var=name: Same as in the run command.

Things like proper reporting of the test case result are not provided, because we are not interested in Kyua receiving them. As long as the information is made available through e.g. stdout and that the tester reports a success/failure code, it's good enough.

Because the body and the cleanup parts need to share the same work directory, the -w flag passed to both parts during debugging must match. Making this happen is left to the caller (Kyua), as the caller may also want to implement different behaviors like keeping the work directory around.