0.7.x manual 05.Coverage reporting

This page is work in progress!!!

Coverage reporting

JCUnit has a reporting mechanism by which you can make sure all the levels of each factor are covered in your test suite, all the states in an FSM are covered, etc.

To use this feature, you need to set reporters attribute in @GenerateCoveringArrayWith annotation appropriately.

Following is an example.

  @RunWith(JCUnit.class)
  @GenerateCoveringArrayWith(
      engine = @Generator(IPO2CoveringArrayEngine.class),
      reporters = {
          @Reporter(value = FSMMetrics.class, args = { @Value("messageDigestStory"), @Value("1")}),
          @Reporter(value = CombinatorialMetrics.class, args = { @Value("2") })
      }
  )
  public static class Example {
    @FactorField(
        levelsProvider = FSMLevelsProvider.class,
        args = { @Value("3") })
    public Story<MessageDigest, MessageDigestExample.Spec> messageDigestStory;

    @Test
    public void test() {
        ...

The line containing FSMMetrics.class in the example is set to report how much an FSM defined by MessageDigestExample class is covered by values assigned to messageDigestStory field during the generate test suite's run.

And the second argument @Value("1") specifies so called "switch coverage", which means the ratio of covered action sequences among all the possible action sequence. 1 means a sequence which has 1 state in between 2 actions.

          @Reporter(value = FSMMetrics.class, args = { @Value("messageDigestStory"), @Value("1")}),

This reporter prints following output.

Action coverage
  getDenominator       - 2
  getNumerator         - 2
  getRatio             - 1.00
State coverage
  getDenominator       - 1
  getNumerator         - 1
  getRatio             - 1.00
Switch coverage (1)
  getDenominator       - 4
  getNumerator         - 4
  getRatio             - 1.00

The second line,

          @Reporter(value = CombinatorialMetrics.class, args = { @Value("2") })

requests JCUnit to print how much possible value pairs are covered by the test suite.

Combinatorial coverage
  getDenominator       - 99
  getNumerator         - 96
  getRatio             - 0.97
Violation ratio
  getDenominator       - 7
  getNumerator         - 0
  getRatio             - 0.00

Violation ratio tells you how many test cases are violating the constraints you gave to the test class.