0.7.x manual 07.Automatic negative test generation

This page is work in progress!!!

Automatic negative test generation

When you are applying combinatorial testing, it is very important to exclude "invalid" test cases. "Invalid" means if SUT is given such a test case, the system can/should not handle it and abort its operation immediately. Because combinatorial testing is a technique to find bugs, where combination of values that system should be able to handle couldn't be handled well, those tests cases must be avoided.

That being said, appropriate error handling is of course important and to be tested. JCUnit has a mechanism to generate test cases that violate constraints given by a user.

Following is an example to generate .

@RunWith(JCUnit.class)
@GenerateCoveringArrayWith(
    checker = @Checker(
        value = SmartConstraintChecker.class,
        args = { @Value(".$QuadraticEquationConstraint") }))
public class QuadraticEquationSolverTest7 {
  public enum QuadraticEquationConstraint implements Constraint {
    A_IS_NON_ZERO {
      public boolean check(Tuple tuple) throws UndefinedSymbol { ... }
      public String tag() { return "aIsNonZero"; }
    },
    DISCRIMINANT_NON_NEGATIVE { ... 
      public String tag() { return "discriminantIsNonNegative"; }
    },
    A_IS_IN_RANGE { ... 
      public String tag() { return "coefficientsAreValid"; }
    },
    B_IS_IN_RANGE { ... 
      public String tag() { return "coefficientsAreValid"; }
    },
    C_IS_IN_RANGE { ... 
      public String tag() { return "coefficientsAreValid"; }
    },;
    public abstract boolean check(Tuple tuple) throws UndefinedSymbol;
    public abstract String tag();
  }

Complete code of this example is found here. QuadraticEquationSolverTest7

If you create a test class like this, JCUnit generates a test suite in a manner where

1. It creates Regular test cases which do not constraint any of constraints defined as enum members defined in QuadraticEquationConstraint
2. Then it tries to generate Violation test cases each of which violates one and only one constraint in the enum.
3. If there are factor-levels that are not covered by test cases generated by step 1 and 2, JCUnit tries to generate test cases for them one by one.

A partial test suite generated from the class in step 1 is like below.

Regular: (a,b,c)=(1,1,0)
Regular: (a,b,c)=(1,0,-1)
Regular: (a,b,c)=(1,-1,-100)
Regular: (a,b,c)=(1,100,1)
Regular: (a,b,c)=(1,-100,100)
Regular: (a,b,c)=(-1,1,1)
Regular: (a,b,c)=(-1,0,100)
Regular: (a,b,c)=(-1,-1,0)
Regular: (a,b,c)=(-1,100,-100)
Regular: (a,b,c)=(-1,-100,-1)
Regular: (a,b,c)=(100,1,-100)
Regular: (a,b,c)=(100,0,0)
Regular: (a,b,c)=(100,-1,-1)
Regular: (a,b,c)=(100,100,0)
Regular: (a,b,c)=(100,-100,1)
Regular: (a,b,c)=(-100,1,100)
Regular: (a,b,c)=(-100,0,1)
Regular: (a,b,c)=(-100,-1,100)
Regular: (a,b,c)=(-100,100,-1)
Regular: (a,b,c)=(-100,-100,0)

Those quadratic equations should all have real solutions. And on the other hand, in step 2 and 3 test cases that do not have real solutions, are not quadratic, or have too big coefficient, will be generated.

Invalid: (a,b,c)=(1,0,100)
Invalid: (a,b,c)=(1,-100,-2147483648)
Invalid: (a,b,c)=(1,2147483647,0)
Invalid: (a,b,c)=(0,1,100)
Invalid: (a,b,c)=(2147483647,1,1)
Invalid: (a,b,c)=(-2147483648,1,0)
Invalid: (a,b,c)=(1,-2147483648,0)
Invalid: (a,b,c)=(1,1,2147483647)

Executing the generated test suite

Generated test cases are inputs to SUT, in this case quadratic equation solver, and we want to verify if it gives us correct solutions for regular test cases and appropriate solutions for violation test cases.

@When notation

Regular test cases and violation test cases have different types of expectation. In regular test cases, we usually expect that a function (or functions) of SUT is executed successfully to the end without errors or exceptions. In violation test cases, we expect the function will be aborted and and appropriate exception thrown or error message shown.

To test those, the first thing we need to do is to identify to which category the current test case being executed belongs. For this, we can use @When notation.

In session 6 of quadratic equation solver test examples, we used @When annotation. Similarly, we can use this annotation to let JCUnit know a condition by which it can determine a certain test method should be invoked.

To specify your constraints whose violation invokes your method, use a string value returned by tag method of them. But give a poind (#) sign at the beginning.

You can use exactly the same syntax for @Condition methods.

  @Test
  @When({ "#aIsNonZero&&#discriminantIsNonNegative&&#coefficientsAreValid" })
  public void printEquationToStdOut() {
    ...
  }

A method annotated with @When like above will be invoked when and only when all the constraints that return #aIsNonZero, #discriminantIsNonNegative, and #coefficientsAreValid are kept.

Regular tests (positive tests)

For regular tests, you want to make sure all the related constraints are satisfied before a test is executed. As already discussed, you can specify the constraints by doing like this.

  @Test
  @When({ "#aIsNonZero&&#discriminantIsNonNegative&&#coefficientsAreValid" })
  public void printEquationToStdOut() {
    ps1.println(String.format("Regular: (a,b,c)=(%d,%d,%d)", a, b, c));
  }

But this is very error prone. You may happen to forget including a member in the constraint enum, you may forget to update the condition when you add a new constraint, etc.

You can instead do this to include all the constraints in the enum,

  @Test
  @When({ "#*" })
  public void solveEquation$thenSolved() {
    ...
  }

Violation tests (negative tests)

To implement violation tests, you want to invoke the method whenever any of given constraints is violated. If you annotate your method with @When using commas, you can achieve it. Also by giving an exclamation (!) to each constraint name, you can invert the condition.

  @Test
  @When({ "!#aIsNonZero", "!#discriminantIsNonNegative", "!#coefficientsAreValid" })
  public void printEquationToStdErr() {
    ...
  }

This @When means if "aIsNonZero", "discriminantIsNonNegative", or "coefficientsAreValid" is violated, this method should be invoked.

As we have seen in regular tests, we can use "#*" here also.

  @Test
  @When({ "!#*" })
  public void printEquationToStdErr() {
    ...
  }

Future works

It might be useful to use @Condition annotated methods as constraints automatically instead of letting users create enums. Issue-#51