Background

In a nutshell, the goal of JCUnit is "Model your application as code and let JCUnit do the rest." as stated at the first line of the repository.

In this section, I'll walkthrough how we implement a JUnit test class usually.

Challenges in JUnit test codes

Following is a JUnit test code that I would write first for a certain software under test, Sut.

  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    // given
    DataSet dataSet = DataSetRepository.load("dataset/directory/A.dat");

    // when
    Sut sut = Sut.createSut(dataSet);
    ResultSet resultSet = sut.performQuery(QueryFactory.create("query:X=x"));

    // then
    assertThat(resultSet, ResultSetMatchers.sizeIsEqualTo(1));
  }

This test method looks clean so far, but the test class that encloses this method will become messier and messier as you try to improve test coverage over your SUT because you will need to keep adding similar but slightly different test methods to try with different data sets and queries.

When we have another dataset B.dat, you will add another set of test methods.

  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    DataSet dataSet = DataSetRepository.load("dataset/directory/A.dat"); //given
    Sut sut = Sut.createSut(dataSet);
    ResultSet resultSet = sut.performQuery(QueryFactory.create("query:X=x"));  //when
    ...
  }

  @Test 
  public void givenDataSetB$whenPerformQueryX$then...() {
    DataSet dataSet = DataSetRepository.load("dataset/directory/B.dat"); //given
    Sut sut = Sut.createSut(dataSet);
    ResultSet resultSet = sut.performQuery(QueryFactory.create("query:X=x"));  //when
    ...
  }

Next you will want to another set of variants of the test methods above for a query Y(query:Y=y).

  @Test 
  public void givenDataSetA$whenPerformQueryY$then1entryReturned() {
    DataSet dataSet = DataSetRepository.load("dataset/directory/A.dat"); //given
    Sut sut = Sut.createSut(dataSet);
    ResultSet resultSet = sut.performQuery(QueryFactory.create("query:Y=y"));  //when
    ...
  }

  @Test 
  public void givenDataSetB$whenPerformQueryX$then...() {
    DataSet dataSet = DataSetRepository.load("dataset/directory/B.dat"); //given
    Sut sut = Sut.createSut(dataSet);
    ResultSet resultSet = sut.performQuery(QueryFactory.create("query:Y=y"));  //when
    ...
  }

Parameterized runner

You've already had 4 test methods, probably you want to refactor this test class. Is it a good idea to extract a method exercise such as following?

  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    assertThat(exercise("dataset/directory/A.dat", "query:X=x"), sizeIsEqualTo(1));    
  }
  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    assertThat(exercise("dataset/directory/B.dat", "query:X=x"), sizeIsEqualTo(1));    
  }
  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    assertThat(exercise("dataset/directory/A.dat", "query:Y=y"), sizeIsEqualTo(2));    
  }
  @Test 
  public void givenDataSetA$whenPerformQueryX$then1entryReturned() {
    assertThat(exercise("dataset/directory/B.dat", "query:Y=y"), sizeIsEqualTo(3));    
  }

  private ResultSet exercise(String dataSetName, String queryString) {
    DataSet dataSet = DataSetRepository.load(dataSetName); //given
    Sut sut = Sut.createSut(dataSet);
    return sut.performQuery(queryString));  //when
  }

Or if you are familiar with Parameterized runner, you make it a bit cleaner.

   @RunWith(Parameterized.class)
   public class ExampleTest {
    @Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][] {     
                 { "dataset/directory/A.dat", "query:X=x", sizeIsEqualTo(1) }, 
                 { "dataset/directory/B.dat", "query:X=x", sizeIsEqualTo(1) }, 
                 { "dataset/directory/A.dat", "query:Y=y", sizeIsEqualTo(2) }, 
                 { "dataset/directory/B.dat", "query:Y=y", sizeIsEqualTo(3) }
           });
    }

    public ExampleTest(String dataSetName, String queryString, Matcher matcher) {
      this.dataSetName = dataSetName;
      this.queryString = queryString;
      this.matcher = matcher
    }

    @Test
    public void test() {
        assertThat(exercise(this.dataSetName, this.queryString), this.matcher);
    }

    private ResultSet exercise(String dataSetName, String queryString) {
      DataSet dataSet = DataSetRepository.load(dataSetName); //given
      Sut sut = Sut.createSut(dataSet);
      return sut.performQuery(queryString));  //when
    }
   }

Right now, factors involved in this test are only 2, but it is quite common situation for you to have several or a dozen of (or even more) factors you want to try with. In such a situation, the number of test cases can easily explode or you need to accept a risk where you are missing important combinations in your test class.

Theories runner

One painkiller for this is to use Theories runner which automatically generates and executes all the possible test cases for all the given values (@DataPoints) of all the given parameters.

Following is a code snippet from farenda.com (simplified by me)

@RunWith(Theories.class)
public class TheoriesAndDataPointsTest {
    @DataPoints("a values")
    public static int[] aValues() {
        return {1, 2};
    }
 
    @DataPoints("b values")
    public static int[] bValues() {
        return {3, 4};
    } 

    @Theory
    public void sumShouldBeCommutative(@FromDataPoints("a values") int a,
                                       @FromDataPoints("b values") int b) {
        System.out.printf("a = %d, b = %d%n", a, b);
        assertEquals(a + b, b + a);
    }
}

This should print following to stdout.

a = 1, b = 3
a = 1, b = 4
a = 2, b = 3
a = 2, b = 4

It generates complete test cases by using Cartesian product. This approach should work

• if you only have a small number of parameters to be tested
• and if each test case takes only small amount of time

If any of them is broken, the approach will not be efficient enough. If you have a lot of parameters, the number of test cases will grow in exponential order, which cannot be tested by any powerful computer in practical time. And if a test case takes a significant amount of time, the total execution time can become unendurably long easily.

On the other hand, if requirements above are met, why don't you use @Parameterized runner with which you can add/remove desired test cases to find acceptable balance between coverage and execution time? Taking a risk to miss important combinations can be mitigated by reviewing the test suite carefully, although it might not be fun and error-prone process, but at least you have a way.

As it will be mentioned later, JCUnit provides another way to address these problems using combinatorial testing technique. (t.b.d.)

Constraints

Next important fact is that not all the possible combinations of parameters used in a test suite are valid in testing. If you have a think of a following example that has six parameters,

Factor	Levels
Platform	Linux, MacOSX, Windows
Java	JavaSE7, JavaSE8, OpenJDK7
Browser	Safari, Firefox, Chrome, InternetExplorer
DBMS	PostgreSQL, MySQL, SQLServer
Application server	Jetty, Tomcat
Web server	Apache HTTP server, IIS

If a parameter platform is set to Linux, the browser can't become InternetExplorer since we usually expect that it only runs on Windows. If you are using @Theories runner you think this is not a problem since all the possible combinations will be tested and you can simply use assumeThat method to exclude those invalid patterns.

(t.b.d.)

Models

(t.b.d.)

References

• 1 "JUnit Theories with DataPoints", farenda.com
• 2 "ecFeed", ecFeed.com