0.7.x manual 04.FSM

This page is work in progress!!!

Introduction

FSM support of JCUnit (FSM/JCUnit) offers you automated 'Model-based testing' functionality for Java classes. The basic idea is to let users model their SUT(System under test)s as finite state machines and the other things should be taken care of by computers.

Following is a diagram picked up from a Wikipedia article about Model-based testing that illustrates a pipeline of "Offline test case generation". Activities executed by JCUnit and artifacts generated by JCUnits are annotated in it.

Users are supposed to provide 3 types of information. Model, test requirements, and IXIT, which stands for "Implementation extra information".

[FIG.0] "Model-based testing pipeline and JCUnit's process"

                                    (1)User
                                    +---------------+
                                    |     Model     |
                                    +---------------+
                                            |       Inside FSM/JCUnit
                         +------------------------------------------+
    (2)User              |                  V  JCUnit               |
    +-----------------+  |         +---------------+                |
    |Test requirements|--|-------->|Test derivation|                |
    +-----------------+  |         +---------------+                |
                         |                  |                       |
                         |                  V                       |
                         |        +-------------------+             |
                         |        |Abstract test suite|(internal data generated by JCUnit)
                         |        +-------------------+             |
                         |                  |                       |
     (3)User             |                  V            JCUnit     |
    +------+             |  +---------------------------------+     |
    | IXIT |-------------|->|Executable test suite compilation|     |
    +------+             |  +---------------------------------+     |
                         |                  |                       |
                         |                  V                       |
                         |        +---------------------+           |
                         |        |Executable test suite|(internal data generated by JCUnit)
                         |        +---------------------+           |
                         |                  |                       |
                         |                  V  JCUnit               |
                         |           +--------------+               |
                         |           |Test execution|               |
                         |           +--------------+               |
                         |                   |                      |
                         |                   V                      |
     (4)User             |            +--------------+              |
     Review--------------|----------->|    Reports   |(Generated by JCUnit)
                         |            +--------------+              |
                         +------------------------------------------+

The rest of this document consists of following sections.

• Modeling FSM: In this section, it will be discussed how to model your SUT as FSM in JCUnit.((1) in the diagram)
• Running tests: How to define 'test requirements'(how should test cases be generated, etc) and 'IXIT'(how should SUT be set up before starting each test case, etc) in JCUnit will be discussed in this section. ((2) and (3) in the diagram)
• Reviewing reports: JCUnit generates report in test execution phase. How to read those reports will be discussed in this section. ((4) in the diagram)
• Inside FSM/JCUnit: The mechanism of FSM support will be discussed in this section.
• Tips: More practical techniques, e.g., how to model nested FSMs, how to implement adapters when necessary, etc., will be discussed.

Author recommends you to follow steps described in this document (especially the first section of it) with your hands to understand ideas behind the product. The procedures might look complicated initially, but once you try it, you should notice it's designed intuitive and straightforward (at least the author tried very hard to make it intuitive and straightforward).

Modeling FSM

Let's model your FSM. If you are creating a finite state machine which has two actions, cook and eat, and only after cook is done, the machine can eat a thing, a state transition diagram for it would be like following.

[FIG.1] "Simplest finite state machine"

                                +--+
                                |  |eat
                                |  V
    +-----+                   +------+
    |  I  |------------------>|COOKED|
    +-----+       cook        +------+

The machine has 2 states, which are I and COOKED. I is the initial state in which the machine is right after its creation. And COOKED is the state to which it moves after an action cook is performed.

If you are going to implement this state machine as a Java program, it might become like this,

[CODE.1] "Implementation of Figure 1 - SUT with a bug."

    public class FSMonster {
        boolean cooked = false;
        public void cook() {
            // Don't we need to check the value of cooked before assigning?
            this.cooked = true;
        }

        public void eat() {
            if (this.cooked) {
                System.out.println("Yummy!");
            } else {
                throw new IllegalStateException("Not yet cooked!");
            }
        }
    }

What should happen if cook is attempted when the machine is already in COOKED state? Unless explicitly described in the state machine diagram, shouldn't it be rejected?

Yes, it should be rejected. This is an intentional bug for explanation of "FSM support feature". This bug itself might be easy to be found, but if you have some experience in software developments, finding/debugging this sort of bugs is sometimes a time consuming, cumbersome, boring task.

How to detect this sort of bugs in your SUT using "FSM feature" of JCUnit will be discussed later in this document.

Listing states and actions

Let's go back to the diagram [FIG.1], as we already saw, there are 2 states and 2 actions,

• States: I, COOKED
• Actions(Input symbols): cook, eat

In JCUnit, to model a finite state machine, you need to implement FSMSpec<SUT> interface. SUT is a class name of your software under test. And inside your implementation, you will use a few annotations, @StateSpec, @ActionSpec, and @ParametersSpec. Only first two will be necessary to model the machine and @ParametersSpec will be discussed later to model an action with parameters.

public static final fields annotated with @StateSpec will be treated as states by JCUnit. Among those fields, a field named I (capital I) has a special semantics, where it is considered 'initial state' of the finite state machine. And you must define it always.

Methods annotated with @ActionSpec are treated as definitions of actions. They must return Expectation<SUT> and their first parameter must be Expectation.Builder<SUT> always. JCUnit validates the types and complain if they do not meet the requirements.

Following is a skeleton of the spec of the FSM.

[CODE.2] "Model in JCUnit of Figure 1 - Skeleton"

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ... },
      @StateSpec COOKED {
        ... },;

      @ActionSpec public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
        ... }
      @ActionSpec public Expectation<FSMonster> eat(Expectation.Builder<FSMonster> b) {
        ... }
    }

Modeling states and actions

Mathematically, a (deterministic) finite state machine (transducer) can be formalized as follows

      Sigma: input symbols
      Gamma: output symbols
      S:     states
      s0:    initial state
      delta: state transition function. delta: S x Sigma -> S
      omega: output function. omega: S x Sigma -> Gamma (Mealy machine)

We have already created an enum class, Spec. This so far modeled S, s0, and Sigma. S is represented by enum fields annotated with @StateSpec. s0 is I. Sigma is modeled as methods annotated with @ActionSpec.

In this sub section, how we can model the rest of them, which are Gamma, delta, and omega will be discussed.

About Gamma, since JCUnit is a software product to test Java programmes, we need to consider exceptions as output symbols not only regular returned values.

And about Sigma, since methods in Java have parameters, we somehow need to take them into considerations. This topic will also be covered in this sub-section.

Modeling states

At first, FSMSpec interface requires you to implement check(SUT): boolean method. The method is responsible for checking if the SUT is in the specified state. Suppose that FSMonster has a method isReady(), which returns true iff it's in COOKED state, you can do following

[CODE.3] "Implementing states"

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        @Override public boolean check(FSMonster fsm) {
          return !fsm.isReady();
        }
      },
      @StateSpec COOKED {
        @Override public boolean check(FSMonster fsm) {
          return fsm.isReady();
        }
      },;
      ...
    }

But this is actually a part of 'IXIT', which should be discussed in [a later section](#Running tests). For now, define it at one level upper and let it return true always.

    public enum Spec implements FSMSpec<FSMonster> {
      ...
      public boolean check(FSMonster fsm) {
          return true.
      }
    }

Modeling actions

As we already mentioned, if it is not explicitly allowed in a state machine diagram, we should think that an operation isn't allowed.

In the figure 1., operations allowed are 'cook' on state 'I', and 'eat' on 'COOKED'. Therefore, we should test if 'eat' on 'I' and 'cook' on 'COOKED' result in errors.

[FIG.2] "Simplest finite state machine"

                                +--+
                                |  |eat
                                |  V
    +-----+                   +------+
    |  I  |------------------>|COOKED|
    +-----+       cook        +------+

The idea that unless it is explicitly allowed, it should result in an error' can be expressed in a following way.

[CODE.4] "Implementing actions (1) Defining default behaviours"

    public enum Spec implements FSMSpec<FSMonster> {
      ...
      @ActionSpec public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
        return b.invalid().build();
      }
      @ActionSpec public Expectation<FSMonster> eat(Expectation.Builder<FSMonster> b) {
        return b.invalid().build();
      }
    }

The parameter b is an instance of Expectation.Builder<SUT>, by which you can instantiate Expectation<SUT>. In this case you are creating an expectation where this operation should fail (invalid).

And then you will override these methods in the states accordingly.

[CODE.5] "Implementing actions (2) Defining state specific behaviours"

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.valid(COOKED).build();
        }
      },
      @StateSpec COOKED {
        @Override public Expectation<FSMonster> eat(Expectation.Builder<FSMonster> b) {
          return b.valid(COOKED).build();
        }
      },;

      @ActionSpec public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
        return b.invalid().build();
      }
      @ActionSpec public Expectation<FSMonster> eat(Expectation.Builder<FSMonster> b) {
        return b.invalid().build();
      }
      @Override public boolean check(FlyingSpaghettiMonster fsm) {
        return true;
      }
    }

Testing values returned by methods

In the example above, we are only able to test SUT's states. But methods can return values. And they must be tested.

To test a returned value by a method, you need to describe your expectation for SUT. You can do it by giving it to Expectation.Builder.

If a method cook() of FSMonster should be returning a string "Cooking spaghetti", then you can do this.

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ...
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.valid(COOKED, CoreMatchers.startsWith("Cooking")).build();
        }
      },
      ...

The method valid(FSMSpec<SUT>, Matcher) of the builder sets expected status of the SUT and a condition to be satisfied by the value returned by the method cook of SUT, in this example FSMonster. The Matcher and CoreMatchers in this example are from org.hamcrest library, which is used in JUnit itself.

About the finite state machine model we are using

As you may noticed, you can test a method which returns a different value when the state machine is in a different state by overriding @ActionSpec annotated method differently in states.

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ...
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.valid(COOKED, CoreMatchers.startsWith("Cooking a dish")).build();
        }
      },
      @StateSpec COOKED {
        ...
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.valid(COOKED, CoreMatchers.startsWith("Cooking another dish")).build();
        }
      },
      ...

This is one sort of transducers called 'Mealy machine'. Its mathematical model can be formalized as follows.

      Sigma: input symbols
      Gamma: output symbols
      S:     states
      s0:    initial state
      delta: state transition function. delta: S x Sigma -> S
      omega: output function. omega: S x Sigma -> Gamma (Mealy machine)

You can refer to Wikipedia articles for definitions of the models (Mealy Machine and Finite state transducer).

Annotations to model FSM

Following is a matrix that summarizes specifications of annotations used to model FSMs.

Annotation	Target	Modifiers	Type
@StateSpec	Field	public static final	Enclosing class
@ActionSpec	Method	public	Expectation<SUT>
@ParametersSpec	Field	public static final	Parameters

A list of behaviours of those annotations follows.

• @StateSpec: Enclosing class of a field annotated by this must implement an interface FSMSpec<SUT>. If you want to define a behaviour of an action which can be seen on a specific state, you can define a method (which is defined in the enclosing class and annotated with @ActionSpec)
• @ActionSpec: The name of the method must be the same as the name of the method you are going to model by it. The first parameter must be Expectation.Builder<SUT>. And the rest of the parameters must exactly be the same as ones of the method to be tested in SUT.
• @ParametersSpec: A field annotated with this defines arguments given to a method which has the same name as it. The first argument will be picked up from the first array, the second argument will be from the second array. This manner will be followed to the last element, respectively.

Testing a method with parameters

Methods have parameters. JCUnit has another annotation @ParametersSpec to define arguments given to actions that represent methods.

                                +--+
                                |  |eat
                                |  V
    +-----+                   +------+
    |  I  |------------------>|COOKED|
    +-----+ cook(pasta,sauce) +------+

If a method cook has 2 parameters pasta and sauce, they can be modeled as following.

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {...},
      @StateSpec COOKED {...},;

      @ActionSpec public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b,
          String pasta,
          String sauce) { ... }
      @ParametersSpec
      public static final Parameters cook = new Parameters.Builder(Object[][] {
          { "spaghetti", "spaghettini", "penne" },
          { "peperoncino", "carbonara", "meat sauce" },
      }).build();
      @ActionSpec public Expectation<FSMonster> eat(Expectation.Builder<FSMonster> b) {
        ... }
    }

You can define arguments that should be given to a method cook as a public static final field whose name is the same as the method.

      @ParametersSpec public static final Parameters cook = new Parameters.Builder(new Object[][] {
          { "spaghetti", "spaghettini", "penne" },
          { "peperoncino", "carbonara", "meat sauce" },
      }).build();

The type of the argument given to the constructer must be Object[][]. And the first element of it should be an array each of whose elements will be given to the method cook as its first argument. Of course the rest of the array will be treated in the same manner.

i.e., for the method cook(String,String), one of { "spaghetti", "spaghettini", "penne" }, e.g., penne will be picked up and given to cook's first argument. For the second parameter, one of { "peperoncino", "carbonara", "meat sauce" }, e.g., meat sauce will be picked up and used as cook's second parameter.

JCUnit will automatically generates combinations of actual arguments from @ParametersSpec. Probably you might get concerned if a method has only several parameters each of which has only several possible values, it results in a thousands of test cases. But it will not happen usually, because JCUnit applies all-pair techniques here. The detail will be discussed as a part of explanation for how you can describe "test requirements" in JCUnit.

Testing a method with parameters and their constraints

Probably you want to define a topLevelConstraint for the parameters you give to an action. It can be done by doing below.

      @ParametersSpec public static final Parameters cook = new Parameters.Builder(new Object[][] {
          { "spaghetti", "spaghettini", "penne" },
          { "peperoncino", "carbonara", "meat sauce" },
      }).setConstraintManager(
        new ConstraintManagerBase() {
          @Override
          public boolean check(Tuple tuple) throws UndefinedSymbol {
            String pasta = (String)tuple.get("p0");
            String sauce = (String)tuple.get("p1");
            if ("penne".equals(pasta) && "carbonara".equals(sauce)) return false;
            return true;
          }
        }
      ).build();

This example precludes a test case for "penne carbonara" which doesn't sound very tasty from the test suite to be generated. p0 and p1 are parameter names assigned to factors you defined.

Or you can do below to do the same thing.

      @ParametersSpec public static final Parameters cook = new Parameters.Builder()
      .add("pasta", "spaghettini", "penne" )
      .add("sauce", "peperoncino", "carbonara", "meat sauce" )
      .setConstraintManager(
        new ConstraintManagerBase() {
          @Override
          public boolean check(Tuple tuple) throws UndefinedSymbol {
            if ("penne".equals(tuple.get("pasta")) && "carbonara".equals(tuple.get("sauce")))
              return false;
            return true;
          }
      })
      .build();

One thing you should be careful here is the names of factor are not associated with parameter variable names of an action method.

That is, if you do following

      @ParametersSpec public static final Parameters cook = new Parameters.Builder()
      .add("sauce", "peperoncino", "carbonara", "meat sauce" )
      .add("pasta", "spaghettini", "penne" )
      .setConstraintManager(
        new ConstraintManagerBase() {
          @Override
          public boolean check(Tuple tuple) throws UndefinedSymbol {
            if ("penne".equals(tuple.get("pasta")) && "carbonara".equals(tuple.get("sauce")))
              return false;
            return true;
          }
      })
      .build();

    @ActionSpec
    public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String pasta, String sauce) {
      return b.invalid().build();
    }

Values assigned to pasta and sauce will be skewed. pasta will be assigned one of "peperoncino", "carbonara", or "meat sauce". And sauce will be assigned one of "spaghetti" or "penne".

Testing exceptions thrown by methods

Same as returned values, we want to test if a method is throwing an appropriate exception. You can do it by writing code as follows,

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ...
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.invalid(NullPointerException.class).build();
        }
      },
      ...

You can even test if the SUT is in intended state after an exception is thrown.

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ...
        @Override public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
          return b.invalid(I, NullPointerException.class).build();
        }
      },
      ...

In the example above, JCUnit will test if the method cook throws NullPointerException and then test if the SUT (FSMonster object) is in state I.

Running tests

Now we have modeled the FSM to be tested. Let's generate tests and run them. If we go back to the diagram in the introduction ([FIG.0]), what we need to do now is to define 'test requirements'(2) and 'IXIT'(3). In this section we will first discuss how to define 'IXIT' and then move to 'test requirements' because FSM/JCUnit has its defaults for 'test requirements' and users do not need to pay attention to them unless they want to.

IXIT

Other Model-based testing solutions, e.g., ModelJUnit and GraphWalker, usually require users to implement adapters to talk to their SUTs. FSM/JCUnit, since it automatically determines a method to be executed in SUT, which is assumed to be a Java object, based on an action's name and its signature in model side, users do not need to write adapters almost at all.

The only thing a user needs to do in order to run the generated test suite is to call a method FSMUtils.perform. The procedure is straightforward and discussed in this subsection. See [Performing a story] chapter. The author believes this characteristic makes it a lot easiew to start trying Model-based testing with FSM/JCUnit.

That being said, implementing ```check(SUT)`` method in each state in the model will be very helpful for some reasons. The background and how to implement the method will be discussed in a chapter [Implementing check method].

Performing a story

Let JCUnit know a field to store information about what path on FSM diagram should be executed.

    @FactorField(levelsProvider = FSMLevelsProvider.class)
    public Story<FSMonster, Spec> main;

Spec is the (enum) class that models our FSM in the previous section. And FSMonster is the class of our SUT. You don't need to initialize this field by yourself, JCUnit will do it for you.

A factor field whose levelsProvider is FSMLevelsProvider is marked FSM field by FSM/JCUnit and must be typed with Story<SUT, SPEC> where SPEC is a spec class that you defined for the SUT FSMonster in the previous section.

If a test class has one or more FSM fields, FSM/JCUnit feature will be activated.
As other regular factors, the field must be public instance member.

In [the next section](#Inside FSM/JCUnit), internal structure of Story<SUT, SPEC> object will be discussed in detail but for now you can consider it is just an object that stores a sequence of events (methods) and expected states after they are given to the FSM.

Now you can perform the story you have declared in the test class by just writing following code.

    @Test
    public void test() throws Throwable {
      FSMonster sut = new FSMonster();
      FSMUtils.performStory(this, "main", sut);
    }

Implementing check method

If there is not an easy (and safe) way to check it, simply you can return true always like following.

    public enum Spec implements FSMSpec<FSMonster> {
      ...
      public boolean check(FSMonster fsm) {
          return true.
      }
    }

But from a debugging perspective, defining check(SUT) method in each state as much as possible is a very good idea.

    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        @Override public boolean check(FlyingSpaghettiMonster fsm) {
          return !fsm.isReady();
        }
      },
      @StateSpec COOKED {
        @Override public boolean check(FlyingSpaghettiMonster fsm) {
          return fsm.isReady();
        }
      },;
      ...
    }

FSM/JCUnit makes sure if the method check(SUT) returns true whenever it performs a scenario. Since even if it doesn't check it, some expectation for returned values or exceptions would be broken later on in case your SUT has a bug, it is not mandatory to implement the method. But you probably want to know from what point SUT's internal state becomes different from the expected one as immediately as possible. Otherwise, you will need to figure out what really happened to the SUT, e.g., oh the returned value became unexpected from this point on. This would mean some internal information of my FSM was broken before this action but this action itself doesn't modify the object's state. Then let's go back to even one previous before it...

On the other hand, without making sure the SUT is in expected state, should we really return true? Yes, it's inevitable at least in some cases.

If the SUT has getState(), it would be good to write something like following,

    @StateSpec I {
      @Override public boolean check(FlyingSpaghettiMonster fsm) {
        return "Initial".equals(fsm.getState().getName());
      }
    },
    @StateSpec COOKED {
      @Override public boolean check(FlyingSpaghettiMonster fsm) {
        return "Cooked".equals(fsm.getState().getName());
      }
    },;

But it is not necessarily the case always, a developer might not draw or might not be able to draw state machine diagram for some reasons, you might be testing a class someone you don't know, etc. And more importantly, the FSM we modeled first [FIG.1] is independent of its actual implementation. Even if "getState()" method is provided, is the method really giving a correct state always? Isn't it what we are very testing? If it is giving a state different from expectation, the SUT might be actually in wrong state. Or it might be a bug where it is not giving a correct state. Either way, we can say that it's a bug. Therefore, it's a good idea to check if the returned state is correct. But even if it is giving an expected state, it might be just deceiving us by a bug. Thus, what we can/should do here is to check SUT's state if it violates any known constraints derived from its other behaviours. In our example above, it is the value returned by isReady method.

Test requirements

In this subsection, how test requirements, e.g., number of test cases in a test suite to be generated, to what extent paths on a SUT's FSM will be covered, etc., can be configured in JCUnit will be discussed.

Switch coverage

In order to test FSMs, state coverage and transition coverage have been widely used7. But it easy to come up with a bug which cannot be detected by test suite which make both of them 100%. E.g., on a certain transition, some internal variable gets broken and FSM reaches a state A. Later on the broken variable will be used and the SUT malfunctions. If there is another transition that makes our FSM's state A, the bug will possibly not be detected.

A Japanese book ソフトウェアテスト技法ドリル(Drills for software testing techniques) discusses this issue and they introduce an idea called 'switch coverage' (pp. 149).

If all the possible 2 adjacent transitions are covered, it will be called "1 switch coverage", because in between those 2 transitions there is 1 state (switch). Similarly, "2 switch coverage" is defined that all the possible 3 successive transitions need to be covered.

But obviously the number of test cases would very quickly explode as N of "N-switch coverage" increases.

And even worse, FSM/JCUnit considers two same actions which have different sets of arguments as two different transitions. This makes the number of transitions very big.

To balance those 2 requirements, which are switch coverage and number of test cases, you can use some parameters.

To tune switch coverage, you can configure number of switches through FSMLevelsProvider's parameter. And for the other, you can configure tuple generation algorithms and their parameters.

Number of switches

You can specify a number of switches through providerParams.

    @FactorField(levelsProvider = FSMLevelsProvider.class, providerParams = { @Param("2") })
    public Story<FSMonster, Spec> main;

Shortly to say, if you specify "2" for this parameter, it means 3 actions (at least) will be executed from a state chosen by FSM/JCUnit as a starting point in a test case. Because during the sequence 2 states are passed through.

By applying combinatorial method to a state machine, FSM/JCUnit generates a test suite with relatively a small (manageable) number. And this means not all the possible paths whose length are the same as the number specified by this parameter are actually executed unless you are giving the same number as a number of all the factors FSM/JCUnit internally creates.

For more details, refer to [Inside FSM/JCUnit](#Inside FSM/JCUnit) section.

TupleGenerators

In this chapter, a few built-in tuple generators of JCUnit and its characteristics when you use them with FSM feature will be discussed.

IPO2TupleGenerator

IPO2TupleGenerator is a default of FSM/JCUnit. And by changing its parameter, you can control the balance between test strength and test suite size.

Following is an example to give 3 instead of default(2) to IPO2TupleGenerator.

      @RunWith(JCUnit.class)
      @TupleGeneration(
          generator = @Generator(value = IPO2TupleGenerator.class, params = @Param("3"))
      )
      public static class TestClass3 {
        ...
      }

But this increases number of test cases and test generation time.

RandomTupleGenerator

All-pair (or t-wise) test generation can be very time consuming process. Some times probably you want to test your SUT more quickly even if you sacrifice coverage on your FSM. In such a situation, you can configure your test class to use RandomTupleGenerator instead of IPO2TupleGenerator, which is used by default.

    @RunWith(JCUnit.class)
    @TupleGeneration(
        generator = @Generator(value = RandomTupleGenerator.class, params = {@Param("100"), @Param("1")})
    )
    public static class TestClass1 extends TestClass {
    }

As shown, you can control the number of test cases in a suite explicitly (100 in this example).

Reviewing reports

When you run a test, JCUnit will generate a report as follows.

     1:Starting(primary#setUp):ScenarioSequence:[I#cook(spaghettini,carbonara)]
     2:  Running(primary#setUp):I#cook(spaghettini,carbonara) expecting status of 'primary' is 'COOKED' and a string starting with "Cooking" is returned
     3:  Passed(primary#setUp)
     4:End(primary#setUp)
     5:Starting(primary#main):ScenarioSequence:[COOKED#cook(spaghetti,peperoncino),COOKED#eat()]
     6:  Running(primary#main):COOKED#cook(spaghetti,peperoncino) expecting status of 'primary' is 'COOKED' and a string starting with "Cooking" is returned
     7:  Passed(primary#main)
     8:  Running(primary#main):COOKED#eat() expecting status of 'main' is 'COOKED' and a string containing "yummy" is returned
     9:  Passed(primary#main)
    10:End(primary#main)

This report is generated from a test suite for a slightly different FSM from the one we have used. Following is the diagram that describes the state machine.

                                +--+
                                |  |eat
                                |  V
    +-----+                   +------+
    |  I  |------------------>|COOKED|
    +-----+ cook(pasta,sauce) +------+
                                |  A
                                |  |cook(pasta,sauce)
                                +--+

The difference is that you can perform cook(pasta,sauce) action even if you are already in COOKED state.

In this section, it will be discussed how you can read this report.

setUp and main

As it will be explained later, a story comprises "setUp" scenario sequence and "main" scenario sequence. "main" is a test case itself which defines what should be done and in what order. On the other had, "setUp" ensures the SUT to be in the first state from which "main" scenario sequence starts.

Let's take a look at the first line of the report. It says

     1:Starting(primary#setUp):ScenarioSequence:[I#cook(spaghettini,carbonara)]

"primary" is a name of a FSM (story) to be tested. And 'setUp' shows that this line is reporting an activity of "setUp" scenario sequence. Then the following scenario sequence will be executed. According to this line, the sequence only contains one scenario. It performs an action cook with arguements "spaghetti" and "carbonara". This procedure is necessary because the "main" scenario sequence starts with cooked state as you see following.

     5:Starting(primary#main):ScenarioSequence:[COOKED#cook(spaghetti,peperoncino),COOKED#eat()]

This line explains the main scenario sequence is going to perform cook with arguments and then perform another eat action again.

Execution report

Following will be output during test case execution. According to the line 5, 2 scenarios are going to be executed in this sequence, cook(pasta,sauce) and eat.

     5:Starting(primary#main):ScenarioSequence:[COOKED#cook(spaghetti,peperoncino),COOKED#eat()]
     6:  Running(primary#main):COOKED#cook(spaghetti,peperoncino) expecting status of 'primary' is 'COOKED' and a string starting with "Cooking" is returned
     7:  Passed(primary#main)
     8:  Running(primary#main):COOKED#eat() expecting status of 'primary' is 'COOKED' and a string containing "yummy" is returned
     9:  Passed(primary#main)
    10:End(primary#main)

line 6 and 6 shows what FSM/JCUnit is going to do and what it is expecting as a result of those scenarios.

Pick up line 6.

     6:  Running(primary#main):COOKED#cook(spaghetti,peperoncino) expecting \
             status of 'primary' is 'COOKED' and a string starting with "Cooking" is returned

It is almost what it is. FSM/JCUnit is going to run a scenario cook with arguments spaghetti and peperoncino. And it expects that the SUT remains COOKED state and the method returns a string that starts with Cooking.

If the SUT's behaviour doesn't meet this expectation we will receive following output.

    Starting(primary#main):ScenarioSequence:[I#cook(spaghetti,peperoncino)]
      Running(primary#main):I#cook(spaghetti,peperoncino) expecting status of 'primary' is 'COOKED' and a string starting with "Cooking" is returned
      Failed(primary#main): Expectation was not satisfied: ['a string starting with "Cooking"' is expected to be returned but 'Creating spaghetti peperontino' was returned.]
    End(primary#main)

    com.github.dakusui.jcunit.fsm.Expectation$Result: Expectation was not satisfied: ['a string starting with "Cooking"' is expected to be returned but 'Creating spaghettini carbonara' was returned.]
        at com.github.dakusui.jcunit.fsm.Expectation$Result.throwIfFailed(Expectation.java:235)
        at com.github.dakusui.jcunit.fsm.ScenarioSequence$Base.perform(ScenarioSequence.java:184)
        at com.github.dakusui.jcunit.fsm.Story.perform(Story.java:26)

As you can see, the message is self-descriptive.

      Failed(primary#main): Expectation was not satisfied: \
          ['a string starting with "Cooking"' is expected to be returned but 'Creating spaghetti peperoncino' was returned.]

In this case, in spite that we are expecting a string which starts with "Cooking", but the returned one was starting with "Creating".

In case a state error is detected, the message will be like this

    [FSM 'primary' is expected to be in 'COOKED' state but not.(actual='FlyingSpaghettiMonster@1197515375(false)')]

The portion actual='FlyingSpaghettiMonster@1197515375(false)' is printed by toString method. So it is a good idea to override toString method in your SUT to return a string that represents the object's internal state.

Inside FSM/JCUnit

In this section, following technical details will be discussed.

• Story/ScenarioSequence/Scenario
• Internal FSM factors

Story/ScenarioSequence/Scenario

Story<SUT, SPEC> is a class that holds a path an FSM should pass through in a test case, generated by FSM/JCUnit. Story has two ScenarioSequences, one of which is setUp and the other is main. main one is a part of a test case, which may start with a different state from I. setUp describes a procedure to reach the state with which the main scenario sequence belonging to the same story.

Both of them are instances of class ScenarioSequence.

A ScenarioSequence object is, as its name suggests, a sequence of scenarios. Then what a Scenario object needs to be answered. It is an object which comprises 3 members, State given, Action when, and Args with.

A State object represents a state the SUT's FSM should be in before the scenario is performed.

Following is a class diagram that illustrates relationships between those classes.

[FIG.3] "Class diagram for Story/ScenarioSequence/Scenario"

                      +----------------+       +--------------+
          setUp +---->|ScenarioSequence|<>---->|Scenario      |
                |    1+----------------+       +--------------+
                |
                |
    +-----+     |
    |Story|<>---+
    +-----+ 1   |
                |                              +--------------+
                |                              |Scenario      |
                |     +----------------+       +--------------+
           main +---->|ScenarioSequence|<>---->|State   given |
                     1+----------------+ 1   * |Action  when  |
                                               |Args    with  |
                                               +--------------+

When a ScenarioSequence is performed, it executes each scenario which belongs to the sequence one by one from the first one to the last. And when a scenario is executed, it figures out the status to which the FSM moves after a when action is executed with with arguments. Then it actually executes a method in SUT whose name is the same as the when action's. The name of when action is derived from the spec class we discussed in the previous section.

Also as discussed in the previous section,

    @Test
    public void test() {
      FSMonster sut = new FSMonster();
      FSMUtils.performStory(this, "main", sut);
    }

Following is an excerpt from CODE.2.

      @ActionSpec public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b) {
        ... }

When FSM/JCUnit finds this declaration, it assumes that there is a method whose name is cook without any arguments in the SUT (i.e., FSMonster) and the method will be called during test executions.

If the method has 1 or more parameters, the action definition would look like following,

      @ActionSpec
      public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String dish) {
        ... }

This can be handled by FSM/JCUnit appropriately. JCUnit looks up a method whose name is cook and its only argument's type is String.

But you are not able to test 2 overloading methods at once. If you are going to do it, action definitions will look like following, which would work.

      @ActionSpec
      public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String dish) {
        ... }

      @ActionSpec
      public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String pasta, String sauce) {
        ... }

Overloading a method

But how can we give parameters to them? Based on what we have discussed, the parameters definitions would look like this,

      @ParametersSpec
      public static final Parameters cook = new Parameters.Builder(new Object[][] {
          { "spaghetti", "spaghettini", "penne" },
          { "peperoncino", "carbonara", "meat sauce" },
      }).build();

      @ParametersSpec
      public static final Parameters cook = new Parameters(new Object[][] {
          { "soup", "primo", "dolce" }
      }).build();

But this results in a compilation error because a Java class can only have one field with a certain name.

To model an overloading method like this one, you need to define a field with a different name and refer to it from a method in your model using parametersSpec attribute.

      @ParametersSpec
      public static final Parameters cook = new Parameters(new Object[][] {
          { "soup", "primo", "dolce" }
      }).build();

      @ActionSpec(parametersSpec="cook1")
      public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String dish) {
        ... }

      @ParametersSpec
      public static final Parameters cook = new Parameters.Builder(new Object[][] {
          { "spaghetti", "spaghettini", "penne" },
          { "peperoncino", "carbonara", "meat sauce" },
      }).build();

      @ActionSpec
      public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String dish) {
        ... }

Internal FSM factors

Suppose your FSM has 3 states and 4 actions. And it's named "myfsm". And you have configured your test class that the number of switches is 1. FSM/JCUnit will create following factors internally.

[FIG.4] "State machine for example of FSM factor expansion"

          pay         drink
      +---------+  +-----------+
      |         |  |           |
      V         |  V           |
    +---+      +----+        +----+
    | I |----->| S0 |------->| S1 |
    +---+ cook +----+eat     +----+
      |                        A
      |                        |
      +------------------------+
                drink/1

And following is a list of definitions of methods mentioned in the previous diagram. Values that a tester want to use in tests as arguments of each method are given in comments.

    public void drink(String beverage)  // beverage can be "tea" or "coffee"
    public void eat(int number)         // silver can be 1 or 2
    public void cook(String pasta, String sauce)
                                        // pasta can be "spaghetti", "spaghettini", or "penne"
                                        // sauce can be "peperoncino", "meat sauce", or "carbonara"
    public void pay(int money)          // money can be 1, 10, or 30.

A state machine defined above will be translated into a set of factors, shown in a following matrix, by FSM/JCUnit.

Factor	Levels
FSM:myfsm:state:0	I, S0, S1
FSM:myfsm:action:0	cook, drink, eat, pay
FSM:myfsm:param:0:0	1, 10, 30, "spaghetti", ..., 2, "tea", "coffee", or VOID
FSM:myfsm:param:0:1	"peperoncino","meat sauce", "carbonara", VOID
FSM:myfsm:state:1	I, S0, S1, VOID
FSM:myfsm:action:1	cook, drink, eat, pay, VOID
FSM:myfsm:param:1:0	(see above)
FSM:myfsm:param:1:1	(see above)

Note that FSM:myfsm:param:0:0: is a union of first arguments of drink, eat, cook, and pay. And even if the values are overlapping, they are not listed twice. That is, 1 is a possible first argument for eat and pay, it appears in the factor only once, though.
After FSM states, actions, and parameters are expanded to those factors, JCUnit will generate test cases. During this generation, FSMConstraintManager checks if each test case matches requirements given by the FSM, e.g., "If 'FSM:myfsm:state:0' is S0 and 'FSM:myfsm:action:0' is 'pay', this test case must be wrong" (actual checking procedure is slightly more complicated than this, because FSM/JCUnit tries to generate test cases which tests if SUT gives an appropriate exception).

Tips

In this section, some useful tips for FSM/JCUnit in real usages.

Nested FSM

In real world, things are nested. In FSMs it is so, too. An object with states has a createXyz method. And then the returned object by the method has its own states, too.

FSM/JCUnit supports this sort of structure. Suppose that we are going to test an FSM that creates another one, and we want to test both of them. [FIG.5] shows an example for this situation.

[FIG.5] "Nested FSM"

    primary FSM
                                +--+
                                |  |eat
                                |  V
    +-----+                   +------+
    |  I  |------------------>|COOKED|
    +-----+ cook(pasta,sauce) +------+
                                |  A
                                |  |cook(pasta,sauce): nested FSM
                                +--+
    nested FSM


     +---+
     |   |toString
     |   V
    +-----+
    |  I  |
    +-----+

In order to implement tests for these FSM, things to be done are described in [CODE.6].

[CODE.6] "Nested FSM example"

    /**
     * Spec of parent FSM. This create a child FSM to be tested when cook method
     * is called on COOKED state.
     */
    public enum Spec implements FSMSpec<FSMonster> {
      @StateSpec I {
        ...
      },
      @StateSpec COOKED {
        ...
        @Override
        public Expectation<FSMonster> cook(Expectation.Builder<FSMonster> b, String dish, String sauce) {
          ////
          //  (*) Building an Expectation object for nested FSM.
          return b.valid(this, new Expectation.Checker.FSM("nested")).build();
        }
      }


    /**
     * Spec of nested FSM, in this case a string, which is immutable and its
     * spec as FSM is trivial.
     */
    public enum NestedSpec implements FSMSpec<String> {
      ...
    }

    @FactorField(levelsProvider = FSMLevelsProvider.class)
    public Story<FlyingSpaghettiMonster, Spec> primary;

    @FactorField(levelsProvider = FSMLevelsProvider.class)
    public Story<String, NestedSpec> nested;

    @Test
    public void test1() {
      FSMonster sut = new FSMonster();
      FSMUtils.performStory(this, "primary", sut, new ScenarioSequence.Observer.Factory.ForSilent());
    }

The line commented "(*) Building an Expectation object for nested FSM" is the trick. By doing this, you are able to let FSM/JCUnit know the returned expectation is for another FSM. And the argument "nested" is a name of the Story field you want to let FSM/JCUnit perform.

If a user creates a cyclic link, what will happen? Don't worry FSMUtils.performStory executes a story at most once by checking Story object's state.

And as usual, you can now perform the story by doing test1().

But it might be a good idea to do following because, as it is mentioned, FSMUtils.performStory method tries 'at most' once each story. It means that if the story doesn't execute cook method on COOKED state, the nested FSM will not be tested in the test case.

      @Test
      public void test2() {
        FSMonster sut = new FSMonster();
        FSMUtils.performStory(this, "primary", sut);
        if (!this.nested.isPerformed()) {
          FSMUtils.performStory(this, "nested", "Cooking spaghetti meat sauce");
        }
      }

Working example is found here.

SUT adapter

FSM/JCUnit requires almost not 'adapter' implementation work, but you can do it if you want.

What you need to do is simple.

• Create an adapter class. Let's call it MyAdapter, here.
• Model your SUT by implementing FSMSpec<MyAdapter> as in normal use cases of FSM/JCUnit. Let's call the model MySpec here.[CODE.7]
• Implement all the methods annotated with @ActionSpec so that they operate actual SUT (but exclude the first parameter Expectation.Builder<MyAdapter> from the parameters).[CODE.8]
• Consider implementing check(MyAdapter) method in MySpec class by delegating to MyAdapter#check(MySpec).

Inside MyAdapter you can do whatever you want. You can use Selenium to manipulate GUI, issue CLI commands, HTTP requests, etc. Another benefit of this approach is that you will have a good Java wrapper API to access your SUT which can be re-used for other purposes like creating admin utilities, value added services on top of your SUT, etc.

[CODE.7] "Spec example"

    public enum MySpec {
      @StateSpec I {
      },
      @StateSpec ... {
      }
      @ActionSpec public Expectation<MyAdapter> initialize(Expectation.Builder<MyAdapter> b) { ... }
      @ActionSpec public Expectation<MyAdapter> perform(Expectation.Builder<MyAdapter> b) { ... }
      @ActionSpec public Expectation<MyAdapter> print(Expectation.Builder<MyAdapter> b, PrintStream ps) { ... }
      @ActionSpec public Expectation<MyAdapter> toString(Expectation.Builder<MyAdapter> b) { ... }

      public boolean check(MyAdapter myAdapter) {
        return myAdapter.check(this);
      }
    }

[CODE.8] "An adapter"

    public class MyAdapter {
      public void initialize() {
        ...
      }

      public void perform() {
        ...
      }

      public void print(PrintStream ps) {
        ...
      }

      public String toString() {
        ...
      }

      public boolean check(MySpec spec) {
          return ...;
      }
    }

Offline testing

Doing offline testing with FSM/JCUnit is simple. You have already had a test case in your test object as @FactorField annotated fields (fsm1 and fsm2 in this example). Keep them and use them later.

    @RunWith(JCUnit.class)
    public class DoubleFSMTest {
      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, TurnstileTest.Spec> fsm1;

      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, TurnstileTest.Spec> fsm2;

Probably it is a good idea to separate the class into two (or more if necessary), one of which is for generating test suite and the other is for later execution (see the example below).

    @RunWith(JCUnit.class)
    public abstract class DoubleFSMTestBase {
      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, TurnstileTest.Spe> fsm1;

      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, TurnstileTest.Spec> fsm2;

    }

    public class DoubleFSMTestGenerator extends DoubleFSMTestBase {
        @Rule
        public               Recorder recorder      = new Recorder();
        ...
    }

    @TupleGeneration(
        generator = @Generator(
            value = Replayer.class,
            params = {
                @Param("FailedOnly"),
                @Param("src/test/resources")
            }
        )
    )
    public class DoubleFSMTestExecutor extends DoubleFSMTestBase {
        public void testFSMs() {
          ...
        }
    }

You can refer to following files for how Recorder and Replayer work.

• Recorder.java
• Replayer.java

Multi-threads

Suppose that you are given 2 (or more) FSM and you need to make sure they can work concurrently without defects.

    @RunWith(JCUnit.class)
    public class ConcurrentTurnstileTest {
      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, Spec> t1;
      @FactorField(levelsProvider = FSMLevelsProvider.class)
      public Story<Turnstile, Spec> t2;

      @Test(timeout = 100)
      public void test1() {
        FSMUtils.performStoriesConcurrently(
            this,
            new Story.Request.ArrayBuilder()
                .add("t1", new Turnstile())
                .add("t2", new Turnstile())
                .build()
        );
      }
    }

In order to test a Turnstile object is thread-safe, you can do try following code fragment.

      @Test(timeout = 100)
      public void test1() {
        Turnstile turnstile = new Turnstile();
        FSMUtils.performStoriesConcurrently(
            this,
            new Story.Request.ArrayBuilder()
                .add("t1", turnstile)           // The same object is manipulated by 2
                .add("t2", turnstile)           // threads concurrently.
                .build()
        );
      }

This test will fail miserably because a turnstile isn't thread-safe generally speaking.

Inside the method, FSMUtils.performStoriesConcurrently, scenarios that belong to main scenario sequences of those multiple FSMs' actions are invoked in 'synchronized' way.

The first scenarios of the sequences are executed at the same time, and the next scenarios will not be executed until all scenarios finish. Once all the first scenario have finished, the second ones will be executed in a same manner, etc.

Factors are organized like a matrix below when test suite is generated internally,

Factor	Levels
FSM:t1:state:0	I, LOCKED
FSM:t1:action:0	coin, pass
FSM:t1:state:1	I, LOCKED
FSM:t1:action:1	coin, pass
FSM:t2:state:0	I, LOCKED
FSM:t2:action:0	coin, pass
FSM:t2:state:1	I, LOCKED
FSM:t2:action:1	coin, pass

If we apply a pairwise test suite generation to this factor space, all the combinations between t1's states and t2's states will be tested, t1's first action and t2's first action, etc. But does number of test cases become 256? No just 4. This is because the turnstile's model limits possible transitions and also because pairwise technique.

Thus, we are able to test possible combinations with a reasonably good coverage and small amount of test cases.

References

• 0 "Wikipedia article about Model-based testing"
• 1 "Wikipedia article about Mealy machine"
• 2 "Wikipedia article about Finite state transducer"
• 3 "ソフトウェアテスト技法ドリル テスト設計の考え方と実際", 秋山浩一, ISBN97804-8171-9360-5, 日科技連, 2010
• 4 "Introduction to Combinatorial Testing", D. Richard Kuhn, Raghu N. Kacker, Yu Lei, CRC Press, 2013
• 5 "ModelJUnit"
• 6 "Model Based Testing (MBT)", hcltech.com
• 7 "Practical Model-Based Testing - a tools approach"
• 8 "GraphWalker"
• 9 "Selenium WebDriver"