Section: Scientific Foundations

Automatic test generation

We are mainly interested in conformance testing, which consists in checking whether a black box implementation under test (the real system that is only known by its interface) behaves correctly with respect to its specification (the reference which specifies the intended behavior of the system). In the line of model-based testing, we use formal specifications and their underlying models to unambiguously define the intended behavior of the system, to formally define conformance and to design test case generation algorithms. The difficult problems are to generate test cases that correctly identify faults (the oracle problem) and, as exhaustiveness is impossible to reach in practice, to select an adequate subset of test cases that are likely to detect faults. Hereafter we detail some elements of the models, theories and algorithms we use.

We use IOLTS (or IOSTS) as formal models for specifications, implementations, test purposes, and test cases. We adapt a well established theory of conformance testing  [30] , which formally defines conformance as a relation between formal models of specifications and implementations. This conformance relation, called ioco compares the visible behaviors (called suspension traces) of the implementation $I$ (denoted by $STraces\left(I\right)$) with those of the specification $S$ ($STraces\left(S\right)$). Suspension traces are sequence of inputs, outputs or quiescence (absence of action denoted by $\delta$), thus abstracting away internal behaviors that cannot be observed by testers. Intuitively, $IiocoS$ if after a suspension trace of the specification, the implementation $I$ can only show outputs and quiescences of the specification $S$. We re-formulated ioco as a partial inclusion of visible behaviors as follows:

In other words, suspension traces of $I$ which are suspension traces of $S$ prolongated by an output or quiescence, should still be suspension traces of $S$.

Interestingly, this characterization presents conformance with respect to $S$ as a safety property of suspension traces of $I$. The negation of this property is charaterized by a canonical tester $Can\left(S\right)$ which recognizes exactly $[STraces\left(S\right).{\Lambda }_{!}^{\delta }\setminus STraces\left(S\right)]$, the set of non-conformant suspension traces. This canonical tester also serves as a basis for test selection.

Test cases are processes executed against implementations in order to detect non-conformance. They are also formalized by IOLTS (or IOSTS) with special states indicating verdicts. The execution of test cases against implementations is formalized by a parallel composition with synchronization on common actions. A Fail verdict means that the implementation under test (IUT) is rejected and should correspond to non-conformance, a Pass verdict means that the IUT exhibited a correct behavior and some specific targeted behaviour has been observed, while an Inconclusive verdict is given to a correct behavior that is not targeted.

Test suites (sets of test cases) are required to exhibit some properties relating the verdict they produce to the conformance relation. Soundness means that only non conformant implementations should be rejected by a test suite and exhaustiveness means that every non conformant implementation may be rejected by the test suite. Soundness is not difficult to obtain, but exhaustiveness is not possible in practice and one has to select test cases.

Test selection is often based on the coverage of some criteria (state coverage, transition coverage, etc). But test cases are often associated with test purposes describing some abstract behaviors targeted by a test case. In our framework, test purposes are specified as IOLTS (or IOSTS) associated with marked states or dedicated variables, giving them the status of automata or observers accepting runs (or sequences of actions or suspension traces). Selection of test cases amounts to selecting traces of the canonical tester accepted by the test purpose. The resulting test case is then both an observer of the negation of a safety property (non-conformance wrt. $S$), and an observer of a reachability property (acceptance by the test purpose). Selection can be reduced to a model-checking problem where one wants to identify states (and transitions between them) which are both reachable from the initial state and co-reachable from the accepting states. We have proved that these algorithms ensure soundness. Moreover the (infinite) set of all possibly generated test cases is also exhaustive. Apart from these theoretical results, our algorithms are designed to be as efficient as possible in order to be able to scale up to real applications.

Our first test generation algorithms are based on enumerative techniques, thus adapted to IOLTS models, and optimized to fight the state-space explosion problem. On-the-fly algorithms where designed and implemented in the TGV tool, which consist in computing co-reachable states from a target state during a lazy exploration of the set of reachable states in a product of the specification and the test purpose  [25] . However, this enumerative technique suffers from some limitations when specification models contain data.

More recently, we have explored symbolic test generation techniques for IOSTS specifications  [29] . The objective is to avoid the state space explosion problem induced by the enumeration of values of variables and communication parameters. The idea consists in computing a test case under the form of an IOSTS, i.e., a reactive program in which the operations on data are kept in a symbolic form. Test selection is still based on test purposes (also described as IOSTS) and involves syntactical transformations of IOSTS models that should ensure properties of their IOLTS semantics. However, most of the operations involved in test generation (determinisation, reachability, and coreachability) become undecidable. For determinisation we employ heuristics that allow us to solve the so-called bounded observable non-determinism (i.e., the result of an internal choice can be detected after finitely many observable actions). The product is defined syntactically. Finally test selection is performed as a syntactical transformation of transitions which is based on a semantical reachability and co-reachability analysis. As both problems are undecidable for IOSTS, syntactical transformations are guided by over-approximations using abstract interpretation techniques. Nevertheless, these over-approximations still ensure soundness of test cases  [26] . These techniques are implemented in the STG tool (see  5.1 ), with an interface with NBAC used for abstract interpretation.