Test Specification Language

In our test specification language, tests are organized in test suites. A test suite is composed of import statements, test scenarios, and test cases, which are describe in the following.

1. Import Statements

An import statement is used for importing elements from the UML model to be tested, such as activities and classes. Listing 1 shows how import statements are defined.

Listing 1. Import statement declaration

In Listing 1, all elements of the package with the name package_name are imported into the test suite.

2. Test Scenarios

Our testing framework allows developers of UML models to test the behavior of UML activities. Thereby, specified test cases are evaluated by executing the activity under test. If the activity under test expects input values or has to run in the context of an object, these input values and context object have to be specified. Therefore, test scenarios may be defined that enable the specification of objects and links between these objects, which may be used as input values or context object. Listing 2 shows the definition of test scenarios.

1 scenario scenario_name [
2   object object_name_one: Class_Name { 
3     property_name_one = 'this is a string';
4     property_name_two = true;
5     ...
6   }
7   object object_name_two: Class_Name { ... }
8   link association_name {
9     source association_end_name_one = object_name_one; 
10    target association_end_name_two = object_name_two;
11  }
12 ]

Listing 2. Test scenario declaration

A test scenario is composed of a name (e.g., scenario_name), object definitions, and link definitions.

An object definition is composed of a name used for referring to the object (e.g., object_name_one), the type of the object (i.e., a previously imported UML class, e.g., Class_Name), and attribute declarations. An attribute declaration is composed of the name of the attribute (e.g., property_name_one), the assignment operator (‘=’), and a value declaration (e.g., ‘this is a string’). Attribute declarations are separated by using a colon (‘;’). In the test specification language, there are three primitive types available for defining attribute values:

• String: An arbitrary sequence of characters declared between single quotation marks (e.g., ‘this is a string’)
• Boolean: Values true and false
• Integer: Signed integers (…, -1, 0, 1, …)

A link is an instance of an association linking two objects with each other. A link definition is composed of the name of the instantiated association (e.g., association_name), a source object assignment, and a target object assignment. A source/target object assignment is composed of the keyword source/target, followed by the names of the respective association end (e.g., association_end_nam_one), the assignment operator (‘=’), and the name of the object that shall be assigned to the association end (e.g., object_name_one). Please note that for multivalued association ends separate links have to be created for each pair of linked objects.

3. Test Cases

Test cases are the main components of a test suite. A test case defines assertions on the behavior of a UML activity. Listing 3 shows how test cases are defined.

1 test test_name activity Activity_Name ([parameter_node_name = 'value', ...]) 
2 [on context_object_name] {
3   ...
4 }

Listing 3. Test case declaration

A test case is composed of a test name (e.g., test_name), the name of the activity under test (e.g., Activity_Name), an optional list of input parameter value assignments (e.g., parameter_node_name = ‘value’), an optional declaration of a context object (e.g., on context_object_name), and a body. Within the body of a test case an arbitrary number of assertions can be declared. There are two types of assertions: order assertion and state assertion.

3.1 Order Assertions

An order assertion is used for asserting the expected order in which the nodes of the activity under test shall be executed. It is composed of the keyword assertOrder and the list of activity nodes in their expected execution order. To specify the relative order of activity node executions, the jokers ‘_’ and ‘*’ can be used. For skipping exactly one node in an order assertion the joker ‘_’ can be used. For skipping zero or more nodes in an order assertion the joker ‘*’ can be used. An example of an order assertion is shown in Listing 4.

Listing 4. Order assertion declaration

In Listing 4, it is asserted that the first executed node in the activity under test is the node with the name node_one, immediately followed by the node named node_two, after which there might be an arbitrary number of nodes executed (expressed by the joker ‘*’) until the node named node_three is executed. Finally, there is exactly one node executed after the node named node_three, which is also the last executed node (expressed by the joker ‘_’).

It is also possible to assert the order of nodes contained by a called activity, i.e., an activity that is called by the activity under test using a call action. An example is depicted in Listing 5.

Listing 5. Order assertion declaration considering called activities

In Listing 5, it is asserted that when executing the activity under test, first the node node_one is executed, then the node node_two is executed, and then arbitrary many other nodes are executed. Thereby, the node node_one is a call action, calling another activity. Within the execution of the called activity, first the node sub_node_one is executed followed by the execution of the node sub_node_two and the execution of arbitrary many nodes.

3.2 State Assertions

A state assertion is used for asserting the states of the tested model during the execution of the activity under test. It is composed of a temporal expression selecting the states to be asserted and a state expression defining expected properties of the selected states. Listing 6 shows how state assertions are defined.

Listing 6. State assertion declaration

3.2.1 Temporal Expressions

A temporal expression consists of a quantifier, a temporal operator, and either a node contained by the activity under test or an OCL constraint. The temporal operator in combination with a specified activity node or alternatively an OCL constraint selects a set of states to be considered by the state assertion. The temporal quantifier defines in which of the selected states, the state expression of the state assertion should be fulfilled.

Our test specification language supports the temporal operators after and until, which define that all states after or until an activity node has been executed or alternatively an OCL constraint evaluates to true shall be considered by the state assertion.

Furthermore, the temporal quantifiers always, eventually, immediately, and sometimes are supported, which define that the state expression of the state assertion should be fulfilled in all states, eventually in the selected set of states, in the very first selected state, or in at least one state.

Listing 7 shows how temporal expressions are defined.

1 [always|eventually|immediately|sometimes] [after|until] action actionA
2     [until action  actionB]
3 [always|eventually|immediately|sometimes] [after|until] constraint constraint_one
4     [until constraint constraint_two]

Listing 7. Time expression declaration

First, the quantifier is defined (e.g., always) followed by the temporal operator (e.g., after). In case actions are used for selecting states, the keyword action followed by the name of the action is defined (e.g., actionA). In case OCL constraints are used for selecting states, the keyword constraint is used followed by the name of the OCL constraint (e.g., constraint_one). If the temporal operator after is used, an additional until part is available, which enables to define that all states until the execution of a certain action (e.g., actionB) or until the fulfillment of an OCL constraint (e.g., constraint_two) shall be considered.

In the following, the meaning of various combinations of temporal operators and quantifiers is explained in more detail based on the examples given in Figure 1. Figure 1 shows the states S1, S2, S3, and S4 resulting from the execution of the actions action1, action 2, action3, and action4. The values a, b, and c represent the results of evaluating conditions on these states.

(1a) In each state starting from the first state until the state resulting from the execution of action2 (i.e., the states S1 and S2), the state expression c should evaluate to false. This state assertion is fulfilled.

(1b) The state expression c should evaluate to true in at least one of the states from the first state until the state where the condition b becomes true (i.e., the states S1, S2, and S3). This assertion is fulfilled.

(2b) From the first state in which the condition a becomes true, until the first state in which the condition b becomes true (i.e., the states S2 and S3), the state expression c should become true in one state and remain true in each following state. This assertion is fulfilled.

(3a) The state expression c should evaluate to true in the state resulting from the execution of action4 (i.e., the state S4). This assertion is fulfilled.

3.2.2 State Expressions

For checking the state of an object provided as input or output of an action of the activity under test, or as input or output parameter of the activity itself, an object state expression can be defined within a test case. In Listing 8, an example of two object state expressions is presented.

1 actionA.result = null;
2 Activity.result = Scenario_Name.object_name_two;

Listing 8. Object state expression declaration

An object state expression, as presented in line 1 of Listing 8, is composed of the name of an action (e.g., actionA), the name of a pin of this action (e.g., result), an operator (e.g., ‘=’), and a value which is compared with the value provided as input or output of the specified pin. As presented in line 2 of Listing 8, also a value provided as input to or output of an activity parameter node can be checked.

Beside the equality operator ‘=’, also the following operators are available:

• Inequality operator (‘!=’),
• Comparison operators for Integer values (‘>’, ‘<‘, ‘>=’, ‘<=’)
• Inclusion operator for lists (includes) checking whether a list of values provided as input to or output of an action or activity contains the declared value
• Exclusion operator for lists (excludes) checking whether a list of values provided as input to or output of an action or activity does not contain the declared value.

For checking the state of an object provided as input to or output of an action or activity, a property state expression can be defined within a test case. In Listing 9 an example of a property state expression is shown.

Listing 9. Property state expression declaration

Similarly as an object state expressions, a property state expression is composed of the name of an action (e.g., actionA), the name of a pin of this action (e.g., result), a double colon separator (i.e., ‘::’), the name of the property to be checked (e.g., property_name), an operator (e.g., ‘=’), and a value which is compared with the value of the property of the object provided as input to or output of the specified action. Again, also an object provided as input to or output of an activity parameter node can be checked. The operators allowed in property state expressions are the same as the operators allowed for object state expressions.

3.2.3 OCL Constraints

Besides state expressions, the test framework also supports the definition of OCL constraints for asserting the execution states of a model under test. Therefore, OCL constraints defined in a separate OCL file can be invoked by state assertions. An example for invoking an OCL constraint is depicted in Listing 10.

Listing 10. Invocation of an OCL constraint in a test case

An OCL constraint is invoked by its name (e.g., constraint_name) and an optional on declaration, which limits the evaluation of the constraint to a single object (e.g., on actionA.result limits the evaluation to the object provided by the pin actionA.result). If the on part is omitted, the constraint is evaluated on each object in the states selected by the state assertion.

An example of a definition of an OCL constraint is shown in Listing 11.

1 package package_name
2 context Class_Name
3 inv constraint_name: property_name = 'aString'
4 endpackage

Listing 11. Specification of an OCL constraint

An OCL constraint is specified in a separate OCL file and has to be defined in the same package as the classes of the UML model under test (e.g., package_name). Each constraint is defined for a context type (e.g., Class_Name) and has a name (e.g., constraint_name) as well as a body (e.g., property_name = ‘aString’).