User Study

In order to assess the ease of use and usefulness of our testing framework, we have performed a user study with eleven participants. The user study was organized in four steps:

  1. An introduction into fUML and our testing framework.
  2. A questionnaire concerning the skills of the participants in using languages relevant for the user study.
  3. Two tasks that had to be completed by the participants with our testing framework.
  4. A questionnaire concerning the subjective opinion of the participants about the ease of use and usefulness of our testing framework.

1. Introduction

At the beginning of the user study, the participants were given a short introduction into fUML and our testing framework. This introduction included:
  • The most important concepts provided by fUML for modeling the structure and behavior of a system.
  • The most important concepts provided by our test specification language for evaluating the functional correctness of UML activities.
  • The content of the test results obtained from executing and evaluating a test suite.

2. Skills Questionnaire

After the introduction, we handed out a skills questionnaire to the participants, in order to assess their previous knowledge on UML, fUML, OCL, and unit testing. A summary of the results of the questionnaire is provided in Table 1.

Language no experience beginner average expert
UML 8 3
UML class diagrams 5 6
UML activity diagrams 8 3
UML action language 3 3 3 2
fUML standard 2 3 3 3
OCL 1 5 2 3
Unit testing (e.g., JUnit) 1 3 6 1
Table 1: Results of the skills questionnaire

Among the participants we had post doctoral, doctoral, and master students with varying levels of skills. All of the participants had good previous knowledge of UML being slightly more experienced with UML class diagrams than with UML activity diagrams. When it comes to fUML, we had a balanced sample of participants. Most of the participants rated their previous knowledge of OCL at the beginner level and their knowledge of unit testing at an average level.

3. Tasks Performed by Participants

For the purpose of performing the user study, we have used a model of an online store application. The participants were given two tasks that they had to complete by using our testing framework.

Task 1. The first task was composed of two correct activities, a brief description of each activity, and requirements on the activities in plain English language. The participants had to define several test cases for verifying the functional correctness of the given activities in terms of the fulfillment of the given requirements.

Task 2. The second task was composed of an activity with two defects, two test cases asserting the correct behavior of the activity, and the test results of both test cases. The participants had to detect the defects and suggest corrections if possible by using the given material (i.e., the activity, test cases, and test results).

We observed the participants while they were performing the tasks in order to find out how easy it is to use our test specification language for defining test cases as well as how useful test results are for locating and correcting defects in activities.

4. Opinion Questionnaire

At the end of the user study, a questionnaire was given to the participants concerning their subjective opinion of the ease of use and usefulness of our testing framework. A summary of the results of this questionnaire is depicted in Table 2.

Task very easy easy medium hard very hard
Read class diagrams 7 4
Read activity diagrams 3 7 1
Write test cases 8 3
Read test cases 3 4 2 2
Read test results 3 4 2 2
Correct activity diagrams 1 3 2 2 3

Table 2: Results of the opinion questionnaire