User Study
In order to assess the ease of use and usefulness of our testing framework, we have performed a user study with eleven participants. The user study was organized in four steps:
- An introduction into fUML and our testing framework.
- A questionnaire concerning the skills of the participants in using languages relevant for the user study.
- Two tasks that had to be completed by the participants with our testing framework.
- A questionnaire concerning the subjective opinion of the participants about the ease of use and usefulness of our testing framework.
1. Introduction
At the beginning of the user study, the participants were given a short introduction into fUML and our testing framework. This introduction included:- The most important concepts provided by fUML for modeling the structure and behavior of a system.
- The most important concepts provided by our test specification language for evaluating the functional correctness of UML activities.
- The content of the test results obtained from executing and evaluating a test suite.
2. Skills Questionnaire
After the introduction, we handed out a skills questionnaire to the participants, in order to assess their previous knowledge on UML, fUML, OCL, and unit testing. A summary of the results of the questionnaire is provided in Table 1.
| Language | no experience | beginner | average | expert |
| UML | 8 | 3 | ||
| UML class diagrams | 5 | 6 | ||
| UML activity diagrams | 8 | 3 | ||
| UML action language | 3 | 3 | 3 | 2 |
| fUML standard | 2 | 3 | 3 | 3 |
| OCL | 1 | 5 | 2 | 3 |
| Unit testing (e.g., JUnit) | 1 | 3 | 6 | 1 |
Among the participants we had post doctoral, doctoral, and master students with varying levels of skills. All of the participants had good previous knowledge of UML being slightly more experienced with UML class diagrams than with UML activity diagrams. When it comes to fUML, we had a balanced sample of participants. Most of the participants rated their previous knowledge of OCL at the beginner level and their knowledge of unit testing at an average level.
3. Tasks Performed by Participants
For the purpose of performing the user study, we have used a model of an online store application. The participants were given two tasks that they had to complete by using our testing framework.
Task 1. The first task was composed of two correct activities, a brief description of each activity, and requirements on the activities in plain English language. The participants had to define several test cases for verifying the functional correctness of the given activities in terms of the fulfillment of the given requirements.
Task 2. The second task was composed of an activity with two defects, two test cases asserting the correct behavior of the activity, and the test results of both test cases. The participants had to detect the defects and suggest corrections if possible by using the given material (i.e., the activity, test cases, and test results).
4. Opinion Questionnaire
At the end of the user study, a questionnaire was given to the participants concerning their subjective opinion of the ease of use and usefulness of our testing framework. A summary of the results of this questionnaire is depicted in Table 2.
| Task | very easy | easy | medium | hard | very hard |
| Read class diagrams | 7 | 4 | |||
| Read activity diagrams | 3 | 7 | 1 | ||
| Write test cases | 8 | 3 | |||
| Read test cases | 3 | 4 | 2 | 2 | |
| Read test results | 3 | 4 | 2 | 2 | |
| Correct activity diagrams | 1 | 3 | 2 | 2 | 3 |
Table 2: Results of the opinion questionnaire