1/28

Seminar Software Quality and Safety

Topic: “Comparing design models resultant from two processes: with and without Architecture – an empirical study“

Author: Lukas Banach

Tutors: Nazim MadhavjiRemo Ferrari

Thursday 17.07.2008

Motivation Task Context Variables Instrument Results

2/28

Overview

• Motivation • Task description• Study context• Static / dynamic attributes• Instrument to measure attributes• Preliminary results• Future Work

Motivation Task Context Variables Instrument Results

3/28

Motivation

• Someone wants to create lower-level models• Does architecture help?

• Do we produce higher quality (faster / more precise, ...) with architecture?• What are the costs and the benefits?

• There is a huge gap between research and the industry

• Useful to say: use architecture because of ...

Motivation Task Context Variables Instrument Results

4/28

Motivation

• Answer research question:

“How do lower-level design models constructed with a software architecture compare to lower-level design models that are not based on an architecture?“

Motivation Task Context Variables Instrument Results

5/28

My task description

• My focus is to establish the methology of study to answer this question

• Dependent variables, instument, statistical analysis• Prepare an empirical study• No study – beyond the scope of this seminar

Motivation Task Context Variables Instrument Results

6/28

Basic idea of an empirical study

Independent variable (1)

Dependent variable (292)Dependent variable (38)Dependent variable (35)Dependent variable (436)Dependent variable (007)

Independent variable (0)

Dependent variable (112)Dependent variable (82)Dependent variable (2)Dependent variable (93)Dependent variable (4)

Binary variable (0/1)

influence

Motivation Task Context Variables Instrument Results

7/28

Where are the models from?

• Participants: ungraduate students• Randomly assigned by the staff

• Their task: model the behaviour of a garage door

• Two types of groups• With and without architecture

Motivation Task Context Variables Instrument Results

8/28

Where are the models from?

• 6 groups ouf of 19• XML data (Models)• Task description• All created documents

Motivation Task Context Variables Instrument Results

9/28

Uppaal design models

• Templates• States• Transitions • Xml – Files

• Simulator• Verifier

Motivation Task Context Variables Instrument Results

10/28

Study variables

• Independent variable (binary)• Architecture or no architecture

• Dependent variables• Static attributes• Dynamic attributes

Motivation Task Context Variables Instrument Results

11/28

Static attributes

• Number of templates• Number of states • Number of transitions• Size of the xml file • Number of available features• Detail level of the available features• Complexity Level of the templates

Motivation Task Context Variables Instrument Results

12/28

Dynamic attributes

• Percentage of accessible states • Percentage of correct scenarios• Deadlocks • Infinite loops

Motivation Task Context Variables Instrument Results

13/28

Instrument to measure attributes

• Xml parser• Counting states, transitions, templates & complexity level• Creates queries to check accessible states (E <>)

• Checklist• available features and detail level

Motivation Task Context Variables Instrument Results

14/28

Instrument to measure attributes - Xml parser

Motivation Task Context Variables Instrument Results

15/28

Instrument to measure attributes

• Verifier• Accessible states (E <>)• Deadlocks (E<> deadlock)

• Scenarios• Available features

• Simulator

Motivation Task Context Variables Instrument Results

16/28

Instrument to measure attributes - verifier

Motivation Task Context Variables Instrument Results

17/28

Instrument to measure attributes - simulator

Motivation Task Context Variables Instrument Results

18/28

Collected dataArchitecture teams No architecture teams

Team1 Team3 Team4 Team11 Team12 Team13

Size 73 KB 19 KB 60 KB 55 KB 39 KB 123 KB

Templates 18 9 16 15 28 23

States 167 46 75 59 72 226

Transitions 317 80 155 155 152 412

Complexity Level 2 (states or transitions >=10) 8 2 6 4 4 12

Complexity Level 1 (states and transitions <10) 10 7 10 11 24 11

E<> Queries (out of Memory) 53/54(0) 1/35(0) 76/76(0) 36/53(17) 35/35(0) 48/48(78)

E<> deadlock no yes o.o.m yes o.o.m o.o.m

Motivation Task Context Variables Instrument Results

19/28

Collected data - features

Architecture teams No architecture teams

Feature Team1 Team3 Team4 Team11 Team12 Team13

1a yes (1) no yes (1) yes (1) yes (1) yes (1)

1b yes (1) no yes (1) yes (1) yes (1) yes (1)

1c yes (1) no yes (1) yes (1) yes (1) yes (1)

1d yes (only with key) no yes (2) yes (1) no no

1e yes (1) no yes (1) yes (1) yes (1) yes (1)

1a: Opening the garage door by user pressing the button.

1e: All operations should involve displaying messages on the control unit which would be located in the garage.

Motivation Task Context Variables Instrument Results

20/28

Collected data - scenarios

1: User presses the garage door button to open the garage. While the door is opening, the user presses the button. The user presses the button again; however, there is an obstacle in the way of the door going down.

Architecture teams No architecture teams

Scenario Team1 Team3 Team4 Team11 Team12 Team13

1 yes no yes no yes yes

2 yes no no no no yes

3 no no no no no no

4 no no no no no no

5 no no yes no no no

Motivation Task Context Variables Instrument Results

21/28

Statistical analysis - total

Total

0,00

100,00

200,00

300,00

400,00

500,00

600,00

Size Templates States Transitions

Architecture teams

No architecture teams

Motivation Task Context Variables Instrument Results

22/28

Statistical analysis - percentage

Percentage

0

10

20

30

40

50

60

Architecture teams No architecture teams

Correct features in %

Correct scenarios in %

Motivation Task Context Variables Instrument Results

23/28

Statistical analysis - complexity

Complexity of architecture groups

Complexity Level 2(states ortransitions >=10)

Complexity Level 1(states andtransitions <10)

Complexity of no architecture groups

Complexity Level 2(states ortransitions >=10)

Complexity Level 1(states andtransitions <10)

Motivation Task Context Variables Instrument Results

24/28

Statistical analysis - correlation

-0,4

-0,2

0

0,2

0,4

0,6

0,8

1

Correlation(#Transition -> Features)

Correlation(#States ->Features)

Correlation(#Transition -> Scenarios)

Correlation(#States ->Scenarios)

Architecture

No architecture

Motivation Task Context Variables Instrument Results

25/28

Statistical analysis - significance

• χ2-Test (Chi-Square-Test)

• H1: The correlation between the attribute “Architecture“ and the number of correct Scenarios is statistically significant.

• H0: The attribute “Architecture“ and the number of correct Scenarios are statistically not correlated• χ2 = 0.1893

Motivation Task Context Variables Instrument Results

26/28

Summary

• Attributes

• Created instrument

• Collect preliminary data

• Statistical analysis

Motivation Task Context Variables Instrument Results

27/28

Future work

• Improve scenarios (yes/no -> stopped at )• Create more queries with the xml parser• For the study:

• Use professional people instead of students

Motivation Task Context Variables Instrument Results

28/28

The end

Questions ?

29/28

χ2

  A nA

S

3.5 4 3.5 3 7

0.116 0.133 0.116 0.1 0.233

nS

11.5 11 11.5 12 23

0.383 0.366 0.383 0.4 0.766

15   15   30

χ2 = (3.5 – 4)² / 4 + (3.5 – 3)² / 3 + (11.5 – 11)² / 11 + (11.5 – 12)² / 12χ2 = 0.1893

χ2 df = 1:5% = 3.84χ2 df = 1:1% = 6.68χ2 df = 1:0.1% = 10.8