Osaka University
An An ExperimentalExperimental Comparison Comparisonof Checklist-Based Readingof Checklist-Based Reading
and Perspective-Based Readingand Perspective-Based Readingfor UML Design Document Inspectionfor UML Design Document Inspection
Giedre Sabaliauskaite*Fumikazu Matsukawa**
Shinji Kusumoto**Katsuro Inoue**
*Graduate School of Engineering Science, Osaka University**Graduate School of Information Science and Technology, Osaka University
2
Osaka University
ContentsContentsBackgroundResearch GoalsExperiment description
Reading techniques Experimental design Data analysis Results
Conclusions and Further Work
3
Osaka University
Software InspectionSoftware InspectionSoftware inspection is an effective and efficient method to
detect defects in early stages of software development life-cycle
Software inspection usually consists of several activities [1]
[1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31.
Planning
Defect detection
Defect collection
Defect correction
Key activity – defects are detected
Reading techniques are used to guide inspectors
4
Osaka University
Reading techniquesReading techniquesAd hoc and Checklist-based Reading are the most popular [1]Ad hoc: no guidance for inspectorsChecklist-based Reading (CBR): ChecklistScenario-based Reading is a more recent approach: ScenarioPerspective-based Reading (PBR) [2] is one of Scenario-based
approaches Software product is inspected from different perspectives (designer,
tester, etc.) Provides inspectors with different Scenario for each perspective
We analyzed CBR and PBR inspection techniques during experiment
[1] O. Laitenberger, J.M. DeBaud, “An encompassing life cycle centric survey of software inspection”, The Journal of Systems and Software, vol. 50, no. 1, 2000, pp. 5-31.[2] V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, M.V. Zelkowitz, “The Empirical Investigation of Perspective-Based Reading”, Empirical
Software Engineering: An International Journal, vol.1 , no. 2, 1996, pp. 133-164.
5
Osaka University
Related researchRelated researchSeveral works are done in the area of Unified Modelling
Language (UML) diagram inspectionOne of the works is described in [1]
Experimental comparison of CBR and PBR 18 subjects (practitioners) and 2 software systems UML diagrams inspected – Class, State, Sequence, Collaboration Results - PBR 3-person teams exhibited higher effectiveness and
lower cost per defect than CBR teams
[1] O. Laitenberger, C. Atkinson, M. Schlich, K. El Emam. An experimental comparison of reading techniques for defect detection in UML design documents. The Journal of Systems and Software, 53: 183-204, 2000.
Authors of different studies came to the conclusion that UML inspections need to be further investigated
6
Osaka University
Research GoalsResearch Goals
Conduct UML diagram inspection experiment in Osaka university Language used during experiment - Japanese
Compare Reading techniques CBR and PBR with respect to Defect detection effectiveness Cost per defect Time spent on inspection
Compare both individual inspector results and team resultsAcquire experience for future inspection
7
Osaka University
Structure of CBR and PBR techniques, Structure of CBR and PBR techniques, used during experimentused during experiment
CBR Diagram-specialized Checklist developed Included 20 questions
PBRThree perspectives: User, Designer, ImplementerThree scenarios (one for each perspective) developedScenarios consisted of several steps, each of them included
Explanations Tasks to perform Questions to answer
8
Osaka University
Assignment of UML diagramsAssignment of UML diagramsAll inspectors were given system requirement description and Use-
Case diagramsDefects were injected into Activity, Class, Sequence and Component
diagrams
DocumentSeminar system
Hospital system
CBR PBR perspective
User Designer Implementer
System requirement description
1 1
Use-Case diagrams 1 1
Activity diagrams 8 7
Class diagrams 1 1
Sequence diagrams 12 7
Component diagrams 1 1
9
Osaka University
Experiment designExperiment designSubjects
59 third year students of the Software Development course, Osaka University
Objects Seminar information system ( 24 pages ) Hospital information system ( 18 pages )
PBR
CBRUser Designer Implementer
Seminar system 7 6 6 11
Hospital system 7 6 6 10
10
Osaka University
DefectsDefectsThree types of defects inserted into UML diagrams
Syntactic – missing or unnecessary information Semantic – misrepresentation of a concept, or unclear
representation Consistency – inconsistency between UML diagrams
In total, 15 defects inserted into the each software system 4 into Activity diagrams 3 into Class diagrams 5 into Sequence diagrams 3 into Component diagrams
11
Osaka University
Experimental HypothesesExperimental Hypotheses
H01: subjects who use PBR technique should spend less time on inspection
H02: subjects who use PBR should have higher cost per defect
H03: individual defect detection effectiveness should be different between CBR and PBR inspectors
H04: team defect detection effectiveness should be different between CBR and PBR inspector teams
12
Osaka University
Data analysisData analysis
Two types of analysis Individual data analysis
Defect detection effectiveness Time spent on inspection Cost per defect
Team data analysis Subjects assigned into 3-person virtual teams Effectiveness of CBR and PBR teams compared
13
Osaka University
Percentage of inspectors, who have detected Percentage of inspectors, who have detected defects relevant to different perspectivesdefects relevant to different perspectives
50%
60%
70%
80%
Allperspectives
User Designer Implementer
CBRPBR
Both CBR and PBR exhibited similar results
14
Osaka University
Defect detection effectiveness (according Defect detection effectiveness (according to defect types)to defect types)
50%
60%
70%
80%
90%
Syntactic Semantic Consistency
CBR
PBR
Syntactic defects: CBR more effective
Semantic and Consistency defects: PBR more effective
15
Osaka University
Defect detection effectiveness (according Defect detection effectiveness (according to UML diagram types)to UML diagram types)
50%
60%
70%
80%
90%
Class D. Activity D. Sequence D. Component D.
CBRPBR
Class Diagrams: CBR more effective
Activity and Component Diagrams : PBR more
effective
Sequence Diagrams: CBR and PBR exhibited
similar effectiveness
16
Osaka University
Time spent on inspection, Cost per defect Time spent on inspection, Cost per defect and Defect detection effectivenessand Defect detection effectiveness
Independent samples t-testMann-Whitney test
• Significant difference in Inspection Time and Cost • No significant difference in Effectiveness
Variables Reading technique Mean SD
Time spent on inspectionCBR 62.9 11.7PBR 51.351.3 15.1
Cost per defectCBR 6.26.2 1.6PBR 10.2 3.5
EffectivenessCBR 70.2 11.5
PBR 69.1 15.3
PBR: 18% (11 min) less time spent on inspection
CBR: 39% (4 min / defect) lower cost per defect
CBR Effectiveness ≈PBR Effectiveness
17
Osaka University
Virtual teamsVirtual teamsSubject assignment into 3-person virtual teams
Any three CBR reviewers are included into team One reviewer using each of the three PBR
perspectives included into teamC9C8C7
C6C5C4
C3C2C1
CBR group
I5D5U5
I4D4U4
I3D3U3
I6D6U6
I2D2U2
I1D1U1
PBR groupGrouping virtual teams into statistically
independent groups CBR groups: 3 teams included (11 inspectors
for Seminar system, 10 subjects for Hospital system)
PBR groups: 6 teams included (7 Users, 6 Designers, 6 Implementers for each systems)
18
Osaka University
Comparisons between CBR and PBR teamsComparisons between CBR and PBR teams
C1 C2 C3
C4 C5 C6
C7 C8 C9
CBR group 1
U1 D1 I1
U2 D2 I2
… … …
U6 D6 I6
PBR group 1
U1 D1 I6
U2 D2 I1
… … …
U6 D6 I5
PBR group 2
…
CBR PBRCom
parisons
…
C1 C2 C3
C4 C5 C6
C7 C8 C10
CBR group 2
…
01016064000!6
!6!6
!3673437310
PCCC
05588352000!6
!6!6
!3673538311
PCCC
Hospital system
Seminar system
Number of comparisons:
19
Osaka University
Team data analysis (1/2)Team data analysis (1/2)CBR and PBR teams were compared with respect to Team defect
detection effectiveness
Software system
Team defects detection effectiveness
CBR>PBR PBR>CBR CBR=PBR
Seminar 54 753 326 68854 753 326 688 585 743 712 544 449 600
Hospital 10 160 482 17610 160 482 176 8 064 149 760
100 defects ofnumber Total
members team by found defects unique ofNumber _ esseffectivenTeam
Number or comparisons, in which either CBR or PBR was more effective, or both techniques showed similar effectiveness
CBR teams more effective than PBR teams
20
Osaka University
Team data analysis (2/2)Team data analysis (2/2)
Comparison typeSoftwaresystem
Defect detection effectiveness
CBR > PBR PBR > CBR CBR = PBR
All defects includedSeminar 76315579687631557968 0 0
Hospital 72338738487233873848 0 0
Syntactic defect omitted
Seminar 69681600006968160000 0 0
Hospital 8512200085122000 0 0
T-test with significance level of 2.5% was used to evaluate in which comparisons there was a significant difference between CBR and PBR team effectiveness
Two types of comparisons made All defects included Syntactic defects omitted
CBR teams more effective than PBR teams
21
Osaka University
Experiment resultsExperiment resultsIndividual inspector results
Inspectors who use PBR spend on average 18% less time on inspection
Cost per defect of Inspectors who use PBR is on average 39% higher
There is no statistically significant difference in defect detection effectiveness between individual CBR and PBR inspectors, however on average
• PBR is more effective for Semantic and Consistency defects• PBR is more effective for Activity and Component diagrams• CBR is more effective for Syntactic defects• CBR is more effective for Class diagrams
Three-person virtual team results CBR teams exhibit higher defect detection effectiveness than
PBR teams
22
Osaka University
Threats to validityThreats to validityInternal validity
Selection: experiment was a mandatory part of the course Instrumentation: objects which we used could have influence
External validity Students were used as subjects Design documents were similar to those which are used in
practice, but the size of systems in industry is usually largerThere were threats to internal and external validity, but they
were not considered large in this experiment
23
Osaka University
Conclusions and Further researchConclusions and Further researchUML diagram inspection experiment was conductedThe results of the experiment indicate that
There is no statistically significant difference in effectiveness of CBR and PBR individual inspectors
CBR virtual teams are more effective than PBR virtual teamsFuture research will be directed to further investigation of
UML diagram inspection A replication of experiment was conducted in July 2002, during
which real team meeting were performed More detail analysis of both individual and team data Research of Fault Content Estimation methods
24
Osaka University
Software Engineering LaboratorySoftware Engineering Laboratory
http://iip-lab.ics.es.osaka-u.ac.jp/http://iip-lab.ics.es.osaka-u.ac.jp/
26
Osaka University
Experiment Variables and HypothesesExperiment Variables and Hypotheses
Independent variablesReading technique (CBR and
PBR)
Dependent variablesNumber of defects foundTime spent on inspectionDefect detection effectivenessCost per defect
Null HypothesesFor individual inspectors
H01: PBR Time > CBR Time
H02: PBR Cost < CBR Cost
H03: PBR Effectiveness = CBR Effectiveness
For 3-person virtual teamsH04: PBR Effectiveness = CBR
Effectiveness
27
Osaka University
Information systems inspectedInformation systems inspectedSeminar information system
System supports the process of a company which organizes seminars
Includes activities from seminar planning to issuing the qualification certificate
Designs relationships among personnel of seminar company, participants, lecturers, etc.
Hospital information system System supports the process of a hospital Includes activities such as patient registration, medical examination,
treatment, prescription of the medicines, etc. Designs the relationships among the personnel of a hospital,
patients, pharmacists, etc.
28
Osaka University
Collected dataCollected dataTwo types of data collected during experiment
Defect data Time data
Number of inspectors
Defects Inspection time (min)
Number of detectable
defects
Average number of
defects detected
max/min Average max/min
Seminar information system
PBR User 7 7 4.43 6 / 3 60.43 90 / 46
PBR Designer 6 6 5.00 6 / 4 65.50 80 / 51
PBR Implementer 6 9 6.50 9 / 5 76.67 95 / 40
CBR 11 15 10.55 13 / 8 74.64 90 / 62
Hospital information system
PBR User 7 7 4.43 6 / 3 48.29 70 / 25
PBR Designer 6 6 3.83 5 / 3 59.17 73 / 30
PBR Implementer 6 9 6.33 7 / 5 63.30 77 / 44
CBR 10 15 10.50 12 / 8 70.10 94 / 60
29
Osaka University
CBR ChecklistCBR ChecklistCHECKLISTLocate the following diagrams: Class Diagrams, Activity Diagrams, Sequence Diagrams and
Component Diagrams. Answer the questions related to corresponding diagrams. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have any comments write them in Comment form.
…
SEQUENCE DIAGRAMS
10. Aren’t there any inconsistencies among Sequence Diagram, Class Diagram and Requirements Specification?
yes no
11. Are all the necessary Objects and Messages defined? yes no
12. Is every Class from Class Diagram included in any of Sequence Diagrams and vice versa?
yes no
13. Is every Use Case from Use-Case Diagram implemented in at least one of Sequence Diagrams?
yes no
14. Are there no redundant elements (Objects of Messages) in Sequence Diagram?
yes no
…
30
Osaka University
PBR ScenarioPBR ScenarioDESIGNER SCENARIOAssume that you are the Designer of the system. The concern of the designer is to ensure that the
designer’s needs are completely satisfied in the following documents: Requirement specification, Class Diagram and Sequence Diagram. During the inspection process you will need to inspect those documents in order to detect defects in Class and Sequence Diagrams from designer’s point of view.
Please perform tasks following Step1 through Step3. For each step you must locate corresponding documents, follow the instructions and answer the given questions. When you detect a defect, mark it on the diagram and fill the necessary information in the Defect registration form. If you have some comments write them in Comment form.
Step 3 Locate the Class Diagram and Sequence Diagrams
Make a list of all Objects included in Sequence Diagrams and answer the following questions.
3.1. Are all the Objects of Sequence Diagrams defined in Class Diagrams?
Are all Classes of Class Diagrams defined in Sequence Diagrams?
3.2. Are all the Messages between objects Sequence Diagram defined as Methods of the corresponding Class in Class Diagram?
3.3. Are there no redundant elements in Sequence diagrams?
32
Osaka University
Software development processSoftware development process The main steps of software development process
1. Development of Use-case diagrams2. Describing system activities in Activity diagrams3. Defining static structure of the system in Class diagrams 4. Modelling dynamic aspects of the system in Sequence diagrams 5. Detailed description of object states in Statechart diagrams6. Development of the Component diagrams
User’s perspective in our experiment covered the second, and partially the third and the fourth steps
Designer’s perspective covered the third and the fourth steps Implementer’s perspective covered the sixth step, and partially the
third and the fourth steps