detecting defects in object-oriented designs: using ...mvz/mswe609/oort1.pdf · 1 detecting defects...

1

Detecting Defects in Object-OrientedDesigns: Using Reading Techniques to

Increase Software Quality

UFRJCOPPE

Department of Computer ScienceExperimental Software Engineering Group Fraunhofer Center - Maryland

Current Research Team:Prof. Victor R. BasiliForrest Shull, Ph.D.

Guilherme H. Travassos, D.Sc. (1)Jeffrey Carver, Graduate Research Assistant

{basili,travassos,carver}@[email protected]

This work is partially supported by UMIACS and by NSF grant CCR9706151

(1) On leave from the Federal University of Rio de Janeiro - COPPE/Computer Science andSystem Engineering Department, partially supported by CAPES - Brazil

DCS/ESEGOO Reading Techniques

UFRJCOPPE

Evolving a Process for Inspecting OODesigns

Outline:y Reading Techniques

x Families of Reading Techniques

y Reading Techniques for OO Designx First Experiment at UMCP

• Lessons Learned from the Initial Study

y The Road Aheadx Observational Studies

• Lessons Learned from the Observational Studies

y Open Questions

2


UFRJCOPPE

Reading Techniques

Why read?

Software practitioners are taught how to write, buttypically not how to read, software documents

Many software processes assume that practitioners knowhow to effectively find the information they need (i.e. howto read) such documents

e.g. inspection process: read a document to find defects


UFRJCOPPE

Reading Techniques

DomainKnowledge

SoftwareArtifacts

OtherDomain

GeneralRequirements

ambiguity

extraneous

incorrect fact

omission

inconsistency

For what?For what?to deal withto deal with

complete,complete,consistent,consistent,unambiguous, and,unambiguous, and,correctcorrect

documents across thedocuments across thesoftware processsoftware process Software Artifacts:

Requirements Documents,Use-casesScenarios descriptionsDesign DiagramsSource Code…

Increasing QualityIncreasing Quality!!

3


UFRJCOPPE

Reading Techniques

Why read?

looking for defects:

understandingreusinganalyzingconstructingmaintainingtestingreasoning

to:

Defect General DescriptionOmission Necessary information about the system

has been omitted from the softwareartifact.

Incorrect Fact Some information in the software artifactcontradicts information in therequirements document or the generaldomain knowledge.

Inconsistency Information within one part of thesoftware artifact is inconsistent with otherinformation in the software artifact.

Ambiguity Information within the software artifact isambiguous, i.e. any of a number ofinterpretations may be derived that shouldnot be the prerogative of the developerdoing the implementation.

ExtraneousInformation

Information is provided that is not neededor used.


UFRJCOPPE

Reading Techniques

z “Software reading techniques” attempt to increase theeffectiveness of inspections by providing proceduralguidelines that can be used by individual reviewers toexamine (or “read”) a given software artifact and identifydefects

z There is empirical evidence that software reading is apromising technique for increasing the effectiveness ofinspections on different types of software artifacts, notjust limited to source code.

4


UFRJCOPPE

Families of Reading Techniques

Reading

Construction Analysis

ReuseMaintenance DefectDetection

Traceability Usability

Test PlanDesign Code

ProjectSourceCode

CodeLibrary

White BoxFramework Black Box

Framework

Design Requirements Code UserInterface

SCR English Screen Shot

Scope-based Defect-based Perspective-based Usability-based

SystemWide

TaskOriented

Inconsistent IncorrectOmissionAmbiguity Tester UserDeveloper

Novice ErrorExpert

PROBLEMSPACE

SOLUTION

SPACE

Requirements

Use-Cases

Technology

Family

General Goal

Specific Goal

Document(artifact)

NotationForm

Technique


UFRJCOPPE

Reading Techniques for OO Design

Reading

Analysis

DefectDetection

Usability

Design Requirements Code UserInterface

SCR English Screen Shot

Defect-based Perspective-based Usability-based

Inconsistent

IncorrectOmission

AmbiguityTester UserDeveloper

Novice ErrorExpert

Technology

Family

General Goal

Specific Goal

Document(artifact)

NotationForm

Technique

PROBLEMPROBLEMSPACESPACE

SOLUTIONSPACE

OO Diagrams

Traceability

Horizontal Vertical

5


UFRJCOPPE


Abstractions of Information:tracing the semantic consistency of pairsof documents

Procedures:For detecting defects in design diagrams

Uses of Information:detect defects

Goal: To develop a family of reading techniques that can beused to read OO high level design artifacts against

1. themselves, and2. requirements descriptions and use-cases

within a domain in order to identify defects among them.


UFRJCOPPE


UML Artifacts:

Fixed_Rate Loan

risk()principal_remaining()

Variable_Rate Loanprincipal_remaining : number

risk()principal_remaing()

Lendername : texti d : t ex tcontact : textphone_number : number

Borrowername : textid : numberrisk : numberstatus : text

risk()set_status_good()set_status_late()set_status_default()borrower_status()set_status()

Bundle

active time period : dateprofit : numberestimated risk : numbertotal : numberloan analyst : id_numberdiscount_rate : numberinvestor_name : textdate_sold : date

risk()calculate_profit()cost()

Loan Arranger

rec_monthly_report()inv_request()generate reports()identify_report_format()verify_report()look_for_a_lender()look_for_a_loan()identify_loan_by_criteria()manually_select_loans()optimize_bundle()calculate_new_bundle()identify_asked_report()aggregate_bundles()aggregate_loans()aggregate_borrowers()aggregate_lenders()format_report()show_report()

Loanamount : numberinterest rate : numbersettlement data : dateterm : datestatus : textoriginal_value : numberprincipal_original : number

risk()set_status_default()set_status_late()set_status_good()discount_rate()borrowers()principal_remaining()

1

1..*

1

1..*

1..*

1..*

1..*

1..*

1..*

0..1

1..*

0..1

Good

Late

monthly report informing payment on time[ payment time <= due time ]receive a monthly report

Default

monthly report informing late payment[ payment time > due time + 10 ]

monthly report informing late payment[ due time < payment time < due time + 10 ]

monthly report informing late payment[ payment time > due time + 10 ]

monthly report informing payment on time[ payment time <= due time ]

Loan State Diagram

Fanny May : Loan Arranger

Borrower : Borrower

A Lender : Specified Lender

Loan : Loan

verify_report()

new_loan(lender, borrowers)

new_

look_for_a_lender(lender)

look_for_a_loan(loan)

look_for_a_

update_loan(lender, borrower)

update_

lender :

new_lender(name,contact, phone_number)

update(lender)

monthly_report(lender, loans, borrowers)

identify_report_format()

Receive Monthly Report

Specified Lender

Investor

Fanny May

Receive Reports

Monthly Report

Investment Request

Request

Generate Reports

Loan Analyst

Loan Arranger Classes Description

Class name: Fixed_Rate Loan Category: Logical View Documentation: A fixed rate loan has the same interest rate over the entire term of the mortgage

External Documents: Export Control: Public Cardinality: n Hierarchy: Superclasses: Loan Public Interface: Operations: risk principal_remaining

State machine: No Concurrency: Sequential Persistence: Persistent

Operation name: risk Public member of: Fixed_Rate Loan Return Class: float Documentation: take the average of the risks' sum of all borrowers related to this loan if the average risk is less than 1 round up to 1 else if the average risk is less than 100 round up to the nearest integer otherwise round down to 100 Concurrency: Sequential

Loan-Arranger Requirements Specification – Jan. 8, 1999

Background

Banks generate income in many ways, often by borrowing money from their depositorsat a low interest rate, and then lending that same money at a higher interest rate in theform of bank loans. However, property loans, such as mortgages, typically have terms of15, 25 or even 30 years. For example, suppose that you purchase a $150,000 house witha $50,000 down payment and borrow a $100,000 mortgage from National Bank forthirty years at 5% interest. That means that National Bank gives you $100,000 to pay thebalance on your house, and you pay National Bank back at a rate of 5% per year over aperiod of thirty years. You must pay back both principal and interest. That is, the initialprincipal, $100,000, is paid back in 360 installments (once a month for 30 years), withinterest on the unpaid balance. In this case the monthly payment is $536.82. Althoughthe income from interest on these loans is lucrative, the loans tie up money for a longtime, preventing the banks from using their money for other transactions. Consequently,the banks often sell their loans to consolidating organizations such as Fannie Mae andFreddie Mac, taking less long-term profit in exchange for freeing the capital for use inother ways.

6


UFRJCOPPE


Target Artifacts:

RequirementsDescriptions

Use-CasesRequirementsSpecification

High LevelDesign

ClassDiagrams

ClassDescriptions

State MachineDiagrams

InteractionDiagrams

(Sequence)Vertical readingHorizontalreading


UFRJCOPPE


looking for defects:

Type of Defect DescriptionOmission One or more design diagrams that should contain some concept from

the general requirements or from the requirements document do notcontain a representation for that concept.

Incorrect Fact A design diagram contains a misrepresentation of a concept describedin the general requirements or requirements document.

Inconsistency A representation of a concept in one design diagram disagrees with arepresentation of the same concept in either the same or anotherdesign diagram.

Ambiguity A representation of a concept in the design is unclear, and could causea user of the document (developer, low-level designer, etc.) tomisinterpret or misunderstand the meaning of the concept.

ExtraneousInformation

The design includes information that, while perhaps true, does notapply to this domain and should not be included in the design.

Table 1 – Types of software defects, and their specific definitions for OO designs

7


UFRJCOPPE

Reading Techniques for OO DesignFirst Experiment at UMCP

Description:Fall/98 course - CMSC 435http://www.cs.umd.edu/projects/SoftEng/ESEG/manual/tbr_package/

Students organized in 15 teams (14 (3 students team) and 1 (2 studentsteam))

System: financial database application (a system responsible for organizing theloans held by a financial consolidating organization, and for bundling them for resale to investors)

Problem features: small system with low number of classes but withsome design complexity due to non-functional performancerequirements

Target programming languages: C++ or Java

Goal: to evaluate the first version of the reading techniques


UFRJCOPPE


Size Measures for the inspected design:Class Name Attrs. WMC DIT NOC CBO State

Dgm.exists?

Seq1Con-tains?

Seq2Con-tains?

Seq3Con-tains?

Seq4Con-tains?

Seq5Con-tains?

Property 5 0 0 0 1Borrower 2 2 0 0 1Lender 3 1 0 0 2 yesLoan 3 3 0 2 4 yes yesFixedRate Loan

0 1 1 0 4

AdjustableRate Loan

0 1 1 0 4

Bundle 5 2 0 0 2 yes yesInvestmentRequest

4 1 0 0 1 yes

Loan Arranger 0 15 0 0 4 yes yes yes yes yesFinancial Org. 1 0 0 0 2 yes yes yes yes yes yesLoan Analyst 1 12 0 0 3 yes yes yes yes yes

number of attributes (Attrs.), Weighted Methods/Class (WMC), Depth of Inheritance (DIT),Number of Children (NOC), and Coupling Between Objects (CBO)

8


UFRJCOPPE


Reader 1

Reader 2

Reader 3

looking for consistencyhorizontal reading

looking for consistencyhorizontal reading

looking for traceabilityvertical reading

Meet as a team to discuss acomprehensive defect list.Each reader is an “expert” ina different aspect

Final list of all defects sentto designer for repairing

The inspection process with OO reading techniques:


UFRJCOPPE


Subset of metrics collected in the feasibility study

When Collected MetricsBefore the study a) Details on subjects’ amount of experience with requirements, design, and codeAfter individual review b) Time spent on review (in minutes)

c) Opinion of effectiveness of technique (measured by what percentage of thedefects in the document they thought they had found)

d) How closely they followed the techniques (measured on a 3-point scale)e) Number and type of defects reported

After team meeting f) Time spent (in minutes)g) Usefulness of different perspectives (open-ended question)

In post-hoc interviews h) How closely they followed technique (open-ended question, to corroborate d)i) Were the techniques practical, would they use some or all of them again (open-

ended question)

9


UFRJCOPPE


Reading Technique for Class diagrams x Class descriptions For each class modeled into the class diagram, do:1)Read the class description to find an associated description to the class.F Underline with a yellow pen the portion of the Class descriptions corresponding to theclassVerify if:1.1)All the attributes are described and with basic types associatedF Underline them with a blue pen1.2)All the behaviors and conditions are describedF Underline them with a green pen1.3)All the class inheritance relationships are describedF Draw a box around them with a yellow pen if there is any1.4) All the class relationships (association, aggregation and composition) are described with multiplicity indicationF Circle each multiplicity indication with a blue pen if it is correct.

First version of a horizontal reading technique. The emphasis is on syntacticchecking, that is, that the OO notation on both diagrams agrees.


UFRJCOPPE


Some results from the experiment:y The definition of design defects by extending an already existing

defect taxonomyy It is important to measure the effectiveness (number of defects

found):x Subjects using vertical reading tended, on average, to report slightly

more defects of omission and incorrect fact (i.e. of types of defectsuncovered by comparisons against the requirements) than those usinghorizontal reading (6.8 versus 5.4 defects)

x Subjects using horizontal reading tended to report more defects ofambiguity and inconsistency (i.e. of types of defects uncovered byexamination of the design diagrams themselves) than subjects usingvertical reading (5.3 versus 2.9)

10


UFRJCOPPE


Some results from the experiment:y developers agreed that using:

x heuristics to construct OO artifacts is goodx some kind of OO reading technique is worthwhile

y some developers said that they would like to use the sametechniques again but, the mechanisms used to instrument themshould be improved. The study allowed us to identify weaknessesin the first version of the techniques that have led to a secondversion

y It was possible to demonstrate that such reading techniques canbe used as part of design inspections, and do help reviewersdetect defects


UFRJCOPPE

Reading Techniques for OO DesignThe Road Ahead:

1. Improving first version of the reading techniques usingthe subjects’ feedback by:

x inserting semantic concerns into the reading techniques toimprove the detection of defects (incorrect fact, ambiguityand extraneous)

x inserting construction concerns into the reading techniques toallow developers argue about design reasoning

11


UFRJCOPPE

Reading Techniques for OO DesignReading Technique for Class diagrams x Class descriptions

…2) Read the class descriptions to find an associated description to each class on the class diagram.Take a look at the class description. Can this description be used to describe the class that you areconsidering at this time? Is it using an adequate abstraction level?F Box the class name in the class description with a blue pen and write in the same number given tothe class on the class diagram.F Mark found classes with a blue symbol (*) on the class diagramMake good use of this time and verify if:•All the attributes are described along with basic typesCan this class encapsulate all these attributes? Are the attributes associated to feasible/possible basictypes? Does it make sense to have these attributes in the class description? Is some attribute missingbetween the two documents or sounding extraneous?F Write down missing and extraneous attributes with a yellow pen•All the behaviors and constraints are describedCan this class encapsulate all these behaviors? Is some behavior missing between the two documentsor sounding extraneous? Are the behavior descriptions using the same abstraction level? Do thebehaviors use the class attributes to accomplish the procedure? Are the constraints reachable using theattributes and behaviors of the class? Do the constraints make sense for this class? Are the constraintsdepending upon some class attributes? Were they described?F Write down missing/extraneous behaviors with a green penF Write down missing/extraneous constraints with a yellow pen

…


UFRJCOPPE


The Road Ahead:2. Assessment issues:

x Better metrics for assessing inspections effectiveness bylooking for a way to take account “false positives” (i.e. itemsthat are reported by reviewers but which should really not beconsidered defects)

x Developing qualitative as well as quantitative means ofassessment. Qualitative methods would help us understandthe process as well as the results.

12


UFRJCOPPE

Reading Techniques for OO DesignThe Road Ahead

y Observational Study:x we use the term “observational” to define a setting in which an

experimental subject performs some task while theexperimenter gathers data about what exactly the subjectdoes.

y Purpose:x to collect data about how the particular task is accomplishedx observational data

• Time taken• Problems encountered

x Inquisitive data• Influence of prior knowledge


UFRJCOPPE

Reading Techniques for OO DesignLessons Learned from the Observational Studies

y Observational datax Time taken:

• time for applying the techniques is influenced by prior experienceof the reader, complexity of the information, order of reading stepsand levels of abstraction

x Problems encountered:• organizational and semantic issues, format of software artifacts

y Inquisitive datax Influence of prior knowledge:

• domain knowledge seemed to be necessary to support horizontalreading

• basic OO knowledge seemed to be necessary

13


UFRJCOPPE

Reading Techniques for OO DesignThe Road Ahead:

3. Improving defect taxonomyx making use of expert opinion (where experts include skilled

practitioners and researchers) to understand if the taxonomycaptures the important types of defects in OO designs

4. Empirically validating the reading techniquesx doing experiments with specified projects and real developmentx Experiment: http://www.cs.umd.edu/class/fall1999/cmsc735/

5. Packaging and replicating the experiments in other placesx identifying the applicability of such reading techniques in different

development environmentsx Technical Report: ftp://ftp.cs.umd.edu/pub/papers/papers/ncstrl.umcp/CS-

TR-4070/CS-TR-4070.ps.Z


UFRJCOPPE


Open Questions:

1. What is the role of domain knowledge for these twosets of reading techniques?

Horizontal x Vertical reading

2. What is the adequate level of automated support thatshould be provided for such techniques?

Some steps are repetitive and mechanical to the readerClerical activities need to be identified