find requirements defects to build better software

BW6 Session 6/5/2013 2:15 PM

"Identifying Requirements Defects"

Presented by:

John Terzakis Intel Corporation

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ [email protected] ∙ www.sqe.com

John Terzakis Intel

John Terzakis has more than twenty-five years of experience developing, writing, and testing software. With Intel for fourteen years, John is currently a staff engineer working with teams on enhancing product requirements to reduce planning and development times, minimize defects, and improve overall product quality. He is a certified Intel instructor for Requirements Engineering courses. John’s prior experience includes director and manager roles with Shiva, Racal InterLan, and Dataproducts. He was also a member of the technical staff at Bell Labs. John is a fellow with the International Academy, Research and Industry Association (IARIA).

Find Requirements Defects to Build Better Software

John Terzakis

Version 1.0

Intel Corporation

[email protected] 5, 2013

Better Software ConferenceLas Vegas, NV

Legal Disclaimers

Intel Trademark Notice:Intel and the Intel Logo are trademarks of Intel Corporation in the U.S. and other countriesand other countries.

Non-Intel Trademark Notice:*Other names and brands may be claimed as the property of others

Copyright © 2013 Intel Corporation. All Rights Reserved.No part of this presentation may be copied without the written

permission of Intel Corporation.2

Agenda

• Defects and Their Impact on Software Projects

• Common Requirements Problems

• Techniques to Find Requirements DefectsTechniques to Find Requirements Defects

• Examples

• Wrap up



Defects and Their Impact on Software Projects



What is a Defect?

A defect is the result of some mistake during work product creation

These mistakes come primarily from two sources:

• Lack of knowledge

• Lack of attention (usually due to schedule pressure)


permission of Intel Corporation.

to schedule pressure)

This session will focus on how to identify requirementsdefects so they don’t propagate into the software

5

The Requirements Defect ProblemArchitecture

Specifications

DesignDocuments

Requirements

Documents

Test Plans

Documentation

Code



Documentation& Help Files

Requirements Defects Propagate Downstream

6

Requirements DefectsRequirements defects account for the vast majority of the total cost of all defects – often 70% or more (Leffingwell & Widrig, 2003)

• Requirements defects are often the most expensive• Requirements defects are often the most expensive defects because requirements form the basis of so many other work products

• Requirements defects account for up to 40% of many projects’ total budget (Leffingwell & Widrig, 2003)



SW teams commonly spend too much time and money developing the wrong thing!

Impact of Requirements Defects on Projects

Poor requirements accounted for 41-56% of errors discovered, and 5 of the top 8 reasons for project failure (The CHAOS Report, 1995)

IBM and Bell Labs studies show that 80% of all product defects are inserted at the requirements definition stage (Hooks and Farry, 2001)

Requirements errors consume from 28% to more than 40% of a typical project's budget (Hooks and Farry, 2001)



122% average schedule overrun, 45% of delivered functions never used (Standish Group Report, 1995)

Relative Cost to Correct a Defect

Phase Relative Cost (avg.)Inspection 1Inspection 1Design & Coding 10xTesting 25xProduction 100x or greater



Correcting defects earlier in the SW lifecycle pays huge dividends

Common Requirements Problems



Common Types of Defects in RequirementsRequirements contain defects because of the following common problems:

• Missing information

• Missing triggers

• Implicit collections

• Lack of a consistent syntax

• Weak words



• Unbounded lists

• Ambiguity

Anything that causes the reader to guess is likely to produce a defect

Missing Information

Requirements that are missing information are incomplete.

They cause the reader to make their own interpretation of intent of the requirement and thus introduce the potential f ffor defects.

Examples:

• The software shall display a list of TBD users.

• The software shall comply with the latest Windows®

S



SDK.

Missing information may be due to schedule pressure or author “laziness”

Missing Triggers

Requirements that state a fundamental property of the software or occur all the time are called ubiquitous.

Most requirements are not ubiquitous. Requirements written ubiquitously may actually be missing triggers (events, states or optional features needed for them to execute).

Examples:

• The software shall sound the alarm



The software shall sound the alarm.

• The software shall blink the low battery light.

Under what states or triggers do these occur?

Lack of a Consistent Syntax

Requirements that lack a consistent syntax cause the reader to potentially miss key information or misinterpret the intent of the requirement.

Examples:

• An invoice shall be created when the customer places an order.

• The software shall turn on the light when there is an AC



gpower failure condition provided the “visible error indicator” option has been selected.

Implicit Collections

Implicit collections are sets of objects within requirements are not explicitly defined anywhere

Without a definition readers may assume an incorrectWithout a definition, readers may assume an incorrect meaning

Example:

• The software must support 802.11 and other network protocols supported by competing applications under



protocols supported by competing applications under Linux.

15

Weak Words

Weak words are subjective or lack a common or precise definition. Examples include:

• Quick, Quickly • Normal #1 overused weak word:S t• Easy, Easily

• Timely• Fast• Frequently• Intuitive• Feel, Feeling

• Reliable• State-of-the-art• Effortless• Friendly, User-friendly• Secure• Immediate

• Support



Example:

• The software shall install quickly and effortlessly under normalconditions using a user-friendly and intuitive UI.

Unbounded Lists

An unbounded list is one that lacks a starting point, an end point, or both. Examples include:

• At least• Including, but not limited to• Or later• Such as

Unbounded lists are impossible to design for or test against

Examples:



p• The SW shall must maintain a list of at least 250 users• The SW shall install on Windows® 7 or later in under 5

seconds

Ambiguity

Ambiguity occurs when a word or statement has multiple meanings or there is doubt about the meaning.

These problems create ambiguity:These problems create ambiguity:• Vagueness• Subjectivity• Incompleteness• Optionality• Under-specification• Under-reference

• Over-generalization• Non-intelligibility• Coordination ambiguity• Passive voice• Time-logic confusion• Incomplete logic



• Under-reference • Incomplete logic

See Backup slides for examples

Ambiguity Exercise

What does “The bank is green” mean to you?

The bank is green (eco-friendly)

The bank is green (paint color)

www.shutterstock.com www.fumare.com



The river bank is green (lush).

The pool table bank is green (the felt color)

. www.on.aol.com www.debtshepherd.com

Techniques to Find Requirements Defects



Techniques for Finding Requirements Defects

The following techniques can be used to find requirements defects:

Ensure requirements use a common Requirements• Ensure requirements use a common RequirementsSyntax

• Test requirements against a checklist of Attributes of Well Written Requirements

• Test requirements using an Ambiguity Checklist



• Ensure that Non-Functional Requirements (NFRs) are Testable

• Test for Missing Triggers

Common Requirements Syntax

[Trigger] [Precondition] Actor Action [Object]

Example:

When an Order is shipped and Order Terms are not “Prepaid”, the system shall create an Invoice.

• Trigger: When an Order is shipped• Precondition: Order Terms are not “Prepaid”• Actor: the system



• Action: create• Object: an Invoice

Identify requirements that do not adhere to the common syntax

Checklist for Well-Written Requirements

Use a checklist that defines the attributes or qualities that a Well-Written Requirement (WWR) must posses:

CompleteCorrectConciseFeasibleNecessary

Prioritized UnambiguousVerifiableConsistentTraceable



Most of these attributes apply equally to a single requirement and the entire set of requirements

Complete

A i t i “ l t ” h it t i ffi i tA requirement is “complete” when it contains sufficient detail for those that use it to guide their work

Every gap forces designers and developers to guess –who do you want specifying your product?



Correct

A requirement is correct when it is error freeA requirement is correct when it is error-free

Requirements can be checked for errors by stakeholders & Subject Matter Experts (SMEs)

Requirements can be checked against source materials for errors

Correctness is related to other attributes – ambiguity, consistency and verifiability



consistency, and verifiability

Concise

A requirement is concise when it contains just the neededA requirement is concise when it contains just the needed information, expressed in as few words as possible

Requirements often lack conciseness because of:•Compound statements (multiple requirements in one)•Embedded rationale, examples, or design•Overly-complex grammar



Overly complex grammar

Feasible

A i t i f ibl if th i t l t d iA requirement is feasible if there is at least one design and implementation for it

Requirements may have been proven feasible in previous products

Evolutionary or breakthrough requirements can be shown feasible at acceptable risk levels through analysis and prototyping



Necessary

A requirement is necessary when at least one of theA requirement is necessary when at least one of the following apply:

• It is included to be market competitive• It can be traced to a need expressed by a customer, end user, or

other stakeholder• It establishes a new product differentiator or usage model• It is dictated by business strategy roadmaps or sustainability



It is dictated by business strategy, roadmaps, or sustainability needs

Prioritized

A requirement is prioritized when it is ranked or ordered according to its importance.

All requirements are in competition for limited resources. There are many possible ways to prioritize:

•Customer Value, Development Risk, Value to the Company, Competitive Analysis, Cost, Effort, TTM

Several scales can be used for prioritization:



•Essential, Desirable, Nice to Have•High, Medium, Low•Other ordinal scales based on cost, value, etc.

Unambiguous

A requirement is unambiguous when it possesses a single interpretation

Ambiguity is often dependent on the background of the reader

Reduce ambiguity by defining terms, writing concisely, and testing understanding among the target audience

Augment natural language with diagrams, tables and algorithms



to remove ambiguity and enhance understanding

Verifiable

A requirement is verifiable if it can be proved that the requirement was correctly implemented

Verification may come via demonstration, analysis, inspection, or testing.Requirements are often unverifiable because they are ambiguous, can’t be decided, or are not worth the cost to verify.



Consistent

A requirement is consistent when it does not conflictA requirement is consistent when it does not conflict with any other requirements at any level

Consistency is improved by referring to the original statement where needed instead of repeating statements.



Traceable

A requirement is traceable if it is uniquely and persistently identified with a Tag

Requirements can be traced to and from designs, tests, usage models, and other project artifacts.

Traceability enables improved• Change impact assessment • Schedule and effort estimation



• Schedule and effort estimation• Coverage analysis (requirements to tests, for example)• Scope management, prioritization, and decision making

Ambiguity Checklist

Utilize a checklist that defines and helps identify ambiguity in requirements:

VaguenessSubjectivityIncompletenessOptionalityUnder-specification

Over-generalizationNon-intelligibilityCoordination ambiguityPassive voiceTime logic confusion



specificationUnder-reference

Time-logic confusionIncomplete logic

Ambiguity Checklist Definitions• Vagueness: is caused by weak words without a precise meaning

• Subjectivity: is caused by weak words that rely on personal experience or opinion

• Incompleteness: comes from insufficient detail, use of TBD, and unbounded lists

• Optionality: is caused by use of should, may, if possible, when appropriate, etc. (Note: use shall and must instead)

• Under-specification: results from use of verbs such as support,



analyze, or respond, or implicit collections of objects

• Under-reference: consists of incomplete or ambiguous references to other documents, standards, requirements, etc.

Ambiguity Checklist Definitions• Over-generalization: is caused by use of universal qualifiers such as

all or every, and even unmodified nouns like users.

• Non-intelligibility: results from poor grammar, complex logic, “and/or” ambiguity, ambiguous negation or enumeration, and missing definitions

• Coordination Ambiguity: results from use of a conjunction between a modified noun and a pure noun (among other cases)

• Passive Voice: occurs when a requirement does not explicitly name an actor

Ti L i C f i i t h l i l diti i d i



• Time-Logic Confusion: exists when a logical condition is used in place of time-related language

• Incomplete Logic: refers to missing logical conditions, such as missing the else of an if-then

Ensure NFRs are Testable

Non-functional requirements are testable if they contain a Scale, Meter and Goal. We can ensure that the NFR has been implemented correctly using these properties.

Scale: The scale of measure used to quantify the statement

Meter: The process or device used to establish location on a Scale



Goal: minimum level required to achieve success

Identify NFRs missing any of these properties

Test for Missing Triggers

Most legitimate ubiquitous requirements state a fundamental property of the software:

• The software shall be distributed on CD-ROM and DVD media.• The software shall prevent Unauthorized Access to patient data• The software shall prevent Unauthorized Access to patient data.

Question requirements that appear to be ubiquitous by looking for unstated triggers or preconditions:

• The software shall wake the PC from standby• The software shall log the date, time and username of failed logins.



The last two requirements are missing triggers

Finding Defects in Requirements:Examples



Example 1

The LED should be on while data is being read from the DVD.

Defect Finding Technique Pass or Fail? Issue(s)

Common Syntax Fail Location of “while data…”

10 Attributes Fails 9 of 10 Not complete, not correct, etc.

Ambiguity Checklist Fails 3 Vagueness, passive voice, ti lit



optionalityTestable NFR N/A

Missing Trigger Pass

Example 2

When the Restaurant Locator app is selected, the app must quickly display a list of the TBD nearest restaurants.quickly display a list of the TBD nearest restaurants.


Common Syntax Pass

10 Attributes Fails 10 of 10 Not correct, not verifiable, etc.

Ambiguity Checklist Fails 3 Incompleteness, subjectivity,



vaguenessTestable NFR Fail What does “quickly” mean?


Example 3

#_of_Alternatives: The thesaurus software shall display at least five alternatives for the selected word using all sources.least five alternatives for the selected word using all sources.


Common Syntax Pass

10 Attributes Fails 8 of 10 Not feasible, not verifiable, etc.

Ambiguity Checklist Fails 3 Incompleteness , under-f li ti



reference, over-generalizationTestable NFR N/A

Missing Trigger Fail

Example 4

The software shall support measured or manual dispensing of water if user mode is selected. Priority: Highwater if user mode is selected. Priority: High


Common Syntax Fail Location of “if user mode…”

10 Attributes Fails 8 of 10 Not complete, not verifiable, etc.

Ambiguity Checklist Fails 3 Under specification, non-intelligibilit incomplete logic



intelligibility, incomplete logicTestable NFR N/A


Example 5

Rx_Access: Only authorized nurses and doctors shall have access to the patient’s prescription drug history.access to the patient s prescription drug history.


Common Syntax Pass

10 Attributes Fails 8 of 10 Ambiguous, not verifiable, not prioritized, etc.

Ambiguity Checklist Fails 2 Coordination ambiguity passive



Ambiguity Checklist Fails 2 Coordination ambiguity, passive voice

Testable NFR N/A


Example 6

If three unsuccessful logins are detected, the software shall log/report the event to the administrator.log/report the event to the administrator.


Common Syntax Pass

10 Attributes Fails 10 of 10 Not complete, not correct, etc.

Ambiguity Checklist Fails 3 Incompleteness, non-intelligibility, ti l i f i



time-logic confusionTestable NFR N/A


Wrap up



Session SummaryIn this session we have:

• Defined requirements defects and shown their impact on software projects

f• Provided examples of common requirements problems

• Introduced techniques to help find requirements defects

• Analyzed examples using the techniques discussed



Final Thoughts

• Requirements defects lead to cost and schedule overrunsRequirement defects can proliferate into designs, tests and code.

• Focus on requirements defect preventionThe earlier a defect is detected, the less it costs to fix.

• Use checklists and rules to find requirements defectsThey can provide good feedback to authors on the exact nature of the

requirements defects.



“An ounce of prevention is worth a pound of cure”

Contact Information

Thank You!

For more information, please contact:

John Terzakis

[email protected]



Backup



Examples of Ambiguity (1 of 3)

• VaguenessThe software must support all current standards for video encoding before launch.

• SubjectivitySubjectivityThe software must be able to easily and seamlessly transfer media between connected devices.

• IncompletenessThe software must support at least 50 concurrent users.

• OptionalityThe software should include as many end-user help mechanisms as possible



possible.• Under-specification

The software must support 802.11a, b, g, and other network protocols supported by competing applications.

• Under-referenceUsers must be able to complete all previously-defined operations in under 5 minutes 80% of the time.

• Over-generalization


Over generalizationAll users must be able to delete all data they have entered.

• Non-intelligibility

The software shall report/log improper access attempts and notify administrators if a user does not respond to warning messages or lock out the account.

Coordination Ambig it



• Coordination Ambiguity

The software shall allow automated updates and deletions.

The software must display categorized instructions and help documentation.

• Passive Voice

When shipping information has been verified, shipping labels must be printed for each container in the order.

Time/Logic Conf sion


• Time/Logic Confusion

If two orders are received from the same customer for the same part, the software shall follow the process described below.

• Incomplete Logic

When automatic calibration fails, the software shall switch to manual calibration.



Not Complete:The software must allow a TBD number of incorrect login attempts.

Complete:When more than 3 incorrect login attempts occur for a

10 Attributes of a WWR: Examples (1 of 5)

When more than 3 incorrect login attempts occur for a single user ID within a 30 minute period, the software shall lock the account associated with that user ID.

Not Correct:The 802.3 Ethernet frame shall be 2048 bytes or less.

Correct:Th 802 3 Eth t f l th h ll b b t 64 d 1518



The 802.3 Ethernet frame length shall be between 64 and 1518 bytes inclusive.

54

Not Concise:The outstanding software written by the talented development team shall display the current local time when selected by the intelligent and educated user from the well designed menu.

Concise:


Concise:when selected by the user from the menu, the software shall display the current local time.

Not Feasible:The software shall allow an unlimited number of concurrent users.

F ibl



Feasible:The software shall allow a maximum of twenty concurrent users

Not Necessary:The software shall be backwards compatible with all prior versions of Windows®

Necessary:The software shall be backwards compatible with Windows® Vista


The software shall be backwards compatible with Windows® Vista SP2 and SP1, and Windows XP SP3.

Not Prioritized:All requirements are critical and must be implemented.

Prioritized:80% of requirements High, 15% Medium and 5% Low.



Ambiguous:The software must install quickly.

Unambiguous:When using unattended installation with standard options, the software shall install in under 3 minutes 80% of the time and under


software shall install in under 3 minutes 80% of the time and under 4 minutes 100% of the time.100% of the time.

Not Verifiable:The manual shall be easy to find on the CD-ROM.

Verifiable:The manual shall be located in a folder named User Manual in the

t di t f th CD ROM



root directory of the CD-ROM.

Inconsistent:#1: The user shall only be allowed to enter whole numbers.#2: The user shall be allowed to enter the time interval in seconds and tenths of a second.

Consistent:


Consistent:#1: The user shall only be allowed to enter whole numbers except if the time interval is selected.#2: The user shall be allowed to enter the time interval in seconds and tenths of a second.

Not Traceable:Th ft h ll t th f th PIN



The software shall prompt the user for the PIN.Traceable:

Prompt_PIN: The software shall prompt the user for the PIN.

find requirements defects to build better software

Technology