Software Quality ManagementJolene Eichorn

AGENDA Pursuit of Quality Quality Assurance Quality Control Quality Improvement Implementing a Quality Management Program Business Value

THE PURSUIT OF QUALITY Quality is something that is taken for granted when you

have it and cursed when you don't Quality is often not seen as a value add  Quality is often neglected to meet schedule constraints Quality builds giants and nearly destroys them (Toyota) Quality does not happen by chance Quality must be premeditated and carefully planned to

be effective  

 What if we were to pursue quality? What would it look like? How would it be defined? How much would it cost ... or would it save?

CONCERN FOR QUALITY What if Quality provided

Competitive advantage? Increased customer satisfaction? Controlled costs?

 What if poor Quality Was one of the most common reasons for schedule

overruns? Was implicated in close to half of all canceled

projects? 

THE COST OF QUALITY• In 1996, there were 200 million calls for technical

support. At an average of about $23 per call, the industry spent about $4.6 billion on these calls. 

• Over the past 7 years, the ratio of support to total employees in hardware and software companies has grown from 1 in 12 to 1 in 6. 

• The average $3 million project costs companies using poor requirements practices an average of $5.87 million per project -- a $2.24 million premium. IAG Consulting's new Business Analysis

Benchmark , 110 projects at 100 companies surveyed

 

BALANCING EFFICIENCY & EFFECTIVENESS

Efficiency Effectiveness

Quality

Fit for purpose - service

Fit for Purpose - product

Cost

Time

Measured

Cycle Time

Reliability

Customer Driven – Value Defined

GOALIncrease the efficiency of

Solution Delivery Life Cycle activities, the effectiveness of

the participants, and the quality of the

deliverables.

DEFINITION OF QUALITY • Quality is the ability of your product or

service to conform to your customer's: wants needs expectations requirements

• Quality must be measurable when it occurs• Quality should be predictable when it occurs • Quality in design means realization of

purpose or fitness for use.

Align with RUP

Software Quality Improvement

Quality

Ensure Software Quality (SQA)

Software Quality Assurance

Software Quality ControlBusiness Value

QUALITY ASSURANCE“... a process for providing adequate assurance that the software products and processes in the product life cycle conform to their specific requirements and adhere to their established plans.” IEEE Standard 12207 “A planned and systematic means for assuring management that the defined standards, practices, procedures, and methods of the process are applied.” CMMI, SEI

“The planned and systematic activities implemented in a quality system so that quality requirements for a product or service will be fulfilled.” ASQ

The focus is on the process used to create the deliverable

QUALITY ASSURANCEIn

pu

ts

COBIT, CMMi, BABOK, PMBOK and other best practice frameworks

SDLC Methodologies

Current processes, standards, policies

Serv

ices

Developing and supporting processes

Define measures

Process management

Define standards, policies, procedures

Prevention activities

Assess quality

Deliver training

Support automation tools

Prevent defects

Outp

uts

Methods and Processes

Assessment and Metrics

Published standards, policies, procedures

Templates

Quality Assessment

Software Quality Plan

Training class

Deploy Automation tools

QUALITY CONTROL“The operational techniques and activities that are used to fulfill requirements for quality.” ISO 8402-1994

“The observation techniques and activities used to fulfill requirements for quality.” ASQ

  

  The focus is on the deliverable itself.

QUALITY CONTROLIn

pu

ts

Requirements

Deployed software

Design documentation

Serv

ices Define test strategy

Create test plans

Configure test environment

Develop test cases

Execute unit, integration, system, performance, user acceptance tests

Conduct inspections

Outp

uts

Test strategy

Test plans

Test environment

Test cases

Test execution results

Defects

Mechanisms:Testing process, test tools

QUALITY IMPROVEMENTChanging a process to improve the reliability of an outcome. Methods for quality improvement include:

Kaizen Six Sigma IEEE TQM  TRIZ  CMMI ITIL

QUALITY IMPROVEMENTIn

pu

ts

Methods and processes

Assessment and metrics data

Published standards, policies, procedures

Templates

Quality assessment

Defect data

Serv

ices Improve product

Reduce variation

Reduce defects

Improve processes

Re-engineer process Outp

uts

Quantifiably improved processes, standards, templates

Mechanisms:COBIT, CMMi, 6, LeanVAL IT, ITIL, TRIZ

QUALITY IMPROVEMENTLeverage COBIT•As a way of aligning IT with the business•As a tool for defining IT process improvements•For control assessments•For maturity assessments•As a basis for performance measurement•As a communication tool•To map IT goals to business goals

Leverage VAL IT•For implementing IT-enabled change and innovation •To optimize returns from tool investments

Leverage CMMi•As a way to guide process improvement.•For helping to integrate traditionally separate organizational

functions.•To set process improvement goals and priorities.•To provide guidance for quality processes.•As a means for providing a point of reference for appraising

current processes. •To explicitly link your organization's activities to your business

objectives.

BALANCING SPEED & QUALITY

Quality

Fit for purpose - service

Fit for Purpose - product

Cost

Time

Measured

Cycle Time

Reliability

Customer Driven – Value Defined

LeanCreate ValueEliminate WasteStrategic Focus

6Improve QualityReduce VariationImprove Predictability

BALANCING SPEED & QUALITYImplementing Lean programs to reduce time and cost may be ineffective without predictable processes built through Quality Management, thereby preventing Lean goals from being met.

In addition, quality process automation can have an enormous impact on efficiency and quality.

QUALITY IMPROVEMENTLEAN PRINCIPLE 5 - PERFECTION

Pursue Continuous Improvement / Strive for Excellence of SE Processes

Do not ignore the basics of Quality: Build in robust quality at each step of the process, and resolve and do not pass along problems. Strive for perfection in each process step without introducing waste Do not rely on final inspection; error proof wherever possible If final inspection is required by contract, perfect upstream processes pursuing 100% inspection pass rate Move final inspectors upstream to take the role of quality mentors Apply basic PDCA method (plan, do, check, act) to problem solving Adopt and promote a culture of stopping and permanently fixing a problem as soon as it becomes apparent.

LEfSE were developed by the Lean Systems Engineering Working Group of INCOSE (International Council for Systems Engineering). Version 1.0 was released at the INCOSE IW, San Francisco, March 1, 2009.

QA VERSUS QC

Quality Assurance

• Defect Prevention• Proactive Process• Customer

Confidence• Preventative

Quality Control

• Defect Detection• Reactive• Producer

Confidence• Corrective

QA VERSUS QCQuality Assurance

Examples• Process Definition• Quality Assessment• Inspection (Mays, 1990)• Tool Selection• Training• Requirements Analysis• Usability Analysis• Early Prototyping• Root Cause Analysis• Process Checklists• Defect Prevention Process

Quality Control

Examples• Walkthrough• Testing• Inspection (Fagan, 1976)• Checkpoints

A QUALITY MANAGEMENT PROGRAM

Ensures that: An appropriate methodology is in place Projects use standards and procedures Independent reviews and audits are performed Documentation is produced during development Change controls are in place An inspection process is leveraged Feedback is provided to team Noncompliance issues are addressed

QUALITY ASSURANCEThe purpose of Quality Assurance is to provide staff and management with objective insight into processes and associated work products.

A QA group that is independent of the project provides objectivity Everyone performing QA activities should be trained in QA Those performing QA activities for a work product should be separate

from those directly involved in developing or maintaining the work product An independent reporting channel to the appropriate level of

organizational management must be made available QA should begin in the early phases of a project to establish plans,

processes, standards, and procedures that will add value to the project and satisfy organizational policies

CMMI, Guidelines for Process Integration and Product Improvement, Mary Beth Chrissis, Mike Konrad, Sandy Shrum

QUALITY ASSURANCE SERVICES

• Customize Project Dashboard• Develop Development Case• Develop Quality Plan• Acquire Tools• Implement and Configure Tools• Launch Development Processes• Prepare Guidelines• Prepare Templates• Process and Tools Support• Tailor Development Process

Development Environment

• Conduct Quality Assessments• Perform Retrospectives• Stage Gate Assessments• Define and Publish Quality Measures

(Dashboard)• Develop Standard Dashboard Reporting• Facilitate Inspections• Facilitate Standards Adherence

Quality

• Configure Test Environment• Maintain Test Data

Test Environment

• Coordinate Software Engineering Process Group

• Develop and Publish Guidelines• Develop and Publish Methodology and

Process• Develop and Publish Standards• Publish Templates

Methodology

• Formal Review Moderator Training• Formal Review Participant Training• Informal Review Training• Process Training• Quality Training• Tools Training• Methodology Training• UML Training

Training

• Process Mentoring• Standards Mentoring• Tools Mentoring• UML Mentoring

Mentoring

QUALITY ASSURANCE ROLES

• Responsible for all process related aspects of the organization, such as:

• Reviewing best practices. Continuously improving processes using quality measures and tools.

• Developing standard methods, processes, guidelines.

• Creating templates. • Training and mentoring.

Methodologist

• Responsible for all process related aspects of the project, such as:

• Assures that the standards, processes, and procedures are appropriate for the project and are correctly implemented.

• Tailoring the process to match the specific needs of the project.

• Educating and mentoring project members on process related issues.

• Ensuring that valuable project experience is harvested and fed back into the process.

• Assisting the Project Manager in planning the project.

Process Engineer

• Supporting the tools used by the project. This includes selecting and acquiring tools, configuring and setting them up, and verifying that they work.

• Understanding the underlying processes used by the project.

Tools Specialist

• Participating in reviews, walk-through, process checks.

• Facilitating or participating in formal inspections, ensuring the process is followed correctly.

• May sample actual requirements.• Defining quality measures. • Ensuring project quality. • Conducting quality assessments and audits

to ensure that documented standards, processes and procedures are followed.

• Establish software metrics in order to measure the effectiveness of the process.

• Note: The CMMi repeatedly states that senior management must "protect individuals performing the SQA role,” given the possibility of management retaliation against an employee who finds non-compliance issues.

Software Quality Analyst

QUALITY CONTROLThe purpose of Quality Control is to validate and verify that software program/application/products:

1. meet the business and technical requirements that guided its

design and development2. works as expected3. can be implemented with the same characteristics

A QC group that is independent of the those that developed, or maintain the work product, provide objectivity.

QC provides an objective, independent view of the software to allow the business to appreciate and understand the risks at implementation of the software.

QA and QC may address the same work product, but from different perspectives.

QUALITY CONTROL SERVICES

• Define test strategy• Estimate test effort• Manage test schedule• Define test plans

Test Planning

• Define functional test cases• Define performance test

scenarios

Test Design

• Create test cases• Execute test cases

Manual Testing

• Create automated tests• Execute automated tests

Automated Testing

• Create load and stress tests• Execute load and stress tests• Tune applications for

performance

Performance Testing

QUALITY CONTROL ROLES

• Conducts manual and automated testing, logs the outcomes and reports defects.

• Takes an active role in the verification process (all the documents or requirements), this includes the requirements inspection process and verifying that documented requirement standards are followed.

Tester

• Identifies and defines the required tests

• Monitors detailed testing progress and results in each test cycle

• Represents stakeholders who do not have direct or regular representation on the project.

Tester Analyst

• Leads in defining the test approach and ensuring its successful implementation

• Identifies the appropriate techniques, tools and guidelines to implement the required tests

• Provides guidance to the test effort

Test Designer

• Leads the overall test effort• Advocate for quality and testing• Provides resource planning and

management, and resolution of issues that impede the test effort

Test Manager

BUSINESS VALUEQuality is a differentiatorOrganizations typically build the software and systems that make them unique, while they purchase applications for automating common business functions that do not demand differentiation from competitors.

The bottom line is:

Consistent delivery of high-quality software will set an organization apart from its competition.

The business value of software quality, Geoffrey Bessin, http://www.ibm.com/developerworks/rational/library/4995.html

BUSINESS VALUEQuality enables innovationIt is quality in development and maintenance of software that makes it possible for the business to react, adapt, and deploy quickly. If there is any breakdown, any inefficiency, any lack of quality, then the organization will fail to reach the market in a timely fashion and the organization will find itself at a competitive disadvantage.

The bottom line is:

High quality software facilitates innovation by increasing predictability, lowering risk and reducing rework, resulting in happier customers and more revenue.

The business value of software quality, Geoffrey Bessin, http://www.ibm.com/developerworks/rational/library/4995.html

AND BEST OF ALL - QUALITY IS FREE!

    

   

*Philip Crosby was so convinced of this  that he titled one of his books Quality is Free, since the major costs of quality are the costs associated with poor quality.   

cost of defect prevention and early reviews

cost of late stage testing and

reworkCost of Quality

Increasing defect

prevention

Reduces rework

When the amount saved by avoiding defects rises faster than the amount invested to avoid defects, quality is free*

BUSINESS VALUE

Troubled ProjectPoorly defined process / inconsistent buy-in

Front-loaded requirements gathering without reassessment or validation

Absent or limited architectural and code quality assurance

Late cycle system testing

Limited post-deployment monitoring and assessment

Excess cost is easy to spot at every stage.

Perfect Project A well-defined process with unanimous buy-in from team members

Focused attention to value-add, detail and quality across the lifecycle

Constant evaluation of end-user feedback to ensure reinvigoration of the product line and satisfied, repeat customers

There is no excess cost -- the process is optimized.

Quality is freeWhat is the cost of high quality? Is high quality free?

The business value of software quality, Geoffrey Bessin, http://www.ibm.com/developerworks/rational/library/4995.html

Improved quality enables teams to deliver more projects on time, at lower cost, with more features. By preventing defects in the system throughout the entire development process, a team eliminates the time and cost required to find and fix those defects later on.

BUSINESS VALUEReworking a software requirements problem once the software is in operation typically costs 50 to 200 times what it would cost to rework the problem in the requirements stage.

 Because potentially a:

  1 sentence requirement

5 pages of design diagrams

500 lines of code

15 pages of user documentation

48 test cases

 Boehm and Papaccio 1988

Currently, over half of all errors are not found until “downstream” or during post sale software use. This occurs even though 80% of development costs are spent on defect removal.

SAVINGS CAN BE HUGE Each hour spent on Inspections avoided an average of 33 hours of maintenance, and inspections were up to 20 times more efficient than testing. - Russell 1991, study done on large programs

Software Quality Inspections have been found to produce net schedule savings of 10 to 30%. - Gilb and Graham 1993

Every major defect found at Inspection will save 9 hours of downstream correction effort. – Reeve, 1991

Most of the 78 organizations that participated in the National Quality Experiment have seen a ROI of 2:1 to 8:1 on investment in Inspections. – O’Neill, 2001

CONCLUSIONUnderstanding the who, what, why, and when of quality is essential in implementing an effective Quality Program. It requires a combination of distinct disciplines:

They are three unique disciplines which, when used together, can improve the efficiency and effectiveness of any organization leading to reduced cost and increased customer satisfaction.

•A proactive defect prevention program

•Assures that the standards, processes, and procedures are appropriate for the project and correctly implemented.

Quality Assurance

•A reactive defect identification program

•Checks that the project follows its standards, processes, and procedures, and that the project produces the required internal and external (deliverable) products.

Quality Control

•A program that uses the result of Quality Control to improve Quality Assurance

Quality Improvement

ABOUT THE AUTHORSolution Delivery Over 20 years of experience in delivering solutions that optimize tool automation

investments, improve productivity, quality, transparency and alignment between business and IT.

Program Turnaround Specializes in requirements quality assessments and disentanglement of challenged

programs.

Capability Maturity Brings practical knowledge of requirements management best practices and promotes

requirements maturity across large organizations.

Quality Works with organizations and project teams to improve capabilities by implementing

Lean Six Sigma process improvements, effective requirements practices, dashboard reports, and formalizing requirements change management.

Contact Information:Jolene Eichorn, Software Quality Practitioner http://www.linkedin.com/in/eichorn

改善CHANGE FOR THE

BETTER