maximize the impact of your qa&t efforts...company if time is wasted in work-arounds for poor...

Within the software industry, business analysts and testing groups are often the first to feel the pinch of budget cuts. The resulting risk to quality becomes an unrecognized expense that impacts both the company producing the software, and their customers alike.

This is an avoidable effect of cost-cutting measures. Businesses can maximize the quality of their projects and preserve timelines and budget by making a seemingly large up-front investment in quality from the very beginning. With a team of individuals with the proper skills, a Quality Plan to guide all quality-related efforts, controls and metrics that ensure business needs are met, and an effective Test Strategy, businesses can maximize the quality of their projects and preserve timelines and budget.

Maximize the Impact of Your QA&T Efforts

Balancing Cost & Coverage with Effective QA&T Architecture

White PaperSaima Prabhu

2 | Maximize the Impact of Your QA&T Efforts

Anatomy of Cost of Poor Quality (COPQ).There are several kinds of costs associated with quality. To

clarify this issue, we

need to define the

different types of costs.

James Harrington,

formerly a Quality

Expert for IBM, defined

Quality Costs in his

book, Poor-Quality

Costs.2

The first division encountered is between

Indirect Quality Costs and Direct Quality

Costs.

Indirect Quality CostsIndirect Quality Costs include costs that an

effective QA&T strategy would have prevented;

such as those caused by missed delivery dates, the

cost and extra effort to fix defects late in the development

process, and costs incurred by the customer due to poor

product quality.

They also include lost sales due to customer dissatisfaction, and

the cost to repair a damaged reputation. Indirect Quality Costs

usually fall under COPQ, and are easily over-looked if proper

metrics are not in place to measure and report them.

Direct Quality CostsDirect Quality Costs are recorded and tracked in the budget,

and can be divided into Costs to Achieve Quality, and Cost Due

to Poor Quality:

Costs to achieve QualityCosts to Achieve Quality are the cost of Prevention and

Appraisal Activities. These costs are included in the budget.

They are also the Quality Costs over which decision-makers

have the most direct control.

• Prevention Activities include peer reviews,

requirements reviews and verification,

design walk-throughs, risk assessment, and

developing a Project Management Plan.

These activities prevent poor quality by

determining the goals and needs of the

end users and stakeholders, and creating a

system that aligns the

development and

testing team’s efforts

with those needs and

goals.

• Appraisal Activities

include testing,

automation, and the

repair of defects detected in the design and development

stages of the Software Cycle.

Costs Due to Poor QualityCosts Due to Poor Quality (COPQ) are the costs which occur

when a low quality software product or service is produced as

the result of earlier decisions about how to invest in the Costs

to Achieve Quality. They include Internal Error and External

Error costs.

“Cost of Poor Quality is the cost that would disappear if no deficiencies existed in the product provided to the customer or in any activities conducted by the firm.”1

–Frank M Gryna

“Indirect Quality Costs usually fall under COPQ, and are easily over-looked if proper metrics are not in place to measure and report them.”

Indirect Quality CostsMissed Delivery DatesExtra Costs for Fixing Defects Late in the Dev ProcessLost SalesCustomer DissatisfactionDamaged Reputations

Direct Quality CostsCosts to Achieve Quality

Prevention ActivitiesAppraisal Activities

Costs Due to Poor QualityInternal Error CostsExternal Error Costs

COPQ


• Internal Error Costs are the costs that arise within your

company before delivery. These include the cost to your

company if time is wasted in work-arounds for poor design,

or employees redirecting efforts from their jobs to work on

problems presented by the defect, missed project deadlines,

refactoring poorly designed code, failed builds, regression

testing of repairs, etc.

• External Error Costs are the costs that occur after a

defective product has shipped. Some examples of this are

customer service costs as well as the cost to produce and

distribute fixes.

A Stitch in Time Saves Nine According to a 2002 report by the National Institute of

Standards and Technology (NIST), the cost of defects increases

as you move from addressing them with Prevention, to dealing

with them as Internal and External costs.3

This is illustrated in Figure 1.

When you consider that 70% of all defects are introduced in the

design phase, it becomes clear how important it is to engage

the QA&T team from the very beginning of the project.3

You have the opportunity to catch most of the defects in your

product at the point where it is least expensive to fix them.

Prevention and Appraisal activities clearly give you the best

QA&T value for your budget.

Defects that are not remedied by Prevention activities cost five

to ten times more by the time you detect them with Appraisal

activities. If the defects are allowed to persist until they become

External costs, the costs incurred by the defect will have grown

by as much as thirty times.3

Ideally, you want to keep your Internal and External costs as

low as possible. This will occur only if your project invests in the

proper Prevention and Appraisal techniques.

Curbing Internal & External Costs with a Test Architecture“Estimates of the economic costs of faulty software in the U.S.

range in the tens of billions of dollars per year and have been

estimated to represent approximately just under 1 percent of

the nation’s gross domestic product (GDP).”3

The Internal and External COPQ have been referred to by

Quality Guru Dr. Joseph Juran as “The gold in the mine.”4

This is an apt analogy, because it is value that is just sitting

there, hidden, waiting to be discovered and reclaimed. The

question is; how do you find and retrieve it, decide which veins

provide the best value and what retrieval methods are the best

investment?

Prevention

0

10

20

30

Appraisal Internal External

Figure 1. The Cost of Poor Quality

Missed opportunities for Quality in the design and production phases lead to dramatic increases in Internal and External costs later on.

COPQ

“You have the opportunity to catch most of the defects in your product at the point where it is least expensive to fix them.”


NIST describes an inadequate test infrastructure as the biggest

barrier for companies in answering these questions. It is

important to begin building an adequate infrastructure from

the start.

Begin with a Quality PlanThe beginning step for lowering costs should be to establish

a Quality Plan with well-designed monitoring and reporting

structures to capture valuable data throughout the product’s

lifecycle.

The data and analysis provided by these structures will produce

a firm foundation for educated decision making. Engaging the

QA&T team early in the process has benefits that extend

well beyond current projects, as the Quality Plan will also

guide future projects.

There are a number of different metrics that

can be chosen to monitor the quality of a

software product. Care should be taken

to align the metrics and reporting

structure chosen with the goals

and needs of the end users and

stakeholders. Additionally, it is

important that the Quality

Plan ensure that whatever

metrics are chosen; the

collection, measurements

and reporting of the metrics

are done consistently.

Create a Robust Test PlanExperienced and technically

sound testers, when armed with

a Quality Plan that is solidly aligned

with well-defined requirements, a

strong Test Plan, and the right testing tools for the job, will

succeed in catching most defects in the stage

where they are introduced.

“An effective QA&T team must have a plan for dealing with the defects that are detected at any point in the development process.”

Templates Checklists Test Case Management

AutomationFrameworks

DecisionStructures

QAT KB SharePointPortals

TFS Labs Tools

PerformanceFrameworks

Training MARQ Process

Supporting Systems

Practice Guidelines Thought Leadership - Team Leadership


A strong Test Plan will have good traceability and intuitive

test scripts. Traceability involves a process for controlled

documentation of the requirements of a project, tests that will

verify the requirements, controlled documentation to ensure

coverage of high risk areas, a process for analyzing the impact

of any changes made to the requirements, and a formal change

process.

Methods chosen for testing need to be tailored to the project,

as well as the resources available to the Quality team. A fairly

simple and straightforward project that contains significant

business risks will require different solutions than a large,

complex project that undergoes frequent additions and

revisions to support new products or functions.

You may use a creative, adaptable method such as session

testing if your requirements are still under development. You

might combine session testing with automation to achieve

consistency as well as flexibility. Automation alone might be

your answer. If your software is not technically complex, but

will have to run reliably, quickly, and under heavy usage, you

might opt for manual testing augmented with performance

and load testing. When these decisions are made with all of

the demands of the project in mind, coverage and cost can be

balanced, and efficiency greatly increased.

The purpose of having a Test Plan is to ensure that the software

complies with the guidelines outlined in your Quality Plan. This

will keep your QA&T efforts firmly grounded in meeting the

needs of the users and the goals of the stakeholders.

A Test Plan will direct resources and appropriate skill sets to

areas most critical for the success of the project. It will allow

you to strike the proper balance between risk and cost, so that

your project can be delivered on time, within budget, and with

the highest possible quality (and lowest possible Internal and

External costs).

An effective QA&T team must have a plan for dealing with

the defects that are detected at any point in the development

process. Regardless of the reason, it is often necessary for a

few defects to be addressed later in the development cycle. As

the software is built, each new element adds complexity that

can give rise to unexpected problems. Performance testing

could show a necessity for reworking code, or a simple change

request could create a need for more testing.

Good defect management systems and a triage process will

help you reduce the turn-around time for fixes, and the number

of problems that develop into expensive Internal and External

costs later on. Automation of testing at the right time can

greatly increase the speed and efficiency of regression and

confirmation testing at this point in the process.


Automation, and the cost of fixing the defects that it

detects, often fall under Appraisal and Prevention costs. It is

understandable that many companies hesitate to take on these

increased costs up front. However, if properly designed and

directed, these efforts will reduce Internal and External costs

down the line.

When defects are detected and fixed, a comprehensive suite of

tests must be run on the application to ensure that the defect

was fixed, and that no new defects were introduced. If you have

a good automation framework in place, the same test suite can

be run quickly and efficiently, with only a few new test scripts

added to cover the changes. This can provide tremendous

savings versus manual testing. In addition to saving time and

money, a good automation framework allows for more frequent

testing of the product. This increases the efficiency in detecting

defects, and increases the quality of the product.

Controlled requirement and technical documents, traceability,

updated test cases, and automation all appear to be a huge

cost on the project initially, but they help conserve resources

spent fixing and testing defects. If Internal and External costs

are monitored and reported, analysis of those metrics can

determine the root causes of COPQ. This knowledge will

suggest improvements for the Prevention and Appraisal actions

of your next release, as well as for similar projects in the future.

With the proper foundation, you will get the right start on

creating a quality product with maximum efficiency.

Maximize the Impact of your QA&T budget with a Quality Architecture. An Adaptive Risk-based Quality (Such as Magenic’s MARQ)

methodology and Practice Guidelines provide a QA&T

infrastructure aimed at identifying and implementing tools to

minimize Internal and External costs, and maximize the impact

of every dollar spent on your individual project, or across your

enterprise.

Only Magenic Delivery Center personnel are trained and

uniquely equipped to implement the MARQ methodology

and Practice Guidelines. Their experience with applying these

concepts will help you design and implement the kind of QA&T

plans needed to fully benefit from quality related activities and

expenses.

The MARQ methodology includes techniques for creating

Quality and Test Plans specifically designed to:

• Begin QA&T activities from the very beginning of the design

phase, where the opportunity for cost savings is the greatest.

• Ground all QA&T activities in ensuring that the needs of the

users and the goals of the stakeholders are met.

• Identify the areas of highest business risk and technical

complexity and direct resources to areas where they will

provide optimal efficiency and the greatest impact.

• Provide a means of collecting and reporting metrics that will

allow for educated decision-making about quality, and provide

a meaningful picture of how quality activities impact costs.

• Create defect management and triage systems to minimize the

cost of fixing defects detected at any phase of the Software

Development Cycle.

• If necessary, create an Automation Framework to reduce

resources used in testing.

“COPQ is often the largest expense a business experiences. It is frequently undercounted due to a lack of metrics needed to properly record, track, and analyze its causes and consequences.”


SummaryCOPQ is often the largest expense a business experiences.

It is frequently undercounted due to a lack of metrics

needed to properly record, track, and analyze its causes and

consequences.

Costs to Achieve Quality are usually well-defined and easily

quantified. Because the most effective use of resources spent to

achieve quality happens at the very beginning of a project, long

before the effects of an inadequate QA&T foundation begin

to appear, there is often a temptation to begin cost-cutting

measures with these seemingly large, up-front expenditures.

However, numerous Quality experts, as well as research from

IBM and NIST have found that investment in these initial costs

will bring better results as well as long-term savings. The key

to obtaining these results is access to an experienced team of

Quality professionals with a full command of the tools of their

trade, and an understanding of when and how to apply them

for the greatest benefit.

Assessment should begin in the planning stage of the project. A

Quality Plan should be created to ensure that quality efforts are

tailored to the demands of the project, and will be applied with

the greatest level of effectiveness and efficiency. Additionally,

the Quality Plan will provide for monitoring, analyzing, and

reporting COPQ throughout the Software Development

Lifecycle, and will provide valuable information for future

decision-making.

A Test Plan should be designed to fit the size, scope, technical

complexity, and business risk associated with the project, as

well as the budget available to be spent on testing. With these

factors in mind, different testing methodologies can be chosen

to appropriately direct resources where they will be most

valuable, and concerns of cost verses coverage can be balanced

for the best possible efficiency and results.

Magenic Delivery Center has been able to implement these

concepts with great success through their MARQ Methodology

and Practice Guidelines. A number of clients in unique and

demanding business environments have experienced both

reductions in Costs to Achieve Quality, as well as reductions in

COPQ. Contact your Magenic Sales Representative to hear how

Magenic Delivery Center can help your business get the best

Quality for your dollar value.

“Costs to Achieve Quality are usually well-defined and easily quantified.”

M

M

A

A

Quality Plan

QPQP

6

6


References1 Brisco, Nat R. & Gryna, Frank M. “Assessing the Cost of Poor

Quality in a Small Business.” Quimpro Quarterly Special. n.d. Web. http://www.qimpro.com/downloads/Assessing%20the%20Cost%20of%20Poor%20Quality%20in%20a%20Small%20Business%20-%20Frank%20Gryna.pdf

2 Harrington, James H. Poor-Quality Costs. Milwaukee, WI: American Society for Quality (ASQ), 1987.

3 “The Economic Impacts of Inadequate Infrastructure for Software Testing” National Institute of Science and Technology. Planning Report 02-3. 2002. Web. http://www.nist.gov/director/planning/upload/report02-3.pdf

4 “The Juran Institute Research on The Cost of Poor Quality.” Juran Institute. 2005. Web. http://www.juran.com/downloads/COPQ%20Research.pdf

Additional ResourcesCampanella, J. Principles of Quality Cost. 3rd ed. Milwaukee, WI: American Society for Quality (ASQ), 1999.

Feigenbaum, A. V.. Total quality control. 3rd ed. New York: McGraw-Hill, 1991.

Juran, J. M., and Blanton A. Godfrey. Quality control handbook. 5th ed. New York: McGraw-Hill, 1998.

Kaner, Cem. “Quality Cost Analysis: Benefits and Risks” 1996. Web. http://www.kaner.com/qualcost.htm

Perry, W. Structured Approach to Testing: Effective Methods for Software Testing. Hoboken, NJ: John Wiley & Sons, 2000.

About the Author - Saima PrabhuSaima Prabhu is a Consulting Manager for the QAT group at Magenic. She has been in IT since 1995. During this time she has worked marketing, sales, production shop floor management, development, quality assurance and release management.

Saima believes that the key to software development is to deliver software on time, to the highest Quality standards, and the exact business needs and budget of the client.

About MagenicFounded in 1995 by the same technical minds that still run the company, Magenic focuses on the Microsoft stack and mobile application development.

Visit us at magenic.com or call us at 877.277.1044 to learn more or to engage Magenic today.

maximize the impact of your qa&t efforts...company if time is wasted in work-arounds for poor...

Documents