2004 11 17 test process improvement
TRANSCRIPT
-
7/29/2019 2004 11 17 Test Process Improvement
1/18
Test processimprovement
The Models and the Practice
Steven Deneir
November 17th 2004
Test process improvementNovember 17th 2004 2
Index
Introduction
Models
Models
Models
Conclusion & Closing
-
7/29/2019 2004 11 17 Test Process Improvement
2/18
Test process improvementNovember 17th 2004 3
Goal of this presentation
Provide an introduction to models that might help during
test process improvement
Test process improvementNovember 17th 2004 4
What is Test Process Improvement?
Continuous: everything changes around us, so should testing
Improvement: where are you now? where do you want to be?
Quality: product quality, process quality (insight, coverage, predictability,etc.)
Efficiency: cost, lead time, effort
Testing: Techniques, Management, Environments, Tools, etc.
Process: Ad-hoc testing, Structured testing, Exploratory testing, etc.
Related to: the development process, the risks of the project, therequirements, the project planning, etc
Continuous Improvement of the quality, and the efficiency
of the testing process, related to the output of the total
software process
-
7/29/2019 2004 11 17 Test Process Improvement
3/18
Test process improvementNovember 17th 2004 5
Why Test Process Improvement?
Changes are all around us: Business of your organisation might be changing
Higher quality demands of customers
Time to market
Cost cutting
Development process changes: CMMI programs
Development delivers faster thanks to more efficient tools
Etc., etc., etc.
The test process must keep up with these changes!
Test process improvementNovember 17th 2004 6
SW-CMM Capability Maturity Model
In 2000, the SW-CMM was upgraded to CMMI
The SEI no longer maintains the SW-CMM model, its
associated appraisal methods, or training materials, nor
does the SEI offer SW-CMM training.
-
7/29/2019 2004 11 17 Test Process Improvement
4/18
Test process improvementNovember 17th 2004 7
SW-CMM Structure
Maturity Levels
Key Process Areas
Common Features
Key Practices
Process
Capability
Goal
Implementation or
Institutionalization
Infrastructure or
Activities
indicate contain
achieveorganized by
addresscontain
describe
ML 1: Initial
ML2: Repeatable
ML 3: Defined
ML 4: Managed
ML 5: Optimizing
Test process improvementNovember 17th 2004 8
SW-CMM and Testing
SW-CMM Level 3
Key process area Software Product Engineering
Activities:
5: Software Testing is performed according to the projects definedsoftware process
6: Integration testing of the software is planned and performed
according to the projects defined software process
7: System and acceptance testing of the software are planned and
performed to demonstrate that the software satisfies its requirements
A dedicated KPA exists for Peer Reviews
-
7/29/2019 2004 11 17 Test Process Improvement
5/18
Test process improvementNovember 17th 2004 9
SW-CMM Summary
Not really helping us with test process improvement
Upgraded to CMMI in 2000
But reminds that some other things should better be in
place
Test process improvementNovember 17th 2004 10
CMMI CMM Integrated
Based on experience of previous models (SE, SW, )
More flexible: 2 representations Staged representation (similar to SW-CMM)
Continuous representation
Common terminology for the different models that are
integrated
Consistent with ISO 15504 (SPICE)
-
7/29/2019 2004 11 17 Test Process Improvement
6/18
Test process improvementNovember 17th 2004 11
CMMI Staged Representation
ML 1: Initial
ML2:Managed
ML 3: Defined
ML 4: Quantitatively
Managed
ML 5: Optimizing
Maturity Levels
Process Areas
Specific Goals Generic Goals
Specific Practices Generic Practices
Commitment
to Perform
Ability
To Perform
Directing
Implementation
Verifying
Implementation
Common Features
Test process improvementNovember 17th 2004 12
CMMI Continuous Representation
Process Areas
Specific Goals Generic Goals
Specific Practices Generic PracticesCapability Levels
CL 0
CL 1
CL 2
CL 3
CL 4
CL 5
Process Areas
-
7/29/2019 2004 11 17 Test Process Improvement
7/18
Test process improvementNovember 17th 2004 13
CMMI and Testing
Still at maturity level 3 of the staged representation
But continuous representation allows addressing a
process area depending the need
2 dedicated process areas Verification: you built it right
Validation: you built the right thing
After I have the presentation I learned that there was a third process
area strongly related to testing: Product Integration which strongly
relates to integration testing.
Test process improvementNovember 17th 2004 14
CMMI VerificationYou Built it Right
SG 1 Prepare for Verification SP 1.1-1 Select Work Products for Verification
SP 1.2-2 Establish the Verification Environment
SP 1.3-3 Establish Verification Procedures and Criteria
SG 2 Perform Peer Reviews SP 2.1-1 Prepare for Peer Reviews
SP 2.2-1 Conduct Peer Reviews
SP 2.3-2 Analyze Peer Review Data
SG 3 Verify Selected Work Products SP 3.1-1 Perform Verification
SP 3.2-2 Analyze Verification Results and Identify Corrective Action
-
7/29/2019 2004 11 17 Test Process Improvement
8/18
Test process improvementNovember 17th 2004 15
CMMI ValidationYou Built the Right Thing
SG 1 Prepare for Validation SP 1.1-1 Select Products for Validation
SP 1.2-2 Establish the Validation Environment
SP 1.3-3 Establish Validation Procedures and Criteria
SG 2 Validate Product or Product Components SP 2.1-1 Perform Validation
SP 2.2-1 Analyze Validation Results
Test process improvementNovember 17th 2004 16
CMMI Summary
More on testing compared to SW-CMM (2 new process
areas: verification & validation)
But still very general
Keeps showing us that maybe we need first to focus on
some other process areas
-
7/29/2019 2004 11 17 Test Process Improvement
9/18
Test process improvementNovember 17th 2004 17
Test Maturity Models
Testability Maturity Model (TMM)
Test Organization Maturity (TOM) Testing Assessment Program (TAP)
Proposition KPA for SW-CMM
Test Process Improvement (TPI)
Software Testing Maturity Model (SW-TMM)
Test process improvementNovember 17th 2004 18
(SW-)TMM Testing Maturity Model
Existing models did not adequately address testing
needs
CMM model widely used basis for the structure
Developed by Illinois Institute of Technology
A lot of research in different types of companies
Gathered the best practices and characteristics of
mature and immature organizations
-
7/29/2019 2004 11 17 Test Process Improvement
10/18
Test process improvementNovember 17th 2004 19
TMM Structure Levels
Testing
CapabilityMaturity Goals
Maturity Sub-goals
Activities
/ Tasks /
Responsibilities
Implementation
and Organizational
AdaptationCritical
Views
User /
ClientDeveloper
/ Tester
Manager
indicatecontain
supported by
achieved by
address
organized by
Test process improvementNovember 17th 2004 20
TMM Maturity
Levels
Level 1: Initial
Level 2: Definition
Test Policy and Goals
Test Planning
Test Techniques and Methods
Test Environment
Level 3: Integration
Test Organisation
Test Training Program
Test Life Cycle and Integration
Control and Monitor
Level 4: Management and Measurement
Peer Reviews
Test Measurement
Software Quality Evaluation
Level 5: Optimisation
Defect Prevention
Quality Control
Test Process Optimisation
-
7/29/2019 2004 11 17 Test Process Improvement
11/18
Test process improvementNovember 17th 2004 21
TMM & CMM(I)
Process Change Management, Technology Change
Management, Defect Prevention55
Software Quality Management and Quantitative Process
Management44
Intergroup Coordination; Peer Reviews34
Organizational Process Focus, Organizational Process
Definition, Organizational Process Training33
Project Tracking and Oversight, Software Quality
Assurance23
Requirements Management, Project Planning, Software
Configuration Management22
Supporting KPACMMTMM
Test process improvementNovember 17th 2004 22
TMM - Summary
An organization wide approach
Strong management commitment required (link with SPI)
Well founded model
(Strong) link with CMM(I)
Provides support all the way
-
7/29/2019 2004 11 17 Test Process Improvement
12/18
Test process improvementNovember 17th 2004 23
TPI Test Process Improvement
Based on experience of a lot of consultants (Sogeti)
Closely related to TMap
Is an assessment model
Test process improvementNovember 17th 2004 24
TPI Structure
Test
Maturity
Matrix
Key Areas
Levels
Improvement SuggestionsCheckpoints
-
7/29/2019 2004 11 17 Test Process Improvement
13/18
Test process improvementNovember 17th 2004 25
TPI Structure
Test
Maturity
Matrix
CheckpointsTest Domains:
Key Areas
L: Lifecycle
T: Techniques
I: Infrastructure
O: Organisation
MaturityLevels
Test process improvementNovember 17th 2004 26
TPI Summary
Highly practical, highly detailed
Focus is more on higher level testing
Project focus, instead of organizational focus
Limited maturity level and scope (some org. need more)
No link with SPI (such as CMMI)
Strongly TMap related
-
7/29/2019 2004 11 17 Test Process Improvement
14/18
Test process improvementNovember 17th 2004 27
IDEAL
SEI: Software Engineering Institute / Carnegie Mellon
University
Originally a life-cycle model for software process
improvement based on the CMM
An organizational improvement model
How to introduce CMM improvements into an
organization?
Test process improvementNovember 17th 2004 28
IDEAL Structure
Source: SEI
-
7/29/2019 2004 11 17 Test Process Improvement
15/18
Test process improvementNovember 17th 2004 29
IDEAL Summary
Remembers very important steps during process
improvement
Helps in planning
Looks at the next improvement cycle
Test process improvementNovember 17th 2004 30
Assessment
TMM: has an accompanying assessment model (TMM-
AM)
TPI: is an assessment model
CMMI has SCAMPI: Standard CMMI Appraisal Method
for Process Improvement
SCAMPI is also ISO 15504 (Spice) compliant
-
7/29/2019 2004 11 17 Test Process Improvement
16/18
Test process improvementNovember 17th 2004 31
SCAMPI What to retain
Practice Implementation Indicators (PII) Direct Artefacts
Indirect Artefacts
Affirmations
Combination of Objective Evidence (OE) necessary to
corroborate multiple sources Instruments
Presentations
Documents
Interviews
Test process improvementNovember 17th 2004 32
ADKAR Model
Intended as a coaching tool to help employees through a
change
Published by Prosci in 1999
Research in more than 700 companies
-
7/29/2019 2004 11 17 Test Process Improvement
17/18
Test process improvementNovember 17th 2004 33
ADKAR Structure
Incentives and rewards
Compensation changes
Celebrations
Personal recognition
Reinforcement to sustain the
changeR
Practice applying new skills or using new processes and tools
Coaching
Mentoring
Removal of barriers
Ability to implement required
skills and behaviorsA
Training and education
Information access
Examples
Role models
Knowledge on how to changeK
Fear of job loss
Discontent with the current state
Imminent negative consequence
Enhanced job security
Affiliation and sense of belonging
Career advancement
Status or change in social standing
Acquisition of power or position
Incentives or compensation
Trust and respect for leadership
Hope in the future state
Desire to participate and
support the changeD
Management communications
Customer input
Marketplace changes
Ready-access to information
Awareness of the need for
changeA Management communications
Customer input
Marketplace changes
Ready-access to information Fear of job loss Discontent with the current state
Imminent negative consequence
Enhanced job security
Affiliation and sense of belonging
Career advancement
Status or change in social standing
Acquisition of power or position
Incentives or compensation
Trust and respect for leadership
Hope in the future state
Training and education
Information access
Examples
Role models Practice applying new skills or
using new processes and tools
Coaching Mentoring
Removal of barriers
Incentives and rewards
Compensation changes
Celebrations Personal recognition
E
mployeephasesofcha
nge
Enablers
Incentives and rewards
Compensation changes
Celebrations
Personal recognition
Reinforcement to sustain the
changeR
Practice applying new skills or using new processes and tools
Coaching
Mentoring
Removal of barriers
Ability to implement required
skills and behaviorsA
Training and education
Information access
Examples
Role models
Knowledge on how to changeK
Fear of job loss
Discontent with the current state Imminent negative consequence
Enhanced job security
Affiliation and sense of belonging
Career advancement
Status or change in social standing
Acquisition of power or position
Incentives or compensation
Trust and respect for leadership
Hope in the future state
Desire to participate and
support the changeD
Management communications
Customer input
Marketplace changes
Ready-access to information
Awareness of the need for
changeA
Test process improvementNovember 17th 2004 34
ADKAR Summary
plan communication about a change
identify why changes are not working
helps individuals to see why they have difficulties with a
change
-
7/29/2019 2004 11 17 Test Process Improvement
18/18
Test process improvementNovember 17th 2004 35
Conclusion & Closing
Test Process Improvement is not easy. The hardest part
are not the test techniques; but probably theimplementation of the required changes
Models are here to help, by providing a roadmap, not by
being the silver bullet
Questions? Now
or later at [email protected]
Thank you for your time and attention!
Test process improvementNovember 17th 2004 36
References
CMMIhttp://www.sei.cmu.edu/cmmi
TMM
http://www.asq.org/pub/sqp/past/vol1_issue4/burnstein.html
TPIhttp://www.sogeti.nl/tpi
IDEALhttp://www.sei.cmu.edu/ideal
ADKARhttp://www.change-management.com/