![Page 1: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/1.jpg)
Is CMMI® a Guarantee of Performance Improvement?
CMMI Portugal: 3rd of October | Braga
Isabel Lopes Margarido [email protected]
João Pascoal Faria, Raul Moreira Vidal – FEUP Marco Vieira – FCTUC
![Page 2: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/2.jpg)
introduction
approach overview
metamodel
procedures
conclusion
agenda
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 2/26
![Page 3: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/3.jpg)
CMMI benefits introduction overview procedures metamodel
conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 3/26
Performance Improvements over Time by Category
Performance Category Median Improvement
Cost 34%
Schedule 50%
Productivity 61%
Quality 48%
Customer Satisfaction 14%
Return on Investment 4.0 : 1
[1]
![Page 4: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/4.jpg)
motivation
[7]
[4]
CMMI performance depends on the implementation method SCAMPI: organisation honesty, appraisal team quality, small percentage of the organisation, limited number of affirmations
[3-6]
[6]
[8]
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 4/26
introduction overview procedures metamodel conclusion
![Page 5: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/5.jpg)
• Overhead [14]
implementation problems
• Metrics definition (collect and analyse data) [12][13][14] • Metrics categorization [14]
• Uncorrelated and meaningless metrics [10][14]
• Understand statistical nature of level 4 [9][10], [14]
• Tools setup [14]
• Not all projects are measurable [14]
• Lack of institutionalisation [6][14]
• Dissemination problems [14]
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 5/26
introduction overview procedures metamodel conclusion
![Page 6: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/6.jpg)
problem statement
CMMI Model high variability of performance • dependence on methods used and quality of implementation
Quality of Implementation difficulties in the selection of implementation methods
Performance Indicators need for a performance evaluation method • CMMI V1.3 more focused on performance • SCAMPI does not measure performance
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 6/26
introduction overview procedures metamodel conclusion
![Page 7: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/7.jpg)
purpose
objectives of the research develop an evaluation framework
select implementation methods select performance indicators
to allow early evaluation of quality of implementation of the model organisational performance impact of process improvement initiatives on organisational performance
beneficiaries
organisations implementing CMMI SEI – easily verify performance improvement from one SCAMPI to the next
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 7/26
![Page 8: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/8.jpg)
Improvement Techniques
concept Quality Principles • CMMI-DEV
Organisation • PP • PMC • …
Quality of Implementation
Operational Practices • Tools • Techniques • Procedures • …
Org. Performance
CMMI - ACQ
CMMI - SVC
CMMI - DEV PMI
ISO
TSP
Agile
Lean
Six Sigma Theory of
Constraints
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 8/26
![Page 9: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/9.jpg)
ML5
ML4
ML3
ML2 PA
SG1 SP1.1 SG2 SP2.1 …
n n
1 n 0 or 1 n
1 n 0 or 1 n
implementation methods
PI
concept
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 9/26
introduction overview procedures metamodel conclusion
![Page 10: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/10.jpg)
approach overview
Metamodel
evaluation framework application Repository
Methods x CMMI practices
Performance Indicators • potential • profiles
(ML, methods)
T
A
I
L
O
R
I
N
G
Procedures
E
X
E
C
U
T
I
O
N
• Quality of Implementation • Organisation Performance • Performance Improvement
including aggregation and rating
-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))
Semaphore
Evaluation
-thresholds
PI
Project Department
*
0..1
*
0..1
Goal/Practice
Method*
0..1
* 0..1*
0..1
Aggregation
Organisation
*
0..1
Aggregation
Process PI Product PI
Leading PI Lagging PI
Base Measure
Performance Indicator (PI)
* *
Influences4
*
*
Used to calculate4
Goal/Practice
CMMI Model
Evaluation Framework
Method
1..*
*
Evaluates
0..**Evaluates4
-ratting
Support
*
*
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 10/26
introduction overview procedures metamodel conclusion
![Page 11: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/11.jpg)
approach overview
quality of implementation
degree of implementation of a practice with a given method
some performance indicators measure it
performance indicators
measure the organisation performance
measure the quality of implementation
their aggregation indicates degree of institutionalisation, necessary for generic goals and high maturity evaluation
8th of June, 2011 11/26 Isabel Lopes Margarido, SEPG Europe – Dublin, Ireland
introduction overview procedures metamodel conclusion
![Page 12: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/12.jpg)
definitions
base measure – measure of a single property/characteristic of product, process, project or resource (attribute) [15]
performance indicator – measure that provides estimate/evaluation of an attribute derived from base measures or other derived measures [15]
leading indicator – anticipates quality, allows forecasting and diagnosis [16-17]
lagging indicator – follows an event or tendency, allows appraising [17]
methods – good practices, procedures, techniques, tools, etc. , that are part of the processes of the organisation
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 12/26
Attribute x
Base Measure 1
Attribute y
Base Measure 2
Derived Measure Performance Indicator
introduction overview procedures metamodel conclusion
![Page 13: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/13.jpg)
Directly Adressed Partially Adressed Supports Not Adressed Unrated
[7]
Process PI Product PI
Leading PI Lagging PI
Base Measure
Performance Indicator (PI)
* *
Influences4
*
*
Used to calculate4
Goal/Practice
CMMI Model
Evaluation Framework
Method
1..*
*
Evaluates
0..**Evaluates4
Support
-rating
*
*
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 13/26
introduction overview procedures metamodel conclusion
![Page 14: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/14.jpg)
-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))
Semaphore
Evaluation
-thresholds
PI
Project Department
*
0..1
*
0..1
Goal/Practice
Method*
0..1
* 0..1*
0..1
Aggregation
Organisation
*
0..1
Aggregation
introduction overview repository procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 14/26
![Page 15: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/15.jpg)
aggregation
1 1… *
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 15/26
![Page 16: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/16.jpg)
rating and aggregation example PI: Schedule Estimation Error (%) Time: 2010
Time
Value
threshold2
0%
threshold1
threshold2
P1 (D1)
P2 (D2)
P3 (D1)
P4 (D1)
P5 (D3)
Org:
D1:
D2:
D3:
Legend: Org – organisation, D1 – department 1; P1 – project 1.
threshold1
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 16/26
introduction overview procedures metamodel conclusion
![Page 17: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/17.jpg)
aggregation example
Legend: Org – organisation, D1 – department 1; P1 – project 1; alt – alternative; opt – optional; ,mandat – mandatory; ^ – AND; v – OR.
P3 P2
N/A
N/A
N/A
N/A
N/A
D2
D1
Org
PI1
PI2
PI3
PI4
PI5
M1 (alt M2) (PI1 ^ PI2)
M2 (alt M1) (PI1 ^ PI5)
M3 (opt) (PI3)
M4 (mandat) (PI4)
SP1 M4
SP2 (M1 v M2)
SP3 (M1 v M2) ^ M3
SP4 (M1 ^ M4)
P1 P4
N/A
N/A
N/A
N/A
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 17/26
introduction overview procedures metamodel conclusion
![Page 18: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/18.jpg)
research status
identified some of the problems in the CMMI
gathered performance indicators and implementation methods for a subset of practices
designed the metamodel and preliminary version of the evaluation framework
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 18/26
![Page 19: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/19.jpg)
future work
analyse organisations data (PI, methods, relations between PI)
define the rationale for tailoring the PI
validate the evaluation framework in organisations
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 19/26
introduction overview procedures metamodel conclusion
![Page 20: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/20.jpg)
summary
data aggregation and its analysis is particularly
important in the implementation of the GG and HML
performance indicators are useful to evaluate the quality of implementation
we also map the CMMI practices with methods that organisations can choose and adapt
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 20/26
introduction overview procedures metamodel conclusion
![Page 21: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/21.jpg)
questions
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 21/26
![Page 22: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/22.jpg)
[1] C. P. Team, "CMMI® for Development, Version 1.3," CMU/SEI CMU/SEI-2010-TR-033, ESC-TR-2010-033, November 2010.
[2] Gibson, Diane L., Goldenson, Dennis R., Kost, Keith, Performance Results of CMMI®-Based Process Improvement, CMU/SEI, 2006.
[3] N. Davis and J. Mullaney, "The Team Software ProcessSM (TSPSM) in Practice: A Summary of Recent Results," CMU/SEI CMU/SEI-2003-TR-014, ESC-TR-2003-014, 2003.
[4] N. Davis and J. McHale, "Relating the Team Software ProcessSM (TSPSM) to the Capability Maturity Model® for Software (SW-CMM®)," CMU/SEI-2002-TR-008, ESC-TR-2002-008, March 2003.
[5] J. McHale and D. S. Wall, "Mapping TSP to CMMI," CMU/SEI CMU/SEI-2004-TR-014, ESC-TR-2004-014, 2005.
[6] R. Radice, "Statistical Process Control in Level 4 and Level 5 Software Organizations Worldwide," presented at the Software Technology Conference, 2000.
[7] R. Charette, et al., "Understanding the Roots of Process Performance Failure," CROSSTALK The Journal of Defense Software Engineering, pp. 18-22, 2004.
[8] M. Schaeffer, "DoD Systems Engineering and CMMI," presented at the CMMI Technology Conference and User Group, 2004.
references
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 22/26
![Page 23: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/23.jpg)
[8] A. Takara, et al., "Problems and Pitfalls in a CMMI level 3 to level 4 Migration Process," presented at the Sixth International Conference on the Quality of Information and Communications Technology, 2007.
[9] C. Hollenbach and D. Smith. (2002) A portrait of a CMMISM level 4 effort Systems Engineering. 52-61.
[10] B. Kitchenham, et al., "Lessons Learnt from the Analysis of Large-scale Corporate Databases," presented at the International Conference on Software Engineering, Shanghai, 2006.
[11] D. Breuker, et al., "Reliability of software metrics tools," presented at the International Conference on Software Process and Product Measurement, Amsterdam, 2009.
[12] M. C. P. A. Goulão, "Component-Based Software Engineering: a Quantitative Approach," Doctoral, Departamento de Informática, Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia, Lisboa, 2008.
[13] I. Lopes Margarido, et al., "What is wrong with the CMMI® High Maturity Levels?," in SEPG Europe, Porto, 2010.
[14] M. Philips, "CMMI V1.3 Planned Improvements," presented at the SEPG Europe 2010, Porto, Portugal, 2010.
references
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 23/26
![Page 24: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/24.jpg)
images http://mitografias.files.wordpress.com/2009/08/superman-flying.jpg –adapted, 24-06-2010
http://api.ning.com/files/hpf*xOTebDs- F23o6FETZ3j*3sNiONWjfXjTJCzprPjU5bS1 WJoGgWBjMPIOiQkm3SbZ41ijncrJ4K2aT-6dM9QURwHK3led/Dissemination2.jpg -26-06-2010
http://blog.pmtech.com.br/wp-content/uploads/Square-Paradox.jpg – 29-04-2011
http://www.signsexpressshop.co.uk/prodpics/1103.gif – 29-04-2011
Benjamin Haas/Shutterstock, http://cynthiayildirim.posterous.com/how-can-we-measure-the-size-of-the-universe – 29-04-2011
http://ryanstephensmarketing.com/blog/wp-content/uploads/2009/10/one_size_fits_all.JPG
http://evolvingwe.com/wp-content/uploads/2010/11/image3.png – 29-04-2011
http://occlink.com/wp-content/uploads/process-picture.jpg
http://www.pastinyala.com/images/customised_software_product.jpg – adapted 29-04-2011
http://www.smartkids.com.br/conteudo/especiais/transito/sinalizacao/semaforo.gif – 21-01-2011
http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010
http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010
http://etablissements.ac-amiens.fr/0801372g/matieres/anglais/images/difficult.gif – 25-05-2011
http://3.bp.blogspot.com/-_Z2dYcXxMmA/TbLat4c6i_I/AAAAAAAAAnk/KlLdgG-dgtw/s1600/whereamigoing.jpg – adapted, 25-05-20111
http://www.articulate.com/rapid-elearning/wp-content/uploads/2008/08/summary-objectives450.gif – adapted, 25-05-2011
http://1.bp.blogspot.com/_C3jJLFkkSKs/Rzb7NQdztyI/AAAAAAAAA1A/H8nECRh_76A/s400/ponte.JPG – 06-06-2011
http://www.veryhappypig.com/blog/results.jpg -06-06-2011
http://www.stampa.unibocconi.it/immagini/LA4_economiaq20100603145905.jpg - 06-06-2011
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 24/26
![Page 25: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/25.jpg)
interested in our research?
get involved!
participate in the surveys and/or in the validation
phase
share your experience and/or opinion
contact:
Isabel Lopes Margarido, [email protected]
http://paginas.fe.up.pt/~pro09003/
copyright:
partially sponsored by:
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 25/26
![Page 26: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)](https://reader034.vdocuments.us/reader034/viewer/2022051816/54590532af7959755d8b54af/html5/thumbnails/26.jpg)
MA – Measurement and Analysis ML – maturity level n or * – many org – organisation PA – Process Area PI – performance indicator proj – project SCAMPI ℠ – Standard CMMI Appraisal Method for Process Improvement PP – Project Planning PMC – Project Monitoring and Control SEI – Software Engineering Institute SG – specific goal SP – specific practice SVC – services SW – software TSP℠ – Team Software Process V – version
acronyms
ACQ – acquisition C – case study CAR – Causal Analysis and Resolution CMMI® – Capability Maturity Model Integrated dep – department DEV – development DoD – Department of Defense (United States of America) FEUP – Faculty of Engineering, University of Porto GDM – goal driven measurement GG – generic goal GP – generic practice G/P – goal or practice ISO – International Organization for Standardization KLOC – thousand lines of code
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 26/26