Process Improvement for Software Projects

CMMI and Lean Six Sigma

USC CSC1 51016 Nov 2015

Rick Hefner, Ph.D.California Institute of Technology

rhefner@Caltech.edu

2

Agenda

• Current Challenges Facing the Software Industry

• Capability Maturity Model Integrated

• Lean Six Sigma

• Corporate Support

What are some of the challenges facing software companies?

3

Software Projects Have Historically Suffered from Mistakes

Reference: Steve McConnell, Rapid Development

People-Related Mistakes 1. Undermined motivation 2. Weak personnel 3. Uncontrolled problem employees 4. Heroics 5. Adding people to a late project 6. Noisy, crowded offices 7. Friction between developers and customers 8. Unrealistic expectations 9. Lack of effective project sponsorship 10. Lack of stakeholder buy-in 11. Lack of user input 12. Politics placed over substance 13. Wishful thinking

Process-Related Mistakes14. Overly optimistic schedules 15. Insufficient Risk Management16. Contractor failure Insufficientplanning 17. Abandonment of planningunder pressure 18. Wasted time during the fuzzy front end 19. Shortchanged upstreamactivities 20. Inadequate design 21. Shortchanged qualityassurance 22. Insufficient managementcontrols 23. Premature or too frequentconvergence 25. Omitting necessary tasks from estimates 26. Planning to catch up later27. Code-like-hell programming

Product-Related Mistakes28. Requirements gold-plating 29. Feature creep 30. Developer gold-plating 31. Push me, pull me negotiation32. Research-orienteddevelopment

Technology-Related Mistakes 33. Silver-bullet syndrome 34. Overestimated savings fromnew tools or methods 35. Switching tools in the middleof a project 36. Lack of automatedsource-code control

Standish Group survey of 13,000 projects (2003)• 34% successes• 15% failures• 51% overruns4

Top Software Engineering Issues in Defense Industry

1. The impact of requirements upon software is not consistently quantified and managed in development or sustainment

2. Fundamental system engineering decisions are made without full participation of software engineering

3. Software life-cycle planning and management by acquirers and suppliers is ineffective

4. The quantity and quality of software engineering expertise is insufficient to meet the demands of government and the defense industry

5. Traditional software verification techniques are costly and ineffective for dealing with the scale and complexity of modern systems

6. There is a failure to assure correct, predictable, safe, secure execution of complex software in distributed environments

7. Inadequate attention is given to total lifecycle issues for COTS/NDI impacts on lifecycle cost and risk

5

“Top Software Engineering Issues In Defense Industry”, NDIA Systems Engineering Division and Software Committee, Sep 2006

Many Approaches to Solving the Problem

• Which weaknesses are causing my problems?• Which strengths may mitigate my problems?• Which improvement investments offer the best return?

People

Product

Technology

Tools

ManagementStructure

BusinessEnvironment

Process

Methods

6

7

Heritage of Standards for Systems Engineering

EIA / IS 632

ISO/IEC 15288

Mil-Std-499BMil-Std-

499A

1994

1994

1994

19982002

1974

(Not Released)Mil-Std-499

1969

(Trial Use)IEEE 1220

1998

(Full Std)

LegendSupersedesSource for

EIA632

1998

EIA/IS 731

SE CM

IEEE 1220

(Full Std)(Interim Standard)

2002

CMMI-

SE/SW/IPPD

(Interim Standard)

(FDIS)

ISO/IEC 19760

2002

(PDTR)

2002

ISO/IEC 15504(FDIS)

Standards for Systems Engineering, Jerry Lake, 2002

8

The Frameworks QuagmireSarah A. Sheard, Software Productivity Consortium

Evolution of the Frameworks Quagmire, Software Magazine

9

Two Complimentary Approaches to Process Improvement

Data-Driven (e.g., Lean Six Sigma)

• Clarify what your customer wants (Voice of Customer)

– Critical to Quality (CTQs)• Determine what your processes

can do (Voice of Process)– Statistical Process Control

• Identify and prioritize improvement opportunities

– Causal analysis of data• Anticipate your customers/

competitors (Voice of Business)– Design for Six Sigma

Model-Driven (e.g., CMMI)

• Determine the industry best practice

– Benchmarking, models• Compare your current practices to

the model– Appraisal, education

• Identify and prioritize improvement opportunities

– Implementation– Institutionalization

• Look for ways to optimize the processes

Capability Maturity Model Integrated

10

How would you ensure best practices are implemented consistently in your

software company?

12

What is the Capability Maturity Model Integrated?

• The CMMI is a collection of industry best-practices for engineering, services, acquisition, project management, support, and process management

– Developed under the sponsorship of DoD– Consistent with DoD and commercial standards

Three Constellations sharing common components and structure

• CMMI for Development - used by engineering organizations

• CMMI for Acquisition - used by buyers (e.g., govt. agencies)

• CMMI for Services - used by service providers (e.g., help desk)

13

Basic Building Blocks – 22 Process Areas

Project Management• Project Planning• Project

Monitoring and Control

• Supplier Agreement Management

• Integrated Project Management)

• Risk Management

• Quantitative Project Management

Engineering• Requirements

Development• Requirements

Management• Technical

Solution• Product

Integration• Verification• Validation

Support• Configuration

Management• Process and

Product Quality Assurance

• Measurement and Analysis

• Decision Analysis and Resolution

• Causal Analysis and Resolution

Process Management

• Organizational Process Focus

• Organizational Process Definition

• Organizational Training

• Organizational Process Performance

• Organizational Performance Management

Implemented byeach project

Implemented bythe organization

14

Expected Practices Provide Guidancefor Implementation and Institutionalization

SG 1 Establish Estimates SP 1.1 Estimate the Scope of the Project SP 1.2 Establish Estimates of Work Product

and Task Attributes SP 1.3 Define Project Lifecycle Phases SP 1.4 Estimate Effort and Cost

SG 2 Develop a Project Plan SP 2.1 Establish the Budget and Schedule SP 2.2 Identify Project Risks SP 2.3 Plan Data Management SP 2.4 Plan the Project’s Resources SP 2.5 Plan Needed Knowledge and Skills SP 2.6 Plan Stakeholder Involvement SP 2.7 Establish the Project Plan

SG 3 Obtain Commitment to the Plan SP 3.1 Review Plans That Affect the Project SP 3.2 Reconcile Work and Resource Levels SP 3.3 Obtain Plan Commitment

GG 2 Institutionalize a Managed Process GP 2.1 Establish an Organizational PolicyGP 2.2 Plan the ProcessGP 2.3 Provide ResourcesGP 2.4 Assign ResponsibilityGP 2.5 Train PeopleGP 2.6 Manage ConfigurationsGP 2.7 Identify and Involve Relevant

StakeholdersGP 2.8 Monitor and Control the ProcessGP 2.9 Objectively Evaluate AdherenceGP 2.10 Review Status with Higher Level

ManagementGG 3 Institutionalize a Defined Process

GP 3.1 Establish a Defined ProcessGP 3.2 Collect Improvement Information

Project Planning – Implementation Project Planning - Institutionalization

15

Practice Ratings for the Organization/Projects

16

How is the CMMI Used for Process Improvement?

www.sei.cmu.edu/ideal/

IDEAL Model

17

Typical CMMI Benefits Cited in Literature

• Reduced costs– 33% decrease in the average

cost to fix a defect (Boeing)– 20% reduction in unit

software costs (Lockheed Martin)

• Faster Schedules– 50% reduction in release

turnaround time (Boeing)– 60% reduction in re-work

following test (Boeing)

• Greater Productivity– 25-30% increase in productivity

within 3 years (Lockheed Martin, Harris, Siemens)

• Higher Quality– 50% reduction of software

defects (Lockheed Martin)

• Customer Satisfaction– 55% increase in award fees

(Lockheed Martin)

Lean Six Sigma

18

19

What is Lean Six Sigma (LSS)?

• Lean Six Sigma is a powerful approach to improving business

• LSS improvement projects are performed by teams

• Teams use a set of tools and techniques to understand problems and find solutions

• Lean Six Sigma integrates tools and techniques from two proven process improvement methods

+

DMAIC: Define-Measure-Analyze-Improve-Control

A proven, structured problem-solving approach for improving existing processes

• Define: Identify the issue causing the perceived problem or improvement opportunity

• Measure: Collect data from the process

• Analysis: Study the data for cause and effect

• Improve: Eliminate cause(s) and determine resulting effects

• Control: Sustain the gains by monitoring the system

20

21

DMAIC Toolkit

• Charter• Kano Model• Voice of the

Customer• Voice of the

Business• Quality

Function Deployment

• Voice of the Process

• Data Collection Methods

• Measurement System Evaluation

Statistical Controls • Control

Charts• Time Series

methodsNon-Statistical Controls• Procedural

adherence• Performance

Management• Preventive

measures

• Design of Experiments

• Modeling• ANOVA• Tolerancing• Robust Design• Systems

Thinking• Decision & Risk

Analysis

• Cause & Effect Diagrams/ Matrix

• Failure Modes & Effects Analysis

• Statistical Inference

• Reliability Analysis

• Root Cause Analysis, including 5 Whys

• Hypothesis Test

Define Measure Analyze Improve Control

22

Lean - Definition

• Lean is the philosophy of improving performance by reducing waste and eliminating bottlenecks

• Used together, the two methodologies become a powerful set of tools for improving business processes "The 8 Wastes - DOWNTIME" by

Source (WP:NFCC#4). Licensed under Fair use via Wikipedia.

23

Organizational Adoption: Roles & Responsibilities

• Champions – Facilitate the leadership, implementation, and deployment

• Sponsors – Provide resources

• Process Owners – Responsible for the processes being improved

• Master Black Belts – Serve as mentors for Black Belts

• Black Belts – Lead major Six Sigma projects– Typically requires 4 weeks of training

• Green Belts – Lead minor Six Sigma teams, or serve on improvement teams under a Black Belt

– Typically requires 2 weeks of training

24

A Typical Lean Six Sigma Project in Aerospace

The organization notes that systems integration has been problematic on past projects (budget/schedule overruns)

A Six Sigma team is formed to scope the problem, collect data from past projects, and determine the root cause(s)

The team’s analysis of the historical data indicates that ineffective peer reviews are leaving significant errors to be found in test

Procedures and criteria for better peer reviews are written, using best practices from past projects

A pilot project uses the new peer review procedures and criteria, and collects data to verify they solve the problem

The organization’s standard process and training is modified to incorporate the procedures and criteria, to prevent similar problems on future projects

Corporate Support

25

26

Organizational Infrastructure Required

Policies, Processes,Templates & Tools

Best-Practice Libraries

Process Group

Audits & AppraisalsMeasurement RepositoriesPredictive Modeling

Developing and maintaining mature processes requires significant time and investment in infrastructure

Process ImprovementTraining Program

Communications

0

5

10

15

20

25

1 11 21 31 41 51 61 71

UCL

_X

Defe

cts

per c

ompo

nent

Component #

0

5

10

15

20

25

1 11 21 31 41 51 61 71

UCL

_X

Defe

cts

per c

ompo

nent

Component #

27

Northrop Grumman Approach:Institutionalizing Our Improvements

InternalBest Practices

ISO/AS9100 Findings

CMMI Appraisal Findings

Policy

Procedures

Process

eToolkit PAL

WorkbenchStartIt! My MS Portal

Tools

Information

Checklists and Guides

Templates and Examples

Disposition

Independent Audits

• Systems/ Software Engineering Process Group

• Working Groups• Program

Management Advisory Board

Lessons Learned & Metrics

Analysis

Customer Comments

Configuration Control Board

ExternalBest Practices

Industry Standards

Six Sigma Projects

We systematically analyze quality and process data and trends to determine

how to improve our processes

We improve our process assets based on internal and external best

practices

Deployed to

programs

Increasing program

efficiency

msCAS

PCDB

28

Lessons Learned

• Multiple improvement initiatives helps encourage a change in behavior as opposed to “achieving a goal”

– Reinforces that change (improvement) is a way of life

• Benefits result from institutionalizing local improvements across the wider organization

– CMMI establishes the needed mechanisms

• Training over half the staff as Lean Six Sigma Green Belts resulted in a change of language and culture

– Voice of Customer, data-driven decisions, causal analysis, etc.– Better to understand/use tools in everyday work than to adopt the “religion”

• CMMI and Lean Six Sigma compliment each other– CMMI can yield behaviors without benefits– Lean Six Sigma improvements based solely on data may miss innovative improvements

(assumes a local optimum)