transition from sw-cmm to cmmi: the benefits continue · 2003-11-19 · joan weszka - 17 lm...

Joan WeszkaLockheed Martin Mission Systems

Systems & Software Resource Center

© Copyright Lockheed Martin Corporation 2003

CMMI® Technology Conference 2003November 19, 2003

.

Transition from SW-CMM® to CMMI®: The Benefits Continue!

® CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

11/19/03 © Copyright Lockheed Martin Corporation 2003 Joan Weszka - 2

Agenda

ContextBackground on Lockheed Martin’s (LM) CMMI®

transition approachBenefits in the Software CMM® and CMMI® eras

LM Systems Integration (Owego, NY)LM Maritime Systems & Sensors – Tactical Systems(Eagan, MN)LM Maritime Systems & Sensors – Radar Systems(Syracuse, NY)LM Maritime Systems & Sensors – Undersea Systems(Manassas, VA)


Context

A number of Lockheed Martin organizations that tracked quantitative process improvement benefits during their SW-CMM® high maturity journey have now transitioned to CMMI®

Experience to date indicates that these benefits have continued with CMMI® implementationBenefits derived are not attributable only to CMMI®

Many initiatives are underway concurrently with CMMI® (and SW-CMM®) deployment


Lockheed Martin’s CMMI® Deployment Approach

Many Lockheed Martin (LM) companies have institutionalized best-of-breed integrated processes (e.g., IPPD)Multiple process models and standards are in use LM identified industry and internal best practices as sources for corporate-wide process requirementsThe LM Integrated Engineering Process (LM-IEP) standard synthesizes these requirements

Lockheed Martin’s Integrated Engineering Process Standard is being deployed under corporate policy.


Key Tenets of Lockheed Martin’s CMMI® Transition

Address CMMI® in the context of your organization’s business requirements

Lockheed Martin’s Integrated Engineering Process (LM-IEP) standard includes CMMI®, in addition to other standards and requirements (e.g., ISO/IEC 15288, ISO 9001:2000)

Adopt an incremental appraisal approach Lockheed Martin Continuous Appraisal Method (CAM) has been successfully deployed with CMMI® and is being extended for use with LM-IEP


LM-IEP, CAM and CMMI® RelationshipCMMI® provides a set of integrated process and appraisal method requirements

LM-IEPLMLM--IEPIEP

CMMI®Process

Requirements

CMMICMMI®®ProcessProcess

RequirementsRequirements

ARC* Class AAppraisal

Requirements*

ARC* Class AARC* Class AAppraisalAppraisal

RequirementsRequirements** Lockheed Martin Integrated Engineering Process

* Appraisal Requirements for CMMI®® (ARC)

CAM

CAM is an ARC Class A appraisal method.


LM CPSs

ANSI/EIA-632

ISO 9001:2000

IEEE 1220

LM HardwareProcess Standard

ISO/IEC-12207

CMMI®® V1.1

ISO/IEC-15288

LM-IEP Context Diagram

Common Source Standards

OrganizationalStandard Process

LM-IEPStandard

Industry Stds

Government Stds

Domain Specific Standards

Industry Stds

Government Stds

Project Specific Standards

ProjectDefinedProcess

In LM-IEP V1.3 issued 4/02

Reference only

In LM-IEP V2.0 issued 1/0303

Integrated Processes

Integrated Work Products

Integrated Methods

Tailored OSP

EPI Process& Methods

Informative only

CPS – Corporate Policy StatementEPI – Engineering Process Improvement

Company or Business Unit


Continuous Appraisal Method (CAM) Design Goals

Minimize appraisal preparation and reduce costIntegrate process improvement with process appraisal activitiesFacilitate appraisal scheduling and minimize disruption for participantsProvide an appraisal environment conducive to process improvement Promote institutionalization

CAM is being used with CMMI® and being extended for appraisal of LM-IEP requirements.


Project Appraisals

Overview of Incremental Appraisal Using CAM

Maintenance Review

Institutionalization focus with minimal project disruption

OSP = Organizational Standard ProcessPA = Process Area

OSPAppraisal

Appraise OSP Fix Weaknesses Re-appraise

Fix Weaknesses Re-appraise Appraise PA 1

...

Planning




Feedback on CAM usage

CAM has been or is being deployed at 12 Lockheed Martin operating units using CMMI®

6 prior CAMs have been completed using EIA/IS 731

Experience with CAM has been positive:More focus on process improvementLess invasive to programsLess stressful to the organizationMore value-add, in-depth findingsMore cost effective


Lockheed Martin Systems Integration Lockheed Martin Systems Integration -- Owego Process CredentialsOwego Process Credentials

Q-100 Award

19901990

19931993 IBM Baldrige Gold Award

New York State Excelsior Award

19941994

19951995 RIT / USA TODAYQuality Cup Finalist

19961996 ISO 9001:1994 /SW CMM Level 3

19971997 SW CMM Level 5

19981998 LM21 Best Practices

20012001 ISO 9001:2000 /SEI CMMI Pilot Site

20032003 CMMI Level 5 /AS 9100A

20002000 Lean / Six Sigma

TQM

20022002

20032003

20022002

20012001

20002000

19981998

19971997

19961996

19951995

19941994

19931993

19901990


Lockheed Martin Systems IntegrationLockheed Martin Systems Integration% Improvement in Software Defects per Million Delivered Source Lines of Code

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Contributors• Inspection Discipline & Effectiveness• Defect Prevention Process• Process Maturity & Compliance• Tool Usage (Automated Checking Tools)• Increased amount of reuse

Improvements Since 199212.4 % Average per Year80.5% Overall

Inte

grat

ed P

roce

sses

SW C

MM

Lev

el 5

CMM

I Le

vel 3

Capa

ble

CMM

I Le

vel 5

in 2

003

NOTE: Post Delivery Defects Are Defined As Defects That Are Tracked for 2 Years After Customer Delivery

SW C

MM

Lev

el 3

2003


0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

120.0%

140.0%

160.0%

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

CMM

I Le

vel 5

in 2

003

SW C

MM

Lev

el 3

SW C

MM

Lev

el 5

Improvements Since 1992:9.8% Average per Year

ISD

Proc

esse

s

SW C

MM

Lev

el 3

Cap

able

Lockheed Martin Systems Integration Lockheed Martin Systems Integration –– Owego Software ProductivityOwego Software Productivity

Software Productivity (All Software including Reuse)

Contributors:•Increased Reuse (Domain Specific)•Process Maturity and Compliance•Process Consistency•Increased use of High Order Language / 4th Generation / Object Oriented

•Use of development and test tools

CMM

I Le

vel 3

Capa

ble

2003


LM Maritime Systems & Sensors Tactical SystemsProcess Improvement Credentials

Oct. 1999 – Attained SW-CMM® level 4 Dec. 2000 – Attained Systems Engineering Capability Model (EIA 731) level 3Jan. 2001

Began focus on integrated process improvementBegan transition to CMMI®

June 2002 – May 2003 CAM AppraisalOSP: Target profile 5 for CMMI®-SE/SW/IPPD/SS Projects: Target profile 3 for CMMI®-SE/SW

August 2003 – SCAMPISM AppraisalAchieved target profile 3 for CMMI®-SE/SW

® SW-CMM is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


LM Maritime Systems & Sensors Tactical SystemsInitiatives

Streamlining of references and guidance documents which supplement the OSP

• Value stream mapping of engineering process/business model

• LM-IEP gap analysis• Self audit process compliance• Airworthiness manual • Risk Management methodology best practice• Upgrade of Process Asset Library (PAL)• Mechanical engineering guidebook • Change management pilot• Defect prevention pilot


LM Maritime Systems & Sensors Tactical Systems Software Productivity

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

1979 1984 1989 1994 1999 2004

Year

Sour

ce S

tate

men

ts p

er H

our

Level 3CMM®

Level 4 CMM®

Level 3 CMMI®


LM Maritime Systems & Sensors Radar SystemsProcess Improvement Credentials

SW-CMM® level 5 (CBA IPISM) in Dec. 1999Systems Engineering Capability Model (EIA/IS 731) level 3+ (CAM) in Dec. 1999Focus on integrated process improvement including hardware began in 2000Transition to CMMI® began in 2000CMMI®-SE/SW/IPPD target profile 4 (CAM) in Nov. 2002

® SW-CMM and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.


LM Maritime Systems & Sensors Radar Systems Functional Excellence Objectives - 2003

LM 21 business excellenceEnhance program performanceImprove productivityContinue to manage the business with quantitative dataImprove quality of the product and processProvide a supportive infrastructure for process improvementDemonstrate continued process maturity leadershipManage technology innovations to support program performanceDocument processes and procedures core to our successDevelop and train employees for current and future assignments


LM Maritime Systems & Sensors Radar SystemsSoftware Productivity and Quality

Lockheed M artin: NE&SS-Syracuse - Softw are Productivity & Q uality Perform ance H istory

0%

100%

200%

300%

400%

500%

600%

700%

800%

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Softw areProductivityGrow th (%)

-100%

-90%

-80%

-70%

-60%

-50%

-40%

-30%

-20%

-10%

0%

Softw are Error RateReduction (%)

Softw are Productivity and Q uality Perform ance Application of Best Practices and Investm ent Has Resulted in S ignificant Im provem ents in Q uality and Cost. As error rates declined,

productivity increased by 80+% .

CM

M

L4 CM

M

L5CM

M

L2 CM

M

L3

E rror Rate

Productivity G row th

CM

Mi

L4

L


LM Maritime Systems & Sensors – Undersea Systems Process Credentials

Systems & Software Engineering*Software CMM® Level 4 (CBA IPISM) - June 1995Software CMM® Level 5 (CBA IPISM ) - February 1999CMMI® & EIA-731 Level 3 (CAM) - October 2001CMMI® Level 5 (CAM) - October 2002

Quality ManagementISO 9001:1994 - September 1995 AS9000 - November 1997 Defense Contract Management Agency (DCMA) ISO 9001 Qualified - December 1997ISO 9001:2000 - December 2000AS9100A – December 2002

Certificate No.10061

Certificate No.7092

Certificate No.9227

Certificate No.E00015

Certificate No.EN10002

*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.

*Assessed programs comprise over 80% of the Undersea Systems development programs, and all parts of the development cycle.

SM CBA IPI is a service mark of Carnegie Mellon University


LM Maritime Systems & Sensors – Undersea SystemsProcess Chronology

1970sTop-down Structured ProgrammingDesign & Code Inspections

1980-2Functional DecompositionSW Engineering WorkshopAdvanced Design Workshop

1983SW Management Workshop

1984Ada Workshop

1985Requirements Inspections

1986FSC Practices & Measurements

1988SW Technology Steering GroupOrganizational Operating Procedures

1990SW Engineering Process Group FormedFirst SW-CMM® Assessment (Level 3)Formal Estimation Procedures

1991Market Driven QualityReuse Focus

1992Defect Prevention Process

1993Integrated TeamsStandard Development EnvironmentIntegrated Process Group

1994Automated Metrics (MAX)Process Coordination Group (PCG)

1995-6Integrated Process LibraryISO 9001 RegistrationSoftware CMM® Level 4)

1997-9ISO 14001 Registration, AS9000 and

DCMA ISO 9001 qualificationSoftware CMM® Level 5

2000-2ISO 9001: 2000 Certification EIA-731 Level 3CMMI® SE/SW/IPPD/SS Level 5


LM Maritime Systems & Sensors – Undersea SystemsSoftware Product Quality (Defect Rate)

0

500

1000

1500

2000

2500

3000

1980 1984 1988 1992 1996 2000

Maj

or D

efec

ts P

er M

illio

n D

eliv

ered

Sou

rce

Stat

emen

ts

CMM®

Level 3

CMM®

Level 4

CMM®

Level 5

CMMI®

Level 5

Our quality improved as our maturity increased.


LM Maritime Systems & Sensors – Undersea SystemsProcess Improvement Return-on-Investment Summary

Note: Other initiatives underway during this period include• ISO 9001 registration, followed by AS9000• Integrated Teaming, and creation of an Integrated Process Library• Integration of Systems Engineering and SW Engineering

SW-CMM® Level 3

SW-CMM® Level 4

SW-CMM® Level 5

CMMI® Level 5

Improvement Metric 1990 1995 1999 2002Quality Defects/MDSS 600 300 150 51Productivity ESS/Labor Month 220 280 340 379Cost & Schedule +- Variance 15% 10% 8% 8%Rework expressed as a %

of industry avg 6% 3% 2% 2%Reuse Percent 68% 75% 82% 82%


0.5373

1.7010

0.4617

2.7931

4.4119

2.5100

0.00

1.00

2.00

3.00

4.00

5.00

6.00

HDHD DDDD CDCD UTUT CTCT SWITSWIT T&IT&I ATAT Life-cycle phasesMaj

or D

efec

ts /

K-E

quiv

alen

t Sou

rce

Stat

emen

ts

In this case, representing an actual program, Latent Defect Rate = 0.51 Defects / Thousand Equivalent Source Statements

We use defect data from formal inspections to project product quality.

We take corrective actionsif projections show a deviation from the goals.

Latent Defect Projection Model(best fit to Exponential, Inflected-S, Rayleigh)

We produce high quality products by monitoring the quality level throughout the program’s performance.

LM Maritime Systems & Sensors – Undersea SystemsProduct Quality


Sigma Defects/MS1 690,000.02 308,537.03 66,807.04 6,210.05 233.06 3.4

0

100,000

200,000

300,000

400,000

500,000

600,000

700,000

800,000

1.00 2.00 3.00 4.00 5.00 6.00

Sigma Number

Def

/Mill

ion

Parts

5 Sigma

Our quality rate is 20 times better than the average industry rates.

LM Maritime Systems & Sensors – Undersea Systems Software Quality

MS = Million Source Statements

Product Quality Level is in Five Sigma Range


Product Productivity Percent ImprovementDelivered Source Lines of Code per Labor Month

All Languages

0%

50%

100%

150%

200%

250%

300%

350%

400%

1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999

Level 1

Level 2

Level 3

Level 4,5

LM Maritime Systems & Sensors – Undersea Systems Software Productivity vs. SW-CMM® Maturity Level


Reference: “A Correlational Study of the CMM® and Software Development Performance”Lawlis, Flowe & Thordahl, CROSSTALK, September 1995

SCATTER PLOTS BETWEEN “1” AND “3” RATINGS ARE INDUSTRY SAMPLES.LOCKHEED MARTIN DATA HAS BEEN ADDED IN RED.

LM Maritime Systems & Sensors – Undersea SystemsCost & Schedule Estimates vs. Actuals at Maturity Level 5

DATA FROM 8 PROGRAMS IN 2Q 2003

Cost Performance Index (CPI) & Schedule Performance Index (SPI)

0

1

2

3

1 2 3 4,50

1

2

3

1 2 3 4,5

+

++++

+

+

++

++

+++

+++=+++= =

==++

+++

++++++

++++

+ ==

====

CPI

SPI

CMM® Rating

+ +++

CMM® Rating


Summary

At Lockheed Martin, benefits derived during SW-CMM® implementation continue to be realized as CMMI® maturity evolvesAllocating benefits to their sources is difficult when implementing multiple models/standards SW-CMM® and CMMI® are viewed as significant (but not sole) contributors to process improvement return-on-investment (ROI) to date


Contact Information

Joan WeszkaLockheed Martin Mission SystemsSystems & Software Resource Center(301) 240-7013 / Fax [email protected]


AcronymsARC – Appraisal Requirements for CMMI®

CAM - Continuous Appraisal MethodCMMI® - Capability Maturity Model IntegrationESLOC or ESS – Equivalent SLOC/SS; a normalized value derived from new development, plus SLOC/SS that are modified, retained, ported, etc.IPPD - Integrated Product and Process DevelopmentLM - Lockheed MartinLM-IEP - Lockheed Martin Integrated Engineering ProcessOSP – Organizational Standard ProcessPA - Process AreaSLOC – Source Line of CodeSS – Source Statement (sometimes called a “Logical SLOC”)

transition from sw-cmm to cmmi: the benefits continue · 2003-11-19 · joan weszka - 17 lm...

Documents