Sigma Tools for Systems Engineering ?· Title: Microsoft PowerPoint - 3808_1002_1119.ppt [Read-Only]…

Download Sigma Tools for Systems Engineering ?· Title: Microsoft PowerPoint - 3808_1002_1119.ppt [Read-Only]…

Post on 21-Aug-2018




0 download


Copyright 2005 Northrop Grumman CorporationPractical Six Sigma Tools for Systems Engineering9th Annual Systems Engineering Conference23-26 October 2006Rick Hefner, Ph.D.Director, Process ManagementNorthrop Grumman CorporationCopyright 2005 Northrop Grumman Corporation1BackgroundSix Sigma has proven to be a powerful enabler for process improvementCMMI adoptionProcess improvement for measurable ROIStatistical analysisThis presentation will focus on practical tools and techniques for use by systems engineersAgendaWhat is Six Sigma?How does it apply to systems engineering?Strategies and lessons learnedCopyright 2005 Northrop Grumman Corporation2Projects Have Historically Suffered from MistakesReference: Steve McConnell, Rapid DevelopmentPeople-Related Mistakes1. Undermined motivation2. Weak personnel3. Uncontrolled problememployees4. Heroics5. Adding people to a late project6. Noisy, crowded offices7. Friction between developersand customers 8. Unrealistic expectations 9. Lack of effective projectsponsorship 10. Lack of stakeholder buy-in 11. Lack of user input 12. Politics placed over substance 13. Wishful thinking Process-Related Mistakes14. Overly optimistic schedules 15. Insufficient Risk Management16. Contractor failure Insufficientplanning 17. Abandonment of planningunder pressure 18. Wasted time during the fuzzy front end 19. Shortchanged upstreamactivities 20. Inadequate design 21. Shortchanged qualityassurance 22. Insufficient managementcontrols 23. Premature or too frequentconvergence 25. Omitting necessary tasks from estimates 26. Planning to catch up later27. Code-like-hell programming Product-Related Mistakes28. Requirements gold-plating 29. Feature creep 30. Developer gold-plating 31. Push me, pull me negotiation32. Research-orienteddevelopment Technology-Related Mistakes33. Silver-bullet syndrome 34. Overestimated savings fromnew tools or methods 35. Switching tools in the middleof a project 36. Lack of automatedsource-code controlStandish Group, 2003 survey of 13,000 projects 34% successes 15% failures 51% overrunsStandish Group, 2003 survey of 13,000 projects 34% successes 15% failures 51% overrunsCopyright 2005 Northrop Grumman Corporation3Many Approaches to Solving the ProblemsWhich weaknesses are causing my problems?Which strengths may mitigate my problems?Which improvement investments offer the best return?PeopleProductTechnologyToolsManagementStructureBusinessEnvironmentProcessMethodsOne solution!Copyright 2005 Northrop Grumman Corporation4Approaches to Process ImprovementData-Driven (e.g., Six Sigma, Lean)Clarify what your customer wants (Voice of Customer)Critical to Quality (CTQs)Determine what your processes can do (Voice of Process)Statistical Process ControlIdentify and prioritize improvement opportunitiesCausal analysis of dataDetermine where your customers/competitors are going (Voice of Business)Design for Six SigmaModel-Driven (e.g., CMM, CMMI)Determine the industry best practiceBenchmarking, modelsCompare your current practices to the modelAppraisal, educationIdentify and prioritize improvement opportunitiesImplementationInstitutionalizationLook for ways to optimize the processesCopyright 2005 Northrop Grumman Corporation5What is Six Sigma?Six Sigma is a management philosophy based on meeting business objectives by reducing variationA disciplined, data-driven methodology for decision making and process improvementTo increase process performance, you have to decrease variationDefects DefectsToo early Too lateDelivery TimeReduce variationDelivery TimeToo early Too lateSpread of variation too wide compared to specificationsSpread of variation narrow compared to specificationsGreater predictability in the processLess waste and rework, which lowers costsProducts and services that perform better and last longerHappier customersCopyright 2005 Northrop Grumman Corporation6A Typical Six Sigma Project in Systems EngineeringThe organization notes that systems integration has been problematic on past projects (budget/schedule overruns)A Six Sigma team is formed to scope the problem, collect data from past projects, and determine the root cause(s)The teams analysis of the historical data indicates that poorlyunderstood interface requirements account for 90% of the overrunsProcedures and criteria for a peer review of the interface requirements are written, using best practices from past projectsA pilot project uses the new peer review procedures and criteria, and collects data to verify that they solve the problemThe organizations standard SE process and training is modified to incorporate the procedures and criteria, to prevent similar problems on future projectsCopyright 2005 Northrop Grumman Corporation7Roles & Responsibilities Organizational ImplementationChampions Facilitate the leadership, implementation, and deployment Sponsors Provide resources Process Owners Responsible for the processes being improvedMaster Black Belts Serve as mentors for Black BeltsBlack Belts Lead Six Sigma projectsRequires 4 weeks of trainingGreen Belts Serve on improvement teams under a Black BeltRequires 2 weeks of trainingCopyright 2005 Northrop Grumman Corporation8Applicability to EngineeringSystem engineering processes are fuzzySystems engineering "parts" are produced using processes lackingpredictable mechanizations assumed for manufacturing of physicalpartsSimple variation in human cognitive processes can prevent rigorous application of the Six Sigma methodology Process variation can never be eliminated or may not even reduced below a moderate level Results often cannot be measured in clear $ savings returned to organizationValue is seen in reduced risk, increased customer satisfaction, more competitive bids, Copyright 2005 Northrop Grumman Corporation9How Six Sigma Helps Process ImprovementPI efforts often generate have little direct impact on the business goalsConfuses ends with means; results measured in activities implemented, not resultsSix Sigma delivers results that matter to managers (fewer defects, higher efficiency, cost savings, )Six Sigma concentrates on problem solving in small groups, focused on a narrow issueAllows for frequent successes (3-9 months) Six Sigma focuses on the customers perception of qualityCopyright 2005 Northrop Grumman Corporation10How Six Sigma Helps CMMI-Based ImprovementFor an individual process:CMM/CMMI identifies what activities are expected in the processSix Sigma identifies how they can be improved (efficient, effective)SG 1 Establish EstimatesSP 1.1 Estimate the Scope of the ProjectSP 1.2 Establish Estimates of Project AttributesSP 1.3 Define Project Life CycleSP 1.4 Determine Estimates of Effort and CostSG 2 Develop a Project PlanSP 2.1 Establish the Budget and ScheduleSP 2.2 Identify Project RisksSP 2.3 Plan for Data ManagementSP 2.4 Plan for Project ResourcesSP 2.5 Plan for Needed Knowledge and SkillsSP 2.6 Plan Stakeholder InvolvementSP 2.7 Establish the Project PlanSG 3 Obtain Commitment to the PlanSP 3.1 Review Subordinate PlansSP 3.2 Reconcile Work and Resource LevelsSP 3.3 Obtain Plan CommitmentExample Project Planning in the CMMI Could fully meet the CMMI goals and practices, but still write poor plans Six Sigma can be used to improve the planning process and write better plansCopyright 2005 Northrop Grumman Corporation11How CMMI Helps Six Sigma Based ImprovementCMM/CMMI focuses on organizational changeProvides guidance on many dimensions of the infrastructureGeneric Practices (all process areas)GP 2.1 Establish an Organizational PolicyGP 2.2 Plan the ProcessGP 2.3 Provide ResourcesGP 2.4 Assign ResponsibilityGP 2.5 Train PeopleGP 3.1 Establish a Defined ProcessGP 2.6 Manage ConfigurationsGP 2.7 Identify and Involve Relevant StakeholdersGP 2.8 Monitor and Control the ProcessGP 3.2 Collect Improvement InformationGP 2.9 Objectively Evaluate Adherence GP 2.10 Review Status with Higher-Level ManagementProcess AreasOrganizational Process FocusOrganizational Process DefinitionOrganizational TrainingOrganizational Process PerformanceOrganizational Innovation and DeploymentCopyright 2005 Northrop Grumman Corporation12Barriers and ChallengesCapturing the first, low hanging fruit makes Six Sigma implementation look easyClearer problems, simpler solutions, bigger payoffsLittle need for coordinationbut later projects are tougherKeeping projects appraised of similar efforts, past and currentFocusing on the pain, not the assumed solutionEngineering process measurements are often difficult to analyzeDirty (or no) data, human recording problemsMay necessitate Define-Measure-Analyze-Measure-Analyze-etc.Must demonstrate the value of quantitative data to managersManagement style - reactive vs. proactive vs. quantitativeLess value in a chaotic environmentMust engage customersCopyright 2005 Northrop Grumman Corporation13Additional ChallengesDifficulty in collecting subjective, reliable dataHumans are prone to errors and can bias dataE.g., the time spent in privately reviewing a documentDynamic nature of an on-going projectChanges in schedule, budget, personnel, etc. corrupt dataAnalysis requires that complex SE processes be broken down into small, repeatable tasksE.g., peer reviewRepeatable process data requires the project/organization to define (and follow) a detailed processCopyright 2005 Northrop Grumman Corporation14Tools & TechniquesCopyright 2005 Northrop Grumman Corporation15DMAIC A Structured Approach to Improving a Process5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVECopyright 2005 Northrop Grumman Corporation16DMAIC DefinePurpose is to set project goals and boundariesEstablishes upfront focus on customer Key products Project charterProcess map List of what is important to customer -- Critical to Quality factors (CTQs)VOCProjectSIPOC5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVECopyright 2005 Northrop Grumman Corporation17Identify Key Stakeholders Early On Develop communication plan based on level of commitment requiredHostileOpposedXUncooperativeXIndifferentXHesitantCompliantOHelp it workOOEnthusiastic SupportRequirements LeadsDevelopersTestersLevel of CommitmentStakeholders (Examples)X = Current Level of CommitmentO = Level of Commitment Necessary for SuccessCopyright 2005 Northrop Grumman Corporation18Suppliers-Inputs-Process-Outputs-Customers (SIPOC)High level as-is process map5 to 7 key steps of main actionUsed to focus on the fundamental elements of the processSidewaysKEY for (xs)Process ParametersNoise ParametersControllable Process ParametersSOP ParametersCritical ParametersINPUTSINPUTS (Xs)OperatorCopy MachinePaperPowerInstructionsDocument to CopySet SizeRequiredPlace Doc intoPosition in or on Copier.Set Light/DarkSettings.Select PaperTray/SourceSet Number ofCopies NeededOUTPUTS (Ys)Copies with:rightnumberrightcontrastcorrectorientationright sizeon rightpaperOUTPUT(xs)N Hinges on LidAuto FeederN Glass Clean(xs)# Copies Button# Copies Required(xs)N Size displaySize buttons(xs)Darker ButtonLighter ButtonAuto Button(xs)8.5 X 11: Landscape8.5 X 11 Portrait8.5 X 14 Landscape8.5 X 14 Portrait11 X 14Trays with adequatesupply of paperProduct (ys)Doc set correctlyDoc set incorrectlyUpside DownProduct (ys)Correct # of CopiesIncorrect # of CopiesProduct (ys)Right size selectedSize too SmallSize too LargeProduct (ys)Right contrastToo LightToo DarkProduct (ys)Correct Tray SelectedIncorrect Tray SelectedPress CopyButton(xs)Copy ButtonN LedsProduct (ys)Machine producingcopiesMachine not producingcopiesRetrieveCopies(xs)Output TrayProduct (ys)Copies availableSUPPLIERS CUSTOMERSCopyright 2005 Northrop Grumman Corporation19Voice of the CustomerCT "critical to" matrix links process or CT tree (columns of the matrix) and product or CTY tree (rows)Critical To Satisfaction (CTS) Critical To Quality (CTQ)Critical To Delivery (CTD) Critical To Cost (CTC) Critical To Process (CTP) -Process parameters which significantly influence a CTQ, CTD, and/or CTC5CT TreeDefineCTQProcessInputThe CT Matrix Structure......CTY Tree (Product Tree)CTX Tree (Process Tree)Copyright 2005 Northrop Grumman Corporation20DMAIC MeasurePurpose is to narrow range of potential causes and establish a baseline capability levelIdentify specific problem(s)Prioritize critical input/process/output measuresValidate measurement systemKey productsCause/effect diagramsFMEAGage R&RData collection planAnalysis results5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVEDataSamplingGage R&RPatternsCapabilityCopyright 2005 Northrop Grumman Corporation21Failure Modes and Effects AnalysisUsed to identify the way in which errors happen; an error mode, the antithesis of functionEmployed as a diagnostic tool in improvementUsed as a prevention tool in designDeals with the three dimensions of an error mode: Severity Detectability FrequencyProcess or Product Name:Prepared by: Page ____ of ____Responsible: FMEA Date ( Orig ) ______________ (Rev) _____________Process Step/Part NumberPotential Failure ModePotential Failure Effects SEVPotential Causes OCCCurrent Controls DETRPNActions Recommended Resp .Actions Taken SEVOCCDETRPNProcess/ProductFailure Modes and Effects Analysis(FMEA)Process/ProductFailure Modes and Effects Analysis(FMEA)Copyright 2005 Northrop Grumman Corporation22Data Collection PlanMeaning of measurement in relation to project, process, or productWhat is counted into or excludedMeasurement validation methodRegular validation during collectionPeriodic validation of samples or aggregates independent of collection tools and methodsFrequency of measurement collectionCalculations used to derive an indirect, aggregated, or accumulated valueHow and where measurements are stored and accessedTools, methods, resources, and assignments requiredOperational Definition and ProceduresData Collection PlanWhat questions do you want to answer?DataWhat Measure type/ Data typeHow measuredRelated conditions Sampling notesHow/where How will you ensure consistency and stability?What is your plan for starting data collection?How will the data be displayed? Measurement Consistency and AccuracyCopyright 2005 Northrop Grumman Corporation23DMAIC AnalyzePurpose is to evaluate data/information for trends, patterns, causal relationships and "root causes" Key productsQuantitative analysis resultsTheory that has been testedDataAnalysis5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVEDoE Organize CausesHypothesis TestingRegressionProcess AnalysisCopyright 2005 Northrop Grumman Corporation24Six Sigma Tool KitQueue 1Queue 2Chi-SquareRegressiont-testANOVAX1YRegression AnalysisStratificationLSL USLProcessSigma = 2.7Capability AnalysisUCLLCLControl ChartsHypothesis Testing01000-100010 20 30UCLXLCLD B F A C E OtherData AnalysisBoxplot ANOVAsCopyright 2005 Northrop Grumman Corporation25Exercise What is Quantitative Management?Suppose your project conducted several peer reviews of similar code, and analyzed the resultsMean = 7.8 defects/KSLOC+3 = 11.60 defects/KSLOC-3 = 4.001 defects/KSLOCWhat would you expect the next peer review to produce in terms of defects/ KSLOC?What would you think if a review resulted in 10 defects/KSLOC? 3 defects/KSLOC?1510501211109876543Observation NumberIndividual ValueMean=7.8UCL=11.60LCL=4.001Copyright 2005 Northrop Grumman Corporation26Exercise What is Required for Quantitative Management?What is needed to develop the statistical characterization of a process?The process has to be stable (predictable)Process must be consistently performedComplex processes may need to be stratified (separated into simpler processes)There has to be enough data points to statistically characterize the processProcesses must occur frequently within a similar context (project or organization)1510501211109876543Observation NumberIndividual ValueMean=7.8UCL=11.60LCL=4.001Copyright 2005 Northrop Grumman Corporation27What Is a Control Chart?A time-ordered plot of process data points with a centerline based on the average and control limits that bound the expected range of variationControl charts are one of the most useful quantitative tools for understanding variationCopyright 2005 Northrop Grumman Corporation28What Are the Key Features of a Control Chart?1510501211109876543Observation NumberIndividual ValueMean=7.8UCL=11.60LCL=4.001Upper Control LimitProcess AverageLower Control LimitTime ordered x-axisIndividual data pointsCopyright 2005 Northrop Grumman Corporation29There are Many Types of Control ChartsIDDefect sperL ineo fCod e2523211917151311975310. performed with unequal sample sizesU Chart of Defect Detected in Requirements DefinitionCopyright 2005 Northrop Grumman Corporation30What is Special Cause and Common CauseVariation?1510501211109876543Observation NumberIndividual ValueMean=7.8UCL=11.60LCL=4.001Common Cause VariationRoutine variation that comes from within the processCaused by the natural variation in the processPredictable (stable) within a rangeSpecial Cause VariationAssignable variation that comes from outside the processCaused by a unexpected variation in the processUnpredictable151050151050Observation NumberIndividual Value_1Mean=7.467UCL=13.36LCL=1.578Copyright 2005 Northrop Grumman Corporation31What Is a Stable (Predictable) Process?IDDefectsp erLineofC ode2523211917151311975310. Chart of Defects Detected in Requirements DefinitionAll data points within the control limits. No signals of special cause variation.Copyright 2005 Northrop Grumman Corporation32What if the Process Isnt Stable?You may be able to explain out of limit points by observing that they are due to an variation in the processE.g., peer review held on Friday afternoonYou can eliminate the points from the data, if they are not part of the process you are trying to predictYou may be able to stratify the data by an attribute of the process or attribute of the corresponding work productE.g., different styles of peer reviews, peer reviews of different types of work products05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #Out of limit points05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #05101520251 11 21 31 41 51 61 71UCL_XDefects per componentComponent #Copyright 2005 Northrop Grumman Corporation33Hearing VoicesVoice of the process= the natural bounds of process performanceVoice of the customer= the goals established for the product/process performanceVoice of the business= process performance needed to be competitiveProcess capability may be determined for the OrganizationProduct lineProjectIndividual Typically, the higher the level of analysis, the greater the variationCopyright 2005 Northrop Grumman Corporation34Common Challenges for EngineeringData are often discrete rather than continuous, e.g., defectsObservations often are scarceProcesses are aperiodicSize of the the object often varies, e.g., software moduleData distributions may not be normalCopyright 2005 Northrop Grumman Corporation35How Do I Address These Challenges?Employ control chart types that specifically deal with discrete data distributions, e.g., u-charts and p-chartsUse control charts that compensate for widely variable areas of opportunityTransform non-normal continuous data to normal data before constructing a control chart Cross check control charts with hypothesis tests where few data points exist Copyright 2005 Northrop Grumman Corporation36Typical Choices in IndustryMost customers care about:Delivered defectsCost and scheduleSo organizations try to predict:Defects found throughout the lifecycleEffectiveness of peer reviews, testingCost achieved/actual (Cost Performance Index CPI)Schedule achieved/actual (Schedule Performance Index SPI)Defect Detection Profile0.0020.0040.0060.0080.00100.00120.00140.00160.00180.00Req'mts Design Code Unit Test Integrate Sys Test Del 90 DaysPhaseDefects/KSLOCAll ProjectsNew ProcessProcess performance Process measures (e.g., effectiveness, efficiency, speed) Product measures (e.g., quality, defect density).Copyright 2005 Northrop Grumman Corporation37How Can Quantitative Management Help?0 5 10 15051015Observation NumberIndividual ValueMean=7.268UCL=11.17LCL=3.3631 2By measuring both the mean and variation, the project/ organization can assess the full impact of an improvementCan focus on reducing the variation (making the process more predictable)Train people on the processCreate procedures/checklistsStrengthen process auditsCan focus on increasing the mean (e.g., increase effectiveness, efficiency, etc.)Train peopleCreate checklistsReduce waste and re-workReplicate best practices from other projectsCan do bothprocess shiftCopyright 2005 Northrop Grumman Corporation38DMAIC ImprovePurpose is to develop, implement and evaluate solutions targeted at identified root causesKey productsCandidate solutionsPilot resultsRisk assessmentImplementation planSolutionsFMEAPilotImplemen-tation5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVECopyright 2005 Northrop Grumman Corporation39DMAIC ControlPurpose is to make sure problem stays fixed and new methods can be further improved over timeProductsControl planProcess documentationKey learningsControlStandardizeDocumentMonitorEvaluateClosure5CONTROL1DEFINE2MEASURE3ANALYZE4IMPROVECopyright 2005 Northrop Grumman Corporation40The Control PlanSystematic tool for identifying and correcting root causes of out-of-control conditionsFocus is on prevention -- not a triggering systemQuestions that must be answered Who owns the process?How will we transition responsibility from the Black Belt to theprocess owner?How do we ensure we maintain the gains?How do we track our results (performance and financial)?Copyright 2005 Northrop Grumman Corporation41Institutionalize Key LearningsCapture knowledge gained from Six Sigma projectResultsKey learningsPotential future projectsCommunicate to rest of organization for knowledge sharing and transferArchive in knowledge management repositoriesCopyright 2005 Northrop Grumman Corporation42Lessons LearnedCopyright 2005 Northrop Grumman Corporation43Mission Success Requires Multiple ApproachesProcess EffectivenessProgram EffectivenessMission Assurance OperationsEffectivenessDashboards for Enterprise-Wide MeasurementCommunications & Best-Practice Sharing Robust Governance Model (Policies, Processes, Procedures)Risk ManagementSystems EngineeringIndependent ReviewsTraining, Tools, & TemplatesCMMI Level 5 for Software, Systems, and ServicesISO 9001 and AS-9100 CertificationSix SigmaCopyright 2005 Northrop Grumman Corporation44BenefitsBased on 18 Northrop Grumman CMMI Level 5 organizationsHaving multiple improvement initiatives helps encourage a change in behavior as opposed to achieving a levelReinforces that change (improvement) is a way of lifeThe real ROI comes in institutionalizing local improvements across the wider organizationCMMI establishes the needed mechanismsCMMI and Six Sigma compliment each otherCMMI can yield behaviors without benefitSix Sigma improvements based solely on data may miss innovative improvements (assumes a local optimum)Training over half the staff has resulted in a change of language and cultureVoice of Customer, data-driven decisions, causal analysis, etc.Better to understand and use the tools in everyday work than to adopt the religionCopyright 2005 Northrop Grumman Corporation45Contact InformationRick Hefner, Ph.D.Director, Process ManagementNorthrop Grumman CorporationOne Space ParkRedondo Beach, CA 90278(310)