© 2017 Minitab, Inc.11/24/2017
Why Projects Fail
ASQ MilwaukeeMay 15, 2017
Paul Sheehy, Technical Training SpecialistJeff Harding, Regional Sales ManagerDennis Corbin, Technical Training Specialist
© 2017 Minitab, Inc.21/24/2017
Agenda
Macro view – Why Projects FailMinitab’s ResponseMicro view – Why Data Analysis FailsCorrecting Common MistakesQuestions
© 2017 Minitab, Inc.31/24/2017
Only 30% of improvementinitiatives succeed.
© 2017 Minitab, Inc.41/24/2017
Lack of Management Support
Project Risk Poorly Assessed
Project Scope Poorly Defined
No Data or Bad Data
© 2017 Minitab, Inc.51/24/2017
Variation in Project Execution
Projects are not Aligned with
Corporate Initiatives
Project Benefits are Hard to
Communicate
Stakeholders have Poor Visibility
© 2017 Minitab, Inc.61/24/2017
Minitab’s Response
Minitab Product Evolution
© 2017 Minitab, Inc.71/24/2017
Introducing Companion by Minitab
Cloud based project management solution focused on continuous improvement methodologies
© 2017 Minitab, Inc.81/24/2017
© 2017 Minitab, Inc.91/24/2017
Introducing Companion by Minitab
Desktop AppProject execution
Web AppCloud‐based dashboard
Standardized Roadmaps
Integrated Tools
Data sharing
Program Impact
Project Visibility
Configurable
© 2017 Minitab, Inc.101/24/2017
© 2017 Minitab, Inc.111/24/2017
• Project Charters• Financial Tracking• Fishbone• FMEA• Process maps• Value Stream maps• Monte Carlo Simulation
• C&E Matrix100+ Forms and Tools
© 2017 Minitab, Inc.121/24/2017
© 2017 Minitab, Inc.131/24/2017
© 2013 Minitab, Inc.
Running With ScissorsASQ – MilwaukeeMay 2017
1 © Minitab Inc., 2017
© 2013 Minitab, Inc.
2 © Minitab Inc., 2017
The best laid schemes o’ mice an’ men, gang oft agley
Project Management
A Macro Problem
(Jeff’s discussion)
© 2013 Minitab, Inc.
Quality of Analysis
Or
A Micro Problem
© 2013 Minitab, Inc.
4 © Minitab Inc., 2017
RUNNING WITH SCISSORS
© 2013 Minitab, Inc.
5 © Minitab Inc., 2017
What if we do everything right
• Get executive buy in.• Select candidates to lead improvement projects. • Put then through extensive crash course in Six Sigma or other
statistically rigorous improvement methodology.• Select good Champions.• Select good projects.
►AND THEN…
© 2013 Minitab, Inc.
6 © Minitab Inc., 2017
The analyses are done by…
► Process improvement leaders (belts) who may have• Received relatively little training time on proper analytical
processes.• Scarce access to qualified statistical help.• Used statistical software (analyze data) infrequently. • Received most of their guidance from others equally trained
(blind leading the blind).• Etc.
► In other words… Running with scissors
© 2013 Minitab, Inc.
7 © Minitab Inc., 2017
Let me count the ways• What’s Normal & Why Care?• Subgroups• Measurement and Not• Capability• Stray Thoughts
© 2013 Minitab, Inc.
8 © Minitab Inc., 2017
Some comments
►I am sure that today’s audience is aware of most of these issues.
►The purpose of this talk is to remind us what kind of thinking and actions may be going on in the trenches.
►These examples have for the most part been culled from real life. (Dragnet)
© 2013 Minitab, Inc.
Normality(and equal variance)
© 2013 Minitab, Inc.
10 © Minitab Inc., 2017
Normality – the monster under the bed
► Early on, Six Sigma presented the following:
► Many took the normality assumption too seriously.
Hypothesis test for means – two independent samples
© 2013 Minitab, Inc.
11 © Minitab Inc., 2017
Example
► A major industrial• In 1997 the Six Sigma Statistician for the company wrote and
presented a paper titled “What’s Normal & Why Care”. It STARTED with a strongly worded conclusion that for t and F tests, only “severely non-normal” data would have a significant effect on the test outcome.
• Many GBs and BBs were stopping their projects claiming their data was not normal.
© 2013 Minitab, Inc.
12 © Minitab Inc., 2017
Two-Sample t-test, Simulations Using Non-Normal DataRandom data from a Chi-Square distribution with 5 df20,000 simulated tests run for n = 5, 8, 10, 15, 20, 25, 30, 40, 50 (both samples)
50403020100
1.00
0.95
0.90
0.85
0.80
X-Data
Y-D
ata
0.95
20100-10
99.9
99
90
50
10
1
0.1
Acceptance Rate vs Sample Size for 2-Sample t Test
Test Population Distribution vs Normal Distribution Normal Probability Plot of Test Population DistributionNormal - 95% CI
SampleSize
AcceptRate(standard)
AcceptRate(Welch's)
0.95230.95010.95290.95600.95090.94740.95010.95290.9487
0.94690.94840.95040.95690.95280.94890.95230.95620.9520
5 810152025304050
Accuracy of p-value:
Standard Test assumes equal variances
Welch’s Test does not assume equal variances
Both tests perform very well, even for samples as small as 5
At n = 5, accept rate is just above 95% for Standard, just below 95% for Welch’s (theoretical rate for both tests is 95%)
Welch’s Test
Standard
Bottom Line:The two-sample t-test is EXTREMELY robust to this type of non-normal data.
© 2013 Minitab, Inc.
From Minitab’s Assistant Menu
13 © Minitab Inc., 2017
© 2013 Minitab, Inc.
14 © Minitab Inc., 2017
Transformations & NonparametricsOne way to safely handle severely non-normal data is to use transforms.
• How well is this taught to GBs, etc.?• If a user only needs this once a year…?
A second alternative is the use of nonparametric tests such as the Wilcoxon or Mann-Whitney (for a one or two sample test of medians).
• These tests also have assumptions (in this case “symmetric data” and “equal shape”).
• Have users been taught these assumptions?
© 2013 Minitab, Inc.
15 © Minitab Inc., 2017
Two actual illustrationsSeveral years ago a Black Belt told me that his Master Black Belt said not to use Xbar-R charts because his data was non-normal.
During a VOC data gathering initiative by Minitab, a Black Belt told us that his MBB would say “Get that out of here, your data is not normal. I don’t even want to look at it and we can’t use it.”
© 2013 Minitab, Inc.
16 © Minitab Inc., 2017
What about Equal Variance(2-sample t)►If you assume equal variances (standard test) the results are quite sensitive to the interaction of sample size and unequal variances.
►If you assume unequal variances (Welch’s test), the results are robust even for huge differences in variation and sample size.
© 2013 Minitab, Inc.
Subgroups,Control Charts, andCapability
© 2013 Minitab, Inc.
18 © Minitab Inc., 2017
Would you believe…
Minitab’s Tech Support has seen the following relative to capability analysis:
• Sort the data by the output value to decrease the within subgroup variation and thus improve the Cpk.
• Users tried different subgroup sizes to see what number gave them a good Cpk.
• Presented capability with NO validation of stability.• Use of Capability Analysis-Normal when data is not normal.• Asked how to “back-calculate” spec limits for a particular z.usl
or z.lsl value. Etc.
© 2013 Minitab, Inc.
19 © Minitab Inc., 2017
Subgroups and Control Charts
Tech support had a call• “ I have an I-MR chart with a lot of points out of control.
I know why these particular points are so extreme, but what can I do to make my chart look better.”
Another issue• Improper selection of subgroups leads to overly
generous (and incorrect) UCL and LCL values on an X-Bar chart. Assume a four cavity mold for soap. We could choose as a rational subgroup the four bars made each time the press cycled and we would get…
© 2013 Minitab, Inc.
20 © Minitab Inc., 2017
UCL=109.19
LCL=96.60
Because the die was improperly set up, additional (assignable) variation was introduced into the subgroup
© 2013 Minitab, Inc.
21 © Minitab Inc., 2017
With correct subgroupingCavity 1 is shown out of control (note the different UCL and LCL). n = 4
UCL=105.41
LCL=94.20
© 2013 Minitab, Inc.
Measurement
© 2013 Minitab, Inc.
23 © Minitab Inc., 2017
Basic Failure Modes for both Gage R&R and Attribute Agreement Analysis
• The measurement is poorly defined (e.g. diameter of dowel).
• A “ringer” is thrown in as one of the appraisers.• Test not done in “actual” conditions (time, noise, light,
etc.)• Conversion of continuous data into attribute data.• Use of continuous Gage R&R to analyze attribute data.• When Gage R&R or Attribute Agreement Analysis do
not “fit”, the analyst claims (incorrectly) that MSA is not needed.
© 2013 Minitab, Inc.
24 © Minitab Inc., 2017
A warning re: Gage R&R
Gage R&R is essentially a random effects ANOVA and as such the parts sampled (usually 10) should be selected randomly. HOWEVER, the AIAG standards that use the ratio of 0 to10% for ideal and 10 to 30% for marginal are based on using the process width (defined as 6σ) as the denominator. If we use the standard deviation obtained from our sample without validating it against some historical or benchmark value we could be seriously off.
© 2013 Minitab, Inc.
25 © Minitab Inc., 2017
Let us set up an example
We have a process that is known to the gods on Mt. Olympus to have a standard deviation of 10 and a mean of 300.
• Sample 1: We tell the supervisor to get us 10 random parts and she pulls them over the course of two days that have similar environmental conditions resulting in a sample with a small process σ of 5.56.
• Sample 2: We need a sample and the process is not currently running so we go to the engineer’s wall of shame and randomly grab 10 parts (note these are extreme highs and lows – all parts more than ±3σ).
• Sample 3: We grab a truly random sample of 10 parts.
© 2013 Minitab, Inc.
26 © Minitab Inc., 2017
Some Data
• 6σ of the measurement system actual but unknown is 12.78
• Sample 1: 6σ of the process is 6 x 5.56 = 33.36
• Sample 2: 6σ of the process is 6 x 35.00 = 210.00
• Sample 3: We will look at the distribution of standard deviations of 1,000 samples of n = 10
6σ MS
6σ ProcessTotal Variation
© 2013 Minitab, Inc.
27 © Minitab Inc., 2017
Raw Data
If you look at all 1,000 sample standard deviations:
The max of 17.56 is 496% of the min of 3.54
© 2013 Minitab, Inc.
28 © Minitab Inc., 2017
Sigma Distribution (n = 10)
© 2013 Minitab, Inc.
29 © Minitab Inc., 2017
Results of Gage/TV
Pulling a representative sample is vital
© 2013 Minitab, Inc.
CapabilitySome interesting issues
• Not validating stability (and even the measurement system) prior to calculating capability. (Small sample size issue).
• Cpk vs. Ppk.• Possible lack of understanding of the defect rate given only a
Cpk or Ppk.• We have seen capability analysis continuous used with
dichotomous data (essentially 0/1).• No gut check before running the computer command.
30 © Minitab Inc., 2017
© 2013 Minitab, Inc.
Some Stray Thoughts
© 2013 Minitab, Inc.
32 © Minitab Inc., 2017
(Un)common sense examples
• In an attribute inspection process, a Black Belt claimed 10 opportunities per part (denominator), but discarded the part without further inspection when any single defect was found.
• With large data sets the AD normality test can prove the slightest departure from normality.
© 2013 Minitab, Inc.
33 © Minitab Inc., 2017
Additional interesting items
• Confusion between paired t and 2-sample t.• Use of a t test to prove equivalence (can’t prove the null).• Chi-Sq vs. a series of proportion tests.• Individual error rate vs. family error rate when doing a series of
tests.• Too much data (prove differences that are not of practical
significance).• Lose time sequence of data.• Convert variable data to discrete data.• Unchecked use of historical data.
© 2013 Minitab, Inc.
Minitab Available Resources:
1. Customized webinars – Feel free to have people reach out to me at my contact information below. Whether they are looking for help on basic training for Minitab or a more advanced topic like DOE I can help pull in the appropriate resources to help get them up to speed on Minitab related topics through a webinar or even onsite visit
2. Techinical and Statistical Support Line – 814-231-2682
a. Most people just think to call our support line for installation and technical questions. Through this line we also have an entire team of Statistical experts ready to help you answer questions about how to do certain analyses.
3. Free Public Webinars - http://www.minitab.com/en-us/support/webinars/
a. This is an ever changing schedule taught by one of our statisticians or trainers on various topics. They are open for anyone to attend are also recorded for review later
4. Support Page
a. This has video tutorials, example data sets, our blog, and documentation to help you learn Minitab or get your questions answered.
Free case study: www.minitab.com/crayola