presentation from january 8, 2008 dinner meeting
TRANSCRIPT
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
1/20
1
Courtney LaneINCOSE WMA Dinner Meeting
January 8, 2008
Its More Than Just Numbers:How to Implement a Qualitative
Risk Analysis Approach
This document is confidential and is intended solely for the use and information of the client to whom it is addressed.
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
2/20
2
P urpose
Describe the difference between qualitative andquantitative risk analysis
Discuss common problems associated with qualitativeanalysis
Examine best practices for implementing a qualitative riskanalysis approach
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
3/20
3
S tandard Risk Management P rocess
DevelopStrategy
This presentation will focus on the analysis sub-process. This sub-processshould be defined in an organizations risk management strategy
Analyze
PlanMonitor
andControl
Identify
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
4/20
4
C omponents of Risk Analysis
Probability of occurrence
Impact to a projects cost, schedule, and technicalobjectives
Timeframe
Handling method
B y identifying these components, project managers will be able to determinethe magnitude of the risk and effectively prioritize handling of the risks
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
5/20
5
D ifference Between Qualitative and QuantitativeAnalysis
Qualitative vs. Quantitative
U se ranges to selectprobability and impactrankings
Absolute numericalestimates are used for impact and probability
Focuses on overall level of individual risks (e.g. high,medium, low)
Allows projects toaggregate individual risksto determine the overalllevel of project risk
Allows programs to quicklyprioritize risks based on anumber of criteria
U ses complicated analysissuch as Monte Carlosimulation to prioritizerisks
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
6/20
6
W hy use Qualitative Analysis over Quantitative?
Lack of historical data
Lack of the necessary time and resources to conductcomplicated analysis
Desire to use criteria for prioritization that is difficult toquantify
Timeframe
Critical dependencies Political considerations
The type of analysis appropriate for an organization is dependent upon therisk management maturity level of the organization
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
7/20
7
Risk Management Maturity ModelLevel C haracteristics1: Chaotic isks are ignored or managed in an ad hoc fashion
2: Reactive Processes definedManagement uncomfortable discussing riskMost efforts spent on resolving issues
3 : Proactive Processes standardized across the organizationManagement comfortable with risk managementRisk management data used in decision making processes
4 : Predictive U se of quantitative analysis to predict future resultsCost/benefit analysis implemented
5 : Optimization Opportunities to exceed project objectives are identified andmanagedS ource: Modified from SEI s Capability Maturity Model
M ost organizations are not at an appropriate maturity level toimplement quantitative risk analysis
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
8/20
8
Top P roblems Related to Qualitative Analysis
#1U sing inappropriate criteria to determineprobability and consequence levels
#2 Trying to quantify qualitative data
#3Relying on risk scores generated by riskmanagement tools to determine a list of toprisks
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
9/20
9
P roblem #1: Inappropriate C riteria
Criteria is often too detailed
Subject matter experts lack the necessary expertise toappropriately select the criteria
Easy to game the analysis since justification of criteriaselection is extremely subjective
P oor criteria definitions make it difficult for risk analysts toaccurately reflect impacts to cost, schedule, and performance
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
10/20
10
S olution: Understand Your Audience
Determine what questions decision makers want the riskmanagement process to answer
Assess the level of expertise of those performing the riskassessments
Select criteria that balances the needs and abilities of both parties
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
11/20
11
Ex amples of Qualitative Analysis C riteriaNegligible
(A)Marginal
(B)
S ignificant(C )
C ritical(D )
C atastrophic(E )
C ost Minimal or noimpact
$2 5K- $5 0K $5 0K- 100 K $100 K- 25 0K > $25 0K
S chedule Schedule slip < 7 days
Schedule slipbetween 7 and 1 4 days
Schedule slipbetween 1 4 and21 days
Schedule slipbetween 21 and3 0 days
Schedule slipgreater than 1month
P erformance Minimal or noimpact
No impact to keyperformanceparameter;
degradedcapabilityperformance
Degradedenterprisecapability
performance
Degradedcapabilityperformance with
impacts to onekey performanceparameter
Degradedcapabilityperformance with
impacts to morethan one keyperformanceparameter
C riteria is too detailed and thus difficult to assess
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
12/20
12
Ex amples of Qualitative Analysis C riteriaNegligible
(A)Marginal
(B)S ignificant
(C )C ritical
(D )C atastrophic
(E )
C ost Minimal or noimpact
10% of overallbudget
S chedule Minimal or noimpact
Additionalresourcesrequired; able tomeet need dates
Minor slip in keymilestones; notable to meet needdates
Major slip in keymilestone or critical pathimpacted
Cant achieve keyteam or major project milestone
P erformance Minimal or noimpact
Acceptable withsome reduction inmargin
Acceptable withsignificantreduction inmargin
Acceptable; noremaining margin
U nacceptable
S ource: Risk Management Guide for DoD Acquisition
C riteria is extremely subjective and could be easily gamed
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
13/20
13
Ex amples of Qualitative Analysis C riteriaNegligible
(A)Marginal
(B)
S ignificant(C )
C ritical(D )
C atastrophic(E )
C ost Minimal or noimpact
Cost impactscontained withinthe project
Cost impactscontained withinthe program
Cost impactscontained withinthe organization
Cost impactsaffect externalpartners andcould jeopardizeprogram
S chedule Minimal or noimpact
Impacts to criticalpath containedwithin the project
Impacts to criticalpath containedwithin the program
Impacts to criticalpath of more thanone program inthe organization
Impacts thecritical path of external partnersand could
jeopardizeprogram
P erformance Minimal or noimpact
Impacts to criticalfunctions of the
project
Impacts to criticalfunctions of the
program
Impacts to criticalfunctions of the
organization
Impacts to criticalfunctions of
external partnersand could jeopardize theprogram
C riteria definitions are easy to assess and impact levels areconsistent across criteria
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
14/20
14
P roblem #2: Quantifying Qualitative D ata
Engineers often feel more comfortable using numberswhen performing analysis
Assign numbers to levels of impact U se specific probabilities instead of ranges
Risk scores often determined by multiplying impact by probability
However, most risk analysis uses uncalibrated ordinalscales
Cannot perform math on uncalibrated scales Gives a false sense of rigor Decisions made on inaccurate data
P erforming math on qualitative data may lead to inaccurateconclusions and a false sense of rigor
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
15/20
15
S olution: D o Not Include Numbers in Analysis
Avoid assigning numbers to levels of impact andprobability
U se probability ranges U se letters to represent levels of impact and probability
U se the highest level of individual impacts to determineoverall impact (dont average impacts)
U se a risk matrix to determine overall level of risk
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
16/20
16
Ex ample Qualitative Risk Analysis
O verall level of riskis Red
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
17/20
17
P roblem #3: O veremphasizing Risk S cores
If problem #2 is not overcome, this becomes a muchlarger problem
Even if numerical scores are not used, rankings could beincorrect
Risk tools can generate a list of top risks based on locations on therisk matrix
Cannot rely solely on the placement on the risk matrix
Dependencies, political issues and timeliness factors also play apart
O veremphasis of risk scores to determine top risks resultsmay result in poor decisions made by senior leaders
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
18/20
18
S olution:
Tailor list of top risks based on the audience
Consider all factors, not just likelihood and consequence,when creating the list
Do not use arbitrary thresholds to determine handlingstrategies
Aim to manage all yellow and red risks
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
19/20
19
S ummary and C onclusion
Select the appropriate type of analysis based on the riskmanagement maturity level of the organization
Typical challenges associated with qualitative analysiscan be overcome:
K now your audiences Avoid using numbers to perform analyses Consider other factors when selecting list of top risks
Q ualitative analysis can provide leaders with enoughinformation to make sound decisions if implemented properly
-
8/9/2019 Presentation from January 8, 2008 Dinner Meeting
20/20
20
C ontact Information
Courtney LaneSenior Consultant
Booz Allen [email protected]
7 03- 96 1-800 9