27 dezfuli nasa risk management
TRANSCRIPT
-
8/17/2019 27 Dezfuli NASA Risk Management
1/21
NASA’s Risk Management Approach
Workshop on Risk Assessment and Safety Decision Making Under Uncertainty
September 21-22, 2010Bethesda, Maryland
Homayoon Dezfuli, Ph.D.NASA Technical Fellow
NASA Headquarters
-
8/17/2019 27 Dezfuli NASA Risk Management
2/21
Outline
• NASA Organization•
• NASA’s Risk Management Approach• Safety Goals and Thresholds for Human Space Flights
2Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
3/21
NASA Organization Structure
NASA Organization
3
-
8/17/2019 27 Dezfuli NASA Risk Management
4/21
Evolution of Risk-related Policy and
Evolution of Policy Documents
• 2002 – Issuance of PRA Procedures Guide• 2004 – Issuance of NPR 8705 “Probabilistic Risk Assessment (PRA) Procedures for
Safety and Mission Success for NASA Programs and Projects”• 2006 – Issuance of NPR 7123.1 “Systems Engineering Processes…”• 2006 – Revision of NPR 8715.3A “NASA General Safety Program Requirements,”
• 2007 – Revision of NPR 7120.5D “Space Flight Project Management Processes…”• 2007 – Reissue of NASA/SP-2007-6105 “NASA Systems Engineering Handbook”• 2008 – Reissue of NPR 8705.2B “Human-Rating Requirements for Space Systems”• 2009 – Issuance of NPD-1000.5 “Policy for NASA Acquisition”• 2009 – Revision of NPR 8000.4A “Agency Risk Management Requirements”• 2009 – Issuance of NASA/SP-2009-569, “Bayesian Inference for NASA Probabilistic
” • 2010 – Issuance of NASA/SP-2010-576 “NASA Risk-informed Decision MakingHandbook”
Emer in themes:
4
Integrated perspective of risk analysis
Scenario-based modeling of riskBetter treatment of uncertaintiesHomayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
5/21
Risk Management Policy
RM Approach
• NASA Policy Directive (NPD) 1000.5 (2009) states: “ It is NASA policy toincorporate in the overall Agency risk management strategy a risk-informed acquisition process that includes the identification, analysis,and management of programmatic, infrastructure, technical,environmental, safety, cost, schedule, management, industry, andexternal policy r isks that might jeopardize the success with which the
” .• NPR 8000.4A (2009), Agency Risk Management Procedural Requirements,
evolves NASA’s risk management approach to entail two complementaryprocesses: – Risk-informed Decision Making (RIDM)
• To risk-inform direction-setting decisions (e.g., space architecture decisions)• To risk-inform the development of credible performance requirements as part of the overall
s stems en ineerin rocess
– Continuous Risk Management (CRM)• To manage risk associated with the implementation of baseline performance requirements
5
RM RIDM + CRM
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
6/21
Motivating Factors for New RM Approach
RM Approach
• To manage risk in a holistic and coherent manner across the Agency – A enc strate ic oals ex licitl drive RM activities at all levels
– All risk types and their interactions are considered collectively duringdecision-making – Implementation of RM in the context of complex institutional relationships
, , , , …
• To better match the stakeholder expectations and the “true” resources
required to address the risks to achieve those expectations – -
making commitments to stakeholders – Having an integrated perspective of risks when analyzing competing
alternatives
• o e er es a s c ose es e ween e se ec e a erna ve an erequirements derived from it – Derivation of achievable requirements through systematic characterization
of uncertainties
6Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
7/21
The RM Process Begins with NASA Strategic
Goals
RM Approach
• Within NASA’s organizationalhierarchy, high-levelo ec ves ra eg cGoals) flow down in the formof progressively more R I D
M P r
o c e s
s
requirements, whose
satisfaction assures that
• RIDM is designed to maintainfocus on strategic goals as
throughout the hierarchy• CRM is designed to manage
“ ”
7
requirements
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
8/21
Definition of Risk in the Context of Mission
Execution er NPR 8000.4
RM Approach
• Risk is the expression of the potential for performanceshortfalls , which may be realized in the future, with respect toachieving explicitly established and stated performancerequirements – The performance shortfalls may be related to any one or more of
the following mission execution domains:
• Safety• • Cost• Schedule
– based on the uncertainty of achieving it
8
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
9/21
What is RIDM and When is it Invoked?
RM Approach
• A risk-informed decision-making process that uses a diverseset of erformance measures some of which are model-based
risk metrics) along with other considerations within adeliberative process to inform decision making. Paragraph A.14of NASA NPR 8000.4A
– Within RIDM, decisions are informed by an integrated riskperspective rather than being informed by a set of individual
“risk” contributions whose cumulative significance is not
– A decision-making process relying primarily on a narrow set ofmodel-based risk metrics would be considered “risk-based”
• design decisions, make-buy decisions, and budgetreallocation (allocation of reserves), which typically involvere uirements-settin or rebaselin of re uirements
9
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
10/21
The RIDM Process as Defined in NPR 8000.4A
RM Approach
• Identification of decisionalternatives (decision context )and considering a sufficientnumber and diversity ofperformance measures to
-
Identification of AlternativesIdentify Decision Alternatives (RecognizingOpportunities) in the Context of Objectives
for decision-making purposes
• Risk analysis is defined broadly
Risk Analysis of AlternativesRisk Analysis (Integrated Perspective) and
Development of the Technical Basis forDeliberation
.analysis of performanceassociated with the alternative
Risk-Informed Alternative SelectionDeliberate and Select an Alternative andAssociated Performance CommitmentsInformed by ( not so lely based on ) Risk
alternative informed by (notsolely based on) Risk Analysis To Requirements Baselining
1010
.
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
11/21
The Continuous Risk Management (CRM) Process
RM Approach
• Is initiated by the results of the RIDM process:• The risk analysis for the selected
alternative•
• Focuses on meeting performancerequirements• By managing performance margins over
requirements are not violated• By “burning down” (over time) the risk of
violating performance requirements•
Continuous Risk Management (CRM)
e e
Risk-Informed Decision Making(RIDM)
Identification of Alternatives
Communicate andDocument
Id nt ye if
A n l y e
a z
P n l a
a c T
k r
C o n
l o r t
Communicate andDocument
Id nt ye if
A n l y e
a z
P n l a
a c T
k r
C o n
l o r t
R I D M
R e s u
l t s
f o r
t
S e
l e c
t e d A l t e r n a
t iIdentify Decision Alternatives (RecognizingOpportunities) in the Context of Objectives
Risk Analysis of AlternativesRisk Analysis (Integrated Perspective) andDevelopment of the Technical Basis for
Deliberation
Risk-Informed Alternative Selection
1111Homayoon Dezfuli, NASA
Deliberate and Select an Alternative and Associated Performance CommitmentsInformed by (not solely based on) Risk
Analysis
-
8/17/2019 27 Dezfuli NASA Risk Management
12/21
RIDM Process StepsBased on NASA/SP-2010-576
RM Approach
Risk-Informed Decision Making (RIDM)NASA/SP-2010-576Version 1.0April 2010
-Step 1 – Understand Stakeholder Expectations
and Derive Performance MeasuresStep 2 – Compile Feasible AlternativesNASA
Risk-Informed Decision Making
Part 2 - Risk Analysis of AlternativesStep 3 – Set the Framework and Choose the
Analysis MethodologiesStep 4 – Conduct the Risk Analysis and
Document the Results
Part 3 - Risk-Informed Alternative SelectionStep 5 – Develop Risk-Normalized Performance
CommitmentsStep 6 – Deliberate, Select an Alternative, andOffice of Safety and Mission Assurance
NASA Head uartersDocument the Decision Rationale
12
To Requirements (Re)Baselining
http://www.hq.nasa.gov/office/codeq/doctree/SP2010576.htm
-
8/17/2019 27 Dezfuli NASA Risk Management
13/21
RIDM Process – Part 1Understand Stakeholder Expectations and Derive Performance
RM Approach
• An objectives hierarchy is constructed by subdividing the top-levelobjectives into more detailed objectives, thereby clarifying the intended
Measures
.
• At the first level of decomposition, the top-level objective is partitionedinto the mission execution domains of Safety, Technical, Cost, andSchedule.
• Within each domain, the objectives are further decomposed untilappropriate quantifiable performance objectives are generated.
13Notional Objectives Hierarchy
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
14/21
RIDM Process – Part 2
Risk Analysis of Alternatives
RM Approach
• The goal is to develop a risk analysis framework that integrates domain-specific performance assessments and quantifies the performancemeasures – s na ys s - pro a st c mo e ng o per ormance
Product ofRisk
Uncertain Conditions Probabilistically - DeterminedOutcomes
Funding
AnalysisRisk Analysis
of an AlternativePerformance Measure 1
Environment
Technology
LimitedData
OperatingEnvironment
…• Safety Risk
• Technical Risk• Cost Risk•
Performance Measure nEtc.
* Performance measures depicted for a single alternative
Design, Test &ProductionProcesses
• Establishing a transparent framework that: – Operates on a common set of performance parameters for each alternative – Consistently addresses uncertainties across mission execution domains and across
14
a ternat ves – Preserves correlations between performance measures
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
15/21
Setting Risk Analysis Framework
RM Approach
• Detailed domain-specific analysis guidance is available in domain-specific guidance documents like the NASA Cost Estimating
, ,
Probabilistic Risk Assessment Procedures Guide
15Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
16/21
RIDM Process – Part 3
Risk-informed Alternative Selection
RM Approach
• Performance measure dfs constitute the fundamental riskanalysis results.
• However, there are complicating factors for performancemeasures that are ex ressed as dfs: – The pdfs for different alternatives may overlap, preventing a definitive
assessment of which alternative has superior performance
PDF(PM(Alt1)) & PDF(PM(Alt2))
0.4
0.6
0.8
1
e r f o r m a n c e
e a s u r e
)
Alternative 1
Alternative 2
– Different pdfs may exceed imposed constraints at different percentiles,
0
0.2
0 1 2 3 4
Performance Measure
P D F ( P M
Good Bad
16
thereby comingling issues of performance with issues of success
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
17/21
Performance Commitment
RM Approach
• Mean values are used in many differentcontexts to compare alternatives, but thisapproach can:
– Produce values that are strongly influenced by
the tail ends of the pdfs – Introduce significant probabilities of falling short
of imposed constraints, even when the mean
• A Performance Commitment is the level ofperformance whose probability of notbeing achieved matches the decisionmaker’s risk tolerance
– Anchors the commitment the decision maker (DM)is willing to make for that performance measure
• - normalized comparison of decisionalternatives, at a level of risk tolerancedetermined by the decision maker.
17Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
18/21
Deliberation of the Merits of Each Alternative in the Context
of Performance Commitments (notional)
RM Approach
Performance commitments are set atperformance measure values that
Risk tolerances given by the shadedareas under the pdfs, on the “bad”
AlternativeA
Direction of Goodness
AlternativeB
PC A1 PC A2 PC A3
AlternativeC
PC B1 PC B2 PC B3
ImposedConstraint
PayloadCapability
Reliability Cost & Schedule
Performance Measures*
PC C1 PC C2 PC C3
18
* These are arbitrary, notional choices
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
19/21
Goals and Thresholds
• Safety performance, like technical performance, schedule and cost is
• We are developing safety goals and thresholds for human space flightto low earth orbit – Safet Goals are desirable safet erformance levels for drivin
safety improvements – Safety Thresholds are criteria for risk acceptability decisions; not
meeting these values is not tolerable – Both goals and thresholds are defined in terms of aggregate risks
• Help designers with safety performance allocation• Help decision makers to deal with safety-related decisions
- Risk acceptance- Risk mitigation- Safety optimization
19
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
20/21
Safety Regimes and Safety Decisions to be
Made
Goals and Thresholds
Standard of “Minimally Safe Level”Less than this would be “ intolerable”
TRESHOLD
Standard of “Optimally and Sufficiently Safe”More than this May have dimin ishing return
GOAL
TOLERABLESAFE ENOUGH INTOLERABLE
Frequency Threshold
(to be met with ≥ X% probability)
Optimization Mitigation
Aggregate Frequency of Scenarios Leading to Loss of Crew
• Don’t proceed withthe acquisition
• Fix design or
• Actively pursue safetyimprovements via risktradeoff studies
• Keep alert forenhancements, butfocus more on
Increase in Decision Flexibility
20
operation to meetthe threshold.
• ct ve y ent yunaccounted-for hazards viaprecursor analysis
maintaining the goodsafety level that hasbeen been achieved
Homayoon Dezfuli, NASA
-
8/17/2019 27 Dezfuli NASA Risk Management
21/21
References• Probabili stic Risk Procedures Guide for NASA Managers and Practi tioners ,
2002.• NPD 1000.5, Policy for NASA Acquisition, 2009.• . , , .
• NPR 8705.2, Human Rating Requirement for Space Systems , 2009.• NASA/SP-2009-569, Bayesian Inference for NASA Probabilistic Risk and
Reliability Analysis , 2009• NASA/SP-2010-576, NASA Risk-Informed Decision-Making Handbook , 2010.• NASA/SP-2011-XXX, System Safety Guidebook (Under development by NASA
Office of Safety and Mission Assurance)
21Homayoon Dezfuli, NASA