Qualitative Risk Assessment

Risk Analysis for Water Resources Planning and Management

Institute for Water Resources

2008

Risk AssessmentWhat can go wrong?How can it happen?How likely is it?What are the consequences?

Want to Improve Your Risk Analysis?

Use simple narratives that answer these questions honestly

Tell story of existing riskTell story of residual or transformed risk

The NeedManage risk intentionallyDo better than has been doneQuantitative risk assessment not always possible or necessaryQualitative risk assessment can be a viable option

Qualitative Risk AssessmentIs formal, organized, reproducible method based on science and sound evidenceFlexibleEasy to explain to othersSupports risk management decision making

Three Sample MethodsEnhanced Criteria RankingOperational Risk Management (Risk Matrix)CARVER + Shock

Enhanced Criteria Based Ranking

CriteriaRatingsAll Possible Combinations of RatingsRankingEvaluate Reasonableness of RankingAdd CriteriaNew Combinations of RatingsNew Ranking

Question?Which lock gates in division present the greatest potential risk to health and safety and therefore should be repaired first?

Step One: CriteriaAssume criteria equally important (or not).Reflect most important aspects of evaluating risk.Define H, M, L scenarios for each criterion. Use three or four evidence-based criteria.

H = Twenty and above Years of Age.M = Ten to Twenty Years of Age.L = Zero to Ten Years of Age.

H = Daily Use-approximately 365 times a year.M = Great than one and less than 365 times a year.L = Annual use-Once a Year.

H = Loss of Life and/or Property.M = Structure Damage.L = Minimal Loss of Property and/or Damage.

GATESCriteria #1: Age

Criteria #2: Frequency of Use.

Criteria #3: Consequence of Failure.

Step Two: RatingUse expert judgment to critically evaluate the available informationDevelop estimates for each “hazard” against criteriaUse letters or numbers but numbers do not represent an absolute measurement of risk only a relative means for comparison

Gate Criteria 1 Criteria 2 Criteria 3

Knightsbridge H L M

Steadly H M M

Redwood M H L

Jackflash M H L

Cantget L L L

Roughjustice H M L

IORR L M H

19 L H L

Step Three: All Possible Combinations

Greatest Risk HHH

HHM, HMH, MHHHHL, HLH, LHH, HMM, MMH, MHMHLM, MHL, HML, LMH, MLH, MMM,

LHMHLL, LHL, LLH, MML, LMM, MLMMLL, LML, LLM

Least Risk LLLThis is for equally weighted criteria. Unequal weightsyield different listings.

Step Four: Rank SubjectivelyEstablish rank according to descending relative riskIdentify subjective clusters.

Gate Rating Ranking

Steadly HMM Greatest Risk

Roughjustice HML

Jackflash MHL

Knightsbridge HLM Moderate

Risk

Redwood MHL

IORR LMH

19 LHL

Cantget LLL Least Risk

Step Five: Add Criteria?Look at rankings, do they make sense?Have you thought properly about this issue?If they do not, perhaps you did not consider all the most relevant criteriaA new criteria may be added to more accurately reflect the assessors rationale for ranking

Step Five: Add Criteria? (cont)Suppose the following was added to our exampleCriterion 4: Cost of emergency repair

H = Major disruptions to navigation or power, much higher costs to repairM = Much higher costs to repairL = Same as scheduled repair

Step Six: New RatingsGates Criteria #4 Rating

New Combined Ranking

Steadly H HMMH

Jackflash H MHLH

Knightsbridge H HLMH

Redwood M MHLM

IORR M LMHM

19 H LHLH

Roughjustice L HMLL

Cantget H LLLH

Step Seven: New RankingGates

New Combined Ranking

Criteria #4 Rating

Steadly HMMH Greatest Risk

Jackflash MHLH Greatest Risk

Knightsbridge HLMH Greatest Risk

Redwood MHLM Moderate Risk

IORR LMHM Moderate Risk

19 LHLH Moderate Risk

Roughjustice HMLL Moderate Risk

Cantget LLLH Least Risk

Operational Risk Management (ORM)

ORMAKA the risk matrixRisk ranking tool

Uses ranges of consequence and likelihoodCombinations created enable assessors to qualitatively estimate a risk

StepsDetermine purpose and use of matrix

Identify the question to be answered

Define consequences of interestIdentify consequence ranges and definitions Identify likelihood ranges and definitions Identify levels of risk in the cells of the matrix

Your DE Has Seen This“Mishap Risk”DOD "Standard Practice For System Safety”MIL-STD-882D 10 February 2000

Consequence Severities

Probability Levels

Risk Assessment Values

Each risk you assess is placed in a cell and managed accordingly

Risk Levels

Another Example

Source: Assessing Environmental Risk, A Lecture to the Irish Environmental Law Association By: L. M. Ó Cléirigh 29June 2004

Risk Matrix

Three AxiomsWeak consistencyBetweennessConsistent coloring3x3 and 4x4 should look like this to minimize problems

Low HighHigh

LowLow High

High

Low

Source: What’s wrong with risk Matrices? By Louis Anthony Cox, Risk Analysis Vol. 28 No.2, 2008

The Risk Management Point ofMatrix

CARVER + ShockVulnerability assessment method developed for Department of Defense CARVER is an acronym Criticality - measure of public health and economic impacts of an attackAccessibility – ability to physically access and egress from target

Recuperability – ability of system to recover from an attackVulnerability – ease of accomplishing attackEffect – amount of direct loss from an attack as measured by loss in productionRecognizability – ease of identifying target

SHOCKTechnique modified to include seventh attribute that combines health, economic, and psychological impacts of an attack

SHOCK attributes of target

Select a ProcessIdentify a critical process or infrastructure and assess vulnerability across nation, orAssess vulnerability of components of a single process

Critical

A target is critical when “loss” would have significant life, health or economic impacts

AccessibilityA target is accessible when an “attacker” can reach it to conduct the attack and then escape the target undetected

RecuperabilityThe time it will take for the specific facility to recover productivity is the target’s recuperability

VulnerabilityVulnerability measures the ease with which sufficient quantities of threat agents can be introduced to achieve the attacker’s purpose once the target has been accessed

EffectEffect is the percentage of system productivity damaged by an attack at a single facility

RecognizabilityRecognizability is the extent to which the target can be identified by an attacker without confusing it with other targets or components

ShockShock combines the measure of the health, psychological, and collateral national economic impacts of a successful attack on the target system

Sandia LabsUser friendly software has been developed for food defense by FDA and Sandia

http://www.cfsan.fda.gov/~dms/carver.html Process diagrams Interviews Results

Take Away PointsNot all risk assessment needs to be quantitativeDevelop a few consistent and well developed techniques for your usage