safety risk impact analysis of an atc runway incursion alert system

Safety Risk Impact Analysis of an Safety Risk Impact Analysis of an ATC Runway Incursion Alert SystemATC Runway Incursion Alert System

Sybert Stroeve, Henk Blom, Bert Bakker

EUROCONTROL Safety R&D Seminar, Barcelona, Spain, 25-27 October 2006

EUROCONTROL Safety R&D Seminar, Barcelona, Spain, 25-27 October 2006 2

Contents

Motivation

Example application

Systemic approach

Risk results

Conclusions


Runway incursion:Recognised as important air traffic safety issue


Complexity of aerodrome operations Complexity of accident risk assessment

Complexity of operationsMany agents (humans/systems)Many interactionsHighly dynamic Performance deviations

Complexity of risk assessmentMultiple agentsDependencies between agentsDynamics of agentsNominal/non-nominal conditions


Three types of accident models (Hollnagel, 2004)

S

F

S

FS

F

“FTA”Causes

“ETA”Consequences

“Pivotal”Event

S

F

S

FS

F

S

F

S

F

S

FS

F

S

F

Causes Consequences

HAZARD Effect A

Effect BEffect C

Effect D

1. Sequential accident models

Accident = sequence of events

e.g. fault trees, event trees, domino theory

2. Epidemiological accident models

Accident = like spreading of disease (latent/environmental conditions, barriers)

e.g. Reason’s Swiss cheese model, Bayesian belief networks


Three types of accident models (Hollnagel, 2004)

3. Systemic accident models

Accident = emergent from variability of a complex system

e.g. control theory, chaos theory, stochastic resonance

Compared to sequential / epidemiological accident models:• No fixed cause-effect relations• Dynamic / non-linear behaviour• Performance beyond event probability• Complex multi-agent interactions


Contents

Motivation

Example application

Systemic approach

Risk results

Conclusions


Active runway crossing operation:Effectiveness of ATC runway incursion alerting?

Human operators

Pilots take-off aircraftPilots taxiing aircraftRunway controllerGround controllers

Technical systems

VHF R/T communicationActive stopbarRunway incursion alertGround radar

Procedures

Crossing clearance byrunway controllerStopbar switchingRead-back


Contents

Motivation

Example application

Systemic approach

Risk results

Conclusions


Safety risk assessment cycle

Determine operation1

Assess risk tolerability6

Assessseverity4

Identify safety

bottlenecks7

Assess frequency5

Construct scenarios3

Identifyhazards2

Identify objective0

Decision making

Operationaldevelopment

Iterate(option)


Risk assessment by combination of two models:Monte Carlo Simulation + Bias & Uncertainty


Monte Carlo simulation model of multi-agent runway incursion scenario

Key aspects of agents, e.g.SA / task performance of operator Flight phase / aircraft performance

Modes within key aspects, e.g.Task: monitoring / alert reaction Flight phase: taxi / take-off

Dynamics within modes, e.g.Task performance timeTake-off acceleration profile

InteractionsBetween modesBetween key aspects of an agentBetween agents


Parameter values in MC simulation model

TypesTechnical systems, e.g. accuracy, availability, update rate, aircraft thrustHuman performance, e.g. task duration, decision parameter, likelihood of misunderstandingContext, e.g. taxiway layout, visibility

SourcesTechnical system specificationsHuman factors literatureIncident databasesInterviews with operational expertsMeasurement data of real operationsMeasurement data of real-time simulationsSimulation results from other relevant models


Performing Monte Carlo simulation

Model implementation in dedicated Delphi / Java software

MC simulation speed-up by risk decomposition MC simulation of conditional collision risks given an event, e.g.

– R/T system not functioning– Alert system not functioning– Pilots taxiing aircraft are lost– Visibility condition

Assess event probabilityCombine conditional risks and event probabilities

MC simulation: about 105 to 107 simulations per condition

ResultsConditional collision risks at various aggregation levelsOverall collision risk


Bias and uncertainty assessment

Types of differences between simulation model & realityNumerical approximationsParameter valuesFormal model structureNon-covered hazardsOperational concept differences

Assessment stepsIdentify differences between simulation model and realityAssess size of each difference Assess risk sensitivity for parameter valuesAssess effect of each difference on the riskCombine the joint effect of the differences on the risk


Contents

Motivation

Example application

Systemic approach

Risk results

Conclusions


Monte Carlo simulation results

10-6

10-5

10-4

10-3

10-2

SA PF taxiing aircraft

Proceedtaxiway

Crossrunway

Proceedtaxiway

Crossrunway

Visibility Unrestricted 400 – 1500 m

Con

ditio

nal c

ollis

ion

risk

(per

take

-off)

Without RIASWith RIAS


Bias and uncertainty assessment:Effects of model-reality differences (examples)

Significant effects (>30%)Type of manoeuvre of taking-off aircraft to avoid collisionConflict decision process by pilots of taking-off aircraftSpeed of taxiing aircraftMonitoring frequency by pilots of taxiing aircraftDeceleration of taking-off and taxiing aircraftTime before braking is initiated by pilots of taking-off aircraft

Small effects (<13%)Acceleration profile during the take-off runPerformance of R/T communication systemsPerformance of surveillance systemsPerformance of runway incursion alert systemTask scheduling of runway controller


10-6

10-5

10-4

10-3

10-2

Monte Carlo simulation + bias & uncertainty results

SA PF taxiing aircraft

Proceedtaxiway

Crossrunway

Proceedtaxiway

Crossrunway

Visibility Unrestricted 400 – 1500 m

Con

ditio

nal c

ollis

ion

risk

(per

take

-off)

Without RIASWith RIAS


Contents

Motivation

Example application

Systemic approach

Risk results

Conclusions


Conclusions

A wide scope safety assessment (including performance of relevant human operators) is needed to evaluate the effectiveness of a runway incursion alert system

Systemic accident models can effectively analyse the dependent dynamics of multiple agents in aerodrome operations (which is difficult by other model types)

The MC simulations indicate that the effectiveness of ATC runway incursion alerting is small in good visibility, but significant in reduced visibility conditions

Bias and uncertainty assessment supports informed decision making by addressing specific aspects of aerodrome operations at a particular airport


Discussion


Step 0: Identify objective

Close co-operation with decision makers

Aim: safety risk assessment for decision support ofimplementationredevelopmentcertification

Safety contextWhat are the safety criteria, target levels of safety?

ScopeBoundaries of the operation?Absolute or relative information?What types of risks?


Step 1: Determine operation

GoalUnderstanding of operational concept by safety assessors Freeze operational concept during assessment cycle Check for holes and inconsistencies (should be repaired by concept developers)

InputDescription of the operation from concept developers

OutputConcise, structured, consistent operational concept

– human operators– technical systems– procedures– environment


Step 2: Hazard identification brainstorm

Shifting the boundary between imaginable and unimaginable hazards

Open-minded and experienced operational expertsPure brainstormingNo analysis / solutions / mitigation

– open atmosphere: promotes creativity of participants– seemingly unimportant hazards trigger more relevant ones– analysis of one hazard may take too much time– hazards outside scope are removed during later analysis


Step 3: Construct scenarios

Cluster B

Condition

Event n

Event m

Hazardoussituation

Conflict

Cluster J,ATCo resolution

Cluster K,Pilot resolution

Hazards' combined effects


Step 4: Identify severities

How severe can the consequences of a scenario be?consequences and their severities often depend on

– conditions, geometry and resolutionusually a spectrum of severities applies

Example severity classesMinor, Major, Hazardous and Catastrophic

Severity assessment usually performed by safety expertsconsultation of and review by operational experts


Step 5: Assess frequency

Assess frequency of each possible severity per scenario

First assessment cycleInterviews with operational expertsIncident/accident databases

Optional subsequent cycleMonte Carlo simulation


Step 6: Assess risk tolerability

For each conflict scenarioindicate identified severity/ frequency combinationsdetermine associated risk tolerability classification

NegligibleUnacceptable

Unacceptable

Unacceptable

Unacceptable

Unacceptable Unacceptable

NegligibleNegligible

NegligibleNegligibleNegligible

Tolerable

Tolerable

Tolerable

Tolerable

Severity

Frequency

Probable

Catastrophic

Extremely

remote

Remote

Extremely

improbable

Hazardous MinorMajorExample


Step 7: Identify safety bottlenecks

In case of (possibly) unacceptable riskidentify which hazards/conditions contribute significantly to the large risk

Bottlenecks give operational developers a clue where they might improve


Contextual Control Mode Model (Hollnagel, 1993)

scrambled

opportunistic

tactical

strategic

subjectively available time

degreeof

control


Uncertainty assessment matrix

NegligibleNegligibleNegligibleNegligibleSmallMinorNegligible

NegligibleNegligibleNegligibleSmallMinorSignificantSmall

NegligibleNegligibleSmallMinorSignificantConsiderableMinor

NegligibleSmallMinorSignificantConsiderableMajorSignificant

SmallMinorSignificantConsiderableMajorMajorConsiderable

MinorSignificantConsiderableMajorMajorMajorMajor

NegligibleSmallMinorSignificantConsiderableMajor

Parameter value uncertaintyRiskuncertainty

Risk

sen

sitivity


Bias assessment matrix

NegligibleNegligibleNegligibleNegligibleNegligibleNegligibleNegligible

NegligibleNegligibleNegligibleNegligibleNegligibleSmallSmall

NegligibleNegligibleNegligibleNegligibleSmallMinorMinor

NegligibleNegligibleNegligibleSmallMinorSignificantSignificant

NegligibleNegligibleSmallMinorSignificantConsiderableConsiderable

NegligibleSmallMinorSignificantConsiderableMajorMajor

UnlikelyInfrequentLess frequentFrequentRegularTypical

Probability assumption does not applyRisk bias

Bia

s du

e to

n

on

-ap

plica

bility

safety risk impact analysis of an atc runway incursion alert system

Documents