caas surveillance follow-up, control, measurement. caas_s… · each caa will have its own ‘sweet...
TRANSCRIPT
CAAS Surveillance Follow-Up, Control,
Measurement
ICAO Risk Based Surveillance Workshop
Lima, Peru
19-20 March 2018
CAAS’ Safety Oversight Approach
2
Surveillance is an important element in safety oversight
PoliciesRules
DevelopmentApprovals
SurveillanceEnforcementEducation
Surveillance Toolbox
3
Surveillance is not only about audits and inspections
Surveillance
Audits
Inspections
Safety Performance
Indicators
Mandatory Occurrence
Reports
Voluntary Reports
Whistleblower Reports
Occurrence Investigations
Interactions during
approvals application
Interactions during
enforcement actions
Response to education
and outreach
Interactions during rules consultation
Surveillance Policies and Procedures
4
Annual Surveillance Workplan
Monthly Surveillance Report & Tracking
Annual Surveillance Reports
Surveillance Activity X
Assess
Prepare
Do
Follow-Up
Report
Adherence is critical for consistency, control, and accountability
Surveillance Activity Y
Assess
Prepare
Do
Follow-Up
Report
Surveillance Activity Z
Assess
Prepare
Do
Follow-Up
Report
Department
Individual
Department
CAAS
Surveillance Follow-Up
5
Follow-up requires individual and departmental attention
• Immediate corrective action
L1 Finding
Corrective Action Notice
• Corrective action within 30 days
L2 Finding
Corrective Action Notice
• No action required, but reply is needed
Observation
Monthly reporting at the departmental levelIndividual’s responsibility to track responses to Corrective Action Notices
Surveillance Follow-Up
6
Feedback mechanisms on safety issues identified through surveillance
brings attention to actions required from all stakeholders
Safety Oversight Management System (SOMS)
7
Information Technology is an enabler for surveillance follow-up and
control
How much surveillance is enough
8
given our limited resources??
How do we know if this year’s surveillance is
better than last year’s?
9
Surveillance Effectiveness
10
Surveillance measures practical drift, which can be L2 or L1
Baseline Performance: ‘As Designed’System
Design
Practical DriftLevel 2 Findings
Level 1 FindingsSurveillance GoalMeasure practical drift
accurately so that corrective actions can be taken.
Ultimately, to prevent accidents.
Typically, effectiveness is measured in number and level of findings. But this is not so straight-forward. Was it 100% surveillance?
Surveillance Effectiveness
11
Each CAA will have its own ‘sweet spot’ for surveillance quantity given its
own limited resources
0%
Just nice
100%Surveillance Activities
Too little Too much
‘Sweet spot’ for a particular CAA
100% surveillance is unsustainable and
undesirable
0% surveillance is neglecting CAA’s duty
Surveillance Effectiveness
12
‘Sweet spot’ depends on internal and external factors
Manpower Expertise Time Money
Size and Complexity of Industry
Regulations
Compliance Culture SMS Implementation
Safety Oversight Effectiveness
13
Maximise surveillance information available sources to provide accurate
picture of ‘practical drift’
Surveillance Information
Audits
Inspections
Safety Performance
Indicators
Mandatory Occurrence
Reports
Voluntary Reports
Whistleblower Reports
Occurrence Investigations
Interactions during
approvals application
Interactions during
enforcement actions
Response to education
and outreach
Interactions during rules consultation
Safety Oversight Effectiveness
14
Integration of all available information on entire spectrum gives clearer
picture of safety oversight effectiveness
Whistleblower Reports
Interactions during
approvals application
Interactions during
enforcement actions
Response to education
and outreach
Interactions during rules consultation
Inspections
Audits
Voluntary Reports
Safety Performance
Indicators
Occurrence Investigations
Mandatory Occurrence
Reports
Occurrence
Proactive Reactive
Thank you
15