target level of safety and reusable safety arguments for ... · 1 x 10-n for all uas vs. vs. vs....
TRANSCRIPT
UNCLASSIFIED
Target Level of Safety and
Reusable Safety Arguments
for UAS Integration into
Civil Airspace
5th Annual Operational
Analysis Workshop
15 June 2011
Russell Wolfe
Acquisition & Technology Development Group Lead
Modern Technology Solutions, Inc.
Wolfe_UAS Safety Criteria_NATO OAW_15June2011
2 UNCLASSIFIED
UAS Airspace Integration
Challenges
• Unmanned Aircraft Systems (UAS) are more widely used than ever before in aviation history but possess no inherent ability to “see and avoid” to prevent collisions as required by regulations.
• There are three key items inhibiting UAS manufacturers and acquisition organizations from defining performance and safety requirements for UAS operations within the NAS: 1. Inability to derive quantitative sense
and avoid (SAA) requirements and standards from existing regulations
2. Lack of a defined Target Level of Safety (TLS) for UAS
3. Limited guidance on what comprises a Safety Case and how to present the substantiating evidence
3 UNCLASSIFIED
Regulatory Guidance 14 CFR Regulations relating to See & Avoid
Sec. 91.111 - Operating near other aircraft.
(a) No person may operate an aircraft so close to another aircraft as
to create a collision hazard.
Sec. 91.113 - Right-of-way rules: Except water operations.
(b) General. When weather conditions permit, regardless of whether an operation is conducted under instrument flight rules or visual flight rules, vigilance shall be maintained by each person operating an aircraft so as to see and avoid other aircraft. When a rule of this section gives another aircraft the right-of-way, the pilot shall give way to that aircraft and may not pass over, under, or ahead of it unless well clear.
The qualitative nature of the current regulations are not suitable for establishing
requirements for unmanned aircraft “sense and avoid” technical solutions
4 UNCLASSIFIED
Justification for establishing
a Target Level of Safety for UAS
FAA SAA Workshop Final Report “Recommendation 5.5: FAA and DOD should charter a group to define the process,
methods and tools for analyzing UAS SAA as set out in this report. This should include
establishing the numerical probability for the Target Level of Safety (TLS) applicable to UAS
SAA and how the TLS should be allocated between operational procedures and systems and
equipment for substantiating compliance.”
National Aeronautics R&D Plan “Goal 6: Develop capabilities for UAS NAS integration (Near Term <5 yrs): • Develop a flight safety case modeling capability including data collection methods
• Define the appropriate target level of safety and the process for evaluation”
OSD UAS Task Force Airspace Integration IPT Charter “Section 2.4.2 - Systems Integration SIPT: The Systems Integration SIPT will be responsible for identifying acquisition solutions for
airspace integration. Initial focus of this SIPT will be, but not limited to:
….
5. Developing a repeatable and quantifiable approach for demonstrating that a UAS can
meet a target level of safety that will satisfy the Department’s requirements for the type of
UAS and class of airspace operations are intended.”
OSD = Office of the Secretary of Defense
5 UNCLASSIFIED
OSD-led Safety Criteria
and Assessment Project
Project Objectives Participants**
Outcome / Results
• Assess various Safety Guidelines and
approaches to determine the best one for
providing a comprehensive, robust collision
risk assessment of UAS NAS operations
• Conduct a series of workshops to review
current risk levels and to determine a
midair collision Target Level of Safety
(TLS) for a range of UAS types in various
airspace environments
• Establishment of a repeatable/quantifiable
safety methodology with reusable safety
arguments
• Guidelines for how to conduct UAS Safety
Analysis
• Recommended midair collision TLS (10-x)
for DoD UAS to routinely access the NAS
• Profile specific safety analysis reports for
LOS, Terminal, Lateral, Vertical, Operating
Area and Dynamic Operations
** DoD participation includes members from OSD, Service
Safety Centers, AW Directorates, Programs of Record
Schedule
6 UNCLASSIFIED
UAS Target Level of Safety Defining how safe the UAS must be
• Purpose: – Conduct a series of workshops to review current risk levels and to determine
a DoD midair collision Target Level of Safety for a range of UAS groups in various airspace environments
• Workshop Objectives: 1. Provide assistance to the OSD AI IPT Safety Methodology activity to provide
consistency between the recommended methodology and evaluation criteria
2. Conduct a thorough review of historical midair and near-midair collision
statistics to provide an unmitigated NAS risk baseline
3. Determine what factors (e.g. airspace class, environmental, UAS Group)
should be used to establish a TLS value(s) and select the appropriate one(s)
4. Establish a methodology to quantitatively establish a TLS value(s) for
anticipated DoD UAS operations
5. Identify a set of tools that can be used to conduct analysis for determining
whether the TLS value(s) has been met
6. Provide a consolidated DoD recommendation to the future FAA-led TLS
Workshops scheduled to kick-off in 2011
7 UNCLASSIFIED
UAS Target Level of Safety Approach
• Approach
– Conduct a series of workshops attended by military, FFRDC and Industry
subject matter experts in the areas of safety, airworthiness and operations
– Identify past efforts that have defined/used a target level of safety approach
and how it was implemented
– Establish any assumptions, terms and definitions that will be used as the
foundation for developing a TLS for military UAS
– Consider all feasible TLS concepts, parameters and units of measure and
determine the best combination to use
– Derive a DoD midair collision TLS for a range of UAS groups in various
airspace environments
– Provide substantiating evidence supporting any TLS value(s)
Recommended TLS Value(s) for
UAS Operations within the NAS
Group 1 Group 2 Group 3 Group 4 Group 5
Class A na na na 10-8 10-8
Class B 10-5 10-7 10-8 10-9 10-9
Class C 10-5 10-5 10-6 10-8 10-8
Class D 10-4 10-4 10-5 10-7 10-7
Class E 10-4 10-4 10-5 10-7 10-7
Class G 10-4 10-4 10-5 10-7 10-7
Define TLS Criteria Derive TLS Value(s)
Yes
No
OwnshipAvoidance
IntruderAvoidance
ATCDirections
Encounter Threshold
Tau > XNMAC|
Encounter
Encounter
OwnshipInducing
IntruderInducing
OwnshipInducing
IntruderInducing
IntruderInducing
OwnshipInducing
Encounter
MAC|NMAC MAC
NMAC
Encounter
Encounter
8 UNCLASSIFIED
TLS Workshop Discussion Topics Potential TLS Concepts
• Should TLS be defined as a “Target” or as a “Threshold”?
• Should TLS be a single value or a range of values?
• Should TLS be established for the SAA system or the UAS?
• Should TLS be defined for the SAA system alone or for all protection layers?
• Do we use an existing collision risk matrix or define a new one?
• Should TLS be based on historical accident statistics for manned aircraft?
VS.
1 x 10-n For all UAS
VS.
VS.
VS.
VS.
Likelihood Qualitative Quantitative
Probable Anticipated to occur one or more times during the entire system/operational life of an item.
Probability of occurrence per operational hour is greater than
1 x 10-5
Remote Unlikely to occur to each item during its total life. May occur several time in the life of an entire system or fleet.
Probability of occurrence per operational hour is less than 1 x
10-5, but greater than 1 x 10-7
Extremely Remote Not anticipated to occur to each item during its total life. May occur a few times in the life of an entire system or fleet.
Probability of occurrence per operational hour is less than 1 x 10-7 but greater than 1 x 10-9
Extremely Improbable So unlikely that it is not anticipated to occur during the entire operational life of an entire system or fleet
Probability of occurrence per operational hour is less than 1 x 10-9
MIL-STD-882
FAA System
Safety Handbook
9 UNCLASSIFIED
TLS Workshop Discussion Topics Potential TLS Parameters
• Class of Airspace
• Surface Population Density
• Altitude
• UAS Group
• UAS Momentum (mass x velocity)
• Positive Control (IFR / VFR)
• Airspace Density
• Maneuverability
10 UNCLASSIFIED
TLS Workshop Discussion Topics Potential Units of Measure
• Examples – midair collisions / flight hour
– loss of separation / flight hour
– NMACs / flight hour
– fatal accidents / mission
– all accidents / mission
– Others
exposure ofunit
eventTLS
11 UNCLASSIFIED
TLS Workshop Findings / Recommendations
• TLS Concept Decisions:
– Target v. Threshold: Decision was that the TLS value(s) should establish the DoD threshold(s) that should be attained to gain access to the NAS
– Single v. Multiple Values: Multiple Values should be derived, however, the exact number or values will be determined by our future analysis results
– SAA Function or UAS : a TLS will be derived for the SAA Function
– SAA System alone v. All protection layers: ALL Protection Layers should be considered in deriving a TLS
– Use existing risk matrix or define a new one: Agreed by all that this is out of scope and a Service decision
– Should TLS be based on historical accident data: Yes
• TLS Parameter Decisions
– Class of Airspace, UAS Group, and IFR/VFR are the preferred TLS parameters
– Airspace Density and Maneuverability should also be considered
• Unit of Measure Decision
– The TLS unit of measure should be Midair Collisions per Flight Hour
12 UNCLASSIFIED
TLS Workshop Findings / Recommendations
• TLS has previously been applied to existing aviation systems – JAA Requirements on Aircraft
Accident Rates (~1980) – Precision Runway Monitor (1989)
• Two existing modeling environments capable of conducting this type of analysis
– MIT/LL: Collision Avoidance System Safety Assessment Tool (CASSATT)
– MTSI: Sense and Avoid Flight Encounter Simulation Toolset (SAFESTTM)
Pilot
response
model
Manned Aircraft
Unmanned Aircraft
Pilot
response
model
Comm
Sense and avoid
system
TCAS
UAS
EnvironmentModels
Dynamic Simulation
Visual
acquisition
Tim
e H
isto
rie
s &
M
etr
ics
TCASComm
Random
Situations
Aircraft
dynamic
model
UAS
dynamic
model
Airspace Models
13 UNCLASSIFIED
TLS Workshop Findings / Recommendations
• Airspace analysis shows large variations in collision risk based upon:
– Region
– Altitude
– Proximity to major airports
– Cooperative/Non-cooperative
• Determined that 6 study areas should be analyzed based on data variations & operational considerations
– Airspace Class
– Altitude
Airspace Group Lateral Distance
(nm)
Vertical Distance
(ft)
A 1 500
B & C ½ 300
D ½ 300
E above 10k ft. 1 300
E below 10k ft. 1 300
G 1 300
14 UNCLASSIFIED
Next Steps TLS Analysis (currently in work)
• Goal of TLS analysis is to provide substantiation to all TLS values
• Analysis conducted using the MIT/LL CASSATT
and MTSI SAFEST modeling & simulation tools – Data verification/comparison run completed
• Quantifying NAS ambient probability of encounter – Analyzing altitude, airspace class, geographic location
– Results will be used in a fault tree analysis and to
identify if multiple TLSs are reasonable
• Upcoming tasks will focus on nodes of fault trees – Fault tree being constructed to aid in deriving TLS
– Leveraging existing fault trees built for similar
purposes
• Probabilistic Risk Assessment – Encounter probability used as Initiating Event (IE)
– Mitigation analysis using Event Sequence Diagram
Yes
No
OwnshipAvoidance
IntruderAvoidance
ATCDirections
Encounter Threshold
Tau > XNMAC|
Encounter
Encounter
OwnshipInducing
IntruderInducing
OwnshipInducing
IntruderInducing
IntruderInducing
OwnshipInducing
Encounter
MAC|NMAC MAC
NMAC
Encounter
Encounter
15 UNCLASSIFIED
Next Steps Develop Re-usable Safety Arguments
• Once the safety approach and TLS values are adopted by DoD, re-usable safety arguments will be developed for six airspace access profiles as defined within the DoD UAS Airspace Integration Plan.
Terminal Area
Operating Area Dynamic
Visual Line of Sight
Vertical Lateral
16 UNCLASSIFIED
Next Steps Operators, Regulators, Developers
• Planning to conduct a series of follow-on activities to ensure TLS values and safety assessment processes can be leveraged and used by operators, regulators and developers. – Working with owner of MIL-STD-882 to integrate Safety Case
recommendations
– Safety Case support to Services
• Navy BAMS
• Cherry Point MCAS
• Cannon AFB
– RTCA SC-203 support
• Plenary and Workgroup meetings
• Co-lead for WG3 FAA SAA Workshop
MASPS
Standards
DevelopmentOrganizations
(e.g. RTCA, ASTM, SAE)
MOPS TSOsDesign Standards
for Manufacturers and Developers
Developers
Service
Safety Centers
Operators
Program of
Record (POR)
Requirements
Common
Safety Case
Methodology
Service Unique
Safety Guidance(Army, Navy, AF)
Capability
Based Assessment
ICDCDD /
CPD
AI
KPP
FAA
Unmanned AircraftProgram Office
RegulatorsNotice for Proposed
Rulemaking (NPRM)
Rule Modification
(14 CFR xxx)
Modify
Advisory Circularsand/or FAA Orders
Document Change
Proposals (DCP)
Formal Rule-
Making Process
Revised Rules
Interpretation
AoA Safety Case
TLS Workshop
Recommendations
AI – Airspace Integration
AoA – Analysis of Alternatives
CDD – Capability Development Document
CPD – Capability Production Document
ICD – Initial Capabilities Document
KPP – Key Performance Parameters
MASPS – Minimum Aircraft System Performance Specification
MOPS – Minimum Operational Performance Standard
TSO – Technical Standard Order
17 UNCLASSIFIED
Summary
• Provided an overview of an on-going OSD-led effort to establish two items essential for UAS airspace integration: – Common Safety Methodology for conducting UAS Safety Analysis
– Target Level(s) of Safety for DoD UAS
• Discussed the TLS Workshop initial findings and recommendations
• Discussed the TLS Analysis Plan currently in work
• Defined what some of the next steps will entail
Establishing the safety criteria and methodology for conducting UAS safety
analysis are critical elements in solving the “see and avoid” challenge.
18 UNCLASSIFIED
Contact Information
• Russell Wolfe Acquisition & Technology Development Group Lead Modern Technology Solutions, Inc. 5285 Shawnee Road, Suite 400 Alexandria, VA 22312
– 703-564-3828