aviation software fundamentals course slides#1 › dasp › docs › regulations ›...
Post on 07-Jul-2020
7 Views
Preview:
TRANSCRIPT
28/02/2020
1
UNCLASSIFIED
UNCLASSIFIED
Aviation Software Certification - Introduction Directorate of Initial Airworthiness (DIA-DASA)
About this Course
• It’s an introduction only• Assumed knowledge:
– little, if anything, about aviation software– you know that software exists and that it is important to the ADF– it is a bonus if you have undertaken System Safety training
• This course will not make you an expert• Course objective is to provide you with basic knowledge to enable you
to:– Appreciate the peculiarities of Aviation Software to be a smarter
customer – Communicate with software engineers on relevant issues– Know where to find guidance on Aviation Software
2
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
Agenda
• Introduction to Aviation Software• Software Integrity Principles• Software Assurance Techniques and Principles• Software Type Design Approvals• Authority Benchmarks for Approval• Authority LOI• Good Practice Problem Reporting• In-service Software Management• Other Topics
– Aeronautical Data and Mission Planning– Electronic Flight Bags– Unmanned Aerial Systems
3
Admin
• Mobile phones• Evacuation Points• Toilets• Course Attendance List• Course Slides• Course Critiques (Training Reform)
4
UNCLASSIFIED
UNCLASSIFIED
28/02/2020
2
Background
• Our Background• Your Background
– Where do you work?– What is your level of software knowledge?– How much software do you touch in your current job?
5
UNCLASSIFIED
UNCLASSIFIED
Directorate of Initial Airworthiness Roles
• Certification of new acquisition aircraft and Major changes to type design
– Define Safety Benchmarks for Aircraft Design
– Processing Airworthiness Issue Papers (AwIP) and communicate risks to Command
– Authority Endorsement of Statement Of Intent and Usage (SOIU)
– Tender Evaluation Assistance
– Certification Programme (CP) and Type Certification Basis (TCB) Agreement
– Authority Recommendations for Military Type Certificate (MTC), Military Supplemental Type Certificate (MSTC), Military Restricted Type Certificate (MRTC) , Military Permit To Fly (MPTF)
6
UNCLASSIFIED
UNCLASSIFIED
Directorate of Initial Airworthiness Roles
• Design Services
– Design Technologies & Standards (Software Systems Integrity)
• Aviation Software
• Aeronautical Data
• Electronic Flight Bags (EFB)
• Mission Planning Systems (MPS)
– UAS Certification
– Aerodromes & Heliports
– Initial Airworthiness Regulation & Promotion
– Design Organisation Approvals
– Type Certification
7
UNCLASSIFIED
UNCLASSIFIED
Directorate of Initial Airworthiness Roles
• Education and Training
– Sponsor the following DASA Training Courses:
• Type Certification
• Aviation System Safety Engineering / Technical Risk Management
• Aviation Software
• E3 Management for Designers/Maintainers
• ADF Crash Protection
• Ship Aviation Interface Certification and Safety
8
UNCLASSIFIED
UNCLASSIFIED
28/02/2020
3
Software Systems Section
9
Software Specialist 1 (SS1)
Software Specialist 1A
Software Specialist 1B
Software Specialist 1C
UNCLASSIFIED
UNCLASSIFIED
Software Systems Roles
• Interpret, prescribe & apply airworthiness standards– AAP 7001.054 Section 2 Chapter 3 Aviation Software– DO-178C Software Considerations In Airborne Systems and Equipment
Certification– Align with international NAA/NMAA best practice
• Assess the competence of a software development organisation (MDOA)• Review Specs and SOIU• Assist with the Design Certification process, particularly:
– Review and provide agreement on Plans for Software Aspects of Certification or equivalent
– Review Aviation Software aspects of a Certification Programme– Inspection of Compliance Documents– Approval of Major Changes to Type Design involving Aviation Software– Training, Education and Promotion
10
UNCLASSIFIED
UNCLASSIFIED
Software Systems Roles
• Sponsor Training– Aviation Software Certification - Introduction (This course)– Aviation Software Certification – Intermediate – Master of Science in Software Engineering (18 months)
11
UNCLASSIFIED
UNCLASSIFIED
Questions?
12
UNCLASSIFIED
UNCLASSIFIED
28/02/2020
4
UNCLASSIFIED
UNCLASSIFIED
Introduction to Aviation SoftwareDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED 14
Computer System
https://www.youtube.com/watch?v=xnyFYiK2rSY
UNCLASSIFIED
UNCLASSIFIED 15
Advantages
Flexibility
DurabilityWeight
Software
UNCLASSIFIED
UNCLASSIFIED 16
Source Lines of Code (SLOC)
• Source lines of code (SLOC) is a software metric used to measure the size of a computer program by counting the number of lines in the text of the program's source code.
• How many SLOC in the program below?
int main (){
int i; cout << "Please enter an integer value: "; cin >> i; cout << "The value you entered is " << i; cout << " and its double is " << i*2 << ".\n"; return 0;
}
28/02/2020
5
UNCLASSIFIED
UNCLASSIFIED 17Approx. 200 LOC
UNCLASSIFIED
UNCLASSIFIED
The Large Hadron Collider
18
Approx. 3 Million LOC
UNCLASSIFIED
UNCLASSIFIED
Android Lollipop (2015)
19
Approx. 16 Million LOC
UNCLASSIFIED
UNCLASSIFIED 20
Approx. 40 Million LOC
28/02/2020
6
UNCLASSIFIED
UNCLASSIFIED 21
If lines of code were printed out on an A4 sheet at 60 lines per page
UNCLASSIFIED
UNCLASSIFIED 22
The C-130J has more than 50 Software Configuration Items and Approx. 600,000 LOC
UNCLASSIFIED
UNCLASSIFIED 23
UNCLASSIFIED
UNCLASSIFIED 24
The Super Hornet has Approx. 1.1 Million LOC
28/02/2020
7
UNCLASSIFIED
UNCLASSIFIED 25
UNCLASSIFIED
UNCLASSIFIED 26
The F-22 Raptor has Approx. 2.2 MLOC
UNCLASSIFIED
UNCLASSIFIED 27
UNCLASSIFIED
UNCLASSIFIED 28
The F-35 Lightning II has Approx. 8 MLOC
28/02/2020
8
UNCLASSIFIED
UNCLASSIFIED 29
There are over 70 applications from 15 different vendors running on the Boeing 787 Dreamliner’s common core computer. It has Approx. 8 MLOC
UNCLASSIFIED
UNCLASSIFIED 30
F-35 & Boeing 787: 8 MLOC 144,000 pages standing at 48ft or 15m tall
This is approx. 4 x the amount of the F-22 Raptor
UNCLASSIFIED
UNCLASSIFIED
5
10
15
0
20
25
30
C-130J‘99
F-18E/F‘99
F-22‘05
B787‘11
F-35’16-18
31
UNCLASSIFIED
UNCLASSIFIED
Don’t forget…
• Aviation Software also includes technologies that are developed using software development paradigms…
• Firmware: “The combination of a hardware device and computer instructions or computer data that reside as read-only software on the hardware device” [12207 & 498]
• Firmware within avionics falls into the definition of Aviation Software and must not be forgotten
• Complex Electronic Hardware: “A hardware item is considered complex if a comprehensive combination of deterministic test and analyses cannot ensure correct functional performance under all foreseeable operating conditions with no anomalous behaviour.” [DO-254]
32
28/02/2020
9
UNCLASSIFIED
UNCLASSIFIED 33
FPGA / EEPROMs
DIS
PLA
YA
RIN
C 4
29Tr
ansm
itter
Line
Driv
er
Har
dwar
e Fa
ults
UNCLASSIFIED
UNCLASSIFIED
Why?
34
• Software is just logic based instruction• Digital Gates do the same thing• Take the following simple example:
A 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
B 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1
C 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1
D 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Z 0 0 0 1 0 0 0 1 0 0 0 1 1 1 1 1
Z = 0;If A == TRUE {
if B == TRUE {Z = 1;
}}If C == TRUE {
if D == TRUE {Z = 1;
}}
AB
CD
Z
UNCLASSIFIED
UNCLASSIFIED
Why?
• That simple example might be safety logic preventing false CMDS ejection
A 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1
B 0 0 0 0 1 1 1 1 0 0 0 0 1 1 1 1
C 0 0 1 1 0 0 1 1 0 0 1 1 0 0 1 1
D 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1
Z 0 0 0 1 0 0 0 1 0 0 0 1 1 1 1 1
ARM
AUTOOFF
Press for Manual Fire
NO CURRENT THREATS
CMDS_Fire = 0;If ARM == TRUE {
if Manual_Fire == TRUE {CMDS_Fire = 1;
}}If AUTO == TRUE {
if RWR_Detect == TRUE {
CMDS_Fire = 1;}
}
ARMManual_Fire
AUTORWR_Detect
CMDS_Fire
35
UNCLASSIFIED
UNCLASSIFIED 36
Chaos Report 2015
28/02/2020
10
UNCLASSIFIED
UNCLASSIFIED
Why is software difficult
RequirementsComplexity
Continuity
Testability
Flexibility ?!?
37
UNCLASSIFIED
UNCLASSIFIED
Flexibility
Software (in theory)
CHANGE 1 CHANGE
2
Software (in practice)
CHANGE 3
CHANGE 4
CHANGE 5
CHANGE 6
Defects
Time
38
UNCLASSIFIED
UNCLASSIFIED
Complexity
39
UNCLASSIFIED
UNCLASSIFIED
Continuity
Electrical Power
Structural Strength
InputO
utpu
t Test
Predict
Test
Predict
Test
Predict
Software
40
28/02/2020
11
UNCLASSIFIED
UNCLASSIFIED
Testability
• 53 binary options• 2^53 combinations • 9007 trillion combinations!
• If it takes 1 second to write, execute and verify a test case; how long will it take to test this simple program?
• 2.856 Million years
41
UNCLASSIFIED
UNCLASSIFIED
Common causes of Software related accidents
Confusing reliability and safety
Relying on redundancy
Re-using software
Wrong requirements
42
UNCLASSIFIED
UNCLASSIFIED
Wrong requirements
• Lufthansa Flight 2904, A320, Warsaw, 1993– 2 dead, 54 injured, aircraft lost
• Braking Logic– Reverse Thrusters: only when
weight on both wheels (>12T)– Spoilers: only when wheel
speed >72 knots– Brakes: not until wheel speed
has reached a calculated value (variable)
Situation: anticipated cross wind and
standing water on runway
Lesson: The software functioned exactly as specified, but an accident still occurred.
43
UNCLASSIFIED
UNCLASSIFIED
Major causes of Software related accidents
Confusing reliability and safety
Relying on redundancy
Re-using software
Wrong requirements
44
28/02/2020
12
UNCLASSIFIED
UNCLASSIFIED
Ariane 5 Accident
45
https://www.youtube.com/watch?v=5tJPXYA0Nec
UNCLASSIFIED
UNCLASSIFIED
Software Re-use
• Arianne-5 reused an Inertial Reference System (IRS) from Arianne-4.
4
5 Both at 33s
Horizontal Distance
Altit
ude
46
UNCLASSIFIED
UNCLASSIFIED
Over-reliance on Redundancy
• The Ariane 5 accident report notes that according to the culture of the Ariane program, only random failures are addressed and they are primarily handled with redundancy.
IRS#1
IRS#2
47
UNCLASSIFIED
UNCLASSIFIED
Major causes of Software related accidents
Confusing reliability and safety
Relying on redundancy
Re-using software
Wrong requirements
48
28/02/2020
13
UNCLASSIFIED
UNCLASSIFIED
Software Configuration
• ECU Software installed / configured incorrectly
• Not tested in simulator due to project delays
49
https://www.youtube.com/watch?v=2DFCJ32ygIE
UNCLASSIFIED
UNCLASSIFIED
C130J & C27 J Software Load Control Issues
50
LRU with Same Part number for 2 different platformsDifferent Software for each Aircraft
Software Part Number not identified in LRU
How do you determine whether the LRU belongs to C130J or C27J?
UNCLASSIFIED
UNCLASSIFIED
Software ‘Silver Bullets’
Formal Methods
Metrics Extensive Testing
BEWARE: SOFTWARE SILVER BULLETS ≠ SAFE DEPENDABLE SOFTWARE
CASE tools
Development Standards
CMMI
Reviews and audits
51
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
Software Dependability Cube
52
28/02/2020
14
UNCLASSIFIED
UNCLASSIFIED 53
Questions?
Software Integrity PrinciplesDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
55
Software Dependability Cube
UNCLASSIFIED
UNCLASSIFIED
Available training in System Safety
• Other courses on System safety are provided by DASA:– DASA System Safety Engineering - Introduction (Level 1)– DASA System Safety Engineering Management – Basic
(Level 2)– DASA System Safety Engineering Management – Applied
(Level 3)
• Offerings under review this year
• Enquiries to dasa.dtsenquiries@defence.gov.au
56
28/02/2020
15
UNCLASSIFIED
UNCLASSIFIED
What is System Safety?
“A systematic, comprehensive evaluation of the implemented system to show that the relevant safety requirements are met”
‐ARP4754A‐
“System safety describes the state of materiel such that any hazards in the design have been identified and evaluated, and associated risks have been eliminated or minimised so far as is reasonably practicable.”
‐SSE Cse‐
57
UNCLASSIFIED
UNCLASSIFIED
What is System Safety?
Systematic approach:• Aircraft Functional Hazard
Assessment (FHA)• Aircraft Fault Tree Analyses (FTAs)• System FHAs• System FTAs• System Failure Modes and Effects
Analyses (FMEAs)• Item FTAs• Item FMEAs
58
UNCLASSIFIED
UNCLASSIFIED
Hazard Probability
• How do we meet Quantitative Safety Criteria given software fails systematically?
59
Hardware
Systematic Failures
Operating Time Statistically Predictable
Failures
Systematic Failures (1 or 0)SoftwareSystem Inputs
System Inputs
UNCLASSIFIED
UNCLASSIFIED
System Safety – Key Requirements
60
28/02/2020
16
UNCLASSIFIED
UNCLASSIFIED
Catastrophic Critical Marginal Negligible
Frequent 1 3 6 10
Probable 2 5 9 14
Occasional 4 8 13 17
Remote 7 12 16 19
Improbable 11 15 18 20
WE CAN’T ASSIGN PROBABILITY TO SOFTWARE HAZARDS, SO WE USE SOFTWARE ASSURANCE
61
Hazard Probability
Rigour applied is dependent on the severity of the failure condition associated with the failure of the software to perform its intended function
Covered shortly!
UNCLASSIFIED
UNCLASSIFIED
System Safety – Key Requirements
• System Safety requirements are detailed in the ADRM (Section 2 Chapter 2)
• 14 Essential requirements• Three key outcomes:
– Design safety objectives are:• Defined• Agreed by the Authority
– Design hazards are identified and controlled– Hazards identified during design and associated controls are
documented
62
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
Software Dependability Cube
63
UNCLASSIFIED
UNCLASSIFIED
What is Software Assurance?
The planned and systematic actions necessary to provide confidence and evidence that a product or process satisfies given requirements• Objectives• Checks and balances
The IdeaThe more important it is that the software should work, the more rigorous the development must be to make sure it does work.
64
28/02/2020
17
UNCLASSIFIED
UNCLASSIFIED
What is Software Assurance?
The planned and systematic actions necessary to provide confidence and evidence that a product or process satisfies given requirements• Objectives• Checks and balances
The IdeaThe more important it is that the software should work, the more rigorous the development must be to make sure it does work.
65
UNCLASSIFIED
UNCLASSIFIED
Increasing software assurance
Higher software assurance is achieved by increasing the degrees of rigour in:– verification of requirements– test coverage– configuration control– independence of software verification process– robustness testing– verification of compliance with standards– traceability
66
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
67
Software Dependability Cube
UNCLASSIFIED
UNCLASSIFIED
• Software Assurance provides confidence an item is correct to it’s requirements [Verification]. How do we confirm the requirements are correct to the intended safety goals [Validation].
• Software safety is the in-depth analysis of the software in a system that is conducted in order to identify software safety requirements to treat hazards.
• May result in derived requirements
Software Safety
68
28/02/2020
18
UNCLASSIFIED
UNCLASSIFIED
How do we do software safety?
– Is there a formal approach?– How do we understand software in the system context?– What other requirements could I possibly need?
69
UNCLASSIFIED
UNCLASSIFIED
Software Safety Process
Identify system
functions
Identify system level
functional failure modes
and associated severities
Identify potential software
failure modes
Define software safety
requirements to detect and handle or prevent
software failure modes
1 2 3 4Define generic
software safety
requirements
5
70
UNCLASSIFIED
UNCLASSIFIED
Software Safety Process
71
UNCLASSIFIED
UNCLASSIFIED
Software Safety Requirements
• Derived requirements from in-depth analysis of Software• Feeds back to System Safety• Generic requirements can be imported based on good practice &
experience:– Implement Lessons Learnt– Treat Common Issues– Not Constrained by Functionality– Check on Completeness of Software Hazard Analysis
72
28/02/2020
19
UNCLASSIFIED
UNCLASSIFIED
Generic Software Safety Requirements
• Inadvertent Jumps– The system shall detect inadvertent jumps within or into safety
critical functions; return the system to a safe state, and, if practical, perform diagnostics and fault isolation to determine the cause of the inadvertent jump.
• Decision Statements– Decision statements in safety-critical computing system functions
shall not rely on inputs of all ones or all zeroes, particularly when this information is obtained from external sensors.
73
UNCLASSIFIED
UNCLASSIFIED
Sources of Generic requirements
• JSSSEH – Joint Software System Safety Engineering Handbook
• Good Developers will have their own list (ie. ways to allocate memory, ways of assigning variables, not using certain language functions)
74
UNCLASSIFIED
UNCLASSIFIED
Questions?
75
Software Assurance TechniquesDirectorate of Initial Airworthiness (DIA-DASA)
28/02/2020
20
UNCLASSIFIED
UNCLASSIFIED
Contents
• Software Assurance techniques: – Software requirements– Software Design– Software Implementation– Verification– Configuration Management– Quality Assurance– Software tools– Model Based Development– Formal Methods
77
UNCLASSIFIED
UNCLASSIFIED
Requirements?
• Specification of “what” should be implemented
– How the system should behave
– Application Domain Information
– Constraints on the System’s Operation
– Specification of a system property or attribute
– Constraints on the development process
78
UNCLASSIFIED
UNCLASSIFIED
Requirements Engineering
• The activities involved in discovering, documenting, maintaining a set of requirements for a computer based system
• Systematic and repeatable techniques
79
UNCLASSIFIED
UNCLASSIFIED 80
Inputs and Outputs
MissionNeeds
DomainInformation Regulation
PreviousSystems
Information
Agreed RequirementsSystem Specification
System Models
Requirements Engineering
Process
28/02/2020
21
UNCLASSIFIED
UNCLASSIFIED
Requirements: The “Right” Way
• There is no magical answer or standard
• Some software development standards
• Requires a systematic approach to ensure “goodness” and quality
81
UNCLASSIFIED
UNCLASSIFIED
Example Requirement Qualities
• Atomic• Complete• Concise• Consistent Correct • Implementation free (not solutionised)
82
UNCLASSIFIED
UNCLASSIFIED
Where Do Requirements Hide
• Requirements Evidence:– System Specification– Aircraft Performance Specification– Functional and Performance Specification– Software Requirements Specification– System/Sub-System Design Document– Interface Requirements Specification– Operational Functional Documents– Prime Item Development Specification– Contractor Item Development Specification– DOORS– Requirements Traceability Matrix
83
UNCLASSIFIED
UNCLASSIFIED 84
Requirements Traceability
REQ1REQ2REQ3
.
.
System Specification
REQ1:1REQ1:2REQ1:3
.
.
High LevelRequirements
REQ1:1.1REQ1:1.2REQ1:1.3
.
.
Low LevelRequirements
#REQ1:1.1 CodeDO….LOOP WHILE (x=y)
Code
1 2 3 4TEST 1.1TEST 1.2TEST 1.3TEST 1.4.
5Test Points
28/02/2020
22
UNCLASSIFIED
UNCLASSIFIED
What Is A Software Review?
• A review of the work product, by one or more colleagues of the author, to evaluate the technical content and/or quality of the work
85
UNCLASSIFIED
UNCLASSIFIED
Review Spectrum
86
Ad Hoc
Desk Check
Walkthrough
Inspection
Level of Effort
UNCLASSIFIED
UNCLASSIFIED
Ad Hoc
• Informal reviews may often be unnecessarily expensive (because of time-wasting through lack of focus), and frequently provide a sense of security which is quite unjustified by the relatively small number of real defects found and repaired.
87
UNCLASSIFIED
UNCLASSIFIED
Desk Check
• Only one peer besides the author examines the work product. This is the cheapest review approach, but its usefulness is reliant upon the skills, knowledge and self discipline of the reviewers.
88
28/02/2020
23
UNCLASSIFIED
UNCLASSIFIED
Walkthrough
• The author leads members of the development team through a software product and the participants ask questions and make comments about defects.
89
UNCLASSIFIED
UNCLASSIFIED
Inspection
• Very formal type of peer review where the reviewers are following a well-defined process to find defects.
90
UNCLASSIFIED
UNCLASSIFIED
Why We Do It
91
UNCLASSIFIED
UNCLASSIFIED
What To Look At
92
Object Code
Software Architecture
Software Requirements
Software Design
Source Code
Test Evidence
28/02/2020
24
UNCLASSIFIED
UNCLASSIFIED
What To Look For
• Traceability. Does each higher level element trace to at least one lower level element and does each lower level element trace to at least one higher level element?
• Compliance. Do the requirements/design/code satisfy the required system functional, performance and safety requirements?
• Accuracy/Consistency. Are the requirements/design elements accurate or is there ambiguity? Is there any conflict between requirements or design elements?
• Conformance to Standards. Has the defined requirements or design language been used? Have dangerous design and coding practices been avoided? Does the code comply with the approved language subset?
• Compatible. Are there any conflicts between the software requirements or design and the target hardware (e.g. 16-bit vs. 32-bit).
• Verifiable. Is it possible to prove through test that a requirement has been satisfied or design element correctly implemented?
93
UNCLASSIFIED
UNCLASSIFIED
Evidence of Reviews
• Required: positive evidence from the review (that the reviewed item satisfies each criteria)
• A procedure that outlines the criteria that must be reviewed is usually sufficient
94
Review SheetItem Under Review: Module X Version: 1.1.12
Reviewers: A. SmithM. Jones
Date: 01 Feb 10
Findings:No issues found.
Checklist: The reviewed source code correctly implements the relevant design element
The reviewed source code conforms to the coding standard.
UNCLASSIFIED
UNCLASSIFIED
Software Reviews Summary
• Software reviews are an effective tool for identifying errors as early in the life cycle as possible
• Software reviews are required in order to comply with software assurance standards
• Software reviews must produce positive evidence of item compliance
95
UNCLASSIFIED
UNCLASSIFIED
Software Verification
96
Reviews
Analysis
Verification
Test
28/02/2020
25
UNCLASSIFIED
UNCLASSIFIED
Definition
• The evaluation of software by observing its execution
• “Testing is about choosing elements from the input space of the software being tested”
97
UNCLASSIFIED
UNCLASSIFIED
Software Testing Attitudes
98
No difference between
testing and debugging
12
34
Testing shows that the software
works
Testing shows that the software
doesn’t work
Testing reduces the risk of using the software
5Testing is a mental
discipline that helps everyone develop
higher quality software
BetterBetter
UNCLASSIFIED
UNCLASSIFIED
Good Practice
99
1 2 3 4
Define a test strategy
Carefully design tests for your system
Build a test environment
Track defects to ensure their resolution
5
Adopt tools to automate testing
Manage Testing Resources
UNCLASSIFIED
UNCLASSIFIED
Design Organisation Software CM Goals
• Planned and repeatable process• Software work products are identified and controlled• Changes to identified software work products are controlled• Development and verification is only performed using relevant software
baselines
100
28/02/2020
26
UNCLASSIFIED
UNCLASSIFIED
Benefits of CM
• Links between object code and the development and verification evidence that proves its acceptability.
• Assures that the software that is loaded to the target environment is the same as the software that was verified.
• Assures that developers will work from the correct version of data.• Prevents unintended changes to software or data.• Tracks problems through to resolution.
101
UNCLASSIFIED
UNCLASSIFIED
CM Key Tasks
• Establish a process (software configuration management plan or equivalent)
• Establish a library for repository for software baselines.
• Work products to be placed under CM should be identified.
• Change requests and problem reports are initiated, recorded, reviewed, approved and tracked in accordance with a documented and repeatable procedure.
• Changes to baselines are controlled in accordance with a documented and repeatable procedure.
• Release of software products is controlled in accordance with a documented procedure.
• Status of CIs is recorded IAW a documented procedure.• Keep records/evidence documenting SCM activities and baselines.• Software baseline audits are conducted according to a documented procedure.• Software life cycle environment configuration is controlled.
102
UNCLASSIFIED
UNCLASSIFIED
IBM Clear Case (example)
103
UNCLASSIFIED
UNCLASSIFIED
Software Quality Assurance
• An independent check that the developer has done what they said they would do
• Types of Involvement– Check that particular product complies with requirements.– Check that transition criteria have been satisfied.– Software Conformity Review.– Periodic Audits.
• QA Personnel must be independent enough to be effective (e.g. answer to management).
104
28/02/2020
27
UNCLASSIFIED
UNCLASSIFIED
Final Thoughts on QA
• QA rarely staffed with experienced software developers/coders, but are generally excellent at record keeping
• QA often not capable of negotiating with developers• QA and development need to be in a partnership• Project and senior management often back the developers and not the
QA team• QA people must be proactive, independent and “drivers”• Address QA at the project reviews
105
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
Software Dependability Cube
106
UNCLASSIFIED
UNCLASSIFIED 107
Software Development
• Your software development process must be sufficiently robust to achieve your software assurance activities.
• Further training on the linkage between Software Development Lifecycles and Software Assurance is available on the Intermediate Software Course.
UNCLASSIFIED
UNCLASSIFIED
Software Life Cycle
108
28/02/2020
28
UNCLASSIFIED
UNCLASSIFIED
The Four Software Airworthiness Requirements
109
UNCLASSIFIED
UNCLASSIFIED
System Requirement
s
Concept of Operations
Subsystem Design
System Construction
Subsystem Test
SystemTest
System Requirement
Software Requirement
Software Design
Code and Compile
Integration/ Unit Test
Software/ System Test
Software Engineering
Operational Use
Verify
Verify
Validate
Software Engineering ≠
Software Programming
Software engineering is the set of controls placed around software programming to assure the right product is produced.
110
UNCLASSIFIED
UNCLASSIFIED 111
Example Lifecycles: SpiralUNCLASSIFIED
UNCLASSIFIED
Example Lifecycles: Agile
112
28/02/2020
29
UNCLASSIFIED
UNCLASSIFIED
Example Lifecycles: Agile variants
113
UNCLASSIFIED
UNCLASSIFIED
Software Development Standards
• MIL-STD-498 - Software Development and Documentation
• IEEE/EIA 12207 - Standard for Information Technology -Software Life Cycle Processes
114
UNCLASSIFIED
UNCLASSIFIED
Questions?
115
Software Type Design ApprovalsDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
28/02/2020
30
UNCLASSIFIED
UNCLASSIFIED
What is Type Certification
117
“The process through which compliance with the airworthiness design requirements contained in the Type Certification Basis is established through the development of the Type Design to meet the operating roles and environment contained in the Statement of Operating Intent and Usage (SOIU)”
• Outcome: A Type Certificate is issued when sufficient design, test and certification activities necessary to establish compliance with specified airworthiness standards, in order to operate safely within approved roles and environments, have been completed.
UNCLASSIFIED
UNCLASSIFIED
Type Certification Course
• Deputy Director Type Certification• Director Initial Airworthiness
• Detailed overview of Type Certification concepts and DASA application and process.
• Contact: dasa.typecert@defence.gov.au
118
UNCLASSIFIED
UNCLASSIFIED
What is an MTC?
“Certificate issued by the Authority … that certifies the aircraft type design complies with the applicable Type Certification Basis when operated within the conditions and limitations specified on the associated Type Certificate Data Sheet”
• Aircraft must be on Defence register
• Defence Registered Aircraft (DRA) must have a valid Type Certificate before being authorised for flight operations (exception: ops under MPTF, some UAS)
• Aircraft must be operated in the roles authorised in OIP and TCDS (and defined in SOIU)
• Applicant is usually the acquisition PO
• An AwB is convened to provide a recommendation to issue (or not to issue) an MTC.
• Issued by DG DASA
• MTC is issued to a Defence organisation
119
UNCLASSIFIED
UNCLASSIFIED
Software Scope in Type Certification
• What aviation software is type certified– Software is regulated in DASRs as a “Part and
Appliance”– Parts and Appliances can be subject to Type
Certification– Scope of Parts and Appliances to which Type
Certification applies is defined in the EMACC– Typically term “Aviation Software” is used to describe
software installed on a Type Certified Product (Aircraft, Propeller, Engine)
120
28/02/2020
31
UNCLASSIFIED
UNCLASSIFIED
Is this Aviation Software?• Picture of in flight entertainment
121
Installed Software. Is it Type Certified?
YES. Subject to Authority prescribed benchmarks.
UNCLASSIFIED
UNCLASSIFIED
Is this Aviation Software?• Picture of Mission Planning System
122
Uninstalled Software. Is it Type Certified?
NO. Good Practice is described in the ADRM.
Note: Some Mission Planning Systems will have installed software components which are subject to Type Certification.
UNCLASSIFIED
UNCLASSIFIED
Is this Aviation Software?• Picture of EFB
123
Electronic Flight Bags (EFBs). Is it Type Certified?
MAYBE. Boundaries are described in the ADRM.
UNCLASSIFIED
UNCLASSIFIED 124
Simulators. Are they Type Certified?
NO. Requirements are described in DASR FSTD.
28/02/2020
32
UNCLASSIFIED
UNCLASSIFIED
Type Certification Process Overview
125125
• Produce/update SOIU
• Identify Applicant and design organisations • Agree TCB Airworthiness codes/standards Tailoring to standards (MCRIs) Command safety/capability position
• Agree Means of Compliance (CCL) Evidence requirements Relief through extant certifications (CRE Δ)
• Agree Authority evidence inspections (LOI)• Agree Declaration of Compliance
• Produce design • Update CPP (if required)
• Produce evidence of compliance Evaluations, inspections, ground tests, etc Flight test
• Authority Inspections• Declaration of Compliance
• Apply for certification or approval• Confirm Continued Airworthiness responsibility
Maintain TC
Start
Define Operating Intent
Authority Agreement (CPP)
Design
Compliance Demonstration
Submission
•Authority assessment (Project DoSA attestation, AwB, etc)
• Issue instrument or approve major change
Certification/Approval
Plan for Software Aspects of Certification
Software Accomplishment Summary, Version Description Document
UNCLASSIFIED
UNCLASSIFIED
Certification Program Plan Contents for Software Projects
• Generic Certification Programme requirements discussed on Type Certification Course
• The TCB, including applicable airworthiness design requirements and standards, exemptions, special conditions, equivalent level of safety, environmental protection, etc.
• Primary Certification Code – FAR 25.1301/1309? MIL-HDBK-516C Section 14 & 15?
• Means of Compliance - DO-178B/C?
• Tailored benchmark?
• MCRIs?
• The Compliance Documents that will be subject to inspection by the Authority or its delegate/s, the proposed Authority Level of Involvement (LOI)
• Software specific elements:
• Plan for Software Aspects of Certification (PSAC)
• Change Impact Analysis (CIA)
126
UNCLASSIFIED
UNCLASSIFIED
Organisation Responsibilities
• An Approved Military Design Organisation – Undertakes the design and ensures safety– Declares compliance to applicable Airworthiness
Requirements– Submits Compliance Documents to the Authority– Requests Approval of the change to type design
• Minor – can be approved if within MDOA scope• Major – submits request for approval to the Authority
127
UNCLASSIFIED
UNCLASSIFIED
Organisation Responsibilities
• The Authority– Assures compliance to applicable Airworthiness
Requirements– Conducts Inspections of Compliance Documents as part
of a defined Authority Level Of Involvement (LOI)– Approves design change requests certifying compliance
requirements have been met for the change to type design
128
28/02/2020
33
UNCLASSIFIED
UNCLASSIFIED
When do Software Certifications take place?
• Initial Certification (Acquisition)• In-Service, Changes to Type Design (DASR 21.A.91)
– Major Change to Type Design– Minor Change to Type Design
• DASR clearly establishes what constitutes a major change to software.
129
UNCLASSIFIED
UNCLASSIFIED
Classification of Software Change to Type Design
Appendix A to GM 21.A.91• Where a change is made to software produced in
accordance with the guidelines of EUROCAE ED12C/RTCA DO–178C 'Software Considerations in Airborne Systems and Equipment Certification', the change is to be classified as major if either of the following apply, and the failure effect is CATASTROPHIC, HAZARDOUS or MAJOR:– the executable code for software, determined to be Level A or Level
B in accordance with the guidelines, is changed unless that change involves only a variation of a parameter value within a range already verified for the previous certification standard; or
– the software is upgraded to or downgraded from Level A, Level B or Level C; or
– the executable code, determined to be Level C, is deeply changed, e.g. after a software re-engineering process accompanying a change of processor.
130
UNCLASSIFIED
UNCLASSIFIED
Questions?
131
Understanding Authority LOI for Aviation Software
In-service Problem Reporting
28/02/2020
34
UNCLASSIFIED
UNCLASSIFIED
Overview
• What is LOI?• Why you need to understand Authority LOI• Terminology• Assessing the requirement for Authority LOI for software Aspects• Conduct of Authority LOI for software aspects• Delegating Software LOI• Role of CVE vs Authority inspector for software audits
133
UNCLASSIFIED
UNCLASSIFIED
What is Level of Involvement
134
• What?– “Selection of the compliance demonstration activities and data
that the [Authority] will investigate and the depth of these investigations during the certification process” – EASA
• ‘Involvement’ in What?– In applicant assessments/Decision making process?– Involved in generating or compiling evidence showing
compliance?– Involvement in the Certification Programme.
• The Certification Programme presented to the Authority must include acceptable provisions to allow the Authority to review required Compliance Documents.
• How:– Negotiated set of evidences and activities in the CP, expanded
in the PSAC
UNCLASSIFIED
UNCLASSIFIED
What is Level of Involvement
135
UNCLASSIFIED
UNCLASSIFIED
Why you need to understand Authority LOI
• Programmatic risks can arise if requirements for Authority LOI are not able to be achieved due to limitations in existing contracts.
• Assist in the ease of development and negotiation of a Certification Programme.
• Prevent misunderstandings arising from role confusion. • Need to understand that the Authority may seek a range of evidences
and time up-front in your software development. • Sourcing software from foreign sub-contractors may often generate a
requirement to send DASA Software SMEs overseas, or require a considered and informed proposal to appoint a delegate.
136
28/02/2020
35
UNCLASSIFIED
UNCLASSIFIED
De-confusing terminology
137
• [TAREG] Compliance Finding: Meant different things to different people, may have referred to:– Authority inspections of selected evidence? – Authority
Inspection delegated from the TAR/DAR– Assessment of CRE deltas?– Involvement in the Design Process?– Supplies Acceptance functions?– Assessment of future Supportability?
• Compliance Demonstration– An activity by the applicant to confirm compliance of a design
to certification requirements. – May involve the creation, compilation, inspection or review of
compliance demonstration evidences.
UNCLASSIFIED
UNCLASSIFIED
De-confusing terminology
138
• Inspection of Compliance Documents: – An [Authority] Inspection of the Compliance Documents
generated by the applicant. – Includes evidence submitted for (or sampled/witnessed by)
Authority review during LOI, or submitted with applications for approval of a Major change (or generated internally for Minor changes).
• Level of Involvement– A review of Compliance Documents by the Authority during the
conduct of the applicant’s Certification Programme. – Identified LOI must be completed prior to applying for approval
of the Major change.
UNCLASSIFIED
UNCLASSIFIED
Assessing LOI
139
• The Authority is not resourced to review all affected compliance evidences for an application.
• A risk based decision made through:– An assessment of the likelihood of an unidentified non-
compliance in an application. Considering:• Novelty,• Complexity,• Organisational Performance
– The criticality (safety affect) of such a non-compliance• The use of a risk based approach, allows the authority to focus it’s
assurance role on the most important aspects of a design change. • The LOI approach is consistent with international convention and
best practice for Airworthiness regulation. • Ensure DASA does not waste reviewing trivial compliance
demonstration activities
UNCLASSIFIED
UNCLASSIFIED
LOI: Likelihood of an unidentified Non-compliance
140
• Step 1: Likelihood– Novel to who? – Complex to who?– Who assesses applicant performance?
28/02/2020
36
UNCLASSIFIED
UNCLASSIFIED
Recall: Oversight vs LOI
141
UNCLASSIFIED
UNCLASSIFIED
Consequence of an unidentified non-compliance
142
• Step 2: Criticality• Intuition DAL A or B Software will usually always require class 3 or
higher
UNCLASSIFIED
UNCLASSIFIED
Assessing LOI: Historical difficulties
143
• Design is already Mature prior to submitting a CPP• Applicant cannot provide key documentation such as the PSAC or
the Change Impact Analysis that is required to undertake the assessment.
• Perception that the PSAC is ‘accepted’ without being provided to the Authority.
• Insufficient data is submitted to assess the CPP: Authority is told requested data is ‘not in the contract’, or other difficulties.
• Not understanding different roles between a CVE Inspection and Authority inspection.
• Data requested by the Authority to complete LOI is ‘not available’.
UNCLASSIFIED
UNCLASSIFIED
So what? What do we look at?
144
• Depending on the project (and certification basis), it may not always make sense to look at the same information.
28/02/2020
37
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities
145
• Why is involvement during the software lifecycle preferable:– Authority review may identify a requirement to generate
additional evidence or undertake actions where findings or deficiencies are identified.
– Authority Inspection is improved by a capacity to witness testing, participate in reviews and witness builds or conformity activities.
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities
146
• As a result of these issues, it is preferable that the Authority is involved upfront and as early as possible, and at key points in the software lifecycle.
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities
147
• There is no mandated policy on how the Authority is to achieve it’s audit outcomes, however where suitable the Authority will align with good international practice. – An alternative may be sought by the Authority or applicant,
depending on the particular situation, certification requirements, or lifecycle of a software item.
• International Practice: Stage Audits or inspections: “Stage of Involvement” or ‘SOI’– Terminology used by the FAA to describe a staged auditing
process– Originally described in the FAA Job Aid (Now historical)
describes an inspection process that might be used by: ‘Authorities’, ‘Designees’ [DERs], and ‘Applicants’.
– Continues to be used now internationally.
UNCLASSIFIED
UNCLASSIFIED
LOI Typical activities: planning audit (SOI 1)
148
1. Assure plans and standards meet DO-178B objectives and address other applicable software policy, guidance and issue papers.
2. Assure that the processes described in the applicant’s plans meet the objectives of DO-178B and address other applicable software policy, guidance, and issue papers
3. Obtain agreement between FAA and applicant on the plans, standards, and proposed methods of compliance
• Note: For Software Aspects, the Authority will typically seek to achieve some or all of the outcomes prior to approving the CPP.
28/02/2020
38
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities – Development Audit (SOI 2)
149
1. Assess implementation of plans and standards for the software requirements, design, and code, and related verification, SQA, and SCM data.
2. Assess and agree to plans and standards changes.3. Assess implementation of new technology and methods to
ensure compliance to plans, standards, and agreements. 4. Assure life cycle data satisfies the certification standards
objectives and other applicable software policy, guidance and issue papers.
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities – Verification Audit (SOI 3)
150
1. Assess implementation of verification and test plans and procedures.
2. Assess completion and compliance of all associated SCM and SQA tasks.
3. Ensure software requirements are verified.4. Ensure robustness testing is planned and is being
performed.5. Ensure analyses (including timing, memory, test coverage,
structural coverage and data and control coupling) are being performed as required by the certification standard.
6. Ensure verification activities satisfy the certification standards objectives.
UNCLASSIFIED
UNCLASSIFIED
LOI Typical Activities – Final Audit (SOI 4)
151
1. Assure final software product meets the certification standards objectives and is ready for certification.
2. Address any open items.
UNCLASSIFIED
UNCLASSIFIED
Focus Areas
152
– If there are novel or more critical aspect of the software design, then the Authority will typically focus it’s review on these components. eg:• Thread traces will be done against affected Safety Critical
Requirements. • Planning review may focus on design standards for novel
techniques. • Potentially a focus on the Change Impact Analysis and
regression testing where the modification to the Computer Software Configuration Item may affect existing safety critical requirements / hazards.
28/02/2020
39
UNCLASSIFIED
UNCLASSIFIED
What else may be audited by software inspectors
153
– Other authority sections have differing LOI requirements– Authority software SMEs may seek involvement in
development assurance objectives and focus on requirements validation activities.
– Authority will apply LOI similar to the above for changes to safety critical Complex Electronic Hardware upgrades.
– Minor Changes to type? – Other Authority approvals: i.e. AUSMTSO Authorisations,
UASOPs, ANSP Certificate, may have a similar process in the future.
UNCLASSIFIED
UNCLASSIFIED
Outcomes of Authority LOI
154
• Authority will provide a report at defined stages (perhaps linked to SOI stages) disclosing the outcomes of these involvement activities.
• Authority will provide a report once LOI is complete, advising that Authority LOI (for software aspects) has been closed.
• Applicant cannot submit a declaration of compliance to the Authority until all defined Authority LOI is executed – as the certification programme is not complete.
UNCLASSIFIED
UNCLASSIFIED
Authority LOI – feedback and findings
155
• Authority may pass on Product Certification Findings and Actions that will need to be closed prior to closing LOI. DIFFERENT FROM ORGANISATIONAL FINDINGS AND ACTIONS
• Decision to not action findings and actions will require production of an MCRI (ESF, exception) and DASA must be presented evidence that the elevated risk level has been accepted by command IAW the principles of the DASP.
• Authority may also pass on observations or issues (items related to other areas of the Certification Programme but not precluding closure of LOI)
UNCLASSIFIED
UNCLASSIFIED 156
(What we don’t want to see)
Authority LOI – feedback and findings
28/02/2020
40
UNCLASSIFIED
UNCLASSIFIED
Executing Authority LOI – Historical difficulties
157
• Nested documentation.– DASA will request a document, but the evidence targeted is in
a subordinate document. Seeking the evidence then causes a delay
• Notes taken by auditor become Intellectual Property of the applicant.
• International travel.• Form 31a submitted concurrently with evidence required to
complete Authority LOI.
UNCLASSIFIED
UNCLASSIFIED
Audits one by Compliance Verification Engineers = LOI?
158
• Short answer: No• Many CVEs may elect to follow (or follow a similar process) to the
SOI concept described earlier, to achieve their assessment. • This shared terminology can lead to significant confusion. • DASA may elect to attend a CVE ‘SOI audit’, rather than to run our
own on-site audit, to fulfil our assurance outcomes.
Applicant “SOI” Reports
UNCLASSIFIED
UNCLASSIFIED
Clarification: FAA DERs as CVEs
159
• Projects will sometimes reach back into parent companies (etc) and make use of individuals with privileges and competencies under different regulatory systems. An example of this are individuals with FAA DERs designation, while employed by a design organisation, are able to conduct Authority inspections on behalf of the FAA.
• While a DER (etc) qualified individual may be appointed as a CVE for a DASR MDOA; they are not conducting assessments on behalf of the Authority.
• No one is conducting assessments on behalf of the Authority unless they have been explicitly notified by the DASA that they are fulfilling this role.
• Intuition: When contracted as a CVE, DERs and other highly competent individuals are not exercising an Authority delegation. However, when such an individual is contracted (and this is outlined to the Authority in the CP), this may result in lower Authority LOI.
UNCLASSIFIED
UNCLASSIFIED
Questions
160
28/02/2020
41
Good Practice for Problem ReportingDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED 162
Problem Reporting: An overview
• Development Lifecycle: Problem Report Management• Management of Open Problem Reports• Linkages between In-service anomalies, Occurrence Reporting
(potentially unsafe conditions) under DASR 21.A.3A and Problem Reporting
• When does a Problem Report require the Authority to issue an Airworthiness Directive under DASR 21.A.3B(b)
UNCLASSIFIED
UNCLASSIFIED 163
Problem Reporting, defining the problem.
• Problem reporting is one of the most important Software Configuration Management activities. – Multiple stakeholders, terminology (classification systems) and
cross domain. – Usually linked to/generates a Change request and is heavily tied in
to CM activities. • Linkage to problems identified in-service• Impacts Holder obligations and Occurrence Reporting system
UNCLASSIFIED
UNCLASSIFIED
Known and
unknownanomalies
Anomalies Identified and RemovedDuring Test
Anomalies Identified and Resolved During Design
Anomalies w
ithin Software over single developm
ent cycle
Problems / Anomalies
Software Assurance moves this line upwards
Increased Rigour identifies and resolvesanomalies or systematic failures that
result from design errors and faults
Anomaliesresident in
operatingsoftware
164
28/02/2020
42
UNCLASSIFIED
UNCLASSIFIED 165
Development Problem Reporting – States and Classification
• The below terminology is the preferred terminology in upcoming FAA/EASA guidance:– Classification schemes
• Potential Safety: assessed at the product, system, or equipment level [More detail later in this section], a PR having an actual potential Catastrophic, Hazardous or Major safety effect on the aircraft, or affecting compliance with operating rules.
• Functional: A PR recording a process non-compliance, deficiency or deviation that cannot result in a potential safety, nor potential functional, impact.
• Documentary: a PR linked to a deficiency in a life cycle data item but not linked to a process deficiency or deviation. This includes typographical or editorial defects in life cycle documents.
• Other: a PR having no potential safety impact, no potential functional impact, and not linked to a process deficiency or a deviation or to a documentary deficiency
UNCLASSIFIED
UNCLASSIFIED 166
• Different stakeholders for different items
• Complex interfaces between aircraft systems.
• All stakeholders have a role in the In-service Problem Reporting Change
• Stakeholders should consider their view when classifying problem reports (i.e. stakeholders at different levels will have a different position over whether a PR is potential safety)
Development Problem Reporting – Stakeholders
UNCLASSIFIED
UNCLASSIFIED 167
• The applicant (i.e. the top level stakeholder), at the time of certification, should have dispositioned all problem reports. All problem reports with a potential safety impact must be resolved.
Development Problem Reporting – StakeholdersUNCLASSIFIED
UNCLASSIFIED 168
• Due to Capability imperatives, projects may seek to advance with “Open” PRs.
• Good Practice:– Good Practice for Problem Reporting is available in EASA AMC 20-
189– Good practice requires the production of an OPR Summary report
(to higher stakeholders or the cert authority) identifying:• The OPR, the affected configuration item(s), summary of the
OPR (understandable by higher level stakeholders), a description of the problem understandable by higher level stakeholders.
Development Problem Reporting – OPRs and certification
28/02/2020
43
UNCLASSIFIED
UNCLASSIFIED
In-Service Problem Reporting
169
UNCLASSIFIED
UNCLASSIFIED 170
In-service Problem Reporting
• Anomalies or occurrences encountered in-service may require design organisations to raise and track problem reports
• Per DASR 21.A.3A(a), the Holder must: “establish a system or network of systems to receive information about occurrences”.
• Problem Reports should be provided to applicable design organisations/OEMs for assistance in assessments, and tracking at an appropriate level.
• Recall: All Open Problem reports (i.e. including those captured during in-service operations), with a potential safety impact on the proposed installation must be resolved for a future modification.
UNCLASSIFIED
UNCLASSIFIED
• The MTC Holder must have a system to carry out the following:– Collect
• Establish a system or network of systems to receive information about occurrences
– Assess/Analyse • Determine whether the occurrence is reportable to DASA• Determine what the occurrence means from a type design
Perspective i.e. whether it is or may be an unsafe condition– Report
• Report to DASA in a Form and Manner stipulated by DASA for reportable and/or unsafe occurrences
– Investigate• Investigate and determine the corrective actions required
171
In-service Problem Reporting and Unsafe ConditionsUNCLASSIFIED
UNCLASSIFIED 172
• Following the identification of an occurrence related to a software/AEH item, the MTC Holder in conjunction with any other the stakeholders should:– Record the PR. – Classify the PR (i.e. classify as ‘Potential Safety’)– Assess whether the occurrence, is a reportable occurrence per
DASR 21.A.3A(b), and report if necessary (potentially unsafe condition is identified) to the Authority and relevant stakeholders
– Actions to resolve any unsafe condition• The Authority, will make a determination regarding whether the situation
disclosed in the Occurrence Report necessitates the issuance of an Airworthiness Directive; to mandate additional actions to mitigate / resolve an unsafe condition.
• [INTUITION]: A ‘Potential Safety’ PR, is not automatically an unsafe condition, but is a reportable occurrence.
In-service Problem Reporting
28/02/2020
44
UNCLASSIFIED
UNCLASSIFIED 173
Questions?
In-Service CM and Load ControlDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
Software Load Control – In-Service
175
UNCLASSIFIED
UNCLASSIFIED
Software Load Control
• Design Organisation, CAMO and Maintenance Organisations – All are responsible
• Software Load Control: Controls must be established to ensure the aviation software configuration is: – approved for installation; – installed in the required configuration; and – verified to be installed correctly.
• DASR 21– 21.A.165 – Obligations of the holder
• CAMO– M.A.201 – Responsibilities
• 145 (AMO)– 145.A.42 – Acceptance of components– 145.A.48 – Performance of Maintenance
176
28/02/2020
45
UNCLASSIFIED
UNCLASSIFIED
C-130J & C-27J Software Load Control Issues
177
LRU with Same Part number for 2 different platformsDifferent Software for each Aircraft
Software Part Number not identified in LRU
How do you determine whether the LRU belongs to C-130J or C-27J?
UNCLASSIFIED
UNCLASSIFIED
On-board interrogation
• “Smart” aircraft may have on-board interrogation capability or a Memory Loader Verifier allowing maintainers to establish current aircraft software configuration during maintenance or when loading software
178
UNCLASSIFIED
UNCLASSIFIED
No interrogation capability
• Legacy aircraft may need to be verified via other means such as an OEM report/attestation from OEM
• There is a need to find out what software is in boxes that return from an OEM to assure that the only the authorised software configuration is installed
179
UNCLASSIFIED
UNCLASSIFIED
Production Approval
• DASR 21.A.307, requires an aviation software item to have an Authorised Release certificate (i.e. DASR Form 1, or equivalent). – This Form 1 certifies that the item was manufactured in conformity
to approved design data and is marked in accordance with DASR 21 Subpart Q.
– DASR 21, Subpart F, and G provide guidance on acquiring a production organisation approval, or producing items without one.
– DASR recognition provides guidance on equivalent instruments.
180
28/02/2020
46
UNCLASSIFIED
UNCLASSIFIED
Aircraft Software Management
• Software may come from– MTC Holder and associated design organisations. – User Modifiable Software (more later)– Vendor supplied. i.e.
• Navigation and Terrain databases• In flight entertainments systems.
• May be loaded on– Removable media– Portable Electronic devices– Ground based servers. – Automatic Test Equipment.
181
UNCLASSIFIED
UNCLASSIFIED
In-Service Software Management:
• Refer to FAA AC 43-216 for the latest guidance.
182
UNCLASSIFIED
UNCLASSIFIED
Special Topics: Field Loadable Software
• Field Loadable software reduces aircraft down-time for maintenance and increases efficiency of maintaining airborne equipment.
• The aircraft system must be designed to allow a software item to be ‘field-loadable’ per DO-178. – This includes in-built protections against corruption or partial
loading at an integrity level appropriate for the FLS. – Protection methods must be implemented to prevent inadvertent
enabling of the field-loading function during cruising or any other safety-critical phase.
– Further guidance is provided in DO-178C section 2.5.5
183
UNCLASSIFIED
UNCLASSIFIED
Special Topics: User Modifiable Software
• User Modifiable software component is that part of the software that may be changed by the user within the modification constraints without certification authority review, If the system requirements provide for user modification.
• User modifiable software must be designed for. Further guidance is available in DO-178C 2.5.2
184
28/02/2020
47
UNCLASSIFIED
UNCLASSIFIED
Special Topics: Cybersecurity (Developing)
• Aircraft Network Security will add additional Software Maintenance management and load control considerations
• Good practice is found in RTCA DO-355• Stay tuned for further developments.
185
UNCLASSIFIED
UNCLASSIFIED
Questions?
186
Aeronautical Data and Mission PlanningDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
UNCLASSIFIED
What is a mission planning system?
• What is a Mission Planning System:– A suite of software applications and associated
hardware that allow maps, charts, weather, intelligence and aircraft performance data to be used in developing navigation solutions, communication settings, flight/mission calculations, etc.
• Examples:– Portable Flight Planning System– Joint Mission Planning System– F35 ALIS
188
28/02/2020
48
UNCLASSIFIED
UNCLASSIFIED
F-35 ALIS
189
UNCLASSIFIED
UNCLASSIFIED
Is this a good idea?
190
Mission Planning and Google Earth?
UNCLASSIFIED
UNCLASSIFIED
Technical Aspects of MPS
• There are hazards associated with MPS that could propagate to the aircraft safety:– Databases may have errors.– Calculations can be incorrect.– Data could be corrupted during transfer.
• MPS must be assured commensurate with the severity of associated hazards.– No different to any other system with safety implications.
• Knowledge of operational processes is required to determine how severe the hazards are.
191
UNCLASSIFIED
UNCLASSIFIED
Technical Aspects of MPS
• DASA position is that MPS are not directly impacting airworthiness– MPS no longer to be in the TCBs– DASA does not certify MPS– Responsibility lies with MAO, through CASG ( Acq & Sus SPOs)
• Important enough that DASA provides guidance– AeroData managed separately, through DASR.ANSPs– ADRM chapter (Synergies with EFB)– Factsheet
192
28/02/2020
49
UNCLASSIFIED
UNCLASSIFIED
Common MPS Hazards
• Erroneous Weight and Balance Calculations• Erroneous Take Off and Landing Data Calculations• Erroneous Fuel Usage Calculations• Incorrect Translation of Waypoint Map Coordinates• Erroneous Data Packages for Guided Weapons• Introduction of Errors into Aeronautical Data
– Transfer, Formatting or Translation– Digital Maps– Navigation Databases– Airfield Databases– Digital Terrain Elevation Data– Obstacle Data
193
UNCLASSIFIED
UNCLASSIFIED
Aeronautical Data Processing
• For each link, assure that:
– errors are not introduced, or
– errors will be detected.
194
Mission Planning System
Database
Real World Data
Reformat
e.g. DAFIF, DTED, Jeppesen
W&BCalculations
(Generate)
TakeoffData
(Generate)
AirfieldDatabase(Transfer)
ChecksumChecksum
Checksum
Assurance
Assurance
Reversal Check
UNCLASSIFIED
UNCLASSIFIED
The Process
• System Safety – use the 7 step Risk Management Process– Identify the data processed by the MPS.– Identify the data criticality and data assurance level.– Identify each MPS component that processes each data
element.– Identify what function that each MPS component
performs for each data element.• Transfer, Format, Manipulation, Generation
195
UNCLASSIFIED
UNCLASSIFIED
The Process
• System Safety – use the 7 step Risk Management Process– Define and Implement Treatments to assure the
detection and handling or absence of errors.• OpEval (Validation)• Increase assurance• Ensure data comes from an authorised source• Verification Tools• Operational restrictions (Refer SIs/Standards
Manual)
196
28/02/2020
50
UNCLASSIFIED
UNCLASSIFIED
Criticality
197
Severity DAL Data Criticality
Definition Data Assurance
LevelCatastrophic A Critical The data, if erroneous, would prevent continued safe
flight and landing or would reduce the ability to cope with adverse operating conditions to the extent that there is a large reduction in safety margins or functional capabilities. There is a high probability when trying to use corrupted critical data that an aircraft would be placed in a life threatening situation.
1
Hazardous B Critical 1
Major C Essential The data, if erroneous, would reduce the ability to cope with adverse operating conditions to the extent that there is a significant reduction in safety margins. There is a low probability when trying to use corrupted essential data that an aircraft would be placed in a life threatening situation.
2
Minor D Essential 2
No Safety Effect
E Routine The data, if erroneous, would not significantly reduce aircraft safety. There is a very low probability when trying to use corrupted routine data that an aircraft would be placed in a life threatening situation.
3
UNCLASSIFIED
UNCLASSIFIED
Treatment Options
Function Data Assurance
Level
Detection and Handling OR Absence
Transfer Critical Digital Error DetectionORFeedback / Read Back Verify
OR Level A or B
Essential OR Level C or D
Routine No requirements.
Format Critical Feedback / Reversibility CheckORIndependent Redundancy
OR Level A or B
Essential OR Level C or D
Routine No requirements.
Manipulate/ Generate
Critical Feedback / Reversibility CheckORIndependent Redundancy
OR Level A or B
Essential OR Level C or D
ANDLogical Consistency Checks
ORSemantic Consistency Checks
Routine No requirements.
198
See notes in Table 3 eADRM S5 C3 for additional information on assurance levels.
UNCLASSIFIED
UNCLASSIFIED
Source of Data Criticality Level examples
• DO-201• DO-276• AERODROME OBSTACLE CHARTS
199
UNCLASSIFIED
UNCLASSIFIED
Other Factors
• There are other, important characteristics of aeronautical data that the MPS may negatively affect:– accuracy– resolution– timeliness– completeness– format
• These considerations may result in additional requirements being placed on the MPS.– e.g. When translating the navigation database, the input
resolution shall be maintained.
200
28/02/2020
51
UNCLASSIFIED
UNCLASSIFIED
Summary
• DASA is not directly involved in the management of MPS– MPS are not considered as directly impacting
airworthiness – Guidance from DASA however exists through ADRM
and (upcoming) factsheet
• System Safety principles are applied, through the 7-step risk management process
201
UNCLASSIFIED
UNCLASSIFIED
Further Reading
• eADRM Section 5 Chapter 3 Mission Planning Systems
• eADRM Section 5 Chapter 4 Electronic Flight Bags• RTCA DO-200B Standards for Processing
Aeronautical Data• RTCA DO-201A Standards for Aeronautical
Information• AC 03/2018 – 7 Step Risk Management Process
202
UNCLASSIFIED
UNCLASSIFIED 203
Questions
Electronic Flight BagsDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
28/02/2020
52
UNCLASSIFIED
UNCLASSIFIED
Introduction
• FAA definition:– Electronic display system intended primarily for flight deck use that
includes the hardware and software necessary to support an intended function
• EASA definition:– An information system for flight deck crew members which allows
storing, updating, delivering, displaying, and/or computing digital data to support flight operations or duties.
• EFBs can be either:– Portable or Integrated
205
UNCLASSIFIED
UNCLASSIFIED
ADF use of EFBs
• EFBs are not typically part of the aircraft type design and are usually considered as Aircraft Support Systems
• Authority has prescribed design requirements for EFB as they may have an effect on the safety of flight– ADRM Section 5 Chapter 4
• The above is in-line with the guidance provided by EASA and the FAA– FAA Advisory Circular 120-76D – EASA - ED 2014/001/R: AMC 20-25
206
UNCLASSIFIED
UNCLASSIFIED
Types of EFBs
• Portable EFBs:– Can not have a worst credible failure condition greater than Minor
(2x.1309)• Commercial electronic hardware is not certified for aviation use and so
its reliability is not defined. This means that a given functionality may be lost any moment
– Used primarily for:• Display of Static Information (Publications, Manuals,
Aeronautical Information, Flight Orders, Checklists etc)• Situational awareness tool (Interactive maps, Chart Viewers,
Mission Planning, and Flight Planning)
• Integrated EFBs– Integrated EFBs are part of the aircraft Type Design and require design
approval and certification as per any other flight display or aircraft instrument. See ADRM Section 2 Chapter 3 – Aviation Software.
207
UNCLASSIFIED
UNCLASSIFIED
EFB Design Requirements
• Conduct System Safety Assessment to determine worst credible failure condition– Portable EFBs must not have a worst credible failure
condition greater than Minor (FAA AC 2x.1309)
• EFB Software– Assured commensurate with the worst credible failure
condition (perform intended functions for portable EFBs)
• EFB Hardware – Adhere to role equipment and portable electronic device
design requirements as per ADRM Section 5 Chapter 6
208
28/02/2020
53
UNCLASSIFIED
UNCLASSIFIED
EFB Software
• Portable EFBs:– Evidence must be documented demonstrating that the
Portable EFB operating system and hosted application software can perform the intended functions and do not provide false or misleading information
– FAA Order 8900.1 Vol.4 Ch.15 provides a good checklist to evaluate the EFBs intended functions
• Integrated EFBs:– Operating system and hosted application software must
comply with the requirements of ADRM Section 2 Chapter 3 – Aviation Software
209
UNCLASSIFIED
UNCLASSIFIED
Software Application Considerations
• Verification program should demonstrate that applications meet the functional requirements and are robust enough to prevent the provision of confusing or misleading information.
• Focus on good practice:– Reduce/eliminate misleading data– Readable– Usable– Robust– Accurate, available and timely provision of data– Compatible with OS
• Applications with NAA/MAA approval should be used when available
210
UNCLASSIFIED
UNCLASSIFIED
EFB Hardware Considerations
• Adhere to role equipment and portable electronic device design requirements as per ADRM Section 5 Chapter 6– Use of aircraft power– Batteries– Environmental Considerations– Mounting Devices– Human Machine Interface– Electromagnetic Compatibility – Stowage
211
UNCLASSIFIED
UNCLASSIFIED
Good Practice – Portable EFBs
• Use supported by Safety Case• Perform intended functions • Software robust• Robust backups or procedures in the event of
failure• Hardware compatible with the target environment• Operational procedures are adequate• Use supported by operational evaluation/workload
assessment
212
28/02/2020
54
UNCLASSIFIED
UNCLASSIFIED
Further Information
• ADRM Section 5 Chapter 4• FAA Order 8900.1- Volume 4 Chapter 15
– Electronic Flight Bag Authorization Process • Advisory Circular (AC) 120-76D
– Authorization for Use of Electronic Flight Bags• Advisory Circular (AC) 20-173
– Installation of Electronic Flight Bag Components• EASA Decision (ED) 2014/001/R
– Airworthiness and Operational Considerations for Electronic FlightBags, Annex II – AMC 20-25
• Civil Aviation Advisory Publication (CAAP) 233-1(1) – Electronic FlightBags
• Air Warfare Centre – Command Systems - ADF EFB intranet webpage
213
UNCLASSIFIED
UNCLASSIFIED
Questions?
214
Software in UASDirectorate of Initial Airworthiness (DIA-DASA)
UNCLASSIFIED
UNCLASSIFIED
DIA-UAS
• UAS mailbox - dasa.uas@defence.gov.au
• DASA website -https://www.defence.gov.au/DASP/DefenceAviationSafetyProgram/InitialAirworthiness/UASCertification.asp
• UAS Regulation – self-paced awareness module -https://www.defence.gov.au/DASP/Docs/Regulations/DefenceAviationRegulationSet/SelfPacedDASRUASAwarenessModule.pdf
216
28/02/2020
55
UNCLASSIFIED
UNCLASSIFIED 217
Certified Specific OpenCategory Certified Specific A Specific B Small UAS
<25kgVery Small
<2kgMicro <0.1kg
Is an Airworthiness Instrument required? Yes.
MTC/MRTCYes.
UASOP
No. Command approval with notification to
DASA
No. Command approval
Is production and design organisation approval required?
Yes May be No No
Operational restrictions No Yes Standard Scenario based Standard Operating Conditions
Populous Overfly Yes Subject to SFARP
Dependent on Standard Scenario No
Is continuing airworthiness organisation approval required?
Yes May be No No
Is maintenance organisation approval required?
Yes May be No No
UAS Operator Requirement Mil Pilot Specified in
UASOPCommand discretion Command discretion
SW SME Involvement
Yes Possible
Software Section Interface to UAS UNCLASSIFIED
UNCLASSIFIED
Questions?
218
top related