weapon systems and cyber testing and evaluation cyber...17 march 2016 elbert michael ruiz, fred...
TRANSCRIPT
![Page 1: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/1.jpg)
17 March 2016
Elbert Michael Ruiz,
Fred Wright, PhD,
Ronald Prado, Douglas Woods
Weapon Systems and Cyber Testing and Evaluation
![Page 2: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/2.jpg)
2
Outline
Challenges/Impetus of Weapon Systems T&E
Methodology: Leveraging Systems Engineering
Lessons Learned and Observations
Needs and Technology Gaps
![Page 3: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/3.jpg)
Impetus for Topic
Risk Management FrameworkSystem Development Life Cycle Activities
• Cyber Security Directive now applies to all DoD IT (including Platform IT)
• Confidentially, Integrity, Authorization, and non-repudiation
• Cybersecurity fully integrated into system lifecycles
• Cybersecurity T&E is conducted throughout the acquisition lifecycle
• Two phases:
• 1) Cooperative Vulnerability and Penetration Assessment,
• 2) Adversarial Assessment
• Includes all digital paths, not just Ethernet
IA Focused Cyber SecurityFocused
3
![Page 4: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/4.jpg)
4
Cyber-Physical System Examples
Cyber-physical systems can also be self-contained (condensed System-of-Systems)
Commercial vehicles exhibit heterogeneous types of vulnerability vectors
Natural extension to apply concepts, techniques, and procedures to military Weapons Systems
Multi-stage attack sequences
Vehicle-to-VehicleCommunications
RSU-to-VehicleCommunications
Keyless Entry
Tire PressureSystem
InfotainmentUnit
Telematics
EngineControl
Unit
Transmission Control Unit
CAN Bus Controller
SecuritySystem
Anti-LockBrakes
On-Board Diagnostics
ClimateControl
http://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/
![Page 5: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/5.jpg)
5
Weapons Systems as Cyber-Physical Systems
![Page 6: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/6.jpg)
6
Outline
Challenges/Impetus of Weapon Systems T&E
Methodology: Leveraging Systems Engineering
Lessons Learned and Observations
Needs and Technology Gaps
![Page 7: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/7.jpg)
Cybersecurity T&E Phases mapped to
Acquisition Life Cycle
Department of Defense
Cybersecurity Test and Evaluation GuidebookVersion 1.0July 1, 2015
Most high-level methodology descriptions call out the the actual assessment phase, but what exactly does it entail?
![Page 8: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/8.jpg)
8
Starting Points - Assessment Methodology
CYBER & RF
Experts
Assessment Report
Attack Tree Analysis
• System Documentation• Block diagrams• Subsystem Detail• Interconnectivity
• Interviews• On-site Examination• Access to system• Risk Man. Framework
documentation
Subsystem Vulnerability Assessment
• Most likely ways in to affect mission
• Ability to Detect,Prevent, React, Recover from Cyber Attack
• Potential vulnerability to disruption, spoofing, malware
• Attack Surfaces• Network Nodes
• Propagation paths
Subsystem Identification
• Delphi Method• Subject Matter Experts
(e.g.)• Embedded SW
Reverse Engr (RE)• RF Protocol RE• Hardware RE
Vulnerability Analysis Deep
Dive
Develop Exploits
Demonstration/Red Team
Events
• Focus on likely paths• System Simulation• System Hardware• System SW• SME capabilities (SW Defined
Radios, fuzzers, spoofers, signal monitors)
• Difficult/Time consuming
• Malware almost exclusively 0-day
• Disrupt/Spoofing easier
• Validates ability of adversary
• Can assess and test mitigations
• Risks and Recommended Actions
T&E CapabilityNeeded
Common Denominator
![Page 9: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/9.jpg)
9
Attack Tree Analysis (ATA)
Paradigm for performing hostile threat risk analysis using a rigorous tree-structured mathematical approach
Techniques first published and described in the early 1990’s
Based on Fault-Tree Analysis methodology developed in the 1960’s/70’s
ATA methods incorporate not only details of the system being defended, but also methods available to the attacker
Attack tree models excel at estimating the risk for situations where events happen infrequently or have never happened before
![Page 10: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/10.jpg)
Attack Tree Example: Backdoor Scenario
10
![Page 11: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/11.jpg)
11
RF links/comms
GPS
Supply chain (hardware and software)
Maintenance interfaces
Reprogramming interfaces
IT enterprise connections
Command and control systems
Mission planning
Training systems
Common Attack Vectors for Weapon Systems
![Page 12: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/12.jpg)
System Cross-Functional Cyber Analysis
12
RF System Vectors
SoftwareVectors
HardwareVectors
Network Vectors
Comms/GPS/etcMaintenance Ports/Anti-Tamper
Auto-pilot/HUDs/Collision Avoidance
TCP-IP/ System Busses
![Page 13: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/13.jpg)
Vulnerability Assessment Methodology for Embedded Systems
13
Operational Assessment
Attack Surface Enumeration /
Risk Assessment
Message Reverse
Engeineering
MessageGeneration
Experiments / Technique
Refinement
Download Code
Initial Disassembly/ Static Analysis
Dynamic Analysis
HardwareReverse
Engineering
Instrument- ation
RF Link Insertion / Propagation
Analysis
Lab or Field Test DemonstrationsSoftware Analysis
Communications Analysis
Software Assessment
Hardware Assessment
Communications Assessment
Exploit Development / Experiments /
Technique Refinement
SDR Development
Phase I Phase II Phase III
Our full 3-phase approach for vulnerability analysis and exploitation
SDR=SW Defined Radio
![Page 14: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/14.jpg)
14
Outline
Challenges/Impetus of Weapon Systems T&E
Methodology: Leveraging Systems Engineering
Lessons Learned and Observations
Needs and Technology Gaps
![Page 15: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/15.jpg)
15
Develop the assessment team Avionics and Electronic Warfare (EW) platform cyber
assessment expertise in developmental stages
Cross-section of embedded system reverse engineering skill-sets
Technically proficient in RF and digital protocol analysis, firmware reverse engineering, embedded vulnerability discovery and exploit
Vulnerability discovery and exploit important step Improves confidence of current and future assessment scoring
Confidence in assessment scoring builds over time (multiple platform assessments)
Commonalities in cross-platform subsystems inform – likely vulnerabilities
Best Practices/ Lessons Learned
![Page 16: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/16.jpg)
16
Start assessments early
System research is inherently time-consuming.
Identification and demonstration of vulnerabilities even more time-consuming
Provide access to the necessary platform experts to the assessment team
Pilots, trainers, maintenance crew, sub-system SMEs
Access to detailed system/subsystem information critical
Message formats, timing, and protocols between platform subsystems
Include all digital paths (1553, serial, Ethernet, RF)
Board schematics and firmware for each relevant subsystem
Best Practices/ Lessons Learned (Continued)
![Page 17: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/17.jpg)
17
Access to System System Integration Labs
Hardware in the Loop Labs
Off-site System/subsystem analysis – Subsystems available to assessors at their facilities
Plan multiple trips for assessors
Reuse past analysis and assessment results and patterns Device evaluations should feed “platform” evaluations
Feed back into hardware and software standards/implementation guides, security patterns and approaches Security is built-in to systems engineering
Best Practices/ Lessons Learned (Continued)
![Page 18: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/18.jpg)
18
Outline
Challenges/Impetus of Weapon Systems T&E
Methodology: Leveraging Systems Engineering
Lessons Learned and Observations
Needs and Technology Gaps
![Page 19: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/19.jpg)
19
Cross-Functional vector tracking/test control (https://kepler-project.org/ & http://ptolemy.eecs.berkeley.edu/ )
Smart Fuzzers
Intelligent protocol/stack reverse engineering tools
Open standards with reference architectures and controls for security (e.g., Future Airborne Capability Environment (FACE™))
Persistent laboratories with these capabilities and weapon systems “stacks” to verify implementations in DT
Including simulations of controlled processes (real-time but not necessarily high-fidelity)
Gaps and Needs
![Page 20: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/20.jpg)
Smart Fuzzer and Automated Protocol Reverse Engineering Concepts
Example protocol analyzer structure using machine learning (e.g., deep learning techniques)
Support for thousands of simultaneous tests-Reuse/share environment assets as appropriate
Analysts can make informed decisions on which bugs to target
![Page 21: Weapon Systems and Cyber Testing and Evaluation Cyber...17 March 2016 Elbert Michael Ruiz, Fred Wright, PhD, Ronald Prado, Douglas Woods Weapon Systems and Cyber Testing and EvaluationImpetus](https://reader035.vdocuments.us/reader035/viewer/2022062920/5f0267fc7e708231d4041ed5/html5/thumbnails/21.jpg)
21
Assessments have been successfully executed and lessons learned are available
Time and cost must be balanced with “depth” of assessment
Need: Integration with systems engineering/design to ensure better security
Need: automation to reduce time and cost
Summary