cybersecurity test and evaluation: the atec perspective - itea · cybersecurity: what does atec...
TRANSCRIPT
U.S. ARMY TEST AND EVALUATION COMMAND
Cybersecurity Test and Evaluation: The ATEC PerspectiveMike Zwiebel, Director of Test Management
27 March 2019
Agenda
1. What does ATEC Need to Know about Cybersecurity2. Cybersecurity Evaluation Data Sources3. Cybersecurity T&E Phases4. Cybersecurity Testing Opportunities5. DEVOPS / Rapid Acquisition: Notional T&E Strategy6. Challenges7. Back-Up Charts
2
Cybersecurity: What Does ATEC Need To Know• U.S. Army Evaluation Center (AEC) must evaluate effectiveness, suitability,
survivability.• Operational survivability concern: Is the system robust and resilient against hostile
cyber activity?o Does the system meet all Federal and DoD cybersecurity regulations,
guidelines, and best practices? o Does the system introduce exploitable cyber vulnerabilities to the systems and
networks with which it interoperates?o Are vulnerabilities introduced to system survivability when integrated into its
end-state employment? o Does the system provide the capability to detect the loss of system or data
integrity, and to restore the system and data to a known good (trusted) state?• Operational system/network cybersecurity capabilities:
o Prevent compromise by threato Mitigate effects when compromisedo Recover system to pre-compromised state Compliance with IA controls
are necessary but not sufficient
3
Cybersecurity: Evaluation Data Sources• Documentation
o DOD Architecture Framework (Operational/System/Technical Views)o Program Protection Plano Contract Language (Request For Proposal)o Technical Reviewso System Engineering Plans
• Risk Management Framework (RMF) Artifacts• Contractor • System Integration Labs (SILs) - PM or AFC Combat Capabilities Development
Command (CCDC)• Test Teams
o CCDC/ Data and Analysis Center (D&AC)/Lethality, Survivability & Human Systems Integration Division (LSH)
o PM ITTS, Threat Systems Management Office (TSMO)• ATEC Test Centers?
4
Cybersecurity: T&E Phases
• Develop Cybersecurity T&E Strategy
• Develop Evaluation Methodology inclusive of Cybersecurity
Outcome: MS A TEMP
• Develop DT&E Framework• Update Cybersecurity T&E
Strategy• Incorporate Cyber Attack
Surface elements into test plans
• Define resources for cybersecurity DT&E
Outcomes: MS B TEMP and input to RFP, PDR,
CDR
• In cooperation with SE develop understanding of system vulnerabilities
• Assess system for vulnerabilities
• Provide feedback to SE
Outcome: Test plans; initial vulnerability
assessments; input to CDR
• Execute adversarial cybersecurity DT&E event within realistic mission environment.
• Use of Cyber ranges
Outcome: Input to DT&E Assessment,
MS C TEMP
• Overt and cooperative review of the system to characterize operational cybersecurity status
• Determine residual risk as well as readiness for the Adversarial Assessment.
Outcome: POA&M for documented
vulnerabilities
Understand Cybersecurity Requirements
Characterize Cyber Attack
Surface
Cooperative Vulnerability Identification
Adversarial Cybersecurity
DT&E
Full RateProduction
Decision Review
Technology Maturation &
Risk Reduction
Engineering & Manufacturing Development
Production and Deployment O&S
SRR SFR CDR TRR SVRASR
MaterielSolutionAnalysis
MDD
DRAFTCDDAOA CDD CPD
IATT
Cooperative Vulnerability and
Penetration Assessment
IOT&E
T&E Phases
OTRRDT&EEvent
Adversarial Assessment
CDD Validation
Dev RFP Release Decision
DT&E Assess-
ment
DT&E Assess-
ment
PDR
ATOA B C
• Full operational test and evaluation of the system’s defensive cyberspace performance in the operational environment.
Outcome: OTA and DOT&E Reports
Phases are iterative and executed as part of the Acquisition continuum.
Developmental Test Integrated DT/OT
Operational Test
This is Cybersecurity “Shift Left”5
6
Network
Environment (Live, Virtual, Collective)
Systems Under Test (SUTs)
Electronic Attack
Cyber Eletromagnetic Activity
ATEC• Test Methodology/Environment• Instrumentation• System Access/Operation• System Knowledge• Evaluation (ESS/Safety)
AFC Lethality Survivability and HSI
Division• Army CVPA lead• Augments TSMO during
Test• Conducts assessment
during Operation Test:o Prevento Mitigateo Recover
PM ITTS Threat Systems Management Office
• NSA certified Red Team• Cyber Aggressors• Validated Cyber Threat
Pilot Studies• ATC• EPG• RTC
National Cyber Range• Operated by TRMC• Provides realistic
Cybersecurity T&E
System Access• Network• Physical• Electronic• Other
Cybersecurity: Testing Opportunities
Test Center Involvement7
Other capabilities
Early Soldier
Involvement
8
Deployment (Refinement by CPTs at Ft. Gordon)
Test and Evaluation(1 to 6 months of testing at the forge facilities with dedicated team)
Capability Development (TBD)
Demonstrations, software and code analysis
Fielding Decision
Dedicated and Independent Test Team & Contractors located at the forge facilities –Enables Continuous, Real Time Data Analysis and Report Writing
Report
DT/ Lab testing
Robust testing in an operational realistic
environment
Limited User Event
TestingFix
Find
Test
DT/ Lab testing
FixFind
Test
Limited User Event
testingFixFind
TestAdditional T&E as required
Ktr/ MIL operators testing (CPT)
In stride assessments & iterative testingKTR MILKTR / MIL
Agile Test Team Focus: Learn, Assess, Find, Identify, Recommend, Fix, and Verify
Entry Point for prototype
DT/OT OT
Month 1
1 2 3 4
Month 2
1 2 3 4
Month 3
1 2 3 4
Month 4
1 2 3 4
Month 5
1 2 3 4
Month 6
1 2 3 4
Month 7
1 2 3 4
Month 8
1 2 3 4
Month 9
1 2 3 4
Month 10
1 2 3 4
Month 11
1 2 3 4
Month 12
1 2 3 4
DEVOPs / Rapid AcquisitionNotional T&E Strategy
• T&E strategy ideally integrated into a dedicated test and training site • Must represent OT-like environment
• Notional schedule aligns with proposed development/fielding needs of end user (e.g., 1 to 6 months T&E window)
• Small continuous events to provide feedback and a tailored evaluation product • Requires a dedicated and independent Test Team; in-stride T&E
• Cybersecurity is embedded throughout the process• Ideally still requires exercises at least annually for OT-like assessment(s)
Capability Drop
Cyber Tabletop Example of a Capability Drop
Learn
Tech Evolution Intel Driven
Recursive process to support simultaneous
capability drops of multiple RDPs
Capability Development, Test and Evaluation, Deployment
Test Concept Brief Emerging Results Brief
DT: Developmental Test OT: Operational Test
Challenges• Cybersecurity as Systems Engineering Discipline • Contractual Language for systems with IT
• Build Cybersecurity into design• Accountability for Cybersecurity findings discovered in testing• Data accessibility
• Reducing introduction of vulnerabilities of integrated systems• Understanding Operational Requirements and Impacts
• “If there is a computer in something, it can be cyber-attacked, and we need to be able to harden it and defend it.” the Pentagon’s Deputy Chief Information Officer for Cybersecurity Mr. Richard Hale
• “The Joint Staff has recently put out a formal requirement document that includes cybersecurity as a key part of the survivability key performance parameter (KPP) for every new system”
• Metrics Defensible Systems
9
Back-Up Charts
10
A B CIATT OTRR
OT Cooperative Vulnerability
and Penetration Assessment
OT Adversarial Assessment
FRP
Phase 5CVPA
Phase 6AA
Proposed Cybersecurity T&E Events
DT Adversarial Assessment
Phase 4Phase 3
DT Cooperative Vulnerability
& Penetration Assessment (CVPA)
Events derived from draft DASD(DT&E) DoD Cybersecurity Test and Evaluation Guidebook, and DOT&E Cybersecurity Operational Test and Evaluation Guidance Memo (01 August 2014)
Phase 2Phase 1Understand
Cybersecurity Requirements
Characterize Cyber Attack
Surface
Developmental Test Integrated DT/OT Operational Test
Analysis phase
Test phase
RMF
Cooperative Vulnerability Identification
Adversarial Cybersecurity
DT&E
11
Cybersecurity T&E Process (4)“Shift Left”
Yesterday’s Intermediate Threat is Today’s Novice Threat
Compliance with IA controls / standards and profiles are
necessary but not sufficient
Fielded systems found to have novice vulnerabilities during OT, which is problematic and costly.
Threat
Program Start
Program OT
12