test and evaluation/science and technology (t&e/s&t) program · distribution statement a....

Test and Evaluation/Science and Technology Program

Distribution Statement A. Approved for public release: distribution unlimited.

Command, Control, Communications, Computers, and Intelligence (C4I) & Software Intensive Systems

Test (C4T) Test Technology Area

Using Artificial General Intelligence in the Operational Evaluation F-35 System of Systems

December 2018

Mr. Kent Pickett (C4T TTA)Dr. Edward Chow (JPL)

2Distribution Statement A. Approved for public release: distribution unlimited.

C4I & Software Intensive Systems Test (C4T)Mission

Develops technologies to test C4I and Software Intensive Systems thatoperate in complex military environments. With emphasis on automatedtesting, analysis (real-time and post-test) and evaluating the increasingmass of structured and unstructured data. C4T is divided into thefollowing three domains:

1. Distributed Testing: This domain will address technologies to: reduce T&E infrastructurebiases, improve T&E cross domain/multi-level security abilities, advance T&E for platformsemploying big data/cloud environments, and create agile/contested/dense communicationenvironments.

2. Test Automation: This domain will address technologies to: advance Test Big DataCollection, Analysis, Reporting, & Visualization; create high fidelity representations ofoperational systems or net-centric environments; and assess next generation of warfightermanaged information objects.

3. Modeling & Simulation: This domain will address technologies to: determine required fidelityin Live and Simulated environments; improve the Validation & Verification and Aggregationtechniques; improve run-time performance for real-time applications, and systems,communications, and environmental representations.


C4T Overview

Innovate T&E: Joint, Early, Often & Agile

Complex Warfare Environments

Battle increasingly sophisticated adversaries in increasingly complex environments*

Innovative approaches to how we fight, posture our force, & leverage our asymmetric strengths & technological advantages*

Distributed Testing (DT)

Modeling & Simulation (M&S)

Test Automation (TA)

• T&E Big Data Rapid Analysis• Automated Testing Utilizing

Virtualization and Cloud Environments

• NextGen Handhelds and Widgets

• Automated Control of Targets

BIG DATA & CLOUD COMPUTING

• Determine Simulation Fidelity• V&V Across Battlespace

Environments• Battlespace Environments

Aggregation• Improve Simulation Run-time

Performance• Representation of Systems,

Communications, and Environments

C4I & Software Intensive Systems Test (C4T)Overview Technology Domains

• Remove Test Infrastructure Biases• Cross Domain Solutions and Multi-Level Security• Assess Big Data Warfighter Systems• Testing Warfighter Systems Employing Agile Comms• Emulate Contested/Dense Communications Environments


Topics For Discussion

• Overview of the Assistant for Understanding Data through Reasoning, Extraction, & sYnthesis (AUDREY) Project

• Solving System of System (SoS) Big Data Problems with Engineering Notes and Non Axiomatic Logic (NAL)

• F-35 Electronic Signature Monitoring (ESM) T&E Use Case• Project Specifications• Summary


Technology Overview

• T&E Need– JSF represents a complex System of Systems (SoS) Need means of identifying and describing complex component interactions

– Joint Strike Fighter (JSF) produces terra bytes of data per day Testers need intelligent way of managing data and isolating problems in this SOS context

– JSF requires developmental test (DT) quality testing in operational test (OT) environment Must conduct DT quality test without extensive support infrastructure and resources

• S&T Challenge– Develop framework for intelligent knowledge management Artificial General Intelligence (AGI)-based knowledge representation Intelligent information retrieval capabilities

– Develop automated insight extraction & workflow synthesis capabilities User-extensible plug-in library Intelligent workflow assembly and dispatch; insight production

– Enable real-time test understanding via automated in-situ analytics Automatic workflow synthesis and analytics dispatch Real-time data catalog and model revision


Project Description

Queries, Rules, Directions

Display, Advice, Alert, and Knowledge

F-35

* DART: Data Acquisition, Recording and Telemetry

Front-End Data/Image Processing and Machine

Learning

DART*

StructuredData

UnstructuredData

RAID

Intelligent Insight Extraction, Workflow

Synthesis, and Knowledge Management

• Real-time Automated Insight Engine for Data to Decision (RAID) will develop an Intelligent, automated assistant for data to decision for F-35

- Learn T&E know-how, experiences, and relationships from testers and analysts- Assist human in processing large amount of test data in complex situations- Use data to empirically validate and improve learned knowledge with human assistance- Use human-like reasoning to identify insights from structured and unstructured data- Enable distributed testers to use shared knowledge to identify critical test information

RAID uses learned knowledge to assist testers to turn data into decisions


T&E Benefits

• Complex System of Systems (SoS) Testing– RAID enhances next generation complex SoS testing with stored subsystem

testing knowledge and revolutionary insight discovery capabilities• Big Data and Large Amount of New Information to Learn

– RAID assists testers in identifying what data and information is most important to focus on

• T&E Life-Cycle Knowledge Management– RAID enables distributed human-level T&E knowledge capture,

understanding, sharing, and reuse throughout the life-cycle of the system.• DT Support during OT

– RAID provides DT quality testing during OT without the expensive DT support personnel and infrastructure

• Collaborations and Innovation– RAID intelligent knowledge sharing encourages collaborations and fosters

innovation between testers in different ranges


S&T BackgroundState-of-the-Art Technologies and Challenges

• Traditional Artificial Intelligence (AI) difficulty with uncertainty– Rule-based AI approaches fail in real-world

problems with missing and contradictory data • Deep Neural Network (DNN) is better with

image, speech, and text but– Requires large training set which is costly in

T&E environment and complete retraining on even slight changes to problem domain

– Does not understand big picture meaning like human

– Problem with real-time testing due to separate training and reasoning phases in DNN

• Machine Learning techniques require experts to do feature engineering– Need expensive Ph.D. level data scientists for

T&E


AUDREY (Assistant for Understanding Data through Reasoning, Extraction, & sYnthesis)

• JPL has created a system called AUDREY, based on Artificial General Intelligence (AGI)technologies, that can solve problems that were previously considered unsolvable by machines.

• AUDREY uses Non Axiomatic Logic (NAL) to achieve neural processing at symbolic level– Bio-inspired Neural Symbolic Processing for higher level cognitive reasoning

• AUDREY can accomplish:– Reasoning and learning new knowledge at the same time – Dealing with missing or contradictory data – Learning from human engineering T&E expertise

Achieves unprecedented levels of reasoning for previously

unsolvable problems

The Evolution of AI


Solving SoS Big Data Problems with Engineering Notes and Non Axiomatic

Logic


Capturing and Applying Human Engineering Expertise

• Artificial Intelligence (AI) has been used for years to help solve difficult problems but it is currently reaching its limits

– A serious problem with traditional AI approaches is that a different solution (rule sets) is needed for even slightly different versions of a problem. These systems are always playing catch-up with evolving problems and are very brittle.

• A new subfield of AI has emerged, called Artificial General Intelligence (AGI), that offers exciting solutions to problems which traditional AI is unable to solve

– Solves the problems traditional AI systems cannot by learning and reasoning at once and overcoming the limitations of one-off solutions to specific versions of a problem

– Works in the presence of missing information and is able to intelligently reason about the unexpected situations.

• AUDREY captures rules based on human experience and applies them in a problem contextual reasoning environment.

– If/Then rule bases are developed from Engineering Notes,” If the flight performance indicates stall, then check flap angles and speed”. These notes are called Tidbits.

– Tidbits are parsed into AUDREY rules but these rules are also given a context profile, indicating under what conditions/problems they apply.


Tidbit Parser Overview

RAID Tidbit ParserWebserver

User Client Desktop

HTTP

Client portal accessible via local browsers

XML

Execute H

unterCortex Graph

Audrey’s Cortex Architecture

Titan Graph

Send

to

Cor

tex

Tidbit Text File


F-35 Electronic Signature Monitoring (ESM) T&E

Use Case


• ESM Testing Environment– System Under Test (SUT) should see/describe controlled RF

emitters (potential Air Defense Radars)– Environment may contain uncontrolled emitters (cell

towers, radio stations, etc)• Available Data

– Time Space Position Information (TSPI) for emitters and SUT

– ESM perception measurements Parametric (frequency, priority, pulse width, etc.) Direction of arrival ID info (emitter type, mode, etc.) Geolocation info (lat, long, height-above-ellipsoid)

• Goal is to determine if SUT perception “matches” truth so that various Measures of Performance (MOPs) can be calculated

ESM T&E


Frequency

PW

PRI

(𝑓𝑓1, 𝑝𝑝1, 𝑤𝑤1)

(𝑓𝑓2, 𝑝𝑝2, 𝑤𝑤2)

• Data analysis scenarios:– Easy: emitter has relatively unique parameters/signature and

ESM points right at it– Medium: two emitters have similar signatures, ESM missing

discriminating parameters, how to break tie?– Hard: controlled emitter has same signature as uncontrolled

emitter; Is ESM getting right one?• Testers use undocumented “human knowledge” or

rules to handle this:– Some rules always work; others only hold in certain situations– Some rules may introduce additional complications requiring “real

thinking” to overcome

ESM T&E Matching Problem


First Resultswith Engineer Supplied Tidbits and RDFB

DIVA Matching

RAID Matching

Tidbit used Matching Expectation

Audrey Rule

RAID F35 automatically scores in same manner as Engineering Expertise would produce in DIVA


Summary

• One of the technologies, AUDREY uses a new form of AI called Artificial General Intelligence (AGI)

• Using AGI AUDREY can accomplish:– Reasoning and learning new knowledge at the same time – Dealing with missing or contradictory data – Learning from human engineering T&E expertise– Focusing T&E analysts on SoS problems areas

• AUDREY is currently being used to automatically analyze F-35 Performance Data and F-16 TSPI Data

test and evaluation/science and technology (t&e/s&t) program · distribution statement a....

Documents