1 rapid: representation and analysis of probabilistic intelligence data carnegie mellon university...

RAPID:Representation and Analysis ofProbabilistic Intelligence Data

Carnegie Mellon UniversityPI : Prof. Jaime G. Carbonell / jgc@cs.cmu.edu / (412) 268-7279

Dr. Eugene Fink / e.fink@cs.cmu.edu / (412) 268-6593Dr. Anatole Gershman / anatoleg@cs.cmu.edu / (412) 268-8259

DYNAMiX TechnologiesPOC: Dr. Ganesh Mani / gmani@dynamixtechnologies.com / (412) 401-0121

Mr. Dwight Dietrich / ddietrich@dynamixtechnologies.com / (724) 940-4304

PAINT PAINT

Carnegie Mellon

FacultyJaime G. CarbonellEugene FinkAnatole Gershman

StudentsBin FuDiwakar PunjaniAndrew Yeager

People

DYNAMiX

PrincipalsDwight DietrichGanesh Mani

EngineersAtul BhandariJeremy HermannVeera Manda

Outline of the presentation

• RAPID functionality

• Preliminary demo

• Architecture and main components

• Integration with REALISM

• Current results and work plan

Analysis of uncertain intelligenceRAPID is a probabilistic reasoning engine for the analysis of dynamically evolving intelligence data.

Intelligence results

RAPID will help:• Identify important holes• Locate most crucial

missing pieces• Insert these pieces

Initial knowledge

Jigsaw analogy:

Knowledge sources:• Public domain• Intelligence• Inferences

Analysis of uncertain intelligenceRAPID will help intelligence analysts to accomplish the following tasks.• Draw probabilistic conclusions from available

intelligence, including uncertain and missing data• Identify potentially surprising developments• Formulate and assess hypotheses• Identify critical uncertainties• Develop strategies for proactive collection of

additional intelligence to resolve uncertainties, based on the analysis of cost / benefit trade-offs

Filtering and processing of

new intelligence

Propagation of inferences

Analysisof key

indicators

Development of intelligence-

collection plans

Massive newintelligence

Intelligencecollection

Analysts

Underlying functionality• Representation of uncertainty:

Novel representation of massive uncertain data,which supports fast matching and inferences

• Inferences from uncertain data:Scalable inference mechanism for reasoningabout uncertain intelligence

• Analysis of critical uncertainties:Assessment of uncertain situations, evaluation of datautility, and identification of important missing data

• Proactive intelligence planning:Evaluation of available probes and constructionof optimized intelligence-collection plans

Preliminary demo

Uncertainty analysisand probe evaluation,integrated into Excel.

Architecture

Advanced analysis of incomplete data,identification of critical uncertainties,evaluation and selection of probes,what-if analysis, and visualization. Excel extension for the analysis

of uncertainty, probes, and proactive data collection

Uncertainty calculus andproactive probe planning

A large-scale database of incomplete anduncertain facts, uncertain inference rules,and hypotheses, which allows scalableplanning of proactive data collection.

Scalable assessment ofuncertain intelligence

Relational database of uncertaindata and inference rules

Uncertain situation assessmentand data-collection planning

An advanced API for integration withother systems.

Optional user interface for the integratedaccess to all system components, whichextends the standard Excel interface.

Analystinterface

Architecture

Proactiveintelligencecollection

Generalintelligencecollection

Processing ofdata streams

Real-time matching of queriesand inference rules against amassive stream of new data

Approved plans forproactive data collection

Fast database operations on astream of newly incoming data,and integration of this streamwith the static database.

Excel extension for the analysis of uncertainty, probes, and

proactive data collection

Analystinterface

Hypotheses,conclusions, and

data-collection plans

Architecture

Analystinterface

Real-time matching of queriesand inference rules against amassive stream of new data

Value-addedreasoning tools

Approved plans forproactive data collection

Value-addedreasoning tools

Uncertainty database

Microsoft Excel

• Representation of probabilitydistributions and qualitativeuncertainty

• Uncertainty arithmetic

Uncertainty analysis

• Representation of data utility• Tracking utility changes

during data collection• Identification of critical

uncertainties

Situation assessment

• Representation of probes• Evaluation of probe utility• Automated selection and

launching of critical probes

Proactive probe planning

What-if analysis of alternativefuture developments and data-collection plans based on anextension of Excel “scenarios”

Contingency planning

Analystinterface

Scalable assessmentof uncertain intelligence

Uncertainfacts

Goals, queries, andhypotheses

Prioritized plans for proactive

data collection

Uncertaininference

Semanticnetwork

Criticaluncertainties

Querymatches

Evaluation ofhypotheses

Inferredfacts

Learnedinference

Conflictdetection

Manual entry, selection, and editing of knowledge

Analystinterface

Value-added reasoning tools

Part of uncertainty database

Knownpatterns

Identification of patternsand their gradual changes

in massive data streams

ARGUS data explorer

Contingency analysisWhat-if analysis of alternative

hypotheses, data-collection plans,and possible future developments

Alternative scenariosand their implications

Markov reasoningSelection of most likelyhypotheses and possible

future developments

Markovmodels

Adversarial searchAnalysis of possible

concealment and disinformation,and plans to prevent them

Adversarial goalsand resources

Identification of syntacticallydifferent words that refer

to the same objects

Entity co-reference

These tools are not essentialfor the core functionality.

The available intelligence data and inference rules are in Excel tables, and in the uncertainty database integrated with Excel.

Analyst interface• Optional extension of the Excel interface

• Visualization and explanation of intelligence data, inferences, and data-collection plans

Integration goalsWe will integrate the text-extraction system developed by HNC / Fair Isaac with the uncertainty-analysis system developed by CMU / DYNAMiX. The integrated system will support the following capabilities.• Extraction of facts, relations, and causal

links from natural-language documents• Evaluation of given hypotheses• Proactive information gathering• Application to the analysis of Iranian

nano-technology plans and capabilities

Inputs and outputs

Output:• Large structured tables of

relevant facts and entities, which include uncertainty

• Inference-rule representation of relations and causal links, also including uncertainty

Input:• Requirements and filters for

the information extraction• Natural-language documents• World-wide web

Output:• Inferences from uncertain data• Exact and approximate

matches for given queries• Hypothesis assessment• Proactive plans for collecting

additional data

Input:• Tables of uncertain facts• Uncertain inference rules• Queries for specific data• Analyst hypotheses

REALISM RAPID

Architecture

Informationrequests

REALISMHNC / Fair Isaac

Structured relations andcausal links

Structuredfacts andentities

Topicfilters

RAPIDCMU / DYNAMiX

Analystinterface

Initial results• Detailed technical plan of uncertain situation

assessment and proactive probe planning:architecture, functionality, and algorithms

• Uncertain intelligence scenario based onpublic data about Iranian nano-technology

• Preliminary prototype of situation assessment tools integrated with a relational database

• Preliminary prototype of a tool for the resolution of entity co-references

• Application of DYNAMiX Data Explorer to the nano-tech conference data provided by PAINT

Current work

• Uncertainty calculus,integrated with Excel

• Proactive probe planning

• Scalable uncertainty assessment,integrated with a relational database

• Initial analyst interface

Prototype of uncertainty calculus March Prototype of probe-planning tools MarchInitial RAPID / REALISM integration MayInitial analyst interface (extended Excel) JunePrototype of uncertainty database July

Short-term plan

Uncertain situation assessmentand proactive probe planning

July 2008

Discrimination among competing hypothesesand identification of critical uncertainties

July 2009

Fully integrated deployable prototype July 2009Advanced proactive-intelligence planningand learning of inference rules

July 2010

Value-added tools, which may include data-stream processing, entity co-reference, adversarial search, and Markov reasoning

July 2011

Fully integrated deliverable system Jan 2012

All versions of RAPID will demonstrate all main capabilities, with increasing functionality over time.

Long-term plan

EvaluationWe expect that RAPID will provide significant advantage over available off-the-shelf tools, such as standard spreadsheets and database systems.

To support this claim, we plan to compare the productivity of analysts using RAPID with that of analysts who perform the same tasks using commercially available tools.

Experimental group:Use of RAPID

Control group:Use of standard tools

EvaluationWe expect that RAPID will provide significant advantage over available off-the-shelf tools, such as standard spreadsheets and database systems.

To support this claim, we plan to compare the productivity of analysts using RAPID with that of analysts who perform the same tasks using commercially available tools.We will view RAPID as success if it consistently outperforms the standard tools, and the analysts report the overall positive experience of using it.

Adjustment of the earlier planWe need to adjust the plan to the new budget.We will deliver the full core functionality, but we propose to reduce the work on value-added tools.

Reduced work• Processing of data streams• Advanced contingency analysis • Analyst interface

Suspended work• Predictive Markov models• Analysis of adversarial actions

Appendices

• Previous work

• Empirical evaluation

• PAINT contributions

ARGUSARGUS project sponsored by DTO/ARDA: Identification and tracking of novel patterns in massive databases and data streams.

CreateBackground

DetectNovelEvents

GenerateProfiles

Re-cluster

UpdateProfiles

HistoricalData

BackgroundModel

NovelEvents

NovelClusters

TrackedEvents

New ProfilesProfiles

Alerts

Analysts

CreateBackground

DetectNovelEvents

GenerateProfiles

Re-cluster

HistoricalData

BackgroundModel

NovelEvents

NovelClusters

New ProfilesProfiles

Alerts

Analysts

ARGUS• Estimate the density function at t0• Grow the cluster for a period of Δt while

reducing the weight of old records

• Estimate the new density function at t0+Δt

• Compare the two estimates

t0 + Δt

Re-clustering

RespiratoryDiseases SARS

Densitychange

RADARRADAR project sponsored by DARPA:Analysis and management of volatile crisis situations based on uncertain data.

Data elicitorParser Optimizer

Processnew data

Update crisis-management

Suggest data-collection strategies

Top-level controland learning

Analysts

We have applied the system to repair a schedule of a conference after a crisis loss of rooms.

Crisis

0.50 Manual

Repair

0.61 Auto w

licitation

0.72 Auto w

licitation

0.93ScheduleQuality

Manual and auto repair

ScheduleQuality

6040 80 100Number of Questions

(Out of 1100)

Dependency of the qualityon the number of questions

Unlike ARGUS…• Represents and analyzes uncertainty• Supports complex inferences

Unlike RADAR…• Scales to massive intelligence datasets• Analyzes complex “external” situations• Develops intelligence-collection plans

Appendices

• Previous work

Evaluation goalsWe expect that RAPID will provide significant advantage over available off-the-shelf tools, such as standard spreadsheets and database systems.

To support this claim, we plan to compare the productivity of analysts using RAPID with that of analysts who perform the same tasks using commercially available tools.

Experimental group:Use of RAPID

Control group:Use of standard tools

Experimental setup

We expect to recruit retired intelligence analysts for the system evaluation, and ask them to perform several tasks based on given uncertain data.

• Identify the data most relevant to given tasks

• Evaluate the validity of given hypotheses

• Find relevant hidden patterns

• Identify critical missing data and propose acost-effective plan for collecting this data

Performance measurementsWe will measure the following main factors to evaluate the performance of analysts:

• Number of high-level tasks completedwithin the experiment time frame

• Accuracy of hypothesis evaluation

• Number and relevance of identified patterns

• Effectiveness and costs of data-collection plans

We will also ask analysts to complete a questionnaire on their overall experience.

Expected results

We will view the proposed work as success if• RAPID consistently outperforms the off-the-

shelf tools in all four performance factors,

• the performance difference for each factor is statistically significant, and

• analysts report the overall positive experience of using the system.

RAPID / REALISM evaluation

Component utility:We will also evaluate the utility of REALISM and RAPID by comparing the productivity of subjects under the following three conditions:• Use of the integrated system• Use of REALISM without RAPID• Use of RAPID without REALISM

Component evaluation:We will measure the following performance factors:• Accuracy and completeness of text extraction• Accuracy of hypothesis evaluation• Effectiveness of data-collection plans• Speed of each system component

Appendices

• Previous work

Main contributions

Feedback

Strategy Generation and Exploration

Dynamic Simulation

Models

Response Options

Representation of massive uncertain knowledgeAutomated discovery of causal relationships

Fast probabilistic integration of all evidenceAnalysis of possible future developments

Identification of critical uncertaintiesPlanning of proactive intelligence gathering

Inputs and outputs

Uncertain intelligenceand analyst opinions:

Massive stream ofstructured records

Specifichypotheses

New learnedrules

Data-searchqueries

Querymatches

Evaluation ofhypotheses

Plans for proactiveintelligence collection

Uncertainsituation

assessment

Inferencerules

Domainknowledge

InputsFrom other PAINT components:• Available intelligence data and its certainty• Hypotheses about unknown factors• Available domain knowledge

From analysts:• Intelligence-analysis tasks and priorities• Hypotheses and related opinions• Responses to RAPID-generated probes• Additional domain knowledge

From other sources:• Databases with available intelligence• Public databases with relevant data

Outputs• Inferences from available uncertain data• Evaluation of given hypotheses• New hypotheses and their certainties• Plans for proactive intelligence collection• Learned inference rules

1 rapid: representation and analysis of probabilistic intelligence data carnegie mellon university...

Documents

completely heterogeneous transfer learning with attention...

game theory - cs.cmu.edu

sm@jgc session three

1985 saez carbonell

artificial intelligence 15-381 heuristic search methods...

logistics - cs.cmu.edu

jgc energy services

jgc infectionprevention-5

carnegie mellon school of computer science 1 nsf-relevant...

carbonell, miguel

sm@jgc session one

gametheory - cs.cmu.edu

jaime carbonell (cs.cmu/~jgc) with vamshi ambati and...

01234' - cs.cmu.edu

artificial intelligence 15-381 web spidering & hw1...

correlation clustering - springermach.0000033116.… ·...

sm@jgc session two

brics - cs.cmu.edu

aaa v carbonell

jgc nuclear services