Fusion and AutomationFusion and Automation Human Cognitive

and Visualization Issues

Jean-Remi DuquetMarc GregoireMaura Lohrenz

Kesh KesavadosElisa Shahbazian

Tore SmestadAmy VanderbiltMargaret Varga

What is Data (& Information) Fusion?What is Data (& Information) Fusion?

The most recent definitions of the JDL DF levels are listed below:

Level 0: Sub-Object Data Association and Estimation (pixel/signal level data association and characterization)

Level 1: Object Refinement (object continuous state (e.g., kinematics) estimation, discrete state (e.g., object attribute type and identity) estimation)

Level 2: Situation Refinement (object clustering and aggregation, relational analysis, communications and contextual estimation from multiple perspective)

Level 3: Significance Estimation (situation implication, event prediction, consequence prediction, opportunities and vulnerability assessment)

Level 4: Process Refinement (adaptive processing through performance evaluation and decision/resource/mission management).

From Steinberg, Bowman and White (1999) Revisions to the JDL DF Model, SPIE vol. 3719, pp. 430-441.

AssumptionsAssumptions

Assume there is a DF system that satisfies our requirements

Rapid prototyping of this DF system could be achieved with the following:  1. A NATO agreed upon set of modular ontologies and an

environment for allies to augment these

2. NATO guidelines to assess the requirement for DF capabilities and DF automation

3. NATO guidelines for selecting DF architecture and interfaces, given the system requirements

4. A mechanism to propose and develop a rule set to guide the selection of choices in the DF design

5. A NATO repository of data/information fusion capabilities and algorithms

6. A tool that guides the developer to put together the DF system based on the input information (wizard)

Visualization ChallengesVisualization Challenges

1. Conveying the measure of uncertainty to the Operator, with minimal clutter

2. Level of Automationa. How to determine when to apply a DF decision automatically

and when to provide the capability as a decision support tool for the Operator

b. Context-based automation

c. User-selected automation

d. Level of awareness management by exception

e. Self-monitoring of DF

3. How should the existence of multiple hypotheses be conveyed to the Operator?

Challenges (continued)Challenges (continued)

3. How should the pedigree information be conveyed and what decision support is necessary for the interpretation of the pedigree? Visualization of the credibility of the source

4. Level of Interaction How should the Operator participate/aid/veto in the DF

decisions?

5. How to visualise and address conflict between DF decision sensor and/or Operator decision?

6. Measuring system effectiveness Maximise the information contents

Design Methodology for the Design Methodology for the Visualization SystemVisualization System

Proposed design based on design processed developed by Buffalo Ontology workshop held in March, 2002 …

World

Information Requirements to support Decision

Making

Ontology

Abstract Relationships

• State Change• Comparison

• Similarity• Absolute Magnitude

Display Instantiations

Known Methods for Knowledge

Elicitation and Requirements

Gathering

Mutually Informing

Problem Context• Physical Environment• Experience• Culture• Hardware • Data Availability• Integration with other

required forms

Constrains display concept

selection

EmpiricalEvaluation

Suite of Representational

Elements

Many to Many

Mapping

From Kesavadas and Llinas, et al. (2002) Ontology Workshop, SUNY Buffalo

World

Sen

sors

FusionProcesses

Display

Person

World

Information Requirements to support Decision

Making

Ontology

Abstract Relationships

•State Change•Comparison

•Similarity•Absolute Magnitude

Display Instantiation

Problem Context•Physical Environment

•Experience•Culture

•Hardware •Data Availability

EmpiricalEvaluation

Suite of Representational

Concepts

World

Information Requirements to support Decision

Making

Ontology

Abstract Relationships

•State Change•Comparison

•Similarity•Absolute Magnitude

Display Instantiation

Display Instantiation

Problem Context•Physical Environment

•Experience•Culture

•Hardware •Data Availability

EmpiricalEvaluation

Suite of Representational

Concepts

From Kesavadas and Llinas, et al. (2002) Ontology Workshop, SUNY Buffalo

RecommendationsRecommendations There are many visualization tools to represent risk,

uncertainty, and rate of change, but no good guidelines for how best to implement these tools optimally (100%) or sub-optimally (70%)!

There is a need for an ontology that accounts for specific fusion and automation needs (e.g., uncertainty).

Need both subjective and objective evaluation criteria. Would help to have a standard (NATO?) testbed for

prototyping purposes. Recommend a NATO archive of available tools, guidelines,

etc. to help rapid prototyping, analysis, and building data fusion systems.

Need an integrated process model that flows smoothly through all stages of data fusion / visualization design.