Collaborative Time Sensitive Targeting

aka

Technologies for Team Supervision

March 31, 2008

Missy CummingsHumans & Automation Lab

MIT Aeronautics and Astronauticshttp://halab.mit.edu

Review of Original ProposalReview of Original Proposal

Project Goal Develop technological support for futuristic unmanned aerial vehicle (UAV) teams

engaged in TST operations (but really any time critical resource allocation task)

Technology to Support the Team Supervisor:

Activity Awareness Displays

Interruption Recovery Tools

Technology to Support UAV Operators:

Activity Awareness Displays

Objective Performance Measures

Teamwork in UAV OperationsTeamwork in UAV Operations

Intelligence Consumers (e.g., Ground Troops)

UAV Operators

Operations CenterGround Crew

Vehicle(s)

Generalizable to Other DomainsGeneralizable to Other Domains

3 “Confederates”as UAV Operators

Team Interface Issues in Time-Critical Team Interface Issues in Time-Critical OperationsOperations

• Increasing reliance on collaboration tools- e.g., email, instant messaging

(chat)- Increased communication

overhead

• Need interfaces that intelligently share activity information to improve planning & coordination in networked teams

- Activity awareness

• Research for technology aids to assist supervisors of teams virtually non-existent

• Measurement of technology interventions very difficult- Objective team performance

metrics are elusive

Boeing Team Test FacilityBoeing Team Test Facility

3 “Confederates”as UAV Operators

Simulated Task EnvironmentSimulated Task Environment

Situation Map Display Mission Status Display Remote Assistance Display

Surveillance support for a ground convoy through a hostile region

• UAV team consists of:

- 1 Mission Commander

- 3 UAV operators, each controlling multiple UAVs

• UAVs have camera sensors only

- Team must coordinate with external strike team to destroy identified threats

Team Supervision Decision AidingTeam Supervision Decision Aiding

UAV Operations Team Interruption recovery

tools for the team supervisor

Interruption Recovery Assistance (IRA) toolInterruption Recovery Assistance (IRA) tool

• Interactive event timeline, containing event “bookmarks”

• Increased recovery time but positive impact on decision accuracy, especially in complex task situations.

• IRA tool tended to provide greater benefits to participants without military experience

• Some experimental/interface issues so additional study needed

• Summer 2008 w/ Waterloo

Current Work: Predictive Tool for Team Current Work: Predictive Tool for Team SupervisionSupervision

UAV Team SupervisorTechnologies for

assessing:

- How well is the team doing?

- When & who should received attention?

Team

• The need for more objective team performance metrics

- Our focus is supervisory control in team settings with embedded layers of automation

• Predicting team states versus actual performance

- Artificial intelligence, behavioral pattern detection and performance correlations

• Alert team supervising agent of sub-optimal team behavior/cognitive strategies

- Relying on human knowledge-based reasoning to determine if an anomalous team state exists

• Two stage approach

- Predict an individual operator’s state transition as a proof of concept

- Adapt model to UAV team setting

Current Work: Team Performance PredictionCurrent Work: Team Performance Prediction

• Evaluation and predictions of team behaviors:- Automatic, continuous, and in real-time

• Only relies on easily observable data- User interaction with machine

- User communication

- Eye tracking?

• Use Bayesian pattern prediction methods applied to individual/team behaviors- Hidden Markov Model

• Not predicting performance per se- Anomalous conditions

- Value of human judgment

Team Prediction Model CharacteristicsTeam Prediction Model Characteristics

• Probabilistic state transitions

• Observables vs. Hidden States- Hidden states are not directly visible, but variables influenced by them are.

- Cognitive states vs. behaviors

• HMM uses the observables to infer:- Most likely hidden state sequence- Based on future sequences, the most likely observables- Machine learns the clustering of behaviors (need lots of data!)

Hidden Markov ModelsHidden Markov Models

Modify Flight Parameters

CommsCamera Control

Waypoint Update

H&S MonitoringTarget Monitoring

Mission Replan

Predicting Individual Behavior Predicting Individual Behavior

Human State Estimation - TRACS Human State Estimation - TRACS

Building the Model Building the Model

Group ofCriteria

IndividualCriterion

Group ofMatches

IndividualMatch

DataCluster

DataItem

Evaluate

BacktrackBrowse

SelectFilte

r

Automatch

Create

1 2 3 4 5 6

8 9 10 11 12 13

15 16 17 18 19 20

22 23 24 25 26 27

28 29 30 31 32 33 34

35 36 37 38 39 40 41

0

7

14

21

GrammarPattern

Recognizer/Predictor

Low Level Input

Future TeamBehavior

ILDs

4 State HMM for Mission Planning Task 4 State HMM for Mission Planning Task

• Unsupervised vs. supervised model training for state recognition

Group ofCriteria

IndividualCriterion

Group ofMatches

IndividualMatch

DataCluster

DataItem

Evaluate

BacktrackBrowse

SelectFilte

r

Automatch

Create

1 2 3 4 5 6

8 9 10 11 12 13

15 16 17 18 19 20

22 23 24 25 26 27

28 29 30 31 32 33 34

35 36 37 38 39 40 41

0

7

14

21

Leveraging Automation

ManualPlanning

Exploration

Predicting an Operator’s Strategies Predicting an Operator’s Strategies

Live Demo! Live Demo!

Validating our HMM Approach Validating our HMM Approach

• Currently only behaviors that entail mouse clicks are observed

- A limit, particularly for monitoring interfaces

• Eye tracking arguably gives us additional data about the information operators access

- Noisy

- Cost-benefit analysis?

• Study will be conducted this summer comparing HMM results with and without eye tracking input

Extending our Approach to UV Teams Extending our Approach to UV Teams

Visualization task

Evaluation/

Choice of UV + engage

Choice of UV

Sector A

Sector B

Sector C

a) Replan Path

b) Replan Target

Extending to task to teams: UV hand-off between Sectors

The Plan The Plan

• Develop HMM using experimental data from individual experiments

- Investigate supervised vs. unsupervised learning

• Develop a team version of the interface

• Update HMM with team experiments to be conducted this summer

Additional Future Work: Designing for Team Additional Future Work: Designing for Team Activity Awareness Activity Awareness

UAV Operations Team

Preliminary UAV Operator Display DesignsPreliminary UAV Operator Display Designs

Operator 1

UAV 4

Reroute/ReassignmentTarget ID Task

3UAV1

UAV 3 UAV 5

Health Status

Fuel level: 40%

Connected

- Searching - Searching Health Status

Fuel level: 70%

Health Status

Fuel level: 70%

Connected Connected 0:00:300:00:050:03:59 UAV3

5

Help

Request sent to:

UAV3

4 - Searching

Assistance Requests Overview

Req 3 (Complete)

Request from: Op 1 Elapsed time: 0m35sDetected by: UAV 3 Target type: 5 (Com Eq)

Request from: Op 3 Elapsed time: 0m30sDetected by: UAV 8 Target type: 5 (Com Eq)

Req 5 Request from: Op 1 Elapsed time: 0m05sDetected by: UAV 2 Target type: 3 (IED-V)

Req 4

ATR Information:

Possible Match: Target Type 5: Comm Equipment

Match Confidence: Medium

Found by UAV2 – Location: 43:00E/24:00N

ATR Decision:

SubmitClassification

Confirm ATR Suggested Classification:

Target Type 5 with Medium Confidence

Provide Alternate Target Classification:

Select Target Type…

Select Confidence Level…

Select Target Priority…

Veto Classification:

Image does not contain known target

Operator 3

Cancel

All

Others

Op 1

mCDR

Op 2

Classification:

Target Type 5 High Confidence

(no classification available)

Elapsed time:

0:00:20

Image Received at:

12:56:40

Request new image

Threat distance from the convoy: 15 miles

Elapsed Time:

Current Time:

1: 25: 30 PM

0: 22: 45

Mission Info:

0: 03: 28

UAV Reassignment

Elapsed Time:

Remaining Time:

0: 02: 37

Routes:

View Filter:

Hazards

Connection Blackouts

Targets Range

All

UAV 4

UAV 5

UAV 6

Sendprivate

All

Op 1

Op 3

mCDR

Strike team

others

Type here…

Op 2 – communication box

ALERTS & Clarification

SendType here…

Share Files

mCDR OthersStrike T Op 1 Op 3

Comm Status

0:03:57

On line

Handoff Available[0:00:30] Conference call

[0:01:20] mCDR to OP1:

[0:01:22] Op1 to Op3:

[0:01:25] Op3 to strike team:

[0:01:29] Op3 to Op2:

4 - Searching

Repair

Repair

RepairRepair

Health Status

Health Status

Health Status

Fuel level: 40%

Fuel level: 70%

Fuel level: 70%

- Target ID5

6

Connected

Connected

DisconnectedSafety route in 0:29:59

Reconnecting in 0:00:49

0:00:18

0:00:04

0:03:57

(0:01:50) UAV 7 (area3) has been shot down

(0:03:00) UAV 4 Fuel Level at 40%

(0:03:40) UAV 6 enter in a difficult area to connect

mCDR

Strike team

OP1

Others

OP3

0:03:57

0:03:57

0:01:53 - Changing shift

0:00:33 - Target task

Comm

Comm

Comm

Comm

0:03:57

Repair

Repair

Repair

Repair

Repair

Comm

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

Very GoodSpeed: X Mb/sMIT Network

MapDisplay

CommunicationsDisplay

Tasking Display:TargetID & rerouting, reassigning UAVs

DeliverablesDeliverables

• 6 Conference papers- ASNE, HSIS, ICCRTS, HFES

• 4+ Technical reports• 3 Undergraduate theses

- 2 pending• 1 Masters thesis• 1 PhD dissertation (pending)• 3 Journal articles in various stages• 1 workshop (CSCW 2006)• IP disclosure pending HMM validation• The Holy Grail of team research???

• Near Term

- Eye tracking study

- HMM-Multi UV experiment

- Extension for supervisor IRA tool

• Mid Term

- Activity awareness displays for operators

- Decision support display for HMM

- Multi-UV connection to predictive model system architecture research (ONR)

Development of CE metric

• Project slated to end December 2008- Requested no-cost extension to JUN 09

- Additional funding

Future PlansFuture Plans

Backup Backup

Simulated Task Environment Display DetailSimulated Task Environment Display Detail

Map Display

Mission Status Display

Simulated Task Software ArchitectureSimulated Task Software Architecture

Large-Screen Wall

Displays

Situation Map Display Mission Status Display

TabletPC Display

Mission Commander Display

Simulation & Collaboration Server

(Grouplab SharedDictionary)

Situation Map Display

Mission Status Display

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

4

(nominal)

(down)

8

1

(reviewing ATR imagery)

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

Potential threat

envelopeTarget strike indicators

Known threat

envelope

Situation Map Display

Mission Status Display

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

Situation Map Display

Mission Status Display

Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness

• UAV status & tasking

• Current & expected convoy safety

• Current & expected operator task performance, relative to convoy safety

Developing Methodology for Deriving Developing Methodology for Deriving Collaborative System Requirements Collaborative System Requirements

Project Goal:

• To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking