recognizing teamwork activity in observations of embodied agents linus j. luotsinen school of...
TRANSCRIPT
![Page 1: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/1.jpg)
Recognizing Teamwork Activity in Observations of Embodied Agents
Linus J. LuotsinenSchool of Electrical Engineering and Computer Science
University of Central Florida
B. Sc. University of Dalarna, Sweden, 2002M. Sc. University of Central Florida, 2004
Ph.D. Dissertation DefenseNovember 06, 2007
![Page 2: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/2.jpg)
Committee Members
School of Electrical Engineering and Computer Science– Dr. Ladislau Bölöni
– Dr. Avelino Gonzalez
– Dr. Kenneth Stanley
Department of Statistics and Actuarial Science – Dr. Liqiang Ni
![Page 3: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/3.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
Covered in candidacy
![Page 4: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/4.jpg)
Introduction
Recognize teamwork activity in a stream of positional agent traces, and annotate them with the recognized actions
Video demonstration
Team merge (t=0)Convoy (t=100) Goal
(t=200)
![Page 5: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/5.jpg)
Applications
Surveillance– Recognize multi-agent activity in surveillance video feeds
Training– Identify discrepancies and deviations from the actions
performed by an expert team
Smarter agents– Model the opponent team to imitate or countermeasure its
actions
Automated annotation– Automatically index large databases for fast content retrieval
After Action Review Digital Video Recorder
![Page 6: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/6.jpg)
Challenges
Observation noise– Position traces can be distorted by inaccuracies in sensors and
localization algorithms
Alignment problems– Movement performed at different location and orientation
Scaling problems– Movement performed at different physical scale
Temporal scaling– Movement happens slower or faster in time
Terrain distortion– Movement distorted because of adaptation to the terrain
![Page 7: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/7.jpg)
Challenges
Movement variants– Movement in alternative ways that map to the same label
Uncertainty regarding the role of the agents in the team
Role count variants– Movement with different number of agents in the team
Agents changing their roles during the team action
![Page 8: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/8.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering– Software tools we developed– Datasets we acquired, segmented and labeled
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 9: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/9.jpg)
Software Tools we Developed
Teamwork Scenario Editor (TSE)– Interactive knowledge engineering tool
– Video editor interface
– Visualize large datasets and geographical areas
All knowledge engineering work in this thesis was performed using the TSE
Acquire the datasets
Find teamwork activities
Label the activities
![Page 10: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/10.jpg)
Datasets we Acquired, Segmented and Labeled
Real-world warfare exercise Recorded over a three days Data collected from hundreds of soldiers and tanks
equipped with GPS devices, laser range finders and laser range detectors
![Page 11: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/11.jpg)
Datasets we Acquired, Segmented and Labeled
Mini-MOUTOTBSAF
Military Operations in Urban Terrain (MOUT)
![Page 12: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/12.jpg)
Military Movement
We extract military movement patterns from the warfare databases
In the military domain the movement techniques and formations are selected based on the situational awareness of the team
Traveling– Enemy contact not expected– Fast movement speed
Traveling overwatch– Enemy contact possible– Medium movement speed– Characterized by continuous movement of lead unit and alternating advancement
of rear units
Bounding overwatch– Enemy contact expected– Slow movement speed– Alternating or successive bounds
![Page 13: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/13.jpg)
Military Movement
Formations are used in combination with movement techniques
Formation is selected based on visibility needs, firepower focus and so on
Column– Leader followed by rear units– Fire power in all directions (flanks, front and rear)
Line– Fire power in the front
Wedge– Fire power in the front and in the flanks
Echelon– Enemy contact expected in the front or in the echeloned flanks– Used when one flanks are secured by obstacles
![Page 14: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/14.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 15: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/15.jpg)
Teamwork Activity Recognition using Hidden Markov Models
A spatio-temporal pattern recognition problem– Recognize teamwork from X and Y coordinates over time for
multiple agents
Why Hidden Markov Models?– Mathematically sound
– Temporal by nature
– Successfully applied in the past (e.g. speech recognition)
Ways of encoding teamwork activity in HMMs– Knowledge engineering
– Learn from a set of representative examples
![Page 16: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/16.jpg)
Workflow Overview
ImportExternal database
Teamwork Scenario Editor
Export
Visualization Identification
Teamwork Activity Recognition
Learning
Teamwork Activity Models
Teamwork Database
Representative Examples
Classification
Real-time data stream
Annotated Teamwork Behavior
![Page 17: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/17.jpg)
The Hidden Markov Model
The HMM consists of a number hidden states
Transition probabilities–
Emission probabilities–
– Gaussian PDF
Initial probabilities–
HMM with 3 hidden states
![Page 18: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/18.jpg)
The Hidden Markov Model
Learning algorithms– Baum-Welch
Optimize by maximizing
– Segmental K-means Optimize by maximizing , where is the optimum hidden
sequence given by the Viterbi decoding algorithm
Classification– Determine the probability that the input sequence was generated
by the HMM
– Forward evaluation algorithm
![Page 19: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/19.jpg)
Baseline Input Format for the HMM + Basic Preprocessing The input is a vector of the agent positions
– VT = {v1, v2, v3,…, vt}– vt = {x1, y1, x2, y2, x3, y3, …, xn, yn}
It is very unlikely that the team action will be repeated in the same location and position!
We perform a pre-processing of the input data which allows us to recognize team actions happening at arbitrary locations and orientation:
– Translation Align with team centroid
– Rotation Align with x-axis
This is not sufficient, many other distorting factors can happen: scaling, terrain distortion, different ordering of the agents – we are dealing with these later in this presentation
![Page 20: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/20.jpg)
Experimental Setup
Test problem– Real-world military warfare exercise
Train and test data– Six activities (extracted using TSE)
– Artificial activities added for testing
![Page 21: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/21.jpg)
Results: Recognition Accuracy
Classification accuracy is 82% Matches the performance of the knowledge engineering
approach
As presented at the AAMAS-07 conference
![Page 22: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/22.jpg)
Results: Real-Time Analysis
HMM with 4 states and 6 classes ~9.4ms
Hidden States Mean Time (ms)
1 2.543
2 4.743
3 7.02
4 9.394
5 12.06
![Page 23: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/23.jpg)
Shortcomings: A Lot of Assumptions!
We assumed that the teamwork activities were performed at the same scale
We assumed that there is no interaction with the environment
Observation input is of fixed arrangement, hence, we assumed that recognition is performed on the same team the HMM was trained for
We assumed that all activities can be modeled using the same number of hidden states in the HMMs
Dimensionality and state space for larger teams will quickly grow out of control
We do not recognize the roles of the agents in the teamwork activity– Who is the leader and who is the follower?
The recognizer is not practical!
![Page 24: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/24.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 25: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/25.jpg)
Team-Oriented Feature Extraction
Improve recognition accuracy and robustness by replacing the “vector of positions” input with a collection of team-oriented features
– The features are extracted through pre-processing– Some of the features replace the existing input vector– Some of the features are the input of the role recognizer (and there is some
overlap)– It requires a more complex recognition workflow (which will be shown later,
together with the role recognition module)
Extract semantically rich features from the agent position traces
Discretization process– Intuitive descriptions of teamwork activities– For classifiers using discrete input
Three feature function classes– Agent-oriented features– Environment-oriented features– Team-oriented features
Calculated over a sliding window in time
Features are matched with the ways humans would understand the scene
![Page 26: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/26.jpg)
Agent-Oriented Feature Functions
Focus on individual agents
Enhance performance of teamwork recognition
Used to recognize (likely) roles
![Page 27: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/27.jpg)
Curvature
The rate at which a curve changes direction
Finite difference approximation with central difference
![Page 28: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/28.jpg)
Environment-Oriented Feature Functions
Agent and team interactions with environment
Environmental objects– Physical or virtual
– Static or dynamic
Domain specific examples– Frontline in war
– Offside line in soccer
– Line-of-scrimmage in football
A1
Obsta
cle
Orientation
Path
t=0
t=1
t=2
Angle
LOS
![Page 29: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/29.jpg)
Team-Oriented Feature Functions
Extract features relative to the team
Specifically designed for teamwork activity recognition
![Page 30: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/30.jpg)
CRPV Discretization
Centroid-Relative Position Vector (CRPV)
Positions are calculated relative the centroid position and orientation
Dimensionality is reduced (compared to previous approach)
Translation, rotation and scale invariant
An evolution is the Role-Relative Position Vector (RRPV)– Privileged agent
![Page 31: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/31.jpg)
Cohesion
Measures the bonding together of the team– Derived from the Principal Component Analysis– PCA: Dimensionality is reduced by restricting attention to the
directions along the scatter cloud that are the greatest
Calculate eigenvalues and eigenvectors from the position-based scatter matrix
– In the 2D case there are two eigenvalues and two eigenvectors
Cohesion is the maximum eigenvalue
CohesionDirection is the direction of the eigenvector with maximum eigenvalue
CohesionGradient is the change in cohesion over a sliding window
![Page 32: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/32.jpg)
Cohesion
Position-based scatter matrix
![Page 33: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/33.jpg)
Team-Oriented Feature Functions
Agent-oriented features can be used by replacing the team with a virtual agent following the team centroid
![Page 34: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/34.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 35: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/35.jpg)
Role-Based Teamwork Activity Recognition
The goal is to improve robustness of the teamwork recognizer
What are the role assignments in the team?
The baseline HMM input we presented previously is a fixed arrangement
– {1, 2, 3, 4} not the same as {2, 3, 4, 1}
Brute force solution – Inefficient: One HMM for each permutation
Role recognition module– Represent teamwork activity based on roles, rather than by agents
![Page 36: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/36.jpg)
Extended Workflow Overview
Import
External database
Teamwork Scenario Editor
Export
Visualization Identification
Teamwork Activity Recognition Framework
Learning
Teamwork Activity Models
Teamwork Data Format
Representative Examples
Classification
Real-time data stream
Annotated Teamwork
Activity
Role Models
Recognition
Mapping
Role Recognition
Team-Oriented Feature Extraction
Role-Oriented Feature Extraction
Assignment
![Page 37: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/37.jpg)
Learning Role Models
Role models are represented by decision trees– Trained from observations with ID3 algorithm
– Pruned to minimize effects of overfitting
– Intuitive (white-box)
– Visualizes the exact features which were used in classification Follow leader uses CRPV
Feature functions for role recognition:– Agent-oriented feature functions
– CRPV feature (assuming that bystander agents are filtered out)
![Page 38: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/38.jpg)
Role Recognition
Calculate the role assignment probability Pr( ai , rj )
– The probability that agent ai plays role rj
Extract observation sequences from the movement trace of each agent
Input each observation in the sequence to the decision tree classifier
The output is a sequence of class frequency vectors Role assignment probability: CRPV
f={2, 4, 0, 0}
NorthWest
f={4, 6, 0, 0}
NorthEast
f={0, 0, 17, 0}
SouthEast
South
f={0, 0, 277, 319}
f={0, 0, 25, 0}
SouthWest
Velocity
North
f={66, 113, 0, 0}
Low
Acceleration
High
f={107, 80, 0, 0}
Decelerating
f={34, 7, 0, 0}
Constant
f={106, 109, 0, 0}
Accelerating
![Page 39: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/39.jpg)
Role Assignment and Mapping
Identify the best match of role to agent assignments by searching the role assignment probabilities– Re-map the team-oriented feature vectors
Multiple role assignment– Each agent can play multiple roles
![Page 40: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/40.jpg)
Role Assignment and Mapping
Unique role assignment– Each agent can play one role
![Page 41: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/41.jpg)
Experimental Setup
Warfare exercise dataset– Same as used previously– Extended with seven activities– Four agents and four roles
Teamwork Activity Recognizer– Gaussian Mixture Model (GMM)– Multiple internal HMMs for each activity to accommodate for complexity
variations in teamwork activities
GPS readings are not always available (e.g. positions from opponents)
Simulate noisy observations– Offset position with a randomly generated number following the Gaussian
distribution multiplied with a noise magnitude
Mean accuracy and standard deviation was calculated using stratified 10-fold cross-validation
Activity Sequences Observations
Traveling column 29 319
Traveling line 30 330
Traveling box 33 363
Bounding overwatch 9 189
Wedge 17 187
Team split 15 192
Team merge 15 177
![Page 42: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/42.jpg)
Results: Creating the Idealized Team Actions
Assume perfect role recognition
Best parameter configuration– Four hidden states
– Three mixture components
![Page 43: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/43.jpg)
Results: Performance Evaluation with Unknown Team-Organization
Shuffled input
![Page 44: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/44.jpg)
Results: Performance Evaluation with Unknown Team-Organization
Accuracy with noise and unknown team-organization– 92.62% with standard deviation 5.53%
– Previous accuracy was 82% (without noise)
Role recognition improves robustness
Team-oriented feature functions improves accuracy
![Page 45: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/45.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 46: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/46.jpg)
Tuning the Performance of Teamwork Activity Recognition
Many components in the recognizer’s workflow are customizable– HMM: Number of hidden states and choice of emission
probability representation
– Feature extraction: Width of sliding window and discretization thresholds
– Role recognition module: Choice of features, parameters in the ID3 learning algorithm and the choice of learning algorithm
The probability density function (PDF) has a significant impact on the accuracy and robustness of the recognizer
Implementation Choice Multi-modal Continuous Parameters
Simple HMM Yes No States (n), Clusters (k)
HMM with Gaussian No Yes States (n)
HMM with GMM Yes Yes States (n), Components (C)
![Page 47: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/47.jpg)
Simple Hidden Markov Model
Discrete (histogram-based) probability density function
Inputs are sequences of discrete values
Vector quantization– Compress observations– Training of a codebook– Cluster assignment
Advantages– Multi-modal distribution
Disadvantages– Extra cluster parameter to train codebook– Can not estimate unseen observations– Requires more training data
![Page 48: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/48.jpg)
Hidden Markov Model with Gaussian PDF
Used in the initial recognizer
Advantages– Continuous
– Can estimate unseen observations
Disadvantages– Unimodal distribution
– Example: How to model U-turn to the left and right in the same HMM?
– Matrix inversion of covariance matrix can be problematic Reduce expressiveness by enforcing diagonal covariance matrices
![Page 49: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/49.jpg)
Hidden Markov Model with GMM PDF
Used in the extended recognizer
Advantages– Continuous
– Can estimate unseen observations
– Multi-modal distribution
Disadvantages– Extra parameter to determine number of mixture components
– Parameter estimation is computationally expensive
![Page 50: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/50.jpg)
Results: Simple HMM
Best recognition accuracy– 60 clusters with 18 hidden
states– Accuracy is 75.05% ± 14.89%
Difficulties with– Bounding overwatch– Team split– Team merge
![Page 51: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/51.jpg)
Results: HMM with Gaussian PDF
Best recognition accuracy – 3 hidden states– Accuracy is 91.29% ± 6.69%
Fails when using more than three states (not enough training data)
– Bounding overwatch– Team split– Team merge
Reduce expressiveness of the PDF
![Page 52: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/52.jpg)
Results: HMM with GMM PDF
Best recognition accuracy – 3 hidden states– 2 mixture components– Accuracy is 93.90% ± 4.68%
![Page 53: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/53.jpg)
Summary
Simple HMM– Performed poorly
– Filters out too much information
HMM with Gaussian PDF– Fails when using the general covariance matrix when n > 3
HMM with Gaussian Mixture Model PDF– Best choice
– Least sensitive to changes in parameter configurations
Implementation Choice Mean Max Min Std. dev.
Simple HMM 62.91% 75.05% 44.52% 5.44%
HMM with Gaussian 83.27% 91.28% 77.76% 4.29%
HMM with GMM 88.30% 93.90% 83.19% 2.17%
![Page 54: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/54.jpg)
Outline
Introduction
Data Acquisition and Knowledge Engineering
Teamwork Activity Recognition using Hidden Markov Models
Team-Oriented Feature Extraction
Role-Based Teamwork Activity Recognition
Tuning the Performance of Teamwork Activity Recognition
Conclusions
![Page 55: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/55.jpg)
Conclusions
We developed a knowledge engineering tool for visualization, identification and extraction of teamwork datasets
We developed a teamwork activity recognizer capable of encoding teamwork activity models from representative observations
We improved accuracy of the recognizer using team-oriented feature functions
The robustness of the recognizer was enhanced by integrating a role-recognition module in the recognition workflow
We studied the importance of probability density function choices in our recognizer
![Page 56: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/56.jpg)
Contributions
Teamwork Scenario Editor
Corpus of labeled teamwork activity
Team-oriented feature extraction
Learning teamwork activity from observation
Role-based teamwork recognition
Performance tuning of the teamwork recognizer
![Page 57: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/57.jpg)
Future Work
Team membership assignment
Larger teams with 100 or more agents
Preliminary studies– Kalman filter
Noise reduction of observation data Fusing information from multiple (uncertain) sources Applied to estimate life-expectancy in a disaster response simulation
– Genetic programming approach to learn agent strategies Applied to a simple foraging game in the single agent domain
![Page 58: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/58.jpg)
Publications Teamwork Recognition of Embodied Agents with hidden Markov models. L. J.
Luotsinen, H. Fernlund and L. Bölöni, IEEE 3rd International Conference on Intelligent Computer Communication and Processing (ICCP07), September 6-8, 2007.
Automatic Annotation of Team Actions in Observations of Embodied Agents. L. J. Luotsinen and H. Fernlund and L. Bölöni, Sixth International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2007), May, 2007, Honolulu, Hawaii.
A Study of the Robustness of Agent Performance in Nine Popular Agent Implementation Paradigms. L. J. Luotsinen, M. A. Khan and L. Bölöni, IEEE 3rd International Conference on Intelligent Computer Communication and Processing (ICCP07), September 6-8, 2007.
A Comparison Study of Twelve Paradigms for Developing Embodied Agents. Ladislau Bölöni, L. J. Luotsinen, Joakim N. Ekblad, T. Ryan Fitz-Gibbon, Charles Houchin, Justin Key, Majid Ali Khan, Jin Lyu, Johann Nguyen, Rex Oleson, Gary Stein, Scott Vander Welde, and Viet Trinh. Software: Practice and Experience, 2007.
Comparing Apples with Oranges: Evaluating Twelve Paradigms of Agency (Book chapter), L. J. Luotsinen, J. N. Ekblad, T. R. F. Gibbon, C. Houchin, J. Key, M. A. Khan, J. Lyu, J. Nguyen, R. Oleson, G. Stein, S. V. Weide, V. Trinh and L. Bölöni, ProMAS-2006, Fourth International Workshop on Programming Multi-Agent Systems, LNAI 4411, pp. 95-114, 2007.
Comparing Apples with Oranges: Evaluating Twelve Paradigms of Agency. L. J. Luotsinen, Joakim N. Ekblad, T. Ryan Fitz Gibbon, Charles Houchin, Justin Key, Majid Ali Khan, Jin Lyu, Johann Nguyen, Rex Oleson, Gary Stein, Scott Vander Weide, Viet Trinh and Ladislau Bölöni, ProMAS-2006, Fourth International Workshop on Programming Multi-Agent Systems, Hakodate, Japan, May, 2006.
A Two-Stage Genetic Programming Approach for Non-Player Characters. L. J. Luotsinen, Joakim N. Ekblad, Annie S. Wu, Avelino J. Gonzalez and Ladislau Bölöni, FuturePlay, The International Academic Conference on the Future of Game Design and Technology, East Lansing, Michigan, October, 2005.
Collaborative UAV Exploration of Hostile Environments. L. J. Luotsinen, Avelino J. Gonzalez and Ladislau Bölöni, 24th Army Science Conference, Orlando, Florida, December, 2004.
![Page 59: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/59.jpg)
Publications (Pending)
Role-Based Teamwork Activity Recognition in Observations of Embodied Agent Actions. L. J. Luotsinen and L. Bölöni, Submitted to: Seventh International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS-2008), May, 2008, Estoril, Portugal.
A Robust Method for Estimating Noisy Measurements Applied to Disaster Response Operations. L. J. Luotsinen, M. A. Khan and L. Bölöni, 2008.
Building a World Model under Communication Constraints for Disaster Response Applications. M. A. Khan, L. J. Luotsinen and L. Bölöni, 2008.
![Page 60: Recognizing Teamwork Activity in Observations of Embodied Agents Linus J. Luotsinen School of Electrical Engineering and Computer Science University of](https://reader036.vdocuments.us/reader036/viewer/2022062309/56649ec15503460f94bcd111/html5/thumbnails/60.jpg)
Questions?