multimodal information exchange and dynamic adaptation

Advanced Decision Architectures Collaborative Technology Alliance

Multimodal Information Exchange and Dynamic Adaptation

Nadine SarterThomas Ferris

Shameem Hameed

University of Michigan


Multimodal Adaptive Displays

• Future battlefield operations will be highly complex and dynamic and require effective information systems/exchange

• Our approach:– Multimodal displays (including vision, audition,

and touch)– Context-sensitive hybrid adaptive/adaptable

information presentation


Research Activities

• Created flexible computer-based simulation platform that supports co-located and remote synchronous collaboration

• Used platform for series of studies on

– Natural patterns and preferences of

modality usage

– Preattentive monitoring of

mission health

– Crossmodal spatial and

temporal links in attention


Natural Patterns of Modality Usage

• People do not necessarily interact multimodally just because a multimodal interface is made available

• Multimodal interaction primarily in the context of spatial tasks and to support complementarity

• Switch modalities mostly for the purpose of recovering from communication breakdowns

• In the context of human-human interaction, most modality combinations were sequential in nature (“contrastive functionality”)

• Modality usage patterns evolve as a function of team coordination and change in response to factors such as scenario demands, the mission phase, and group dynamics.


Crossmodal Links in Attention

• Crossmodal spatial and temporal links

– The modality & location of a stimulus in one modality may facilitate/hinder processing of subsequent stimulus in different modality

– Effects manifest only within a certain time interval between stimuli (SOA – Stimulus Onset Asynchrony)

– Related information should be co-located… (the binding “problem”)


Previous XSL Studies

• Spartan laboratory environments• Simple and artificial cues and tasks

• Do we see these effects in more complex environments with real-world tasks and stimuli?

SOA

~100-300ms e.g., Spence & Driver, 1997


Rightspeaker

Leftspeaker

Remote eye-trackingcamera

Joystick for UAV

FBCB2 sharedmap display

Satellite datauplink displaySatellite data

uplink display

Thermal detectionsystem displayThermal detectionsystem display

PeriscopeDisplay

Tactors wornon wrists

Tactors wornon wrists

Push-to-talk radiobutton strappedto index finger

Push-to-talk radiobutton strappedto index finger

Tactors strapped to outsides of thighs

Tactors strapped to outsides of thighs

Experimental Setup


Method

• 12 cadets and 3 graduates from the University of Michigan Army ROTC program (7 females and 8 males)

• Each participant played Stryker vehicle commander (VC) for the first of a convoy of vehicles in a simulated night-time rendezvous mission

• Throughout the mission, participants were presented with 48 visual, auditory, or tactile targets, either in isolation (‘uncued’ trials, n=24) or preceded (various SOAs) by an ipsilateral (same-side, n=12) or contralateral (opposite-side, n=12) peripheral cue in a different modality


Main Findings

• Confirm that crossmodal spatial links affect performance in more complex settings

• Cuing effects were larger and response times longer and varied to a higher degree than in earlier research

• Crossmodal asymmetries:– Ipsilateral crossmodal cuing was beneficial only for auditory cuing of

visual targets but not vice versa

– Significantly faster responses for contralateral tactile cuing of auditory targets but not vice versa

– Visual-tactile cue-target combinations showed a similar trend favoring contralateral presentations

• Faster response times for contralateral presentations in some cases may be the result of IOR


Cueing of Visual Target

Thermal detection display RTs

808.33

683.93

799.00876.80

715.71

0.00

200.00

400.00

600.00

800.00

1000.00

1200.00

1400.00

uncued ipsi auditory contra auditory ipsi tactile contra tactile

RT

(m

s)

IOR???


Main Findings

• Confirm that crossmodal spatial links affect performance in more complex settings

• Cuing effects were larger and response times longer and varied to a higher degree than in earlier research

• Crossmodal asymmetries:– Ipsilateral crossmodal cuing was beneficial only for auditory cuing of

visual targets but not vice versa

– Significantly faster responses for contralateral tactile cuing of auditory targets but not vice versa

– Visual-tactile cue-target combinations showed a similar trend favoring contralateral presentations

• Faster response times for contralateral presentations in some cases may be the result of IOR


Context-Sensitive Display Design

• Hybrid approach – “Delegation”– Combine positive aspects of adaptive

(system-initiated) & adaptable (user-controlled; management-by-exception) interfaces

– Combine/negotiate among multitude of drivers related to operator, cues, and environment


Mental Workload

• Heart rate– Confounded by physical component &

mental stress• Heart rate variability

– Variation in time interval between consecutive heart beats

– Relatively stable index of mental workload• Shows shifts from rest state to task state• Shows varying levels of workload in task

state– Power spectrum analysis

From Rowe et al, 1998

(Workload)


Availability/Appropriateness of Modality

• Modality availability– Modality may have become temporarily or

permanently unavailable due to ambient conditions/events (e.g., explosion/ambush)

• Modality appropriateness– Nature and type of information conveyed

– Certain modalities more appropriate than others for certain types of information

• e.g., spatial information (geographic location) is best conveyed visually


Challenges

• Weight assignments for modalities may need to be adjusted

• Deadlock arbitration module – Start with “hard” constraints (detectability,

availability) – Then, choose modality with high

appropriateness index– If all those are same, consider previous cue

modality and timing


Experiment – Final Step

• Simulation and hybrid interface have been implemented

• Experiment is designed and under IRB review – will be conducted in Fall

• Will examine feasibility and effectiveness of the approach and compare to “static” information presentation


Tactons

Structured, complex tactile signals which communicate abstract messages(Brewster & Brown, 2004)

Text

Visual icons

Braille

Tactons


Tactile “Cues” vs. Tactons

Tactile “Cues” Tactons

One or few parameters modulated

Literal, critical to maintain intuitive mapping

Very limited information content

Usually multiple parameters modulated

Abstract patterns represent larger concepts or messages

Conscious processing required to decode message

Examples: Interruption management, patient monitoring


Conclusion

• Much more complex picture emerges for effective multimodal information presentation– Requiring careful choice of modality pairings,

location, timing – Calling for context-sensitive presentation of

information– Power of tactons far from being exploited

• Beware of guidelines (“adapt modalities to user preferences”) but see, for example, Jones and Sarter (2009) for guidance…

multimodal information exchange and dynamic adaptation

Documents