uncertainty, action, and interaction eric horvitz microsoft research may 2002 uncertainty, action,...

Uncertainty, Action, Uncertainty, Action,

and Interactionand Interaction

Eric HorvitzEric HorvitzMicrosoft ResearchMicrosoft Research

May 2002May 2002

Toward Mixed-Initiative User InterfacesToward Mixed-Initiative User Interfaces

Designs that assume Designs that assume from the ground upfrom the ground up that that user may guide, collaborate with automated user may guide, collaborate with automated service to achieve desired resultsservice to achieve desired results

User

Automation

Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction Endow system with ability to infer the likelihood Endow system with ability to infer the likelihood

of a user’s goals, intentionsof a user’s goals, intentions

Attempt to scope precision of action to match Attempt to scope precision of action to match goals and uncertainties goals and uncertainties

Determine the expected value of action given Determine the expected value of action given costs and benefits of actioncosts and benefits of action

Consider status of a user’s attention in timing of Consider status of a user’s attention in timing of actionaction

Allow for dialog at appropriate times to resolve Allow for dialog at appropriate times to resolve key ambiguitieskey ambiguities

Provide efficient means for agent–user Provide efficient means for agent–user collaboration to refine guessescollaboration to refine guesses

Allow efficient Allow efficient directdirect invocation and invocation and terminationtermination

Seek innovative designs that maximize benefit Seek innovative designs that maximize benefit of service, minimize the cost of poor guessesof service, minimize the cost of poor guesses

Allow for natural assumptions of shared Allow for natural assumptions of shared memory of recent interactionsmemory of recent interactions

Continue to learn by observingContinue to learn by observing

Principles of Mixed-Initiative InteractionPrinciples of Mixed-Initiative Interaction

Key goal: Provide the user with clear Key goal: Provide the user with clear advance toward goals advance toward goals

Automated, flexible scoping of Automated, flexible scoping of automated service to precision automated service to precision matching task uncertainty, context matching task uncertainty, context

Automated Scoping and Precision of Automated Scoping and Precision of ServiceService

Prefer automation to do less, but do it correctly

Automated Reasoning about the Automated Reasoning about the Uncertainty of a User’s GoalsUncertainty of a User’s Goals

Automated reasoners must guess about a Automated reasoners must guess about a user’s user’s goalsgoals and and desiredesire for services for services

Good guesses can be quite valuableGood guesses can be quite valuable

……but guessing wrong can be costlybut guessing wrong can be costly

Even valuable automation can be distracting Even valuable automation can be distracting and steal user’s scarce attentional resourcesand steal user’s scarce attentional resources

Minimizing Cost of Guessing WrongMinimizing Cost of Guessing Wrong

Seek design innovation: Advice / assistance Seek design innovation: Advice / assistance valuable when right, but errors with minimal valuable when right, but errors with minimal low costlow cost Natural gestures for declining service Natural gestures for declining service Avoid grabbing focus Avoid grabbing focus Alternate channel overlay: NASA Vista display managerAlternate channel overlay: NASA Vista display manager Nondistracting, simple guessing: Vellum gridpoint guessesNondistracting, simple guessing: Vellum gridpoint guesses

More graceful interaction with potentially More graceful interaction with potentially focused userfocused user

Better timing of services in sync with Better timing of services in sync with availability of attentionavailability of attention

Probability, Utility, & Probability, Utility, & Mixed Initiative InteractionMixed Initiative Interaction

Perspective for designPerspective for designSpecific functions, layering of componentrySpecific functions, layering of componentry Foundations of intelligenceFoundations of intelligence

??*&(#))(@%+%%$#*%$#*&%*&(^*^

Infrastructure, fabric for UI innovationInfrastructure, fabric for UI innovation

ProbabilitiesProbabilities

Infer likelihoods of key uncertainties, take ideal actions

Uncertainty and HCIUncertainty and HCI

• User queryUser query

• User activity User activity

• Content at focusContent at focus

• Data structures Data structures

• User locationUser location

• User profileUser profile

• Vision, speech, soundVision, speech, sound

*Utility-directed actionUtility-directed action

Meshing learning & reasoning with UI designMeshing learning & reasoning with UI design

Beliefs & IntentionsBeliefs & Intentions What does a user believe? What are the user’s goals?What does a user believe? What are the user’s goals?

AttentionAttention What is the user’s workload? What is a user attending to? What is the user’s workload? What is a user attending to?

What What will will a user attend to? What a user attend to? What shouldshould a user attend to? a user attend to? PreferencesPreferences

What does the user like and dislike—and how much?What does the user like and dislike—and how much?

InitiativeInitiative What is the cost and benefit of interaction, interruption, What is the cost and benefit of interaction, interruption,

intervention?intervention? What is the right mix of user / system initiatives?What is the right mix of user / system initiatives?

Critical UncertaintiesCritical Uncertainties

LumièreLumière Project Project

User’s ProfileUser’s ProfileUser’s GoalsUser’s Goals

User’s NeedsUser’s Needs

User ActivityUser Activity

Actions + Words Actions + Words Goals Goals

Joint work with J. Breese, D. Heckerman, K. Rommelse, D. Hovel, et al.

Studies with Human Subjects Studies with Human Subjects

ChallengesChallenges

Architectures for intelligent user Architectures for intelligent user interactioninteraction

Reasoning over timeReasoning over time Sensing activity from systems and Sensing activity from systems and

applicationsapplications Integration of probabilistic information Integration of probabilistic information

retrievalretrieval Models of a user’s competencies over Models of a user’s competencies over

timetime

Big PictureBig Picture

ControlControlNewNew

Perceptions, Perceptions, InteractionsInteractions

EventsEvents UncertainUncertainInference aboutInference about

User, WorldUser, World

ComputationComputationof Ideal UI Actionof Ideal UI Action

Ideal Ideal ActionsActions

EventsEventsSynthesisSynthesis

LearningLearningModelsModels

Inference about a User’s Time-Inference about a User’s Time-Dependent GoalsDependent Goals

TimeTime

Goalt-n

Ej,t-n Ei,t-n

Goalto

Ej,to Ei,to

Goalt-1

Ej,t-1 Ei,t-1

Profile

Profile

Profile

Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”

Skill Catogories

Co

mp

eten

cy

Representing and Updating a Representing and Updating a Persistent “Competency Terrain”Persistent “Competency Terrain”

User’s Skills

Co

mp

eten

cy

TimeTime

Toward a “peripheral nervous system” for Toward a “peripheral nervous system” for sensing user activity sensing user activity SDK with event abstraction languageSDK with event abstraction language Compiler for defining filters for user activityCompiler for defining filters for user activity

Sensing Context and ContentSensing Context and Content

Atomic EventsAtomic Events Modeled EventsModeled Events

TimeTime

EventSource 1

EventSource 2

EventSource n

Eve Event-Specification

Language

Abstraction of Events

Overall LumiOverall Lumièère Architecturere Architecture

BayesianBayesianInferenceInference

EventsEvents

TimeTime

• QueryQuery

• ActionsActionsEvent SynthesisEvent Synthesis

Control SystemControl System

• Probability user Probability user

desires assistancedesires assistance

Lumiere Inference and ActionLumiere Inference and Action

InitiativeInitiative

User vs. system initiativeUser vs. system initiative Allowing fluid collaboration via a mix of Allowing fluid collaboration via a mix of

initiativesinitiatives Toward Toward principles of mixed-initiative principles of mixed-initiative

interactioninteraction Projects: Projects: Lookout, DeepListener, QuartetLookout, DeepListener, Quartet

Reasoning about initiative is a Reasoning about initiative is a

high-payoff opportunity area high-payoff opportunity area

for HCI, Ubicomp, IUIfor HCI, Ubicomp, IUI

Learning by watchingLearning by watching Costs-benefit analysis of initiativeCosts-benefit analysis of initiative Minimize disruption: Prefer Minimize disruption: Prefer doing less,doing less,

but doing it correctlybut doing it correctly

Initiative & Interaction: LookoutInitiative & Interaction: Lookout

?? Critical decision: Critical decision: Do nothing.Do nothing. Ask? Ask? Just do it?Just do it?

Joint work with Andy Jacobs

Real-TimeProbabilistic Inference

Cost--Benefit Analysis

User Actions / Context

UI / Service

• Watch user’s behavior• Store cases, timing info• Learn model from data

Learning and Real-Time BehaviorLearning and Real-Time Behavior in Lookout in Lookout

Lookout in Handsfree ModeLookout in Handsfree Mode

Preferences and InitiativePreferences and Initiative

Expected utility as fundamental in Expected utility as fundamental in decisions about servicesdecisions about services

A: Computer takes action i

A: No action i

D: User desires action iD: User does not desire action i

Service

User’s Desire

u(A,D)

u(A,D)

u(A,D)

u(A,D)

Act

No act

Desired Undesired

Action

No Action

P*

1.0

1.0

0.0

u(A,D)

u(A,D)

u(A,D)

u(A,D)

p(D|E)

eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = jj uu((AAii,,DDjj) ) pp((DDjj||EE))eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + ) + pp((DD||EE) ) uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )

eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) ) eueu((AA) = ) = pp((DD||EE) ) uu((AA,,DD) + [1 - ) + [1 - pp((DD||EE)] )] uu((AA,,DD) )

Preferences and InitiativePreferences and Initiative

1.0

1.0

0.0

u(A,D)

u(A,D)

u(A,D)

u(A,D)

p(D|E)

Action

No Action

P*

User rushed

Initiative and ContextInitiative and Context

Utility of outcomes as function of context,Utility of outcomes as function of context, uu((AA,,D,D,))

1.0

1.0

0.0

u(A,D)

u(A,D)

u(A,D)

p(D|E)

Action

No Action

P*

No Action

u(A,D)

User rushed

Increase in Amountof Screen Real Estate

u(A,D)




1.0

1.0

0.0

u(A,D)

u(A,D)

u(A,D)

u(A,D)

p(D|E)

Action

No Action

P*

No Action

u(A,D)

u(A,D)

Increase in Amountof Screen Real Estate

u(A,D)

User rushed


1.0

1.0

0.0

u(A,D)

u(A,D)

u(A,D)

u(A,D)

p(D|E)

Action

P*

Ask

No Action

Expected value of engaging the user in dialogueExpected value of engaging the user in dialogue

Engaging in Dialog about InitiativeEngaging in Dialog about Initiative

Varying Precision of ServiceVarying Precision of Service

Consider contributions across Consider contributions across a spectrum of precision a spectrum of precision

Assume user will refine partial resultsAssume user will refine partial results Under uncertainty, trade off reduced precision for Under uncertainty, trade off reduced precision for

higher accuracyhigher accuracy

ApptDay

Week

Timing of InitiativeTiming of Initiative

Timing is critical: consider patterns of attentionTiming is critical: consider patterns of attention Record length of message and dwell time before Record length of message and dwell time before

calendar invokedcalendar invoked Perform regressionPerform regression

0

2

4

6

8

10

0 500 1000 1500 2000 2500

Length of original message (bytes)Ob

serv

ed d

wel

l bef

ore

act

ion

(sec

)

Conversational ArchitecturesConversational Architectures ProjectProject

DeepListenerDeepListener Bayesian ReceptionistBayesian Receptionist QuartetQuartet

Why do people find it more difficult and Why do people find it more difficult and frustrating to converse with a spoken dialog frustrating to converse with a spoken dialog

system than with a person?system than with a person?

QuestionQuestion

Several answersSeveral answers

• Poor recognition of wordsPoor recognition of words

• Meaning too difficult to captureMeaning too difficult to capture

• Lack of precise user modelsLack of precise user models

• Different social and personality Different social and personality dynamicsdynamics

Interpreting spoken language abounds with Interpreting spoken language abounds with uncertaintyuncertainty

IntuitionsIntuitions Despite uncertainty in humanDespite uncertainty in human––human conversation human conversation

people manage to understand each other quite people manage to understand each other quite well.well.

People consider the source of their uncertainties People consider the source of their uncertainties and pursue actions to resolve confusions.and pursue actions to resolve confusions.

RecognitionRecognition LanguageLanguage Context, topic, meaningContext, topic, meaning Frank troubleshootingFrank troubleshooting

GoalGoal: Models and inference methods that seek : Models and inference methods that seek mutual understanding under uncertainty given mutual understanding under uncertainty given inescapably unreliableinescapably unreliable components. components.

GroundingGrounding

People resolve uncertainties through a People resolve uncertainties through a process of process of groundinggrounding

Process by which participants establish Process by which participants establish and maintain the mutual belief that their and maintain the mutual belief that their utterances have been understood well utterances have been understood well enough for current purposesenough for current purposes

-Clark & Schaefer, 1987 -Clark & Schaefer, 1987

DeepListenerDeepListener

Utility-directed clarification dialogUtility-directed clarification dialog Formal model of “understood well enough”Formal model of “understood well enough” Development environmentDevelopment environment Assessment toolsAssessment tools Focus: Spoken command and control Focus: Spoken command and control

systemssystems

Stakes, Likelihoods, and Stakes, Likelihoods, and Clarification ActionsClarification Actions

Consider stakes of real-world action being Consider stakes of real-world action being consideredconsidered

Should I format your hard drive? Should I format your hard drive?

Should I try to schedule that?Should I try to schedule that?

Should I demolish the King Dome Should I demolish the King Dome nownow??

Consider uncertaintiesConsider uncertainties Consider expected utility of alternative “repair” Consider expected utility of alternative “repair”

actionsactions Costs and benefits of real-world action vs. alternative Costs and benefits of real-world action vs. alternative

dialog repair actionsdialog repair actions

Infer likelihoods of alternative Infer likelihoods of alternative spoken spoken intentionsintentions Likelihoods of different Likelihoods of different spoken intentionsspoken intentions given acoustics given acoustics Optionally condition on goals inferred by user model Optionally condition on goals inferred by user model

external to the speech systemexternal to the speech system

Compute Compute clarificationclarification or real-world or real-world actions with highest expected utilityactions with highest expected utility

Fuse multiple attempts with Bayesian Fuse multiple attempts with Bayesian model that considers confidencesmodel that considers confidences Consider history of utterances within a sessionConsider history of utterances within a session No reason to start over at each turn! ..Leverage what was No reason to start over at each turn! ..Leverage what was

heard beforeheard before

ApproachApproach

Decision ModelDecision Model

Speaker’s Goal(t-1)

User’s SpokenIntention(t-1)

Dialog or Domain-Level

Action(t-1)

ASR Candidate 1Confidence(t-1)

Utility(t-1)

User Actions(t-1)Content at Focus (t-1)

ASR Candidate nConfidence(t-1)

Context

. . .

ASR ReliabilityIndicator(t-1)

External

User Model




Action(t-1)


Utility(t-1)

User Actions(t-1)Content at Focus (t-1)


Context

...

Speaker’s Goal(t)

User’s SpokenIntention(t)


Action(t)

ASR Candidate 1Confidence(t)

Utility(t)

User Actions(t)Content at

Focus (t)

ASR Candidate nConfidence(t)

Context

...



Dynamic Model for Reasoning Over Dynamic Model for Reasoning Over Multiple TurnsMultiple Turns

Dialog Actions under ConsiderationDialog Actions under Consideration

Perform Perform real-world actionreal-world action (e.g., (e.g., implode the King implode the King Dome nowDome now))

Ask for repetitionAsk for repetition to clarify to clarify

Note hesitationNote hesitation or reflection and try again or reflection and try again

Note potential Note potential overhearingoverhearing of noise and inquireof noise and inquire

Note inattentionNote inattention of user and try to acquire user’s of user and try to acquire user’s attentionattention

Don’t perform action and just Don’t perform action and just go awaygo away

Note problem with conversational interaction and Note problem with conversational interaction and attempt to attempt to troubleshoottroubleshoot

Example: DeepListener for handling Example: DeepListener for handling

confirmation, negationconfirmation, negation

DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System

Dynamic Bayesian network modeling and Dynamic Bayesian network modeling and inference inference

MS command and control speech systemMS command and control speech system Backchannel animations: MS AgentBackchannel animations: MS Agent

DeepListener: SDK and Real-Time DeepListener: SDK and Real-Time Clarification Dialog SystemClarification Dialog System

Accruing Evidence Over Repeated UtterancesAccruing Evidence Over Repeated Utterances

Beliefs and Actions for Clarification DialogBeliefs and Actions for Clarification Dialog

Clarification

Beliefs about Spoken IntentionBeliefs about Spoken Intention

yes

no overheard

yes

no overheard

Inferred beliefs

Expected Utility of Alternate ActionsExpected Utility of Alternate Actions

Repeat

Again

EngageNoise

Tshoot Disengage

Attention

Repeat

Again

EngageNoise

Tshoot Disengage

Attention

Expected utility of actions

Preference elicitationPreference elicitation For developers (assessment “at factory”)For developers (assessment “at factory”) For users! Prototypical patterns, assessment For users! Prototypical patterns, assessment

wizard, direct detailed assessmentwizard, direct detailed assessment

Toward a more general SDK for command Toward a more general SDK for command and control dialog (e.g., telephony systemsand control dialog (e.g., telephony systems

Preference Assessment and EncodingPreference Assessment and Encoding

Assessing Preferences on OutcomesAssessing Preferences on Outcomes




Action(t-1)


Utility(t-1)


External UserModel(t-1)

.. .

Speaker’s Goal(t)

User’s SpokenIntention(t)

ASR Candidate 1Confidence(t)

ASR Candidate nConfidence(t)

ExternalUser Model(t)

.. .


Action(t)

Utility(t)


ASR ReliabilityIndicator(t)

Troubleshooting Conversation Failures Troubleshooting Conversation Failures Over Multiple StepsOver Multiple Steps

Expected utility of taking action to repair listening situationas function of details of multi-turn dialog “history”

Assessing Frustration with Assessing Frustration with Number of StepsNumber of Steps

yesTshoot

reflect

Repeat

overheard

Engage

no

When Troubleshooting is the When Troubleshooting is the Best ActionBest Action

yes

Tshoot

reflect

Repeat

When Troubleshooting is the When Troubleshooting is the Best ActionBest Action

Considering Attentional IssuesConsidering Attentional Issues

Toward Continuous Listening SystemsToward Continuous Listening Systems

Beyond cumbersome “push-to-talk” Beyond cumbersome “push-to-talk” systemssystems

Discriminating target of speechDiscriminating target of speech Understanding conversation Understanding conversation

maintenance statusmaintenance status Allowing user barge-inAllowing user barge-in

DeepListener StatusDeepListener Status

Application development environmentApplication development environment Mobile telephonyMobile telephony Desktop applicationsDesktop applications Managing “subdialog”Managing “subdialog”

Evaluation: Tradeoffs—steps vs. stakesEvaluation: Tradeoffs—steps vs. stakes Learning by watchingLearning by watching

44 S is proposing activity to L L is considering S’s proposal of

ConversationConversation

33 S is signaling that p for L L is recognizing that p from S

IntentionIntention

22 S is presenting signal to L L is identifying signal from S

SignalSignal

11 S is executing behavior for L L is attending to behavior from S

ChannelChannel

QuartetQuartet:: Multilevel Grounding & Extended Sensing Multilevel Grounding & Extended Sensing

LevelLevel

Maintenance ModuleMaintenance Module

Maintenance ModuleMaintenance Module

User’s ResponseLatency

Used Nameof System

Stressed Nameof System

Repeat SimilarLogical Form

Eye Gaze onSystem

Time Since LastUser Speech

MaintenanceStatus (t)

MaintenanceStatus (t-1)

SignalAccuracy (t-1)

Time Since LastSystem Action

Others Presentin Room

Calendar ShowsMeeting

Telephone in Use

Overall Parse Fit

UtteranceComplexity

Syntactic SketchScore

NumNon-Terminals

Num PhrasalHeads

Final UtteranceConfidence

SignalAccuracy (t)

InterferenceEvent

Num HypothesesPer Word

Energy Floor

Signal Identified

User’s Focus ofAttention (t-1) User’s Focus of

Attention (t)

General ASRQuality

Is Trained forUser

Microphone Type

ThresholdSetting

Custom GrammarFor Domain

Channel Level

Signal Level

Linguistic EvidenceLinguistic Evidence

Nature, correctness of NL parse Nature, correctness of NL parse as evidence sourceas evidence source

Visual EvidenceVisual Evidence

Evidence for user attentionEvidence for user attention

Acoustical EvidenceAcoustical Evidence

Vision and GroundingVision and Grounding

Considering Visual and Considering Visual and Linguistic CluesLinguistic Clues

“If you could take me to the next slide that would be great.”

“I can look away and talk about the computer.”


… and even talk about going to the next page.”


Bayesian ReceptionistBayesian Receptionist

Continue to Continue to gather gather information information

User’s Goal

Goal 1 Goal n

Subgoal 11

Subgoal 1x1 Subgoal 1xy

Subgoal 1x

Level 0

Level 1

Level 2

Return to previous level Return to previous level of analysisof analysis

Progress to next level of Progress to next level of precision without precision without confirmationconfirmation

Progress to next level Progress to next level of precision after of precision after confirmationconfirmation

EVI

EVI

EVI

Take action in worldTake action in world

Initial utterance, observations

World action

Open request for information

Richer Models of InitiativeRicher Models of InitiativeConversational Architectures ProjectConversational Architectures Project

Joint work with Tim Paek

Goal 1 Goal nLevel 0

User’s Goal

Subgoal 11


Subgoal 1x

Level 1

Level 3

VOI

VOI

VOI

Bayesian Models and DialogBayesian Models and Dialog

VOI

User’s Goal

Goal 1 Goal n

Subgoal 11


Subgoal 1x

Level 1

Level 3

VOI

VOI

VOI

Bayesian Models and DialogBayesian Models and Dialog

Level 0

DARPA ISAT Study:DARPA ISAT Study: 2000-2001 2000-2001

Foundations of Augmented CognitionFoundations of Augmented Cognition

(E. Horvitz and M. Pavel, co-chairs)(E. Horvitz and M. Pavel, co-chairs)

Multiple meetings culminating at NAS, Wash Multiple meetings culminating at NAS, Wash DC (August 2001)DC (August 2001)

Augmented Cognition programAugmented Cognition program

Toward Augmented CognitionToward Augmented Cognition

Divided attentionDivided attention Visual search Visual search Concept attainmentConcept attainment MemoryMemory VisualizationVisualization Judgment & decision makingJudgment & decision making Action under limited time and informationAction under limited time and information ComprehensionComprehension Training & educationTraining & education

Apply computation to support / augment human cognition

Results in Cognitive PsychologyResults in Cognitive Psychology

Characterization of limitations in human cognitionCharacterization of limitations in human cognition

Human capabilities static, computational capabilities Human capabilities static, computational capabilities have grown rapidlyhave grown rapidly

Human abilitiesHuman abilities

Computational Computational resources, prowessresources, prowess

Augmented Augmented CognitionCognition

TodayToday

Toward Augmented CognitionToward Augmented Cognition

ExistingExisting

Psychological Psychological

Results on Results on

Cognitive LimitationsCognitive Limitations

TargetTarget

Cognitive TasksCognitive Tasks

MemoryMemory ConceptConcept

AttainmentAttainmentDivided Divided

AttentionAttention

Ab

ilities & efficien

ciesA

bilities &

efficiencies

* * New Cog. Psych. New Cog. Psych. Research Research

* HCI, Aug. Cognition * HCI, Aug. Cognition ResearchResearch

??

HCI and Augmented CognitionHCI and Augmented Cognition

Designing services with knowledge of Designing services with knowledge of cognitive limitationscognitive limitations

Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context

Value of probabilistic reasoning for Value of probabilistic reasoning for assessing goals, states, contextassessing goals, states, context

Key ArenasKey Arenas

AttentionAttention AttentionAttention MemoryMemoryMemoryMemory

Judgment & Decision makingJudgment & Decision makingJudgment & Decision makingJudgment & Decision making

Visualization Visualization & Display& Display


Learning Learning & Training& Training


Language Language & Interaction& InteractionLanguage Language

& Interaction& Interaction

Neurobiological issuesNeurobiological issuesNeurobiological issuesNeurobiological issues

Meeting FociMeeting Foci

AttentionAttention AttentionAttention MemoryMemoryMemoryMemory









Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,

Reasoning & ApplicationsReasoning & Applications

Advances in Computation:Advances in Computation:Sensing, Learning, Sensing, Learning,

Reasoning & ApplicationsReasoning & Applications

Perspective onPerspective on

ApplicationsApplications

Perspective onPerspective on

ApplicationsApplications

Recent Sample Results from the LabRecent Sample Results from the Lab

UVA/CMUUVA/CMU: Memory, : Memory, spatialization, and contextspatialization, and context

MSRMSR: Alerting, performance, and : Alerting, performance, and memorymemory

UVA/CMU: UVA/CMU: Memory Efficiency and SpatializationMemory Efficiency and Spatialization

(D. Proffitt, R. Pausch, et al.)(D. Proffitt, R. Pausch, et al.)

Significant memory boosts for Significant memory boosts for word-pair trials for single vs. word-pair trials for single vs. multiple screen learningmultiple screen learning

fMRI analysis of differential fMRI analysis of differential activityactivity

InfocockpitInfocockpit

Experiment: Word-Pair RecallExperiment: Word-Pair Recall

TaskTask 10 cue words and 10 target words. 3 lists. Cue words 10 cue words and 10 target words. 3 lists. Cue words

same for all lists. Target words varied from list to list. same for all lists. Target words varied from list to list. Next day: Recall pairs (not told ahead of time)Next day: Recall pairs (not told ahead of time)

Two conditionsTwo conditions Infocockpit condition Infocockpit condition 3 monitors, 3 large projection screens w/ contextual 3 monitors, 3 large projection screens w/ contextual

imageimage & surround sound. Word lists displayed on & surround sound. Word lists displayed on different monitors.different monitors.

Control Control Standard desktop computer.Standard desktop computer.

Recall EnhancementRecall Enhancement

Difference fMRIDifference fMRI

MSR: Studies of Divided Attention, MSR: Studies of Divided Attention, Alerting, NotificationAlerting, Notification

(M. Czerwinski, E. Cutrell, E. Horvitz)(M. Czerwinski, E. Cutrell, E. Horvitz)

Psychological Studies of AttentionPsychological Studies of Attentione.g., Interruption & recoverye.g., Interruption & recovery

(With Mary Czerwinski & Ed Cutrell)(With Mary Czerwinski & Ed Cutrell)

Differing Costs of DisruptionDiffering Costs of Disruption

High-level characterization of taskHigh-level characterization of task

Planning Execution Evaluation0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Interrupted Phase

Relevant vs. Irrelevant AlertsRelevant vs. Irrelevant Alerts

Total Resume0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Task Timing

Relevant

Irrelevant

Probing Deeper Structure of DisruptionProbing Deeper Structure of Disruption

Probing “reorientation cost”Probing “reorientation cost” Visual reorientation (e.g., re-acquire position in list Visual reorientation (e.g., re-acquire position in list

being searched)being searched) Conceptual reorientation (e.g., re-acquire context, Conceptual reorientation (e.g., re-acquire context,

goal)goal)

??


ChallengeChallenge: Identify target book title in a 3 : Identify target book title in a 3 page list of titles with and without cursor page list of titles with and without cursor markermarker

Two targetsTwo targets .5 cases: Verbatim title.5 cases: Verbatim title .5 cases: Gist: .5 cases: Gist: ee.g., .g., ““A book about Ramses II and A book about Ramses II and

the Nilethe Nile.”.” Remind meRemind me button button DesignDesign

2 (title v. gist search) 2 (title v. gist search) x 2 (marker v. no marker) x 2 (marker v. no marker) x 2 (notification vs. no notification trial) x 2 (notification vs. no notification trial) x 8 (replications per condition) x 8 (replications per condition) [[16 participants, 16 participants, 64 trials per session]64 trials per session]


Notifications Notifications Mimicked sound, onset of MSN Messenger 2.0Mimicked sound, onset of MSN Messenger 2.0 Simple multiplication, division problemsSimple multiplication, division problems

Dependent variables Dependent variables Total task timeTotal task time Time to switch to a notificationTime to switch to a notification Number of reminders requested Number of reminders requested Time spent on a notificationTime spent on a notification


Cognitive reorientation dominates spatial Cognitive reorientation dominates spatial reorientation.reorientation.

Reminders requested more for notifications Reminders requested more for notifications under higher cognitive loadunder higher cognitive load

Significant relationship between need for Significant relationship between need for reminder and reminder and timingtiming of the interruption of the interruption Participants more likely to request a reminder when Participants more likely to request a reminder when

disrupted early in taskdisrupted early in task

Sample Results on Sample Results on Disruption and ReorientationDisruption and Reorientation

Augmented Cognition: OpportunitiesAugmented Cognition: Opportunities

AttentionAttention Integrate consideration of results on Integrate consideration of results on

divided attention and disruption in divided attention and disruption in monitoring systems.monitoring systems.

e.g.,e.g., Control if, when, and how information about Control if, when, and how information about a monitored system or situation is presented to a monitored system or situation is presented to operators focusing on another, more central task.operators focusing on another, more central task.

Employ results from visual search and Employ results from visual search and

attention in rendering policiesattention in rendering policies

e.g., Develop automated display layout and e.g., Develop automated display layout and optimization considering time-criticality, content.optimization considering time-criticality, content.

Memory and LearningMemory and Learning Harness results on memory and contextual Harness results on memory and contextual

cuescuese.g.,e.g., Automated sensing / generation of cues Automated sensing / generation of cues during learning, coupled with re-generation of during learning, coupled with re-generation of multiple contextual clues during remembering.multiple contextual clues during remembering.

Harness results on ideal spacing for enhanced Harness results on ideal spacing for enhanced learninglearning

e.g.,e.g., Infer ideal time for reinforcement with Infer ideal time for reinforcement with repetition in a training system.repetition in a training system.


Judgment and Decision MakingJudgment and Decision Making Consider errors of framing and well-Consider errors of framing and well-

characterized biases of judgment, action characterized biases of judgment, action under uncertainty.under uncertainty.

e.g., Guide visual representation of actions, e.g., Guide visual representation of actions, alternatives, and outcomes in a manner that alternatives, and outcomes in a manner that debiases stereotypical errors of judgment in debiases stereotypical errors of judgment in decision making under uncertainty.decision making under uncertainty.


Potential ApplicationsPotential Applications

Information filtering, triage, and alertingInformation filtering, triage, and alerting Mixed-initiative agent-operator interactionMixed-initiative agent-operator interaction Context-sensitive computingContext-sensitive computing Intelligent remindingIntelligent reminding Managing attention and disruptionManaging attention and disruption Automated visualization in control systemsAutomated visualization in control systems Learning and training systemsLearning and training systems Human-errorHuman-error——aware systemsaware systems Automated sensor fusion and decision Automated sensor fusion and decision

supportsupport Judgment de-biasing systemsJudgment de-biasing systems

Presentations on Key ArenasPresentations on Key Arenas

AttentionAttention (Sperling) (Sperling)

Memory Memory (Landauer)(Landauer)

Judgment & Decision makingJudgment & Decision making (Fischhoff) (Fischhoff)

Visualization & DisplayVisualization & Display (Ellis) (Ellis)

Language & InteractionLanguage & Interaction (Oviatt) (Oviatt)

Learning & TrainingLearning & Training (Carroll) (Carroll)

Neurobiological issuesNeurobiological issues (Levy) (Levy)

Breakout GroupsBreakout Groups Key models & findings on limitations from Key models & findings on limitations from

psychologypsychology

Opportunities for real-world applications Opportunities for real-world applications

Challenges per psychological understandingChallenges per psychological understanding

Challenges per computation, display, HCIChallenges per computation, display, HCI

Take an optimistic perspectiveTake an optimistic perspective

Take a pessimistic perspectiveTake a pessimistic perspective

Evaluation metricsEvaluation metrics

Attention Attention Attention Attention MemoryMemoryMemoryMemory




Learning Learning & Training & Training

Learning Learning & Training & Training




AttentionAttention AttentionAttention

Research challenges Most results (e.g., AOC) focus on short durations (< 500 msec)

Need to understand attention over longer time scales

Links among low-level components, conceptual issues with scaling up to perception, attention, action for larger time scale strategies

Attention and disruption in realistic environments

Opportunities Handling divided attention, attention-guided access, attention-centric rendering and visualization, context-sensitive “attentional security” for learning & performance

Potential Applications Context-sensitive alerting systems, automated monitoring & guiding of attention, attending to anomalies, ideal mixed-initiative interaction

Memory & LearningMemory & LearningMemory & LearningMemory & Learning

Research challenges Basic research, e.g., understanding complex interactions among component processes of memory, links to attention, etc.

Understanding state of knowledge, best strategy per this state

User interfaces, user interaction in real-world tasks and systems

Opportunities Significant results have not seen application!

Specific policies that promote forgetting and impair performance during training actually enhance long-term retention!

e.g., spacing, variation, contextual interference, intermittent feedback

Can make training qualitatively a more difficult experience

Potential Applications Context-sensitive reminder systems

Enhanced situation monitoring, training


Research challenges Understand fixation on representations, premature closure vs. paralysis of analysis, propagating implications of key changes in assumptions, considering reliability of sources, influence of stress and fatigue; cognitive processes of situation assessment

Dynamics of belief about processes, uncertainty about spatial relations given streams of information

Decisions, judgment in gaming setting with opponent

Opportunities Harness well-characterized results on biases of judgment

Use of representations for communicating about, understanding situations; sensitivity of variables; value of different kinds of modeling effort

Potential Applications Decision making companions that expand considerations, pose reformulatations, debias, assist in real-time, training

Rendering for ideal fusion of information, judgment, action

Visualization & DisplayVisualization & DisplayVisualization & DisplayVisualization & Display

Research challenges Understanding how people integrate information from multiple components of scene into usable concepts

Basic research needed on how to map distinct findings about visual attention, concept attainment to control of display, layout for tasks

Opportunities Map cognitively deliberate serial analyses into fast perceptual recognition Harness knowledge about emergent features, gestalt principles, redundant coding, popout effects (e.g., use of line orientation, length, width, size, number, terminators, intersection, closure, color, intensity, flicker, direction of motion, depth cues, lighting

Exploit findings on timing, animation, scene complexity

Harness knowledge about biases in representation

Potential Applications Displays that adapt to task at hand, employing perceptual and cognitive principles to highlight most important information

Cognitively efficient representations of spatial relationships, dynamics, uncertainty, reliability

Language & InteractionLanguage & InteractionLanguage & InteractionLanguage & Interaction

Research challenges Handling disfluency speech, variations such as hyperarticulation, speech under stress, in mobile settings

Handling signal variability

Handling ambiguity

New languages, principles for mixed-initiative interaction

Opportunities Ideal designs for multimodal input, based on stereotypic patterns of gesture and language

Error reduction, speech interaction with minimal cognitive load

Potential Applications Natural interfaces that allow variable, noisy gesture and language

Automated detection of misunderstandings in communications

Enhanced multimodal interfaces for efficient interaction

Some Findings and Some Findings and RecommendationsRecommendations

Perspective, focus of Augmented Cognition as Perspective, focus of Augmented Cognition as distinguished from much of psychologically-oriented HCI distinguished from much of psychologically-oriented HCI

Interest, enthusiasm among participantsInterest, enthusiasm among participants

Mathematical psychologists traditionally have been Mathematical psychologists traditionally have been focused mainly on basic mechanisms as disjoint from real-focused mainly on basic mechanisms as disjoint from real-world applications, usage.world applications, usage.

Many current psychological models and research results Many current psychological models and research results are not necessarily well-adapted, matched for augmented are not necessarily well-adapted, matched for augmented cognition work.cognition work.

Need to map some basic results at msec time scale, Need to map some basic results at msec time scale, micromechanisms to broader strategic, planning level of micromechanisms to broader strategic, planning level of perception, analysis, decision making, learningperception, analysis, decision making, learning

New research in psychology will be valuable in support of New research in psychology will be valuable in support of Augmented Cognition efforts—some focused in the Augmented Cognition efforts—some focused in the context of real-world taskscontext of real-world tasks

Criticality of interdisciplinary workCriticality of interdisciplinary work––but potential difficulties but potential difficulties with building interdisciplinary teams for augmented with building interdisciplinary teams for augmented cognitioncognition

Some low hanging fruit, but also some very difficult Some low hanging fruit, but also some very difficult challenges in science and technology in this arenachallenges in science and technology in this arena

Even applications of some simple well-characterized Even applications of some simple well-characterized limitations could go a long way in enhancing performancelimitations could go a long way in enhancing performance

Potential value of composing interdisciplinary academic Potential value of composing interdisciplinary academic advisory group in support of potential DARPA program advisory group in support of potential DARPA program

Some Findings and Some Findings and RecommendationsRecommendations

The timing is right for building practical The timing is right for building practical systems via pursuit of a deeper science of systems via pursuit of a deeper science of

humanhuman––computer symbiosiscomputer symbiosis

But…But…

Research and innovation will face Research and innovation will face significant challenges and impediments, significant challenges and impediments,

both technically and professionally.both technically and professionally.

Assuming current predictions Assuming current predictions of future importance as same of future importance as same

as the actual future importanceas the actual future importance

All events

All events

Events assessed as important

All events


Events forgotten

All events


Events forgotten

Sensed

SummarySummary

Key uncertainties and human-computer Key uncertainties and human-computer interaction…interaction…

Beliefs & intentionsBeliefs & intentions PreferencesPreferences InitiativeInitiative AttentionAttention

Representations & machinery for Representations & machinery for reasoning under uncertainty provide a reasoning under uncertainty provide a

rich fabric for developing valuable rich fabric for developing valuable software services & experiences.software services & experiences.

uncertainty, action, and interaction eric horvitz microsoft research may 2002 uncertainty, action,...

Documents

user attend

user modelingsmart cockpit

users attention

precision of action

users workload

timing of action

benefit of interaction

benefit of service