models of human performance

Models of Human Models of Human PerformancePerformance

CSCI 4800CSCI 4800

Spring 2006Spring 2006

KraemerKraemer

ObjectivesObjectives

Introduce theory-based models for Introduce theory-based models for predicting human performancepredicting human performance

Introduce competence-based models Introduce competence-based models for assessing cognitive activityfor assessing cognitive activity

Relate modelling to interactive Relate modelling to interactive systems design and evaluationsystems design and evaluation

What are we trying to What are we trying to model?model?

Seven Stage Action ModelSeven Stage Action Model[Norman, 1990][Norman, 1990]

Form intentionDevelop plan

Perform action

Object in world

Evaluate against goalInterpret object

Perceive state of object

GOAL OF PERSON

Describing Problem SolvingDescribing Problem Solving

Initial StateInitial State Goal StateGoal State All possible intervening statesAll possible intervening states

– Problem SpaceProblem Space Path ConstraintsPath Constraints State Action TreeState Action Tree Means-ends analysisMeans-ends analysis

Problem SolvingProblem Solving

A problem is something that doesn’t solve easilyA problem is something that doesn’t solve easily

A problem doesn’t solve easily because:A problem doesn’t solve easily because:– you don’t have the necessary knowledge or,you don’t have the necessary knowledge or,– you have misrepresented part of the problemyou have misrepresented part of the problem

If at first you don’t succeed, try something elseIf at first you don’t succeed, try something else

Tackle one part of the problem and other parts may Tackle one part of the problem and other parts may fall into placefall into place

ConclusionConclusion

More than one solutionMore than one solution

Solution limited by boundary Solution limited by boundary conditionsconditions

Representation affects strategyRepresentation affects strategy

Active involvement and testingActive involvement and testing

Functional FixednessFunctional Fixedness

Strategy developed in one version of Strategy developed in one version of the problemthe problem

Strategy might be inefficientStrategy might be inefficient

X ) XXXXX ) XXXX Convert numerals or just ‘see’ 4Convert numerals or just ‘see’ 4

Data-driven perceptionData-driven perception

Activation of Activation of neural structures neural structures of sensory system of sensory system by pattern of by pattern of stimulation from stimulation from environmentenvironment

Theory-driven perceptionTheory-driven perception

Perception driven by memories and Perception driven by memories and expectations about incoming expectations about incoming information. information.

KEYPOINTKEYPOINT

PERCEPTION involves a set of active PERCEPTION involves a set of active processes that impose: processes that impose:

STRUCTURE, STRUCTURE,

STABILITY, STABILITY,

and and MEANING MEANING

on the worldon the world

Visual IllusionsVisual Illusions

Old Woman or Young girl?

Rabbit or duck?

http://www.genesishci.com/illusions2.htm

InterpretationInterpretation

Knowledge of what you are “looking at” Knowledge of what you are “looking at” can aid in interpretationcan aid in interpretation

JAJA CKAN CKAN DJI DJI LLW LLW ENTU ENTU PTH PTH EHIEHI LLTLLT OFEOFETCHTCH APAAPA ILOILOFWAFWA TERTER

Organisation of information is also usefulOrganisation of information is also useful

Story GrammarsStory Grammars

Analogy with sentence grammarsAnalogy with sentence grammars– Building blocks and rules for combiningBuilding blocks and rules for combining

Break story into propositions Break story into propositions

“ “Margie was holding tightly to the string of her beautiful new Margie was holding tightly to the string of her beautiful new balloon. Suddenly a gust of wind caught it, and carried it balloon. Suddenly a gust of wind caught it, and carried it into a tree. It hit a branch, and burst. Margie cried and into a tree. It hit a branch, and burst. Margie cried and cried.”cried.”

Story GrammarStory Grammar

Story

Setting Episode

Event

Reaction

Internal response

Overt response

ChangeOf state

Event

Event

Event

Event Event Event [sadness]

[1]

[2] [3] [4]

[5]

[6]

InferencesInferences

Comprehension typically requires our Comprehension typically requires our active involvement in order to supply active involvement in order to supply information that is not explicit in the information that is not explicit in the texttext

1. Mary heard the ice-cream van coming1. Mary heard the ice-cream van coming

2. She remembered her pocket money2. She remembered her pocket money

3. She rushed into the house.3. She rushed into the house.

Inference and RecallInference and Recall

Thorndyke (1976): recall of Thorndyke (1976): recall of sentences from ‘Mary’ storysentences from ‘Mary’ story– 85% correct sentence85% correct sentence– 58% correct inference – 58% correct inference –

sentence not presentedsentence not presented

– 6% incorrect inference6% incorrect inference

Mental ModelsMental Models

Van Dijk and Kintsch (1983)Van Dijk and Kintsch (1983)– Text processed to extract propositions, Text processed to extract propositions,

which are held in working memory;which are held in working memory;– When sufficient propositions in WM, then When sufficient propositions in WM, then

linking performed;linking performed;– Relevance of propositions to linking Relevance of propositions to linking

proportional to recall;proportional to recall;– Linking reveals ‘gist’Linking reveals ‘gist’

Semantic NetworksSemantic Networks

ANIMAL

Has SkinCan moveEatsBreathes

BIRDCan fly

Has WingsHas feathers

FISHHas finsCan swimHas gillsCANARY

Is YellowCan sing

Collins & Quillian, 1969

Levels and Reaction timeLevels and Reaction time

A canary is a canary

A canary is a bird

A canary is an animal

A canary is a fish

A canary can sing

A canary can fly

A canary has skin

A canary has gills

Collins & Quillian, 1969

0.9

1

1.1

1.2

1.3

1.4

1.5

0 1 2 False

Levels of Sentences

Mean

Reacti

on

Tim

e (

s)

Property

Category

CanariesCanaries

Different times to verify the Different times to verify the statements:statements:– A canary is a birdA canary is a bird– A canary can flyA canary can fly– A canary can singA canary can sing

Time proportional to movement Time proportional to movement through networkthrough network

Scripts, Schema and FramesScripts, Schema and Frames

Schema = chunks of knowledgeSchema = chunks of knowledge– Slots for information: fixed, default, optionalSlots for information: fixed, default, optional

Scripts = action sequencesScripts = action sequences– Generalised event schema (Nelson, 1986)Generalised event schema (Nelson, 1986)

Frames = knowledge about the properties Frames = knowledge about the properties of thingsof things

Mental ModelsMental Models

PartialPartial

Procedures, Functions or System?Procedures, Functions or System?

Memory or Reconstruction?Memory or Reconstruction?

ConceptsConcepts

How do you know a chair is a chair?How do you know a chair is a chair?

A chair has four legs…does it? A chair has a seat…does it?

Prototypes, Typical Features, and Prototypes, Typical Features, and ExemplarsExemplars

PrototypePrototype ROSCH (1973): people do not use feature sets, but ROSCH (1973): people do not use feature sets, but

imagine a PROTOTYPE for an objectimagine a PROTOTYPE for an object

Typical FeaturesTypical Features ROSCH & MERVIS (1975): people use a list of ROSCH & MERVIS (1975): people use a list of

features, weighted in terms of CUE VALIDITYfeatures, weighted in terms of CUE VALIDITY

ExemplarsExemplars SMITH & MEDIN (1981): people use an EXAMPLE to SMITH & MEDIN (1981): people use an EXAMPLE to

imagine an objectimagine an object

Representing ConceptsRepresenting Concepts

BARSALOU (1983)BARSALOU (1983)– TAXONOMICTAXONOMIC

Categories that are well known and can be recalled Categories that are well known and can be recalled consistently and reliablyconsistently and reliably

– E.g., Fruit, Furniture, AnimalsE.g., Fruit, Furniture, Animals Used to generate overall representation of the worldUsed to generate overall representation of the world

– AD HOCAD HOC Categories that are invented for specific purposeCategories that are invented for specific purpose

– E.g., How to make friends, Moving houseE.g., How to make friends, Moving house Used for goal-directed activity within specific event Used for goal-directed activity within specific event

framesframes

Long Term MemoryLong Term Memory

Procedural Procedural – Knowing howKnowing how

DeclarativeDeclarative– Knowing thatKnowing that

Episodic vs. SemanticEpisodic vs. Semantic– Personal eventsPersonal events– Language and knowledge of worldLanguage and knowledge of world

Working MemoryWorking Memory

Limited CapacityLimited Capacity7 7 ++ 2 items (Miller, 1965) 2 items (Miller, 1965)4 4 ++ 2 chunks (Broadbent, 1972) 2 chunks (Broadbent, 1972)Modality dependent capacityModality dependent capacity

Strategies for coping with limitationStrategies for coping with limitationChunkingChunking InterferenceInterferenceActivation of Long-term memoryActivation of Long-term memory

Central executive

Articulatory control process

Auditory word presentation

Visual word presentation

Phonological store

Visual Cache

Inner scribe

Baddeley’s (1986) Model of Working Memory

Slave SystemsSlave Systems

Articulatory loopArticulatory loop– Memory ActivationMemory Activation– Rehearsal capacityRehearsal capacity

Word length effect and Rehearsal speedWord length effect and Rehearsal speed

Visual cacheVisual cache– Visual patternsVisual patterns– Complexity of pattern, number of elements etcComplexity of pattern, number of elements etc

Inner scribeInner scribe– Sequences of movementSequences of movement– Complexity of movementComplexity of movement

TypingTyping

Eye-hand span related to expertiseEye-hand span related to expertise Expert = 9, novice = 1Expert = 9, novice = 1

Inter-key intervalInter-key interval Expert = 100msExpert = 100ms

StrategyStrategy Hunt & Peck vs. Touch typingHunt & Peck vs. Touch typing

KeystrokeKeystroke Novice = highly variable keystroke timeNovice = highly variable keystroke time Novice = very slow on ‘unusual’ letters, e.g., X or ZNovice = very slow on ‘unusual’ letters, e.g., X or Z

Salthouse (1986)Salthouse (1986)

InputInput– Text converted to chunksText converted to chunks

ParsingParsing– Chunks decomposed to stringsChunks decomposed to strings

TranslationTranslation– Strings into characters and linked to Strings into characters and linked to

movementsmovements ExecutionExecution

– Key pressedKey pressed

Rumelhart & Norman (1982)Rumelhart & Norman (1982)

Perceptual processes Perceptual processes – Perceive text, generate word schemaPerceive text, generate word schema

ParsingParsing– Compute codes for each letterCompute codes for each letter

Keypress schemataKeypress schemata– Activate schema for letter-keypressActivate schema for letter-keypress

Response activationResponse activation– Press defined key through activation of Press defined key through activation of

appropriate hand / fingerappropriate hand / finger

Schematic of Rumelhart and Norman’s Schematic of Rumelhart and Norman’s connectionist model of typingconnectionist model of typing

middlering index

little thumb

Left hand

middleindex ring

thumb little

Right hand

Response system

activation

j a z z

jazzWord node, activated fromVisual or auditory stimulus

Keypress node, breakingWord into typed letters;Excites and inhibits nodes

AutomaticityAutomaticity

Norman and Shallice (1980)Norman and Shallice (1980)Fully automatic processing controlled by Fully automatic processing controlled by

SCHEMATASCHEMATA

Partially automatic processing controlled by Partially automatic processing controlled by either Contention Scheduling either Contention Scheduling

Supervisory Attentional System (SAS)Supervisory Attentional System (SAS)

Supervisory Attentional System Supervisory Attentional System ModelModel

Perceptual System

SupervisoryAttentional

System

Effector System

Contentionscheduling

Triggerdatabase

Control schema

Contention SchedulingContention Scheduling

Gear changing when driving involves many Gear changing when driving involves many routine activities but is performed routine activities but is performed ‘automatically’ – without conscious ‘automatically’ – without conscious awarenessawareness

When routines clash, relative importance is When routines clash, relative importance is used to determine which to perform – used to determine which to perform – Contention SchedulingContention Scheduling

e.g., right foot on brake or clutche.g., right foot on brake or clutch

SAS activationSAS activation

Driving on roundabouts in FranceDriving on roundabouts in France– Inhibit ‘look right’; Activate ‘look left’Inhibit ‘look right’; Activate ‘look left’– SAS to over-ride habitual actionsSAS to over-ride habitual actions

SAS active when:SAS active when:Danger, Choice of response, Novelty etc.Danger, Choice of response, Novelty etc.

Attentional Slips and LapsesAttentional Slips and Lapses

Habitual actions become automaticHabitual actions become automatic SAS inhibits habitSAS inhibits habit PerserverationPerserveration

When SAS does not inhibit and habit proceedsWhen SAS does not inhibit and habit proceeds

DistractionDistraction Irrelevant objects attract attentionIrrelevant objects attract attention Utilisation behaviour: patients with frontal lobe Utilisation behaviour: patients with frontal lobe

damage will reach for object close to hand even when damage will reach for object close to hand even when told not totold not to

Performance Operating Performance Operating CharacteristicsCharacteristics

Resource-dependent trade-off Resource-dependent trade-off between performance levels on two between performance levels on two taskstasks

Task A and Task B performed several Task A and Task B performed several times, with instructions to allocate times, with instructions to allocate more effort to one task or the othermore effort to one task or the other

Task DifficultyTask Difficulty

Data limited processesData limited processesPerformance related to quality of data and Performance related to quality of data and

will not improve with more resourcewill not improve with more resource

Resource limited processesResource limited processesPerformance related to amount of resource Performance related to amount of resource

invested in task and will improve with more invested in task and will improve with more resourceresource

POCPOC

Data limited Data limited Resource limitedResource limited

Cost

Task A

Task B

P

M Task A

Task B

P

MCost

Why Model Performance? Why Model Performance?

Building models can help develop theoryBuilding models can help develop theory– Models make assumptions explicitModels make assumptions explicit– Models force explanationModels force explanation

Surrogate user:Surrogate user:– Define ‘benchmarks’Define ‘benchmarks’– Evaluate conceptual designsEvaluate conceptual designs– Make design assumptions explicitMake design assumptions explicit

Rationale for design decisionsRationale for design decisions

Why Model Performance?Why Model Performance?

Human-computer interaction as Applied Human-computer interaction as Applied ScienceScience– Theory from cognitive sciences used as basis Theory from cognitive sciences used as basis

for designfor design

– General principles of perceptual, motor and General principles of perceptual, motor and cognitive activitycognitive activity

– Development and testing of theory through Development and testing of theory through modelsmodels

Types of Model in HCITypes of Model in HCI

SystemSystem ProgramProgram UserUser ResearcherResearcher DesignerDesigner

ProgramProgram XX

UserUser XX XX

ResearcheResearcherr

XX XX XX XX

DesignerDesigner XX XX XX

Whitefield, 1987

Task ModelsTask Models

Researcher’s Model of User, in terms of Researcher’s Model of User, in terms of taskstasks

Describe typical activitiesDescribe typical activities

Reduce activities to generic sequencesReduce activities to generic sequences

Provide basis for designProvide basis for design

Pros and Cons of ModellingPros and Cons of Modelling

PROSPROS– Consistent description through (semi) formal Consistent description through (semi) formal

representationsrepresentations– Set of ‘typical’ examplesSet of ‘typical’ examples– Allows prediction / description of performanceAllows prediction / description of performance

CONSCONS– Selective (some things don’t fit into models)Selective (some things don’t fit into models)– Assumption of invariabilityAssumption of invariability– Misses creative, flexible, non-standard activityMisses creative, flexible, non-standard activity

Generic Model Process?Generic Model Process?

Define system: {goals, activity, tasks, Define system: {goals, activity, tasks, entities, parameters}entities, parameters}

Abstract to semantic level Abstract to semantic level Define syntax / representationDefine syntax / representation Define interactionDefine interaction Check for consistency and completenessCheck for consistency and completeness Predict / describe performancePredict / describe performance Evaluate resultsEvaluate results Modify modelModify model

Device and Task ModelsDevice and Task Models

Device ModelsDevice Models

Buxton’s 3-state device modelBuxton’s 3-state device model

State0

State1

State2

ApplicationApplication

State0

State1

State2

Out of range

Pen on

Pen off Button up

Button down

select drag

Different pointing devicesDifferent pointing devices

DeviceDevice State0State0 State1State1 State2State2

TouchscreeTouchscreenn

XX

PenPen XX XX XX

JoystickJoystick XX XX

MouseMouse XX XX

ConclusionsConclusions

Models abstract aspects of Models abstract aspects of interactioninteraction– User, task, systemUser, task, system

Models play a variety of roles in Models play a variety of roles in designdesign

Hierarchical Task AnalysisHierarchical Task Analysis

Activity assumed to consist of TASKS Activity assumed to consist of TASKS performed in pursuit of GOALSperformed in pursuit of GOALS

Goals can be broken into SUBGOALS, Goals can be broken into SUBGOALS, which can be broken into taskswhich can be broken into tasks

Hierarchy (Tree) descriptionHierarchy (Tree) description

Hierarchical Task DescriptionHierarchical Task Description

1 .0S w itch on O H P

2 .0C h eck p ro jec tion

3 .0P lace fo il on O H P

4 .0F ocu s p ro jec tion

0 .0P resen t O H P s lid es

The “Analysis” comes from The “Analysis” comes from plansplans

PLANS = conditions for combining tasksPLANS = conditions for combining tasks Fixed SequenceFixed Sequence

– P0: 1 > 2 > exitP0: 1 > 2 > exit Contingent Fixed SequenceContingent Fixed Sequence

– P1: 1 > when state X achieved > 2 > exitP1: 1 > when state X achieved > 2 > exit– P1.1: 1.1 > 1.2 > wait for X time > 1.3 > exitP1.1: 1.1 > 1.2 > wait for X time > 1.3 > exit

Decision Decision – P2: 1 > 2 > If condition X then 3, elseif P2: 1 > 2 > If condition X then 3, elseif

condition Y then 4 > 5 > exitcondition Y then 4 > 5 > exit

Reporting Reporting

HTA can be constructed using Post-it notes HTA can be constructed using Post-it notes on a large space (this makes it easy to edit on a large space (this makes it easy to edit and also encourages participation)and also encourages participation)

HTA can be difficult to present in a HTA can be difficult to present in a succinct printed form (it might be useful to succinct printed form (it might be useful to take a photograph of the Post-it notes)take a photograph of the Post-it notes)

Typically a Tabular format is used:Typically a Tabular format is used:

Task Task numbernumber

TaskTask PlanPlan CommentsComments

Redesigning the Interface to a Redesigning the Interface to a medical imaging systemmedical imaging system

Original DesignOriginal Design

Menu drivenMenu driven

Menus accessed by first Menus accessed by first letter of commandletter of command

Menus arranged in Menus arranged in hierarchyhierarchy

Problems with original designProblems with original design

Lack of consistencyLack of consistencyD = DOS commands; Delete; Data file; DateD = DOS commands; Delete; Data file; Date

Hidden hierarchyHidden hierarchyOnly ‘experts’ could useOnly ‘experts’ could use

Inappropriate defaultsInappropriate defaultsSetting up a scan required ‘correction’ of Setting up a scan required ‘correction’ of

default settings three or four timesdefault settings three or four times

Initial design activityInitial design activity

Observation of non-technology workObservation of non-technology workCytogeneticists inspecting chromosomesCytogeneticists inspecting chromosomes

Developed model of taskDeveloped model of taskHierarchical task analysisHierarchical task analysis

Developed design principles, e.g.,Developed design principles, e.g.,Cytogeneticists as ‘picture people’Cytogeneticists as ‘picture people’Task flowTask flowTask mappingTask mapping

Task ModelTask Model

Work flows between specific activitiesWork flows between specific activities

Patient details Administration

Set up

Reporting

Microscope

Cell sample

Analysis

First “prototype”First “prototype”

Layout related to task model

‘Sketch’ very simple

Annotations showmodifications

Second prototypeSecond prototype

Refined layout

‘Prototype’ usingHyperCard

Initial user trials comparedthis with a mock-up of the original design

Final ProductFinal Product

Picture taken from companybrochure

Initial concepts retained

Further modifications possible

Predicting Transaction Predicting Transaction TimeTime

Predicting Performance TimePredicting Performance Time

Time and error are ‘standard’ Time and error are ‘standard’ measures of human performancemeasures of human performance

Predict transaction time for Predict transaction time for comparative evaluationcomparative evaluation

Approximations of human Approximations of human performanceperformance

Unit TimesUnit Times

From task model, define sequence of From task model, define sequence of tasks to achieve a specific goaltasks to achieve a specific goal

For each task, define ‘average time’For each task, define ‘average time’

Quick ExerciseQuick Exercise

Draw two parallel lines about 4cm apart Draw two parallel lines about 4cm apart and about 10cm longand about 10cm long

Draw, as quickly as possible, a zig-zag line Draw, as quickly as possible, a zig-zag line for 5 secondsfor 5 seconds

Count the number of lines and the number Count the number of lines and the number of times you have crossed the parallel of times you have crossed the parallel lineslines

Predicted resultPredicted result

About 70 linesAbout 70 lines

About 20 cross-oversAbout 20 cross-overs

Why this prediction?Why this prediction?

Movement speed limited by biomechanical Movement speed limited by biomechanical constraintsconstraints– Motor subsystem change direction @ 70msMotor subsystem change direction @ 70ms– So: 5000 / 70 = 71 oscillationsSo: 5000 / 70 = 71 oscillations

Cognitive / Perceptual system cycles:Cognitive / Perceptual system cycles:– Perceptual @ 70msPerceptual @ 70ms– Cognitive @ 100msCognitive @ 100ms– Correction takes 70+70+100 = 240msCorrection takes 70+70+100 = 240ms– 5000/240 = 215000/240 = 21

Fitts’ LawFitts’ Law

Paul Fitts 1954Paul Fitts 1954 Information-theoretic account of Information-theoretic account of

simple movementssimple movements Define the number of ‘bits’ processed Define the number of ‘bits’ processed

in performing a given taskin performing a given task

Fitts’ Tapping TaskFitts’ Tapping Task

W

a

Fitts’ LawFitts’ Law

Movement Time = a + b (logMovement Time = a + b (log22 2A/W) 2A/W)

Hits

60

40

20

0 Log2 (2A/W)

1. A = 62, W = 152. A = 112, W = 73. A = 112, W = 21

1 = 5.3 2 = 4.5 3 = 3.2

54

43

21

a

b

a = 10b = 27.5

Alternate VersionsAlternate Versions

MT = a + b logMT = a + b log22 (2A/W) (2A/W)

MT = b logMT = b log22 (A/W + 0.5) (A/W + 0.5)

MT = a + b logMT = a + b log22 (A/W/+1) (A/W/+1)

a and b are “constants”a and b are “constants”

Data derived from plotData derived from plot

Data as predictors?Data as predictors? aa bb

MouseMouse10301030

108108

-107-107

9696

392392

223223

TrackballTrackball7575

282282300300

347347

Potential ProblemsPotential Problems

Data-fitter rather than ‘law’Data-fitter rather than ‘law’

‘‘Generic value’: a+b = 100Generic value’: a+b = 100

Variable predictive power for devices?Variable predictive power for devices?– From ‘mouse data’ we get:From ‘mouse data’ we get:

(assume A = 5 and W = 10) (assume A = 5 and W = 10) loglog22(2A/W) (2A/W) 0.3 0.3339ms, 150.5ms and 34.9ms (!!)339ms, 150.5ms and 34.9ms (!!)

Hick – Hyman LawHick – Hyman Law

William Hick 1952William Hick 1952

Selection time, from a set of items, is Selection time, from a set of items, is proportional to the number of itemsproportional to the number of items

T = k logT = k log22 (n+1), (n+1), Where k = a constant (intercept+slope)Where k = a constant (intercept+slope)

Approximately 150ms added to T for each itemApproximately 150ms added to T for each item

Example of Hick-Hyman LawExample of Hick-Hyman Law

Search Time (s)

4

3

2

1

0 2 3 4 5 6 7 8 10 12

words

numbers

Landauer and Nachbar, 1985

Keystroke Level ModelsKeystroke Level Models

Developed from 1950s ergonomicsDeveloped from 1950s ergonomics

Human information processor as linear Human information processor as linear executor of specified tasksexecutor of specified tasks

Unit-tasks have defined timesUnit-tasks have defined times

Prediction = summing of times for Prediction = summing of times for sequence of unit-taskssequence of unit-tasks

Building a KLMBuilding a KLM

Develop task modelDevelop task model

Define task sequenceDefine task sequence

Assign unit-times to tasksAssign unit-times to tasks

Sum timesSum times

Example: cut and pasteExample: cut and paste

Task Model: Select line – Cut – Select insertion point Task Model: Select line – Cut – Select insertion point – paste– paste

Task One: select lineTask One: select linemove cursor to move cursor to start of linestart of linepress (hold) buttonpress (hold) buttondrag cursor todrag cursor toend of lineend of linerelease buttonrelease button

Times for MovementTimes for Movement

H: homing, e.g., hand from keyboard to mouseH: homing, e.g., hand from keyboard to mouse– Range: 214ms – 400msRange: 214ms – 400ms– Average: 320msAverage: 320ms

P: pointing, e.g., move cursor using mouseP: pointing, e.g., move cursor using mouse– Range: defined by Fitts’ LawRange: defined by Fitts’ Law– Average: 1100msAverage: 1100ms

B: button pressing, e.g., hitting key on B: button pressing, e.g., hitting key on keyboardkeyboard– Range: 80ms – 700msRange: 80ms – 700ms– Average: 200msAverage: 200ms

Times for Cognition / PerceptionTimes for Cognition / Perception M: mental operationM: mental operation

– Range: 990ms – 1760msRange: 990ms – 1760ms– Average: 1350msAverage: 1350ms

A: switch attention between parts of displayA: switch attention between parts of display– Average: 320msAverage: 320ms

R: recognition of itemsR: recognition of items– Range: 314ms – 1800msRange: 314ms – 1800ms– Average: 340msAverage: 340ms

Perceive change:Perceive change:– Range: 50 – 300msRange: 50 – 300ms– Average: 100msAverage: 100ms

Rules for Summing TimesRules for Summing Times

How to handle multiple Mental units:How to handle multiple Mental units:

– M before Ks in new argument stringsM before Ks in new argument strings– M at start of ‘cognitive unit’M at start of ‘cognitive unit’– M before Ps that select commandsM before Ps that select commands– Delete M if K redundant terminatorDelete M if K redundant terminator

AlternativeAlternative

What if we use ‘accelerated scrolling’ What if we use ‘accelerated scrolling’ on the cursor keys?on the cursor keys?– Press Press key and read scrolling numbers key and read scrolling numbers– Release key at or near numberRelease key at or near number– Select correct numberSelect correct number

M H

Pe

P P’ P

Critical Path ModelsCritical Path Models

Used in project managementUsed in project management

Map dependencies between tasks in Map dependencies between tasks in a projecta project– Task X is dependent on task Y, if it is Task X is dependent on task Y, if it is

necessary to wait until the end of task Y necessary to wait until the end of task Y until task X can commenceuntil task X can commence

ProcedureProcedure

Construct task model, taking into account Construct task model, taking into account dependenciesdependencies

Assign times to tasksAssign times to tasks

Calculate critical path and transaction timeCalculate critical path and transaction time– Run forward passRun forward pass– Run backward passRun backward pass

ExampleExample

M H

R

P P’ P

M = 1.35H = 0.32P = 0.2R = 0.34

1 2 3 4

M1.35

H0.32

P0.2 5

P’0.2

R0.34

6

P0.2

Critical Path TableCritical Path Table

ActivityActivity DuratioDurationn

ESTEST LSTLST EFTEFT LFTLFT FloatFloat

MM 1.351.35 00 00 1.351.35 1.351.35 00

HH 0.320.32 1.351.35 1.351.35 1.671.67 1.671.67 00

PP 0.20.2 1.671.67 1.671.67 1.871.87 1.871.87 00

RR 0.340.34 1.671.67 1.731.73 2.012.01 2.072.07 0.060.06

P’P’ 0.20.2 2.072.07 2.072.07 2.272.27 2.272.27 00

PP 0.20.2 2.272.27 2.272.27 2.472.47 2.472.47 00

ComparisonComparison

‘‘Summing of times’ result:Summing of times’ result:– 2.61s2.61s

‘‘Critical path’ result:Critical path’ result:– 2.47s2.47s

R allowed to ‘float’R allowed to ‘float’

Other time-based modelsOther time-based models

Task-network modelsTask-network models– MicroSAINTMicroSAINT– Unit-times and probability of transitionUnit-times and probability of transition

Prompt50ms

Speak word[300 9]ms

System response[1000 30]ms

p

1-p

Models of CompetenceModels of Competence

Performance vs. CompetencePerformance vs. Competence

Performance ModelsPerformance Models– Make statements and predictions about Make statements and predictions about

the time, effort or likelihood of error the time, effort or likelihood of error when performing specific tasks;when performing specific tasks;

Competence ModelsCompetence Models– Make statements about what a given Make statements about what a given

user knows and how this knowledge user knows and how this knowledge might be organised.might be organised.

Sequence vs. Process vs. Sequence vs. Process vs. GrammarGrammar

Sequence ModelsSequence Models– Define activity simply in terms of sequences of Define activity simply in terms of sequences of

operations that can be quantifiedoperations that can be quantified Process ModelsProcess Models

– Simple model of mental activity but define the Simple model of mental activity but define the steps needed to perform taskssteps needed to perform tasks

Grammatical ModelsGrammatical Models– Model required knowledge in terms of Model required knowledge in terms of

‘sentences’‘sentences’

Process ModelsProcess Models

Production systemsProduction systems

GOMSGOMS

Production SystemsProduction Systems

Rules = (Procedural) KnowledgeRules = (Procedural) Knowledge

Working memory = state of the worldWorking memory = state of the world

Control strategies = way of applying Control strategies = way of applying knowledgeknowledge


Architecture of a production system:Architecture of a production system:

Rule base Working Memory

Interpreter

The Problem of ControlThe Problem of Control

Rules are useless without a useful way to Rules are useless without a useful way to apply themapply them

Need a consistent, reliable, useful way to Need a consistent, reliable, useful way to control the way rules are appliedcontrol the way rules are applied

Different architectures / systems use Different architectures / systems use different control strategies to produce different control strategies to produce different resultsdifferent results

Forward ChainingForward Chaining

A C

AB

C

AB If not C then GOAL

If A then B

If A and B then not C


If not C then GOAL

If A then B

Backward ChainingBackward Chaining

A C

AB

C

AB If A then B


Need: not C

Need B

If not C then GOAL

Need GOAL


If not C then GOAL

If A then B


A simple A simple metaphormetaphor

Docks

Ships


Ships must fit the correct Ships must fit the correct dockdock

When one ship is docked, When one ship is docked, another can be launchedanother can be launched

Production RulesProduction Rules

IF conditionIF conditionTHEN actionTHEN action

e.g., e.g., IF ship is dockedIF ship is docked

And free-floating shipsAnd free-floating shipsTHEN launch shipTHEN launch shipIF dock is freeIF dock is free

And Ship matchesAnd Ship matchesTHEN dock shipTHEN dock ship

The Parsimonious Production The Parsimonious Production Systems Rule NotationSystems Rule Notation

On any cycle, any rule whose conditions are On any cycle, any rule whose conditions are currently satisfied will firecurrently satisfied will fire

Rules must be written so that a single rule will Rules must be written so that a single rule will not fire repeatedlynot fire repeatedly

Only one rule will fire on a cycleOnly one rule will fire on a cycle

All procedural knowledge is explicit in these All procedural knowledge is explicit in these rules rather than being explicit in the interpreterrules rather than being explicit in the interpreter

Worked Example: Worked Example: The Tower of HanoiThe Tower of Hanoi

3

2

1

A B C

4

5

Possible Steps 1Possible Steps 1

Disc 1 from a to cDisc 1 from a to c

Disc 2 from a to bDisc 2 from a to b

Disc 1 from c to aDisc 1 from c to a


Disc 2 from b to cDisc 2 from b to c



3

2

1

A B C

4

5



Disc 1 from c to bDisc 1 from c to b


Disc 1 from b to aDisc 1 from b to a




3

2

1

A B C

45




Disc 2 from b to cDisc 2 from b to c



Disc 1 from c to bDisc 1 from c to b


Disc 4 from b to c

Disc 1 from a to c

Disc 2 from a to b

Disc 1 from c to b

Disc 3 from a to c

Disc 1 from b to a

Disc 2 from b to c

Disc 1 from a to c

Simon’s (1975) goal-recursive Simon’s (1975) goal-recursive logiclogic

To get the 5-tower to Peg C, get the 4-tower to Peg B, then To get the 5-tower to Peg C, get the 4-tower to Peg B, then movemoveThe 5-disc to Peg C, then move the 4-tower to Peg CThe 5-disc to Peg C, then move the 4-tower to Peg C

To get the 4-tower to Peg B, get the 3-tower to Peg C, then To get the 4-tower to Peg B, get the 3-tower to Peg C, then movemoveThe 4-disc to Peg B, then move the 3-tower to Peg BThe 4-disc to Peg B, then move the 3-tower to Peg B

To get the 3-tower to Peg C, get the 2-tower to Peg B, then To get the 3-tower to Peg C, get the 2-tower to Peg B, then movemoveThe 3-disc to Peg C, then move the 2-tower to Peg C, The 3-disc to Peg C, then move the 2-tower to Peg C,

To get the 2-tower to Peg B, move the 1-disc to Peg C, then To get the 2-tower to Peg B, move the 1-disc to Peg C, then move move The 2-disc to Peg B, then move the 1-disc to Peg AThe 2-disc to Peg B, then move the 1-disc to Peg A

Production Rule 1Production Rule 1

SUBGOAL_DISCSSUBGOAL_DISCS

IFIF the goal is to achieve a particular configuration of the goal is to achieve a particular configuration of discs discs

AndAnd Di is on Px but should go to Py in the Di is on Px but should go to Py in the configurationconfiguration

AndAnd Di is the largest disc out of placeDi is the largest disc out of place

AndAnd Dj is on PyDj is on Py

And And Dj is smaller than DiDj is smaller than Di

AndAnd Pz is clear OR has a disc larger than DjPz is clear OR has a disc larger than Dj

THENTHEN set a subgoal to move the Dj tower to Pz and Di to set a subgoal to move the Dj tower to Pz and Di to PyPy

Production Rule 2Production Rule 2

SUBGOAL_MOVE_DISCSUBGOAL_MOVE_DISC

IFIF the goal is to achieve a particular configuration of the goal is to achieve a particular configuration of discs discs

AndAnd Di is on Px but should go to Py in the Di is on Px but should go to Py in the configurationconfiguration

AndAnd Di is the largest disc out of placeDi is the largest disc out of place

AndAnd Py is clearPy is clear

THENTHEN move Di to Pymove Di to Py

Goals Operators Method SelectionGoals Operators Method SelectionCard, Moran and Newell, 1983Card, Moran and Newell, 1983

Human activity modelled by Model Human Human activity modelled by Model Human ProcessorProcessor

Activity defined by GOALSActivity defined by GOALS

Goals held in ‘Stack’Goals held in ‘Stack’

Goals ‘pushed’ onto stack Goals ‘pushed’ onto stack

Goals ‘popped’ from stackGoals ‘popped’ from stack

GoalsGoals

Symbolic structures to define desired Symbolic structures to define desired state of affairs and methods to state of affairs and methods to achieve this state of affairsachieve this state of affairs

GOAL: EDIT-MANUSCRIPTGOAL: EDIT-MANUSCRIPT top level goaltop level goal

GOAL: EDIT-UNIT-TASKGOAL: EDIT-UNIT-TASK specific sub goalspecific sub goal

GOAL: ACQUIRE UNIT-TASKGOAL: ACQUIRE UNIT-TASK get next stepget next step

GOAL: EXECUTE UNIT-TASK do next stepGOAL: EXECUTE UNIT-TASK do next step

GOAL: LOCATION-LINEGOAL: LOCATION-LINE specific stepspecific step

OperatorsOperators

Elementary perceptual, motor or Elementary perceptual, motor or cognitive acts needed to achieve cognitive acts needed to achieve subgoalssubgoals

Get-next-lineGet-next-line

Use-cursor-arrow-methodUse-cursor-arrow-method

Use-mouse-methodUse-mouse-method

MethodsMethods

Descriptions of procedures for Descriptions of procedures for achieving goalsachieving goals

Conditional upon contents of working Conditional upon contents of working memory and state of taskmemory and state of task

GOAL: ACQUIRE-UNIT-TASKGOAL: ACQUIRE-UNIT-TASKGET-NEXT-PAGEGET-NEXT-PAGE if at end of manuscriptif at end of manuscript

GET-NEXT-TASKGET-NEXT-TASK

SelectionSelection

Choose between competing Methods, if Choose between competing Methods, if more than one more than one

GOAL:EXECUTE-UNIT-TASKGOAL:EXECUTE-UNIT-TASK

GOAL:LOCATE-LINEGOAL:LOCATE-LINE[select:[select: if hands on keyboard if hands on keyboard

and less than 5 lines to move and less than 5 lines to move USE CURSOR KEYS USE CURSOR KEYS

elseelseUSE MOUSE]USE MOUSE]

ExampleExample

Withdraw cash from ATMWithdraw cash from ATM– Construct task modelConstruct task model– Define production rulesDefine production rules

Task ModelTask Model

Method for goal: Obtain cash from ATMMethod for goal: Obtain cash from ATMStep1: access ATMStep1: access ATM

Step2: select ‘cash’ optionStep2: select ‘cash’ option

Step3: indicate amountStep3: indicate amount

Step4: retrieve cash and cardStep4: retrieve cash and card

Step5: end taskStep5: end task


((GOAL: USE ATM TO OBTAIN CASH)((GOAL: USE ATM TO OBTAIN CASH)

ADD-UNIT-TASK (access ATM)ADD-UNIT-TASK (access ATM)

ADD-WM-UNIT-TASK (access ATM)ADD-WM-UNIT-TASK (access ATM)

ADD-TASK-STEP (insert card in slot)ADD-TASK-STEP (insert card in slot)

SEND-TO-MOTOR(place card in slot)SEND-TO-MOTOR(place card in slot)

SEND-TO-MOTOR (eyes to slot)SEND-TO-MOTOR (eyes to slot)

SEND-TO-PERCEPTUAL (check card SEND-TO-PERCEPTUAL (check card in)in)

ADD (WM performing card insertion)ADD (WM performing card insertion)

ADD-TASK-STEP (check card insertion)ADD-TASK-STEP (check card insertion)

DELETE-UNIT-TASK (access ATM)DELETE-UNIT-TASK (access ATM)

ADD-UNIT-TASK (enter PIN)ADD-UNIT-TASK (enter PIN)

Problems with GOMSProblems with GOMS

Assumes ‘error-free’ performanceAssumes ‘error-free’ performance– Even experts make mistakesEven experts make mistakes

MHP gross simplifies human MHP gross simplifies human information processinginformation processing

Producing a task model of non-Producing a task model of non-existent products is difficultexistent products is difficult

Task Action GrammarTask Action Grammar

GOMS assumes ‘expert’ knows GOMS assumes ‘expert’ knows operators and methods for tasksoperators and methods for tasks

TAG assumes ‘expert’ knows simple TAG assumes ‘expert’ knows simple tasks, i.e., tasks that can be tasks, i.e., tasks that can be performed without problem-solvingperformed without problem-solving

TAG and competenceTAG and competence

CompetenceCompetence– Defines what an ‘ideal’ user would knowDefines what an ‘ideal’ user would know

TAG relies on ‘world knowledge’TAG relies on ‘world knowledge’– up vs downup vs down– left vs rightleft vs right– forward vs backwardforward vs backward

Task-action GrammarTask-action Grammar

Grammar relates simple tasks to Grammar relates simple tasks to actionsactions

Generic rule schema covering Generic rule schema covering combinations of simple taskscombinations of simple tasks

TAGTAG

A ‘grammar’ A ‘grammar’ – mapsmaps

Simple tasksSimple tasks

– Onto Onto ActionsActions

– To form To form an interaction languagean interaction language

– To investigate To investigate consistencyconsistency

ConsistencyConsistency

Syntactic: use of expressionsSyntactic: use of expressions

Lexical: use of symbolsLexical: use of symbols

Semantic-syntactic alignment: order of termsSemantic-syntactic alignment: order of terms

Semantic: principle of completenessSemantic: principle of completeness

ProcedureProcedure

– Step 1: Write out commands and their structuresStep 1: Write out commands and their structures

– Step 2: Determine in commands have consistent structureStep 2: Determine in commands have consistent structure

– Step 3: Place command items into variable/feature Step 3: Place command items into variable/feature relationshiprelationship

– Step 4: Generalise commands by separating into task Step 4: Generalise commands by separating into task features, simple tasks, task-action rule schemafeatures, simple tasks, task-action rule schema

– Step 5: Expand parts of task into primitivesStep 5: Expand parts of task into primitives

– Step 6: Check to ensure all names are uniqueStep 6: Check to ensure all names are unique

ExampleExample

Setting up a recording on a video-Setting up a recording on a video-cassette recorder (VCR)cassette recorder (VCR)

Assume that all controls via front Assume that all controls via front panel and that the user can only use panel and that the user can only use the up and down arrowsthe up and down arrows

Feature list [for a VCR]Feature list [for a VCR]

PropertyProperty Date, Channel, Start, EndDate, Channel, Start, End ValueValue numbernumber FrequencyFrequency Daily, WeeklyDaily, Weekly RecordRecord on, offon, off

Simple tasksSimple tasks

SetDate [Property = Date, Value = US#, Frequency = Daily]SetDate [Property = Date, Value = US#, Frequency = Daily]

SetDate [Property = Date, Value = US#, Frequency = Weekly]SetDate [Property = Date, Value = US#, Frequency = Weekly]

SetProg[Property =Prog, Value = US#]SetProg[Property =Prog, Value = US#]

SetStart[Property = start, Value = US#, Record = on]SetStart[Property = start, Value = US#, Record = on]

SetEnd[Property = start, Value = US#, Record = off]SetEnd[Property = start, Value = US#, Record = off]

Rule SchemaRule Schema

1. Task[Property = US#, Value] 1. Task[Property = US#, Value] SetValue [Value] SetValue [Value]

2. Task[Property = Date, Value, Frequency = US#] 2. Task[Property = Date, Value, Frequency = US#] SetValue SetValue [Value] + press “[Value] + press “ | | ” until Frequency = US#” until Frequency = US#

3. Task[Property = Start, Value] 3. Task[Property = Start, Value] SetValue [Value] + press SetValue [Value] + press “Rec”“Rec”

4. SetValue [Value = US#] 4. SetValue [Value = US#] press “ press “ | | ” until Value = US#” until Value = US#

5. SetValue[Value = US#] 5. SetValue[Value = US#] use “ use “ | | ” until Value = US#” until Value = US#

Architectures for Architectures for CognitionCognition

Why Cognitive Architecture?Why Cognitive Architecture?

Computers architectures:Computers architectures:– Specify components and their Specify components and their

connectionsconnections– Define functions and processesDefine functions and processes

Cognitive Architectures could be Cognitive Architectures could be seen as the logical conclusion of the seen as the logical conclusion of the ‘human-brain-as-computer’ ‘human-brain-as-computer’ hypothesishypothesis

Why do this?Why do this?

Philosophy: Provide a unified understanding of the mindPhilosophy: Provide a unified understanding of the mind Psychology: Account for experimental dataPsychology: Account for experimental data Education: Provide cognitive models for intelligent Education: Provide cognitive models for intelligent

tutoring systems and other learning environmentstutoring systems and other learning environments Human Computer Interaction: Evaluate artifacts and Human Computer Interaction: Evaluate artifacts and

help in their designhelp in their design Computer Generated Forces: Provide cognitive agents Computer Generated Forces: Provide cognitive agents

to inhabit training environments and gamesto inhabit training environments and games Neuroscience: Provide a framework for interpreting data Neuroscience: Provide a framework for interpreting data

from brain imagingfrom brain imaging

General RequirementsGeneral Requirements Integration of cognition, perception, and actionIntegration of cognition, perception, and action

Robust behavior in the face of error, the Robust behavior in the face of error, the unexpected, and the unknown unexpected, and the unknown

Ability to run in real timeAbility to run in real time

Ability to LearnAbility to Learn

Prediction of human behavior and performancePrediction of human behavior and performance

ArchitecturesArchitectures

Model Human Processor (MHP)Model Human Processor (MHP)– Card, Moran and Newell (1983)Card, Moran and Newell (1983)

ACT-RACT-R– Anderson (1993)Anderson (1993)

EPIC EPIC – Meyer and Kieras (1997) Meyer and Kieras (1997)

SOARSOAR– Laird, Rosenbloom and Newell (1987)Laird, Rosenbloom and Newell (1987)

Model Human ProcessorModel Human Processor

•Three interacting subsystems:

•Perceptual•Auditory image store•Visual image store

•Cognitive•Working memory•Long-term memory

•Motor

Parameters of MHPParameters of MHP

CapacityCapacity DecayDecay CycleCycle

Long-term memoryLong-term memory XX XX

Working memoryWorking memory 2.5 – 9 chunks2.5 – 9 chunks 5 – 226s5 – 226s

Auditory image storeAuditory image store 7 – 17 letters7 – 17 letters 70-70-1000ms1000ms

Visual image storeVisual image store 4.4-6.2 letters4.4-6.2 letters 900-900-3500ms3500ms

Cognitive processorCognitive processor 50-200ms50-200ms

Motor processorMotor processor 25-170ms25-170ms

Perceptual processorPerceptual processor 30-100ms30-100ms

Average data for MHPAverage data for MHP

Long-term memory:Long-term memory: ?? Working memory:Working memory: 3 – 7 chunks, 7s 3 – 7 chunks, 7s Auditory image store: 17 letters, 200msAuditory image store: 17 letters, 200ms Visual image store: 5 letters, 1500msVisual image store: 5 letters, 1500ms Cognitive processor: 100msCognitive processor: 100ms Perceptual processor: 70msPerceptual processor: 70ms Motor processor: 70msMotor processor: 70ms


Simple description of cognitionSimple description of cognition

Uses ‘standard times’ for predictionUses ‘standard times’ for prediction

Uses production rules for defining Uses production rules for defining and combining tasks (with GOMS and combining tasks (with GOMS formalism)formalism)

Adaptive Control of Thought, Adaptive Control of Thought, Rational (ACT-R)Rational (ACT-R)

http://act.psy.cmu.eduhttp://act.psy.cmu.edu

Adaptive Control of Thought, Rational Adaptive Control of Thought, Rational (ACT-R)(ACT-R)

ACT-R symbolic aspect realised over ACT-R symbolic aspect realised over subsymbolic mechanismsubsymbolic mechanism

Symbolic aspect in two parts:Symbolic aspect in two parts:– Production memoryProduction memory– Symbolic memory (declarative memory)Symbolic memory (declarative memory)

Theory of rational analysisTheory of rational analysis

Theory of Rational AnalysisTheory of Rational Analysis

Evidence-based assumptions about Evidence-based assumptions about environment (probabilities)environment (probabilities)

Deriving optimal strategies (Bayesian)Deriving optimal strategies (Bayesian)

Assuming that optimal strategies reflect Assuming that optimal strategies reflect human cognition (either what it actually human cognition (either what it actually does or what it probably ought to do)does or what it probably ought to do)

Notions of MemoryNotions of Memory

Procedural Procedural – Knowing howKnowing how– Described in ACT by Production RulesDescribed in ACT by Production Rules

DeclarativeDeclarative– Knowing thatKnowing that– Described in ACT by ‘chunks’Described in ACT by ‘chunks’

Goal StackGoal Stack– A sort of ‘working memory’A sort of ‘working memory’– Holds chunks (goals)Holds chunks (goals)– Top goal pushed (like GOMS)Top goal pushed (like GOMS)– Writeable Writeable


Knowing how to do XKnowing how to do X– Production rule = set of conditions and Production rule = set of conditions and

an actionan action

IF it is rainingIF it is rainingAnd you wish to go outAnd you wish to go out

THEN pick up your umbrellaTHEN pick up your umbrella

(Very simple) ACT(Very simple) ACT

Network of propositionsNetwork of propositions

Production rules selected via pattern Production rules selected via pattern matching. Production rules coordinate matching. Production rules coordinate retrieval of chunks from symbolic memory retrieval of chunks from symbolic memory and link to environment.and link to environment.

If information in working memory matches If information in working memory matches production rule condition, then fire production rule condition, then fire production ruleproduction rule

ACT*ACT*

Declarativememory

Proceduralmemory

Working memory

Retrieval Storage Match Execution

OUTSIDE WORLD

Encoding Performance

Addition-Factsix U (4); T (1); H (0)

eight

addend1 sum

addend2

Knowledge Representation

1618 +_____34_____1

Goal buffer: add numbers in right-most columnVisual buffer: 6, 8Retrieval buffer: 14

Symbolic / Subsymbolic levelsSymbolic / Subsymbolic levels

Symbolic levelSymbolic level– Information as chunks in declarative memory, Information as chunks in declarative memory,

and represented as propositionsand represented as propositions– Rules as productions in procedural memoryRules as productions in procedural memory

Subsymbolic levelSubsymbolic level– Chunks given parameters which are used to Chunks given parameters which are used to

determine the probability that the chunk is determine the probability that the chunk is neededneeded

– Base-level activation (relevance)Base-level activation (relevance)– Context activation (association strengths)Context activation (association strengths)

Conflict resolutionConflict resolution

Order production rules by preferenceOrder production rules by preference Select top rule in listSelect top rule in list Preference defined by:Preference defined by:

– Probability that rule will lead to goalProbability that rule will lead to goal– Time associated with ruleTime associated with rule– Likely cost of reaching goal when using Likely cost of reaching goal when using

sequence involving this rulesequence involving this rule

ExampleExample

Activity: Find target and then use mouse Activity: Find target and then use mouse to select target:to select target:

Hunt_FeatureHunt_FeatureIF goal = find target with feature FIF goal = find target with feature FAND there is object X on screenAND there is object X on screenTHEN move attention to object XTHEN move attention to object X

Found_targetFound_targetIF goal = find target with feature FIF goal = find target with feature FAND target with F in location LAND target with F in location LTHEN move mouse to L and clickTHEN move mouse to L and click

ExampleExample

Model reaction time to targetModel reaction time to target– Assume switch attention linearly increases with Assume switch attention linearly increases with

each new positioneach new position– Assume probability of feature X in location y = Assume probability of feature X in location y =

0.530.53– Assume switch attention = 185msAssume switch attention = 185ms

Therefore, reaction time = 185 X 0.53 = Therefore, reaction time = 185 X 0.53 = 98ms per position98ms per position

Empirical data has RT of 103ms per positionEmpirical data has RT of 103ms per position

ExampleExample

Assume target in field of distractorsAssume target in field of distractors– P = 0.42P = 0.42– Therefore, 185 x .42 = 78ms per Therefore, 185 x .42 = 78ms per

positionposition

Empirical data = 80ms per positionEmpirical data = 80ms per position

LearningLearning

Symbolic levelSymbolic level– Learning defined by adding new chunks Learning defined by adding new chunks

and productionsand productions

Subsymbolic levelSubsymbolic level– Adjustment of parameters based on Adjustment of parameters based on

experienceexperience


ACT uses simple production systemACT uses simple production system

ACT provides some quantitative ACT provides some quantitative prediction of performanceprediction of performance

Rationality = optimal adaptation to Rationality = optimal adaptation to environmentenvironment

Executive Process Interactive Executive Process Interactive Control (EPIC)Control (EPIC)

ftp://ftp.eecs.umich.edu/people/kierasftp://ftp.eecs.umich.edu/people/kieras

Executive Process Interactive Control Executive Process Interactive Control (EPIC) (EPIC)

Focus on multiple task performanceFocus on multiple task performance

Cognitive Processor runs production Cognitive Processor runs production rules and interacts with perceptual rules and interacts with perceptual and motor processors and motor processors

EPIC parametersEPIC parameters

FIXEDFIXED– Connections and mechanismsConnections and mechanisms– Time parametersTime parameters– Feature sets for motor processorsFeature sets for motor processors– Task-specific production rules and perceptual Task-specific production rules and perceptual

encoding typesencoding types FREEFREE

– Production rules for tasksProduction rules for tasks– Unique perceptual and motor processorsUnique perceptual and motor processors– Task instance setTask instance set– Simulated task environmentSimulated task environment

EPICEPIC

Task environment

Auditory

Visual

Speech

Manual

DISPLAYPERCEPTUALPROCESSORS

Auditory

Visual

Speech

Manual

Long-termmemory

Productionmemory

ProductionRule interpreter

Workingmemory

Tactile

Production MemoryProduction Memory

Perceptual processors controlled by Perceptual processors controlled by production rulesproduction rules

Production Rules held in Production Production Rules held in Production MemoryMemory

Production Rule Interpreter applies Production Rule Interpreter applies rules to perceptual processesrules to perceptual processes

Working MemoryWorking Memory

Limited capacity (or duration of 4s) Limited capacity (or duration of 4s) and holds current production rulesand holds current production rules

Cognitive processor updates every Cognitive processor updates every 50ms50ms

On update, perceptual input, item On update, perceptual input, item from production memory, and next from production memory, and next action held in working memoryaction held in working memory

Resolving ConflictResolving Conflict

Production rules applied to executive Production rules applied to executive tasks to handle resource conflict and tasks to handle resource conflict and schedulingscheduling

Conflict dealt with in production rule Conflict dealt with in production rule specificationspecification– LockoutLockout– InterleavingInterleaving– Strategic response deferentStrategic response deferent

ExampleExample

Task one

Stimulus one

Perceptual process

Cognitive process

Response selection

Memory process

Response one

Task two

Stimulus two

Perceptual process

Cognitive process

Response selection

Memory process

Response two

Executive process

Move eye to S2Enable task1 + task 2

Wait for task1 complete

Task1end

Task2 permission

Trial end


Modular structure supports parallelismModular structure supports parallelism

EPIC does not have a goal stack and does EPIC does not have a goal stack and does not assume sequential firing of goalsnot assume sequential firing of goals

Goals can be handled in parallel (provided Goals can be handled in parallel (provided there is no resource conflict)there is no resource conflict)

Does not support learningDoes not support learning

States, Operators, And States, Operators, And Reasoning (SOAR)Reasoning (SOAR)

http://www.isi.edu/soar/soar.htmlhttp://www.isi.edu/soar/soar.html

States, Operators, And Reasoning States, Operators, And Reasoning (SOAR)(SOAR)

Sequel of General Problem Solver (Newell Sequel of General Problem Solver (Newell and Simon, 1960)and Simon, 1960)

SOAR seeks to apply SOAR seeks to apply operatorsoperators to to statesstates within a within a problem spaceproblem space to achieve a to achieve a goalgoal. .

SOAR assumes that actor uses all SOAR assumes that actor uses all available knowledge in problem-solvingavailable knowledge in problem-solving

Soar as a Unified Theory of Soar as a Unified Theory of CognitionCognition

Intelligence = problem solving + Intelligence = problem solving + learninglearning

Cognition seen as search in problem Cognition seen as search in problem spacesspaces

All knowledge is encoded as All knowledge is encoded as productions productions a single type of knowledgea single type of knowledge

All learning is done by chunking All learning is done by chunking a single type of learninga single type of learning

Young, R.M., Ritter, F., Jones, G. 1998 Young, R.M., Ritter, F., Jones, G. 1998

"Online Psychological Soar Tutorial" "Online Psychological Soar Tutorial"

available at: available at: http://www.psychology.nottingham.ac.uk/staff/http://www.psychology.nottingham.ac.uk/staff/Frank.Ritter/pst/pst-tutorial.htmlFrank.Ritter/pst/pst-tutorial.html

SOAR ActivitySOAR Activity

Operators:Operators: Transform a state via some action Transform a state via some action

State:State: A representation of possible stages of A representation of possible stages of progress in the problemprogress in the problem

Problem space:Problem space: States and operators that States and operators that can be used to achieve a goal.can be used to achieve a goal.

Goal:Goal: Some desired situation. Some desired situation.

SOAR ActivitySOAR Activity Problem solving = applying an Operator to a Problem solving = applying an Operator to a

State in order to move through a Problem Space State in order to move through a Problem Space to reach a Goal. to reach a Goal.

Impasse = Where an Operator cannot be applied Impasse = Where an Operator cannot be applied to a State, and so it is not possible to move to a State, and so it is not possible to move forward in the Problem Space. This becomes a forward in the Problem Space. This becomes a new problem to be solved.new problem to be solved.

Soar can Soar can learnlearn by storing solutions to past by storing solutions to past problems as problems as chunkschunks and applying them when it and applying them when it encounters the same problem againencounters the same problem again

SOAR ArchitectureSOAR ArchitectureChunkingmechanism

Production memory

Pattern ActionPattern ActionPattern Action

Decision procedure

Working memoryManager

Preferences Objects

Conflict stack

Working memory

ExplanationExplanation

Working MemoryWorking Memory– Data for current activity, organized into Data for current activity, organized into

objectsobjects Production MemoryProduction Memory

– Contains production rulesContains production rules Chunking mechanismChunking mechanism

– Collapses successful sequences of Collapses successful sequences of operators into chunks for re-useoperators into chunks for re-use

3 levels in soar3 levels in soar

Symbolic – the programming levelSymbolic – the programming level– Rules programmed into Soar that match Rules programmed into Soar that match

circumstances and perform specific actionscircumstances and perform specific actions Problem space – states & goalsProblem space – states & goals

– The set of goals, states, operators, and The set of goals, states, operators, and context.context.

Knowledge – embodied in the rulesKnowledge – embodied in the rules– The knowledge of how to act on the The knowledge of how to act on the

problem/world, how to choose between problem/world, how to choose between different operators, and any learned chunks different operators, and any learned chunks from previous problem solvingfrom previous problem solving

How does it work?How does it work?

A problem is encoded as a current state A problem is encoded as a current state and a desired state (goal)and a desired state (goal)

Operators are applied to move from one Operators are applied to move from one state to anotherstate to another

There is success if the desired state There is success if the desired state matches the current statematches the current state

Operators are proposed by productions, Operators are proposed by productions, with preferences biasing choices in with preferences biasing choices in specific circumstancesspecific circumstances

Productions fire in parallelProductions fire in parallel

ImpassesImpasses

If no operator is proposed, or if there is a If no operator is proposed, or if there is a tie between operators, or if Soar does not tie between operators, or if Soar does not know what to do with an operator, there is know what to do with an operator, there is an impassean impasse

When there are impasses, Soar sets a new When there are impasses, Soar sets a new goal (resolve the impasse) and creates a goal (resolve the impasse) and creates a new statenew state

Impasses may be stackedImpasses may be stacked When one impasse is solved, Soar pops up When one impasse is solved, Soar pops up

to the previous goalto the previous goal

LearningLearning

Learning occurs by chunking the Learning occurs by chunking the conditions and the actions of the conditions and the actions of the impasses that have been resolvedimpasses that have been resolved

Chunks can immediately used in Chunks can immediately used in further problem-solving behaviourfurther problem-solving behaviour

The Switchyard videoThe Switchyard video


It may be too "unified"It may be too "unified"– Single learning mechanismSingle learning mechanism– Single knowledge representationSingle knowledge representation– Uniform problem stateUniform problem state

It does not take neuropsychological It does not take neuropsychological evidence into account (cf. ACT-R)evidence into account (cf. ACT-R)

There may be non-symbolic intelligence, There may be non-symbolic intelligence, e.g. neural nets etc not abstractable to the e.g. neural nets etc not abstractable to the symbolic levelsymbolic level

Comparison of ArchitecturesComparison of Architectures

ACT-RACT-R EPICEPIC SOARSOARTypeType HybridHybrid SymbolicSymbolic SymbolicSymbolic

TheoryTheory Rational Rational analysisanalysis

Embedded Embedded cognitioncognition

Problem Problem solvingsolving

BasisBasis Cog. Psy.Cog. Psy. HCIHCI AIAI

LTMLTM Productions; Productions; factsfacts

Productions; Productions; factsfacts

ProductionsProductions

WMWM Goal stackGoal stack Working Working memory; memory; sensory storessensory stores

Working Working memorymemory

LearningLearning YesYes NoNo YesYes

The Role of Models in The Role of Models in DesignDesign

User Models in DesignUser Models in Design

BenchmarkingBenchmarking

Human Virtual MachinesHuman Virtual Machines

Evaluation of conceptsEvaluation of concepts

Comparison of conceptsComparison of concepts

Analytical prototypingAnalytical prototyping

BenchmarkingBenchmarking

What times can users expect to take What times can users expect to take to perform taskto perform task

– Training criteriaTraining criteria– Evaluation criteria (under ISO9241)Evaluation criteria (under ISO9241)– Product comparisonProduct comparison

Human Virtual MachineHuman Virtual Machine

How might the user perform?How might the user perform?

– Make assumptions explicitMake assumptions explicit– Contrast viewsContrast views

Evaluation of ConceptsEvaluation of Concepts

Which design could lead to better Which design could lead to better performance?performance?

– Compare concepts using models prior to Compare concepts using models prior to building prototypebuilding prototype

– Use performance of existing product as Use performance of existing product as benchmarkbenchmark

Reliability of ModelsReliability of Models

Agreement of predictions with Agreement of predictions with observationsobservations

Agreement of predictions by different Agreement of predictions by different analystsanalysts

Agreement of model with theoryAgreement of model with theory

Comparison with TheoryComparison with Theory

Approximation of human information processingApproximation of human information processing

Assumes linear, error-free performanceAssumes linear, error-free performance

Assumes strict following of ‘correct’ procedureAssumes strict following of ‘correct’ procedure

Assumes only way correct procedureAssumes only way correct procedure

Assumes actions can be timedAssumes actions can be timed

KLM ValidityKLM Validity

Predicted values liewithin 20% of observed values

Comparison of KLM predicted Comparison of KLM predicted with times from user trialswith times from user trials

Total time(s)

25

20

15

101 2 3 4 5 6 7

Trial number

CUI: P = 15.84s mean = 15.37sError = 2.9%

GUI: P = 11.05s mean = 8.64sError = 22%

Inter / Intra-rater ReliabilityInter / Intra-rater Reliability

Inter-rater:Inter-rater:– Correlation of several analystsCorrelation of several analysts– = 0.754= 0.754

Intra-rater:Intra-rater:– Correlation for same analysts on several Correlation for same analysts on several

occasionsoccasions– =0.916=0.916

Validity:Validity:– correlation with actual performancecorrelation with actual performance– = 0.769= 0.769

Stanton and Young, 1992

How compare single data How compare single data points?points?

Models typically produce a single Models typically produce a single predictionprediction

How can one value be compared How can one value be compared against a set of data?against a set of data?

How can a null hypothesis be How can a null hypothesis be proved?proved?

Liao and Milgram (1991)Liao and Milgram (1991)

A-D-*sd A-D A-D+*sd A A+D-*sd A+D A+D+*sd D

Defining termsDefining terms

A = Actual values, with observed A = Actual values, with observed standard deviation (sd)standard deviation (sd)

D = Derived valuesD = Derived values = 5% (P < 0.05 to reduce Type I = 5% (P < 0.05 to reduce Type I

error)error) = 20% (P<0.2 for Type II error)= 20% (P<0.2 for Type II error)

Acceptance CriteriaAcceptance Criteria

Accept Ho if: A-D+ Accept Ho if: A-D+ *sd < D< A+D- *sd < D< A+D- *sd*sd

Reject Ho if: D < A-D- Reject Ho if: D < A-D- *sd *sd

Reject Ho if: D > A-D+ Reject Ho if: D > A-D+ *sd *sd

Analytical PrototypingAnalytical Prototyping

Functional analysisFunctional analysis Define features and functionsDefine features and functions Development of design concepts, e.g., sketches and Development of design concepts, e.g., sketches and

storyboardsstoryboards

Scenario-based analysisScenario-based analysis How people pursue defined goalsHow people pursue defined goals State-based descriptionsState-based descriptions

Structural analysisStructural analysis Predictive evaluationPredictive evaluation Testing to destructionTesting to destruction

Analytical PrototypingAnalytical Prototyping

Functional analysisFunctional analysis

Scenario-based analysisScenario-based analysis

Structural analysisStructural analysis

Rewritable RoutinesRewritable Routines

Mental modelsMental models– Imprecise, incomplete, inconsistentImprecise, incomplete, inconsistent

Partial representations of product Partial representations of product and procedure for achieving subgoaland procedure for achieving subgoal

Knowledge recruited in response to Knowledge recruited in response to system imagesystem image

Simple ArchitectureSimple Architecture

Current State

Action to change

machine state

Rewritable Routines

Goal State

Possible States

Relevant State

Next State

Global Prototypical RoutinesGlobal Prototypical Routines

Stereotyped Stimulus-Response Stereotyped Stimulus-Response compatibilitiescompatibilities

Generalisable product knowledgeGeneralisable product knowledge

State-specific RoutinesState-specific Routines

Interpretation of system imageInterpretation of system image– Feature evolutionFeature evolution

Expectation of procedural stepsExpectation of procedural steps

Situated / Opportunistic planningSituated / Opportunistic planning

Describing InteractionDescribing Interaction

State-space diagramsState-space diagrams

Indication of system imageIndication of system image

Indication of user actionIndication of user action

Prediction of performancePrediction of performance

State-space DiagramState-space Diagram

0

Waiting for: Raise lid

Waiting for: Play Mode

Waiting for: Enter

Waiting for: Skip forward

Waiting for: Skip back

Waiting for: Play Waiting for: Stop

Waiting for: Off

Task: Press ‘Play’ Time: 200msError: 0.0004

State 1

• State number•System image•Waiting for…•Transitions

Defining ParametersDefining Parameters

Activity (times)Activity (times) ErrorError P(noviceP(novice))

P(expert)P(expert)

Recall Plan (1380ms)Recall Plan (1380ms) Wrong planWrong plan 0.260.26 0.0030.003

Select (360ms)Select (360ms) Select Select wrong itemwrong item

0.020.02 0.00040.0004

Press (200ms)Press (200ms) Fail to pressFail to press 0.00040.0004 0.00040.0004

Read (180ms)Read (180ms) MisinterpretMisinterpret 0.160.16 0.090.09

Developing ModelsDeveloping ModelsP=0.997P=0.74

P=0.003P=0.26

P=0.9996P=0.9996

P=0.9996P=0.9996

P=0.9996P=0.9996

P=0.0004P=0.0004

P=0.0004P=0.0004

P=0.0004P=0.0004

P=1P=1

P=1P=1

P=1P=1

Recall plan:1380ms

Press play:200ms

Press Playmode:200ms

Wrong plan:1380ms

Cycle through menu:800ms

Switch off:

300ms

Press Enter:0ms

Press Other Key:200ms

Press Playmode:200ms

Press Play:0ms

Start:0ms

ResultsResults

0500

100015002000250030003500400045005000

'novice' t1 t3 t5 'expert'

What is the point?What is the point?

Are these models useful to Are these models useful to designers?designers?

Are these models useful to theorists?Are these models useful to theorists?

Task Models - problemsTask Models - problems

Task models take time to developTask models take time to develop– They may not have high inter-rater They may not have high inter-rater

reliabilityreliability– They cannot deal easily with parallel They cannot deal easily with parallel

taskstasks– They ignore social factorsThey ignore social factors

Task Models - benefitsTask Models - benefits

Models are abstractions – you always Models are abstractions – you always leave something outleave something out

The process of creating a task model The process of creating a task model might outweigh the problemsmight outweigh the problems

Task models highlight task Task models highlight task sequences and can be used to define sequences and can be used to define metricsmetrics

Task Models for TheoristsTask Models for Theorists

Task models are engineering Task models are engineering approximationsapproximations– Do they actually describe how human Do they actually describe how human

information processing works?information processing works?Do they need to?Do they need to?

– Do they describe cognitive operations, Do they describe cognitive operations, or just actions?or just actions?

Some Background ReadingSome Background Reading

Dix, A et al., 1998, Human-Computer Interaction Dix, A et al., 1998, Human-Computer Interaction (chapters 6 and 7) London: Prentice Hall(chapters 6 and 7) London: Prentice Hall

Anderson, J.R., 1983, The Architecture of Cognition, Anderson, J.R., 1983, The Architecture of Cognition, Harvard, MA: Harvard University PressHarvard, MA: Harvard University Press

Card, S.K. et al., 1983, The Psychology of Human-Card, S.K. et al., 1983, The Psychology of Human-Computer Interaction, Hillsdale, NJ: LEAComputer Interaction, Hillsdale, NJ: LEA

Carroll, J., 2003, HCI Models, Theories and Carroll, J., 2003, HCI Models, Theories and Frameworks: towards a multidisciplinary science, Frameworks: towards a multidisciplinary science, (chapters 1, 3, 4, 5) San Francisco, CA: Morgan (chapters 1, 3, 4, 5) San Francisco, CA: Morgan KaufmanKaufman

models of human performance

Documents

flya canary

birda canary

skina canary

singa canary

canarya canary

fisha canary

animala canary

reaction timea canary