the growth of cognitive modeling in human computer interaction since goms judith reitman olson and...

The Growth Of Cognitive Modeling in Human Computer Interaction

Since GOMS

Judith Reitman Olson and Gary M. Olson

The University of Michigan

Presenters: Tosin Aiyelokun and Norman Makoto Su

Outline Cognitive Modeling Introduction to GOMS GOMS Extensions Modeling Specific Components GOMS Limitations Summary

Cognitive Modeling: Definition A theory that produces a computational model

of how people perform tasks and solve problems by using psychological principles and empirical studies.

Cognitive Modeling: Research Methods

EmpiricalEmpirical MethodsMethods

ProgrammingProgrammingTechniquesTechniques

Formal AnalysisFormal Analysis

Philosophy,

Logic,

Linguistics,

Mathematics,

Computer

Science

Experimental Psychology, Neuroscience

Artificial Intelligence

Cognitive Modeling: Role Limits the design space Answers specific design decisions Estimates total task time Estimates training time Identifies complex, error-prone stages of the

design A means of testing current psychological

theories

Cognitive Modeling: Human Information Processor (HIP)

ReceptorsReceptors

(perception)(perception)

EffectorsEffectors

(motor actions)(motor actions)

ProcessorProcessor

MemoryMemory

External World

HIP

The Human Processor Model

Perceptual Processor-sensory input (audio & visual)-code info symbolically -output into audio and visual image storage (WM buffer)

Cognitive Processor-input from sensory buffers-access LTM to determine response -output response into WM

Motor Processor-input response from WM-carry out response

Cognitive Modeling: Applications GOMS

Today’s presentation Soar

Integrated architecture for knowledge-based problem solving, learning and interacting with external environments.

ACT-R Atomic Components of Thoughts - Rational

GOMS: Overview Formal representation of routine cognitive

skill. A description of knowledge required by an

expert user to perform a specific task. Provides a description of what the user must

learn.

GOMS: Classification Provides a predictive, descriptive and

prescriptive model Predictive

Predicts the time it will take user to perform the tasks under analysis

Descriptive Represents the way a user performs tasks on a system

Prescriptive Guides the development of training programs and help

systems

GOMS: Definition GOMS models user’s behavior in terms of:

Goals What the user wants to do.

Operators Specific steps a user is able to take and assigned a specific execution

time.

Methods Well-learned sequences of subgoals and operators that can accompli

sh a goal.

Selection Rules Guidelines for deciding between multiple methods.

GOMS: A Family of Models Keystroke-Level Model (KLM) Card, Moran, and Newell (CMN-GOMS) Natural GOMS Language (NGOMSL) Cognitive-Perceptual-Motor GOMS

(CPM-GOMS)

GOMS: Keystroke-Level Model (KLM) Simplest GOMS technique

The basis for all other GOMS techniques Predicts execution time

Requires analyst-supplied methods Assumes that routine cognitive skills can be decompose

d into a serial sequence of basic cognitive operations and motor activities, which are: K: A keystroke (280 msec) M: A single mental operator (1350 msec) P: Pointing to a target on a small display (1100 msec) H: Moving hands from the keyboard to a mouse (400 msec)

Top-level Goal: Edit Manuscript (move “quick brown” to before “fox”)

Subgoal: Highlight text

Operators: Move-mouse Click mouse-button

Type characters (keyboard shortcuts)

Methods: 1. Delete-word-and-retype (retype method) 2. Cut-and-paste-using-keyboard-shortcuts (shortcuts method) 3. Cut-and-paste-using menus (menus

method)

Selection Rules: If the text to be moved is one or two characters long, use retype method

Else, if remember shortcuts, use shortcuts method

Else, use the menus

method

KLM Example

Description Operator Duration (sec)

Mentally Prepare M 1.35

Move cursor to “quick” P 1.10

Double-click mouse button K 0.40

Move cursor to “brown” P 1.10

Shift-click mouse button K 0.40


Move cursor to Edit Menu P 1.10

Click mouse button K 0.20

Move cursor to Cut menu item P 1.10



Move cursor to before “fox” P 1.10



Move cursor to Edit menu P 1.10


Move cursor to Paste menu item P 1.10


TOTAL PREDICTED TIMETOTAL PREDICTED TIME 14.9014.90

Method Used

Cut-and-paste-using-menus

1

2

3

4

5

M=1.35P=1.10K=0.20

GOMS: Card, Moran, and Newell (CMN-GOMS)

Subgoal invocations and method selection are predicted by the model given the task situation

In program form – analysis is general and executable

Predicts operator sequence and execution time Based directly on the Model Human Processor

Description Duration (sec)

GOAL: MOVE-TEXT

…….GOAL: CUT-TEXT

……………GOAL: HIGHLIGHT-TEXT

………………..MOVE-CURSOR-TO-BEGINNING 1.10

………………..CLICK-MOUSE-BUTTON 0.20

………………..MOVE-CURSOR-TO-END 1.10

………………..SHIFT-CLICK-MOUSE-BUTTON 0.48

………………..VERIFY-HIGHLIGHT 1.35

……….........GOAL: ISSUE-CUT-COMMAND

…………………MOVE-CURSOR-TO-EDIT-MENU

…………………PRESS-MOUSE-BUTTON 0.10

…………………MOVE-MOUSE-TO-CUT-ITEM 1.10

…………………VERIFY-HIGHLIGHT 1.35

…………………RELEASE-MOUSE-BUTTON 0.10

………GOAL: PASTE-TEXT

…………….GOAL: POSITION-CURSOR-AT-INSERTION-POINT

…………………MOVE-CURSOR-TO-INSERTION-POINT 1.10

…………………CLICK-MOUSE-BUTTON 0.20

…………………VERIFY-POSITION 1.35

…………….GOAL: ISSUE-PASTE-COMMAND

…………………MOVE-CURSOR-TO-EDIT-MENU

…………………PRESS-MOUSE-BUTTON 0.10

…………………MOVE-MOUSE-TO-PASTE-ITEM

…………………VERIFY-HIGHLIGHT 1.35

…………………RELEASE-MOUSE-BUTTON 0.10


CMN-GOMSCMN-GOMS

Extending GOMS: Grammars Explicitly represent knowledge a user needs to

translate from goals to actions. Task-Action-Grammar (TAG) by Payne and

Green Model of content knowledge rather than a full

system to generate user performance estimation. However, we can measure by number of rules.

Extending GOMS: Grammars TAG Example for EMACS:

Task[Direction, Unit] Symbol[Direction] + Letter[unit] Symbol[forward] “cntl” Symbol[backward] “meta” Letter[word] “W” Letter[character] “C”

Task: Move one word forward. Task[forward, word]

Symbol[forward] + Letter[word] “cntl” + “W” “cntl-W”

Extending GOMS: Production Systems Like grammars but models a goal stack and

working memory. Tedious to write but can be fed into a program

to automatically check for completeness and accuracy.

Can predict errors and learning time behavior.

Extending GOMS: Production Systems Production to see if a closing

JOIN statement is needed:

Rule 1: (StartUp.SeeifJoinNeededIF ((GOAL SeeIfJoinNeeded)

(NOT(NOTE SeeingIfJoinNeeded TRUE))THEN ((Add NOTE SeeingIfJoinNeeded TRUE)

(Add STEP CountTables)))

Rule 2: (CountTables((DoTask Count NumberOfTables *NumberOfTables)(Add NOTE NumberOfTables *NumberOfTables)(Delete STEP CountTables)(Add Step AddJoinNote)))

Insert intoWorking Memory

Delete from Working Memory

Extending GOMS: Learning How to estimate time to learn? One solution: Soar (UMICH)

From the FAQ: “Soar has also been used for modeling learning in many of these tasks; however, learning adds significant complexity to the structuring of the task…”

Extending GOMS:Natural GOMS Language (NGOMSL) Structured natural language notation Based directly on the Cognitive Complexity T

heory (Kieras and Polson) Allows GOMS to model working memory (WM) a

nd setup subgoals Unlike CMN-GOMS, provides quantitative predicti

ons about time to learn each new piece of a task.

Description

Method for goal: Move text

Step 1. Accomplish goal: Cut text

Step 2. Accomplish goal: Paste text

Step 3: Return with goal accomplished

Method for goal: Cut text

Step 1. Accomplish goal: Highlight text

Step 2. Retain that the command is CUT, and accomplish goal: Issue a command


Method for goal: Paste text

Step 1. Accomplish goal: Position goal at insertion point

Step 2. Retain that the command is PASTE, and accomplish goal: Issue a command


Selection rule set for goal: Highlight text

If text-is word, then accomplish the goal: Highlight word

If text-is arbitrary, then accomplish goal: Highlight arbitrary text

Return with goal accomplished

Method for goal: Highlight word

Step 1. Determine position of middle of word

Step 2. Move cursor to middle of word

Step 3. Double-click mouse button

Step 4. Verify that correct text is selected

Step 5. Return with goal accomplished

Description Duration (sec)

Method for goal: Highlight arbitrary text

Step 1. Determine position of beginning of text

1.20

Step 2: Move cursor to beginning of text 1.10

Step 3: Click mouse button 0.20

Step 4: Determine position of end of text. 0.00

Step 5. Move cursor to end of text 1.10

Step 6. Shift-click mouse button 0.48

Step 7. Verify that correct text is highlighted 1.20


Method for goal: Position cursor at insertion text

Step 1. Determine position of insertion point 1.20

Step 2. Move cursor to insertion point 1.10

Step 3. Click mouse button 0.20

Step 4. Verify that correct point is flashing 1.20

Step 5. Return with goal accomplished

Method for goal: Issue a command

Step 1. Recall command name and retrieve from LTM the menu name for it, and retain the menu name

Step 2. Recall the menu name, and move cursor to it on Menu bar

1.10

Step 3: Press mouse button down 0.10

Step 4: Recall command name, and move cursor to it

1.10

Step 4: Recall command name, and verify that it is selected

1.20

Step 5: Release mouse button 0.10

Step 6: Forget menu name, forget command name and return with goal accomplished


Extending GOMS: Parallel Processes Cognitive processes are not always sequential

Clerks imprinting checks often realize an error two checks past

When typing, you often realize an error while typing the next sentence or letters

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) Predicts a substantially shorter execution time

than the other models. Allocates less time for “prepare for action” type o

perations. Allow parallel processes.

Requires analyst-supplied methods. Uses Critical Path Analysis to investigate

parallel processes

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) Collect-call example1, operator hits a “collect-call”

key and says “Thank you” to customer:

You can save time by repositioning the key for faster access in the sequential example, but not in the parallel example.

1Courtesy of Newman, Lemming’s TAO (Toll & Assistance Operator) study

Extending GOMS: Cognitive-Perceptual-Motor (CPM-GOMS) Critical Path: a connected sequence that represents the

greatest total time and therefore determines the overall time for a task.

Critical Path1 below is 400 + 280 + 2000 + 280 = 2.96 seconds

1Courtesy of Newman, Lemming’s TAO (Toll & Assistance Operator) study

GOMS Family: SummaryKLM CMN NGOMSL CPM

Architectural BasisSimple Cognitive Architecture

Model Human Processor

Cognitive Complexity Theory

Model Human Processor, assume expertise in use

Goal Hierarchy Implicit Explicit Implicit Implicit

Models Learning/Transfer No No Yes No

Models Parallel Processes No No No Yes

Assigned Mental TimeYes, use operator M

No YesYes, but very short for expert users

Notation UsedPrimitive Operators

Programming Language

Natural Language Schedule/PERT chart

Modeling Specific Components 3 general classes

Memory & Cognition Motor movements Perception

Goals

Intention Evaluationexpectation

Execution

Mental Activity

Physical Activity

Perception

Interpretation

7 steps1 of user activities involved

in computer-based tasks

Action Specification

1Norman, D. (1986)


Goals

Intention Evaluation

MEMORY:Retrieve a unit from long term memory

expectation

COGNITION:Execute a mental step

Choose among methods

Mental Activity

Memory & Cognition: Memory Retrieval GOMS provides

modeling of Memory Retrieval Time to retrieve next

unit of information Moving information

from long-term memory into working memory

1

2

≈ 1350 msec

@MAX(D2…D12)

Memory & Cognition: Execution of a Mental Step GOMS allows explicit representation of mental

steps of a task (the “Cognitive Processor”):

Retrieval of goal

Find the max value in a column

Select a method to achieve the goal

Retrieval of motor movements necessary

to execute the command Execution of each of the chosen commands

Use the “MAX” formula

Type the formula

≈ 70 msec

Memory & Cognition: Method Decision Hick’s Law

T = k log2(n+1), k ~ 150 msecn = # of choices

Time to make a decision is roughly proportional to the log of the number of choices

However, determining T is problematic Spreadsheet task where parameters in a formula are can b

e indicated via numerous methods Hick’s law predicts 200 msec Real time is 2 seconds → Order of magnitude difference!

Memory & Cognition: Method Decision Choice is a complex task that requires many

cognitive steps Steps differ task to task


Goals



expectation



Execution

Mental Activity

Physical ActivityMOTOR MOVEMENTS:KeystrokePoint Move hands

Motor: Key Input Parameters of keyboard input based on

Skill of the typist Best Typist (120 wpm): 80 msec Worst Typist: 1200 msec

Predictability & continuity of the text to be typed Typing random letters: 500 msec

Motor: Mouse Movement Fitts’s Law is a robu

st predictor of mouse movement

Sometimes distance metric is not clear-cut Nested menus

Motor: Applying Fitts’s Law Fitts’s law recommends

Larger target sizes Smaller distances to targets Usage of corners and edges (they have “infinit

e” height and width) Macintosh menus are faster than Windows/Unix style

menus because they lie on the screen edge

Motor: Applying Fitts’s Law

Target size grows as distance from cursor’s position increases

Borders for shorter selectiontime

Fittsized Menus

Motor: Fisheye Model Provide local context

against a global context Focuses on screen space

versus user’s attention 3 properties

Focal point Distance from focus, D Level of detail, LOD

Degree of Interest Function to determine

whether to display an item or not and its size

Motor: Fisheye Menu Good for browsing task

s Allows one to present e

ntire menu without having to use hierarchies or scrolling

Longer learning curve http://www.cs.umd.edu

/hcil/fisheyemenu/fisheyemenu-demo.shtml

Motor: Hand Movements Switching between keyboard and mouse

≈ 360 msec Differences in times due to distance from hom

e position on keyboard and the size of the targets Joystick ≈ 260 msec Arrow keys ≈ 210 msec


Goals



expectation



Execution

Mental Activity

Physical ActivityMOTOR MOVEMENTS:KeystrokePoint Move hands

Perception

Interpretation PERCEPTION:PerceiveSaccade

Perception Recognition or perception

Measure the time to respond to stimuli Responding to lights Recognizing words

Saccade: fast movement of eye, head, etc. Measure the time to move and take in information

in each jump Eye jerking around, scanning or moving to the next

location

Perception An example: spreadsheet perception

Looking for cell addresses and retrieving data

230 msec130 msec, store row label

230 msec130 msec, store col label

230 msec1350 msec, retrieve row & col label

Total: 2300 msec

Summary of Cognitive ParametersRetrieve from memory 1200 msec

Execute a mental step 70 msec

Choose among methods 1250 msec

Enter a keystroke 230 msec

Point with a mouse 1500 msec

Move hands to mouse 360 msec

Perceive 100 msec

Make a saccade 230 msec

GOMS Limitations: User Skill Level Nonskilled or casual users

Current GOMS is best applied towards skilled users.

Transfer and training from one system to another. Example: Transfer GUI operational skills betwee

n MacOS, Windows and KDE.

GOMS Limitations: Errors Even skilled performers make mistakes Errors are probably caused by an overload of

Working Memory (WM) or the goal stack. The higher the WM load, the more errors.

How to model users adapting to errors arising from interface design? Users write down critical cell in a spreadsheet so that it can

be scanned quickly the next time.

Lotus IFPS

WM Load 19 14

Error (%) 14 6

GOMS Limitations: Parallel Processes Must be careful of over simplifying assumptio

ns: “A character on the same hand cannot be initia

ted with a cognitive operator until the motor processor execution of the previous character is complete” – John (1998) Psych literature doesn’t support the above!

Critical path method requires fine grained characterizations of task dependencies and parameters.

GOMS Limitations: Cognitive Processes Cognitive processes are unclear

Gestalt principles exist, but many more factors exist which influence user’s interpretation of screen

GOMS Limitations: Other Problems Usability Functionality Fatigue Mental Workload Individual differences

Some Work Has Addressed This

Topic

Straightforward Extension Seems

Possible

Cognitive Science Does Not Inform Us

Requires Another Kind of

Modeling

Nonskilled users − X − −

Learning X X − −

Errors X X − −

Cognitive Processes

X X X −

Parallel Processes

X X X −

Mental Workload − X − −

Functionality − − − X

Fatigue − − X −

Individual Differences

X − − −

Acceptance − − − X

Fit to organizational

life

− − − X

Outline Cognitive Modeling Introduction to GOMS GOMS Extensions GOMS Limitations Modeling Specific Components Summary

Summary In HCI, GOMS is, by far, the most detail oriented modeling

method for user activities. Great for getting quantitative and qualitative metrics. Easy to explain results. Once constructed, easy to modify in later design iterations.

Readily available tools are scarce, those that do exist have a high learning curve (e.g. CPM-GOMS)

Not as easy as heuristic analysis, walkthroughs or guidelines. Only works for goal-directed tasks. However, GOMS has been highly successful in applications

where human interaction performance is of utmost importance.

Summary Simulations of airplanes and helicopters in simulated

theatres of war (STOWs) with SOARS Sun’s webpage, CAD, word processors, mobile phone

input methods, etc. Project Ernestine: Adding new, “improved” workstati

ons for Telephone Operators CPM-GOMS revealed that the new workstations w

ould have cost an additional $2 million a year to operate!

*Figures from The Precis of Project Ernestine (CHI ’92) by Wayne D. Gray et al.

Summary GOMS model is a predictive, descriptive and

prescriptive model1. Predicts the time it will take user to perform a

task

2. Describes the way a user performs tasks on a system

3. Prescribes ways to develop of training programs and help systems

Questions or Comments?

the growth of cognitive modeling in human computer interaction since goms judith reitman olson and...

Documents

specific task

growth of cognitive

expert user

specific execution time

motor goms cpmgomsgoms

routine cognitive skills

shortcuts methodelse

level model klmcard