1 a biologically useful memory mechanism for the rapid deployment of visual attention ken nakayama
TRANSCRIPT
1
A biologically useful memory mechanism for the rapid deploymentof visual attention
Ken Nakayama
2
seeing = visibility X attention
+
R
3
Metaphors for vision
• Camera: It’s like a picture• Hand: it’s more active
Attention is the hand
How is it controlled?
4
Backgroundsome robust examples of attention
• Change blindness
• Inattentional blindness (even at fovea)
• Attentive tracking “hand and fingers”
5
Rensink flicker experiment• method: alternate two pictures
– Ask subjects to identify changes
• If we were aware of everything in picture, should be easy
• raise your hand when you see the change don’t tell others
At Nissan CBR
6
QuickTime™ and aAnimation decompressor
are needed to see this picture.
Rensink et al., Simons and Levin Change Blindness
7
QuickTime™ and aAnimation decompressor
are needed to see this picture.
8
summary of Rensink flicker experiment
contrary to our phenomenological experiencewe are not aware of everything in our visual world
large changes can escape our notice
Conclusion: seeing requires attention
9
~ 0.5 seconds
attention asan inertia-lesshand
can change size and positionapprox 4-6 times/sec
10
Other functions of attention
• Guidance of motor behavior
• Foraging for food
11
12
13
The intensity of predation depends . .. on the use of specific searching images. This implies that the birds perform a highly selective sieving operation on the visual stimuli that reach their retina . . . birds can only use a limited number of different search images at the same time.
L. Tinbergen(1960)
14
Marian Dawkins (1971)Shifts of ‘attention’ in chicks during feeding
15
Search image widespread?
• Chicks (Dawkins, 1971)
• Pigeons (Reid and Shettleworth, 1992))
• Blue Jays (Bond)
• Bumble Bees (1992)
• Butterflies (Stanton, 1984)
Is it a mental image or could it be something else?
16
Claim
There exists a primitive distributed memory system (seen in our human experiments) that could account for shifts of attention attributed to search images
17
It is part of a fast transient attentional system
(Nakayama and Mackeben, 1989
Vision Research)
18
msec
msec
19
Isolation of thetransient component
Keep location constant
20
Is the transient component due to its activation by a
sensory transient ?
NOsuch sensory transients not
necessary
analogy to “action potential ?”
21
Deploy transient attentionwithout a localsensory transient
Decoys
22
decoy cueing
normalcueing
23
Transient attention is very fast, rises to peak within 100
msec.
Can transient attention learn to go quickly to the appropriate position
on a larger object ?
Is it fast AND flexible
Kristanjansson, Mackeben & Nakayama, 2001
24
cue
stimulus display
fixat ion
cue target relation
fixed
variable
25
50
60
70
80
90
rommsubjects
fixedvariable
learning,
Attention can be effectively deployed toa location within anobject
26
Is the learning a property ofsustained or transient attention ?
27
50
60
70
80
90
0 100 200 300 400 500 600
Cue lead time
AK
C ue lead tim e
Pe
rce
nt
corr
ect
vary cuelead time
Keep target position within the cue constant
28
Learning
• How fast does it occur ?• How stable is it ?
• Method: use quasi random streaks of cue target regularities
29
1
2
3
4
5
time
sequence of cues
50
60
70
80
90
0 1 2 3 4 5 6 7 8 9
P o s itio n in "s tre a k"
A K
A MA M H
% c
orre
ct
Build-up of learningst
reak
30
+
+
+
40
50
60
70
80
90
1 2 3 4 5 6 7 8
NH
AK
position in sequence
color ?st
reak
31
20
30
40
50
60
70
0 1 2 3 4 5 6 7 8 9
NH
AK
Position in streak
perc
ent
corr
ect
local shape ?
32
Learning (summary so far)
• Is very rapid and is temporary• Can be linked position within an object• Can linked to a color within an object• Can linked to a local shape within an object
Are there things “attention” can’t learn ?
33
can it learn a 2nd order relation?
+
+
color and position
++
shape and position
34
0
10
20
30
40
50
60
random consistent
++
+
++
+
random
consistent
AKAMH
2 subjects
35
50
60
70
80
90
0 100 200 300 400 500 600
Cue lead time
AK
C u e le a d t im e
Pe
rce
nt
corr
ect
Part of a fast mechanism of Attentional deployment , reaches peak with 100 msec
36
+
+
+
n-1
n
Identify shape ofthe odd colored target
Maljkovic and Nakayama
Different paradigm to study the same process
position and colorof the targetcan change
37
.
baseline
1 2 3 4 5 6 7 8 9 10 11480
500
520
540
560
580
600
620
640
660
KN
+ +
+. .1 2 nrepeat color
38
4 65321530
550
570
590
610
Order in Sequence
Repeat targetposition
++ ++
1 2 n
. .
39
Attention can learn colors, shapes and locations
Is attention, (or are we)
learning what to expect, thenforming a search image in ourminds ?
Approach:manipulate target color uncertainty
40
prediction
.
0 .2 .4 .6 .8 1
changeeverytrial
nochange
Probability of color change
?
41
VM
400
420
440
540
560
580 KN
0 .2 .4 .6 .8 1
changeeverytrial
nochange
Probability of color change0 .2 .4 .6 .8 1
42
make expectancy very explicitmaximizing the possible use ofsearch images, pitting it againstrepetition
••
•
1 2 1 2 1 2 1 2
Double alternation paradigm
Target color over trials
43
••
•
21
Trialsequence:
order in sequence
repetition
reac
tio
n t
ime
predicted outcomes
1 2 1 2 1 2 1 2
Search image(expectancy)
44
1 2 1 2 1 2 1 2
45
Learning is passive, mechanistic, piecemeal
(color, position)Its not expectancy,
Its not a search image
Its not under conscious control
46
Its not just a linkage to the previous targets but active
inhibition to non-targets
47
.
+ +
distractor color constant
.
+ +
distractor color varies
560
580
600
620
640
654321654321
660
620
640
KN
Distractor varies
distractorsame
SS
48
Fine grain temporal analysis of the learning mechanism
~ 2nd order reverse correlation
What is the influence of a singletrial in the past ?
49
+
n
+
+
n-1
+
sameor different ?
+
n- i
sameor different ?
What is influence ofa single trial inthe past ?
current trial
past future
nn-i
time
50
.
+7+5+3+1-1-3-5-7-9-11-13-15550
560
570
580
590
600
610
FuturePast
EFB
Same Color
Different Color
Same Color
Different Color KN
+7+5+3+1-1-3-5-7-9-11-13-15610
620
630
640
650
660
670
51
.
FuturePast
+7+5+3+1-1-3-5-7-9-11-13-15
EFB
-50
-40
-30
-20
-10
0
10
-15 -13 -11 -9 -3
KN
-50
-40
-30
-20
-10
0
10
+7+5+3+1-1-5-7
52
Linkage is not confined to color
Probably any salient feature
(spatial frequency, for example), will do
53
.
-35
-45
-25
0
15
5
-5
-15
-25
-20
-15
-10
-5
10
0
10
5
+7+5+3+1-1-3-5-7-9-11-13-15
-15 -13 -11 -9 -3
VM
GP
FuturePast
+7+5+3+1-1-5-7
54
Learning restricted to what attracts attention
(features, positions)
Not the fine details that attention allows one to
process
55
.
FuturePast
-15 -13 -11 -9 -3
KN
-50
-40
-30
-20
-10
0
10
+7+5+3+1-1-5-7
-25
-20
-15
-10
-5
10
0
10
5
-15 -13 -11 -9 -3
VM
+7+5+3+1-1-5-7
56
.
1614121086420620
640
660
680
700
720
740
3 5421620
640
660
680
700
720
740
Time (sec) Order in Sequence
KN KN
1sec 2sec3sec
Is the build up and decay over timeor over events?
57
Independence of features/location
58
Trial n - i
trial n
Effect of target and distractor position in the past
59
KN
560 580 620 560600 580620
Previous Position
600
influence of trial n-1
60
Independence of color and location learning
61
hypothesized properties of the memory system
Time (trials) Time (trials)
gradedsummateslinearity (superposition)has independent components (features and locations)
62
RC
location1
focalattention
location2
++
-
++
leakybucket
63
What is learned ?What is the reinforcer ?
Simple identifiers of places where attention just wentand didn’t go ?
Memory kernel function
64
Generalizations beyond the measure of attention
• Speeds eye movements (human and monkey)
• Speeds motor behavior (manual pointing)
65
Saccadic eye movements(in human and in monkey)
66
+
+
+
time+
trial n
trial n - 1
Task: make a saccade to the odd colored target
Human eye movements
67
sacc
adic
late
ncy
54321200
210
220
230VM
position in samecolor sequence
McPeek, Maljkovic & Nakayama
Cumulative effects of learning
-15
-10
-5
0
5
-10 -8 -6 -4 -2 0 2 4
late
ncy
diff
eren
ce
past future
Memory kernel
68
Learning of speeded saccades in monkey
Position in same color sequence
Sac
cadi
c la
tenc
y
Robert McPeek and Ed KellerSmith Kettlewell
69
Learning generalizes to manual pointing
+
until the response
+
Trial 1 1200ms
Measure RT withTouch sensitive screen
(Song & Nakayama, 2003)
70
Learning generalizes to manual pointing
+
until the response
+
Trial 1 1200ms
+
Trial 2 1200ms
+
until the response
Measure RT withTouch sensitive screen
(Song & Nakayama, 2003)
71
Touching target with finger
1 2 3 4 51 2 3 1 2 ….. …..
525
550
575
600
625
650
675
700
1 2 3 4 5 6 7
sequence in order
AH
baseline
425
450
475
500
525
550
1 2 3 4 5 6 7 8
sequence in order
JHS
baseline
RT
(ms)
Position within “Same color” Sequence
Target:
72
Implications for foraging
• Don’t need a cognitive concept like a search image
• Low level temporary, passive, graded connection strengths (plus and minus) may be sufficient
73
Marian Dawkins (1971)Shifts of ‘attention’ in chicks during feeding
74
Anatomical locale/mechanism?
Object centerednot retinotopic cortex
Independent featuressimple 2 layered network
75
Relationship to other learning systems
• Is it a completely specialized sub-system for attention ?
• OR
• Is it the germ for short term memory more generally ?
X
Hypothesis: it’s a biologically conserved memorysystem for the rapid deployment of visualattention.
76
A biologically useful and conservedmemory mechanism for the rapid deploymentof visual attention (and other possible functions
Ken Nakayama
Manfred MackebenVera MaljkovicRobert McPeekArni KristjanssonJoo-Hyun Song