the dynamics of attending and scene analysis

By

Mari Riess Jones

The Dynamics of The Dynamics of Attending Attending andand Scene AnalysisScene Analysis

Thanks to Thanks to

Ralph BarnesRalph Barnes

Devin McAuleyDevin McAuley

Heather MoynihanHeather Moynihan

Noah MacKenzieNoah MacKenzie

Amandine PenelAmandine Penel

Jennifer PuenteJennifer Puente

Nate VaughanNate Vaughan

Special Thanks Special Thanks to to

Edward Edward LargeLarge

I. Scene Analysis: A Neo Gestalt view

II. Scene Analysis: A dynamic perspective

III. The Changing Scene: Tracking in real time

OverviewOverview

I. Scene Analysis:I. Scene Analysis: A Neo Gestalt view A Neo Gestalt view

What’s a Scene?What’s a Scene?

The term scenescene is taken from vision. Two cases are most common:

1. A stationary viewer

2. A moving viewer

The scene is the static visual layout.The scene is the static visual layout.

Sound ScenesSound Scenes

A common assumption:

We use the proximal sound stimulus to recover the location and/or identity of a distal, vibrating, object within a scene.

This is the source identity principle.

Bregman’s TheoryBregman’s Theory

of of Scene AnalysisScene AnalysisBregman builds on this idea:Bregman builds on this idea:

Goal of auditory perception… Goal of auditory perception… is identification of one is identification of one or more sources of a or more sources of a proximal mixture of sounds. proximal mixture of sounds.

If an unfolding sound pattern is perceived as a structurally intact , i.e. as one streamone stream,then it must come from ONE ONE source.source.

Other patterns? Other patterns?

If a pattern is perceived to break up, i.e. form N different subjective units, streamsstreams, ,

then, it must …

come from N separate sources.

Patterns NOT perceived as intact….

Time

high

low

Another source?

Example: A sound Example: A sound patternpattern

frequencyA stream ?One source?

The Big QuestionThe Big Question

What makes a pattern seem seem structurally intact?

Or… What determines a stream?stream?

Bregman’s AnswerBregman’s Answer

Involves a two stage theory

in which

Neo Gestalt principles are critical.

The Neo Gestalt ViewThe Neo Gestalt View

I. Primitive processes Innate, automatic, effortless, Determine coherent perceptual groups at fast rates.

II. Schema driven processes Learned, voluntary, effortful, Determine attention to groupings at slow rates.

Two Stages

Auditory Scene PerceptionAuditory Scene Perception Stage I: Primitive Processes Operate on Gestalt grouping principles:

proximity in frequency and time; similarity (e.g., timbre); continuity; exclusive allocation… and so forth

These are domain general (I.e. speech, music, natural environs,etc.)

Proximity vs ContinuityProximity vs Continuity

Tougas and Bregman (1985)

In a series of famous studies pitted proximityproximity against continuity. continuity.

Goal: To assess which principle dominated, and to dispute a trajectory explanation.

Bouncing versus Crossing trajectories

Time in seconds

Possible Percepts

ConclusionConclusion

Proximity in frequency dominates

People hear bouncing patterns

Rhythm PerceptionRhythm Perception

Speeding up a sequence can break it , perceptually, into interleaved interleaved streams streams

This changes its perceived This changes its perceived rhythm! rhythm!

A stream usually conveys a rhythmrhythm

RhythmRhythm

is perceived to have a ‘is perceived to have a ‘galloping galloping ‘‘rhythmrhythm

This slow melody…This slow melody…

Time

Rhythm perception… depends on the relationshiprelationship of frequency change (f ) to time change (t).

Hi Hi frequencyfrequency

Lo Lo frequencyfrequency

f

t 2 t

But… That Rhythm But… That Rhythm DisappearsDisappears

In a In a fasterfaster melody with the same tones. melody with the same tones.

Now we hear two melodies (streams) each with an isochronousisochronous rhythmrhythm..

A Neo Gestalt A Neo Gestalt ExplanationExplanation

Builds on Gestalt proximity:

1.1. Temporal proximityTemporal proximity is heightened at fast rates.

2. But frequency proximity frequency proximity dominates.

3.3. Two streamsTwo streams form based on pitch proximity: A high tone stream and a low tone stream.

II.II. Scene Analysis:Scene Analysis:A Dynamic PerspectiveA Dynamic Perspective

Auditory PatternsAuditory Patterns

Most of our environment comprises

sources that turn on and off and

modulate over time to project pattern

structure that is….

motion-like and rhythmicmotion-like and rhythmic

Important Pattern Properties Important Pattern Properties

1. Motionlike properties: Rates of change.

[ e.g., f/ t ]

2. Rhythmic properties: Relative timing.

[e.g., ti/tk ]

Motionlike TrajectoriesMotionlike Trajectories

Pitch VelocityPitch Velocity: f / t

In 1976, I proposed that the rate-of-change of frequency (log scale) may be perceived as a sort of velocitylocity. .

Time

high

low

Determinants of Pitch Determinants of Pitch Velocity Velocity

frequency

A sudden increase in rate-of-change in an isochronous rhythm is perturbing.

f

t

f/ t

Motionlike Trajectory Motionlike Trajectory limits on pitch velocity?limits on pitch velocity?

f log

FAST t Slow

Intact Serial Percepts

Vel

ocity

Lim

it

(Jones, 1976)

RATE

SMALL

LARGE

Stimulus patterns

streamed

Rhythmic Properties: Rhythmic Properties: Relative TimingRelative Timing

flog

FAST t =IOI SLOW

Lim

it

RATE

Slower patterns

Alternative View: Alternative View: A focus on timeA focus on time

How do rhythmic properties of a pattern affect real time responding to it?

In the remainder of this talk, I address this question.

General AssumptionsGeneral Assumptions

Connecting through time Connecting through time

Pattern timing determines how living Pattern timing determines how living things things connect connect with a changing scene. with a changing scene.

Living thingsLiving things

Animate creatures possess Animate creatures possess internal internal periodicitiesperiodicities that can that can

synchronize to patterns. synchronize to patterns.

EEnvironment nvironment

Scenes often project dynamic patterns Scenes often project dynamic patterns with with relative timing that is relative timing that is quasi- quasi- periodicperiodic. .

Great Gray OwlGreat Gray Owl

Horned Horned LarkLark

CardinalCardinal

t = IOI = 700 t = IOI = 700 msms

Some quasi-periodic patternsSome quasi-periodic patterns

Internal Periodicities: Internal Periodicities: Biological Oscillations Biological Oscillations

Most studied are circadian Most studied are circadian rhythms; oscillatory periods of ca. rhythms; oscillatory periods of ca. 24 hours. 24 hours.

Many internal periodicities exist; Many internal periodicities exist; periods of 1 periods of 1 ms, several minutes, years !. ms, several minutes, years !.

ConnectingConnecting

“How do internal oscillations of an organism connect with pattern timing? “

One answer: By One answer: By synchronizing. synchronizing.

More precisely: By entrainmentMore precisely: By entrainment

Biological EntrainmentBiological Entrainment

Pittendrigh & DaanPittendrigh & Daan (1976) . (1976) . Circadian rhythms of 24 Circadian rhythms of 24 hours hours synchronize to synchronize to light/dark cycles. light/dark cycles.

Daan et al. Daan et al. (2001) recently (2001) recently updated updated this two oscillator this two oscillator model. model.

SummeSummerr

WinterWinter

From Daan et al. J.Biological Rhythms (2001)

Example: Two Entraining OscillationsExample: Two Entraining Oscillations

M Phase locked

E Phase locked

Shorter Periods?Shorter Periods?

In speciesIn species

with refined use of fast sound with refined use of fast sound patterns,patterns,

Can we postulate,Can we postulate,

internal oscillations with periods internal oscillations with periods lessless than the circadian than the circadian period ?period ?

Generalizing…..Generalizing…..

Long Long natural natural periodsperiods

Respond to Respond to light light changes changes

Most Biological Most Biological

OscillatorsOscillators

Alternatively Alternatively

Other Other OscillatorsOscillators

ShortShort natural periods natural periods

Respond to Respond to soundsound changeschanges

Background:Background: Rhythmic (periodic) ExpectanciesRhythmic (periodic) Expectancies

Quasi-periodicities inQuasi-periodicities in

Rhythmical sound patternsRhythmical sound patterns

Induce Induce Periodic expectanciesPeriodic expectancies

Quasi-periodicities inQuasi-periodicities in

Rhythmical sound patternsRhythmical sound patterns

Induce Induce Periodic expectanciesPeriodic expectancies

Perhaps such expectancies reflect Perhaps such expectancies reflect entrainments entrainments to to patterns of stimulus IOIs?patterns of stimulus IOIs?

Back to Synchrony: Back to Synchrony: AA simple premise simple premise

When we attend , When we attend , an internal activity is an internal activity is elicited … elicited … that co-occurs that co-occurs with the attended object. with the attended object.

This activity involves This activity involves Internal oscillations Internal oscillations

(Jones, 1976). (Jones, 1976).

Where elements appear… then disappear

Attending to Dynamic Attending to Dynamic Patterns Patterns

Time Time

Attention must occur in synchrony with elements.

Dynamic AttendingDynamic Attending

Selective attending in time is Selective attending in time is pacedpaced by stimulus timing. by stimulus timing.

Postulates that :Postulates that :

Attentional focus in time becomes Attentional focus in time becomes phase-locked phase-locked to elements. to elements.

III. The Changing Scene:III. The Changing Scene:Tracking sounds in real Tracking sounds in real

timetime

Attending oscillates in Attending oscillates in timetime

Resources are periodically distributed as internal oscillationsinternal oscillations. .

Time Pulses of Pulses of attention attention energyenergy

Inter-onset-interval, IOI

Expected point in time

PhasePeriod

Formalized as an Oscillator Formalized as an Oscillator ModelModel :

by by Ed Large Ed Large (Large & Jones, 1999)(Large & Jones, 1999)

Stimulus timing paces the oscillator. Oscillator period and phase determine this entrainment .

Periodic Periodic ExpectanciesExpectancies

Stimulus IOI

Phase, iPeriod, Pi

Tones, i

Period and Phase Period and Phase change: change: They They draw draw attending rhythms attending rhythms to “fit” a time to “fit” a time pattern. pattern.

Period changes

Phase changes

The adaptiveadaptive oscillator

The entrained oscillator gravitates to an attractor state of:

1. Phase Synchrony:1. Phase Synchrony: A zero time difference between an sound onset and a pulse peak. 2. Period matching:2. Period matching: A zero time difference between an IOI and oscillator period (limit cycle).

A simple dynamical system.

Relative Phase, Relative Phase, i i

Gives observed - expected times

ii is the order variable in this system. It summarizes the state of the system at any instant i.

i+1 i+1 = = ii + IOI + IOIi+1i+1/P/Pii - - F(F(ii))

The coupling term, F, here is sinusoidal: F(F()= {1/2)= {1/2}sin 2}sin 2 => =>

-

Phase Attractor: Synchrony

-.25 .25

Relative Phase,

i

Phase Response Curve, PRC

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

-0.5 -0.25 0 0.25 0.5

Ph

ase C

orr

ecti

on

,

Entrainment Region

{ { = = 1.0 }1.0 }

Theoretical

Phase Attractor: synchrony

-.25 .25

What does this approach What does this approach purchase purchase

??

A few experimentsA few experiments….….

Preview: Preview: Our Experimental Our Experimental

DesignsDesigns Will involve a context rhythm, given by tones:

Followed by a comparison, C, (of some sort)

Time Judgment

Comparison IOI Pitch Judgment

Comparison tone

C

Time Judgment TasksTime Judgment Tasks RationaleRationale

If relative timing in sound arrays entrains attending,

then the context rhythm should

Shape temporal expectancies

Affect time judgments.

Example Study:Example Study:

Judgment TaskJudgment Task: Is a comparison time interval “shorter”, “same” or “ longer” ?

““Ignore context IOIs“Ignore context IOIs“

Context IOI

Standard IOI

Comparison IOI

Context IOIs are 600 ms. This rhythm should synchronize attending.

+/-T

Standard Ending Varies:Standard Ending Varies:

Expected Standard Ending: = 600 ms Unexpected Standard Endings:

Early = 579 ms

Late = 621 ms

Very Early = 524 ms

Very Late = 676 ms

23A comparison IOI is always yoked to standard IOI

STANDARD IOI

ExpectedEnding

ExpectedEnding:

“Same”

Phase

COMPARISON IOI

Period

Modeling Oscillatory Attending

Context IOIs

600 600 msms

600600

When a standard ends as expected, When a standard ends as expected, the oscillator’s period is the oscillator’s period is accurately preserved accurately preserved

….as a memory of the standard ….as a memory of the standard IOI.IOI.

The Reason: The Reason: The Attentional The Attentional PulsePulse

Pulse location depends on rhythmic context

Expected time

15

Pulse Pulse locationlocation

Very Early Expected Very Late

Standard Ending Time

PC

One Prediction:

A quadratic accuracy Expectancy ProfileExpectancy Profile:

Attentional Attentional PulsePulse

Predicted and ObservedPredicted and ObservedExpectancy Profiles Expectancy Profiles (Large & Jones 1999)(Large & Jones 1999)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Very Early Early Expected Late Very Late

Standard Ending Time

PC

PredictedObserved

P < .01

Subsequent Subsequent ExperimentsExperiments

Over a dozen experiments later (e.g., Barnes & Jones, 2000; McAuley & Jones, 2002) We Find:

• The profile vanishes with no context; sharpens with longer sequences .

• Holds for different rates.

• Reveals harmonic scaling of timing.

Different Induction Different Induction RatesRates

Three conditions: different induction rates.

Condition Condition Induction IOIs Standard Induction IOIs Standard durationsdurations

1. 300 ms 300 ms 524, 600, 676 ms

2. 500 ms 500 ms 524, 600, 676 ms

3. 600 ms 600 ms 524, 600, 676 ms

Isochronous sequencesIsochronous sequences

This task required same vs different duration judgments

Observed Expectancy Profiles Observed Expectancy Profiles Three Three

Time Judgment Conditions Time Judgment Conditions

0.4

0.5

0.6

0.7

0.8

0.9

1

524 600 676

Duration of Standard

Ag

300

500600

Conclude Conclude Back to Relative timing

1. Ratios Ratios of induction IOIsinduction IOIs to expected expected standardsstandards seem influential.

2. 2. Harmonic ratiosHarmonic ratios of 1:1 and 1:2 yield different profiles than the 500 ms with a more complex ratio.

Pitch JudgmentsPitch Judgments

We generalized this approach to attending to tone frequency.

Does rhythmic context affect pitch judgments?

StandarStandard d pitchpitch

ComparisoComparison n pitch pitch

TaskTask: Is a comparison pitch: “same”, “higher”, ”lower” than a

standard pitch?

““Ignore distracting Ignore distracting pitches!”pitches!” “higher”

same”

“lower”

A Pitch Judgment Task A Pitch Judgment Task

Critical IOICritical IOI

21

Independent Variable: Critical IOI

RationaleRationale

In a rhythmic context, In a rhythmic context, attending is paced in time. attending is paced in time.

Pitch judgments will be best Pitch judgments will be best for for temporally expectedtemporally expected tones: tones: When critical IOI = context IOI.When critical IOI = context IOI.

Very Early Expected Very Late

Onset Time of Comparison

PC

Similar Predictions

AA quadratic expectancyquadratic expectancy profile

Attentional Pulse

Experiment 1: Expectancy Profile

0.5

0.55

0.6

0.65

0.7

0.75

0.8

Very Early Early On Time Late Very Late

Onset Time of Comparison

Prop

ortio

n Cor

rect

P<.005

Related ExperimentsRelated Experiments

Found a:

flattened expectancy profile

with irregular context timing

Recent findingsRecent findings

Suggest the importance of Suggest the importance of higher-order time higher-order time structurestructure … …

In focal attending over time In focal attending over time levels levels

An Accent Rhythm An Accent Rhythm

21

Accent Timing

Accented Tones

Higher time level : 4 IOIs

Lower time level: IOI

Accent Rhythm is VariedAccent Rhythm is Varied

21

RegulaRegularr

IrregularIrregular

Accented Tones

Standard Standard pitch pitch

Comparison Comparison pitchpitch

Same Pitch Judgment Same Pitch Judgment TaskTask

Critical IOI is Varied Critical IOI is Varied

21

Two Oscillators?Two Oscillators?

Remember those two circadian Remember those two circadian oscillators (Morning, Evening)?oscillators (Morning, Evening)?

Well, Well, these are these are different!different!

Shorter periods Shorter periods Sensitive to Sensitive to

soundsound

Two Different OscillatorsTwo Different Oscillators

Two oscillators: Different periods

1. Lower-order oscillator, period ~ period ~ IOIIOI..

2. Accent oscillator, period ~ 4 IOIsperiod ~ 4 IOIs

If the Accent oscillator dominates, then only only the Regular Rhythm will give an Expectancy Profile.

Which dominates?

50

60

70

80

90

Very Early Early On Time Late Very Late

Regular Rhythm

Irregular RhythmPCPC

P<.001

Two Oscillators Two Oscillators In listening to sound patternsIn listening to sound patterns

Differential responses to Differential responses to sounds: sounds: AcAccented versus cented versus Unaccented Unaccented

Flexible attendingFlexible attending:: DDominance of one oscillator in ominance of one oscillator in focal attending. focal attending.

Final CommentsFinal Comments

The dynamics of sound patterns together the dynamics of attending may explain real time tracking of events.

Assumptions about rhythm and attentional synchrony can enhance theoretical power.

Entrainment to dynamic contexts may play a role in judgments of both time and pitch.

Thank you !

Thank you !

Two routes to SynchronyTwo routes to Synchrony

1. Anticipatory attending: 1. Anticipatory attending:

Attending heightens Attending heightens priorprior to an element. to an element.

Oscillator Oscillator period period is important. is important.

2.2. Reactive attending:Reactive attending:

If anticipation fails, If anticipation fails,

Attending shifts Attending shifts afterafter an element occurs. an element occurs.

Oscillator Oscillator pphasehase is important. is important.