Spatiotemporal integration of optic flow and running speed in V1Marius Pachitariu1, Adil Khan2, Jasper Poort3, Ivana Orsolic2, Georg Keller4, Sonja Hofer2, Maneesh Sahani1*, Thomas Mrsic-Flogel2*

1Gatsby Computational Neuroscience Unit, UCL, UK 2 Biozentrum, University of Basel, Switzerland 3 University College London, UK 4 Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland * equal contribution

Summary

Background

Neurons in mouse primary visual cortex (V10 respond to both optic flow (abottom-up, external input) and the running speed of the animals (atop-down, internal signal). How are these signals combined?

Recent results have suggested that these signals may be combined mostly inan additive fashion, as the sensory cue integration framework suggests.

However, a small proportion of neurons (13%) respond to large mismatchesbetween optic flow and running speed, in agreement with the predictivecoding framework.Our results

We measure Calcium signals in L2/3 of mouse primary visual cortex fromGcamp6 expressing neurons, while mice are running through a virtualcorridor.

We show here that neural responses are driven by the history of optic flowspeeds (for example by acceleration).

Additionally, we show that this tuning exists for both the running-relatedsignal and the purely optic signals.

Finally, we show that in a large fraction of the population the purely opticand running-related signal are highly anti-correlated while in a differentpopulation they are highly correlated.

We distinguish optic signals from running signals by fitting models in anopen-loop condition where the optic flow speed is unrelated to the runningspeed.

Neural responses in V1 are tuned to speed

Reward zone

Run-up

Licks

open loop: opticflow in VR equalsrunning speed

closed loop: opticflow is unrelatedto running speed

Calcium traces of V1 neurons in virtualreality are tuned to speed.

Time (s)

ne

uro

n n

um

be

r

8 16 24 32

5

10

15

20

25

30

35

40

Running speed

8 16 24 32

0

1

2

3

Time (s)

run

nin

g a

nd

flo

w s

pe

ed

Example neuronsMonotonic tuning curves

0.05 0.1 0.15

0.04

0.06

0.08

0.1

instantaneous speed

dF

/F

0.05 0.1 0.15

0.02

0.03

0.04

0.05

instantaneous speed

dF

/F

Non-monotonic tuning curves

0.05 0.1 0.15

0.1

0.15

0.2

0.25

0.3

instantaneous speed

dF

/F

0.05 0.1 0.15

0.025

0.03

0.035

0.04

instantaneous speed

dF

/F

Responses are affected by the past optic flow speeds

Rn(t, x) = s(x) + Ioptic flow+

+ Irunning speed

Ioptic flow =

T∑t′=0

vof (t− t′)fof (t′)

Irunning speed =

T∑t′=0

vr(t− t′)fr(t′)

fof (t′) =

∑i

wi exp(−t′/τi)

M1= corridor visual features onlyM2= M1+instantaneous speedM3= M1+filtered speed

(two filters)M4= M1+filtered speed

(four filters)

0

0.02

0.04

0.06

0.08

0.1

Exp

lain

ed v

aria

nce

(tes

t)

N = 739 cells

0

0.05

0.1

0.15

0.2

Exp

lain

edmv

aria

ncem

(tes

t)

Nm=m253moutmofm739,mmostmtunedmcells

0.01 0.1 1−0.3

−0.2

−0.1

0

0.1

0.2

0.3

0.4

variance explained by visual features + speed

de

lta

va

ria

nce

exp

lain

ed

visual features alone

full model

Example neurons

0 300 600 900−10

−5

0

5

10

time lag (ms)

filte

r w

eig

ht

Typical filters on the history of speed

What are the prototypical filtershapes?

0 300 600 900

−0.6

−0.4

−0.2

0

0.2

time lag (ms)

filte

r w

eig

ht

Svd on population of filters

Most variance of filter shapes infirst two dimensions

1 2 3 40

1

2

3

4x 10

4

SVD dimension

variance

Opposite tuning to recent and pastspeed

−100 −50 0 50 100−100

−50

0

50

100Correlation −0.707

weight on fast filter (tau = 120 ms)

we

igh

t o

n s

low

filt

er

(ta

u =

60

0 m

s)

Funding from the Gatsby Charitable Foundation, MarieCurie Actions FP7 and the Wellcome Trust

contact: marius@gatsby.ucl.ac.uk

Linear/monotonic tuning to the filtered history of speedexplains nonlinear/non-monotonic tuning to speed

Example neurons 1-4

0.05 0.1 0.15

0.04

0.06

0.08

instantaneous speed

dF

/F

0.05 0.1

0.040.060.08

0.10.120.14

flow input Iof

dF

/F

0 300 600 900−0.02

−0.01

0

0.01

Time (ms)

filte

r w

eig

ht

0.05 0.1 0.15

0.025

0.03

0.035

0.04

instantaneous speed

dF

/F

0.02 0.040.01

0.02

0.03

0.04

0.05

flow input Iof

dF

/F

0 300 600 900

−4

−2

0

x 10−3

Time (ms)

filte

r w

eig

ht

0.05 0.1 0.15

0.06

0.08

instantaneous speed

dF

/F

0 0.05 0.1 0.150.020.040.060.080.1

0.120.14

flow input Iof

dF

/F

0 300 600 900

−10

−5

0

5x 10

−3

Time (ms)

filter

weig

ht

0.05 0.1 0.150.05

0.1

instantaneous speed

dF

/F

0.05 0.10.02

0.04

0.06

0.08

0.1

flow input Iof

dF

/F

0 300 600 900

0

10

20

x 10−4

Time (ms)

filter

weig

ht

Responses in open loop are tuned both to optic flow andrunning speed separately

0

0.05

0.1

0.15

0.2

Exp

lain

ed=v

aria

nce=

(tes

t)

Full=running=model,=incremental=optic=flow=modelsN===255=out=of=739=cells=

M1: corridor visual features + fullrunning model(4 filters)

0

0.05

0.1

0.15

0.2

Exp

lain

ed=v

aria

nce=

(tes

t)

Full=optic=flow=model,=incremental=running=modelsN===166=out=of=739=cells=

M1: corridor visual features + fulloptic flow model(4 filters)

Filter examples for history of running speed and optic flow

filte

rw

eigh

t

0 300 600Time (ms)

0 300 600Time (ms)

0 300 600Time (ms)

0 300 600Time (ms)

0 300 600Time (ms)

0 300 600Time (ms)

Responses in open-loop agree well with tuning in closed-loop

Fit model in open-loop, use it to predict total speed inputs in closed-loop.How well does it agree with a model fit in closed-loop?

Rel

ativ

eco

ntri

buti

onin

open

-loo

pan

dcl

osed

-loo

p

−1 −0.5 0 0.5 10

0.2

0.4

0.6

0.8

1

CorrelationOofOspeedOinputsopenOloopOvsOclosedOloopOmodels

σop

enOlo

op/O(σ

open

Oloop

+Oσ

clos

edOlo

op) 0 50

100

numberOofOcells

OnOaverageOopenOloopOfiltersOexplainasOm

uchOvarianceOinOclosedOloop

0

100

200

300

400

num

berO

ofOc

ells

MostOcellsOhaveOsimilarOresponsesunderOopenOloopOandOclosedOloop

How are running-related and optic-flow inputs combined in V1?

One line of previous work (Saleem et al, 2013) suggests additively(consistent with sensory cue integration framework).Another line of work (Keller et al, 2012) suggests negatively (consistent withpredictive coding framework).

Fit model in open-loop, use it in close-loop to predict running inputs Ir andoptic-flow inputs Iof . What is the correlation between Ir and Iof?

Rel

ativ

eco

ntri

buti

onof

flow

and

runn

ing

White noise controls: is there a bias due to fitting linearmodels from correlated predictors?

Are predictive-coding and cue integration neurons just over-fitted? NO.

Mismatch and integration neuronshave more signal variance

−1 −0.5 0 0.5 10

0.5

1

1.5

2

σru

n+

σflo

w

Correlation of running Irand flow I

of

Cells with more optic flow variancealso have more running variance

10−4

10−2

100

10−4

10−2

100

σrun

σflow

We replaced all neural traces with white noise (but smoothed it with 300msGaussian kernel like we do the true dFs).

0

10

20

30

40

50

60

num

ber

of c

ells

White noise control

−1 −0.5 0 0.5 1

Correlation of running Irand flow I

of

Discussion

Acceleration signals in V1 are likely important behavioural cues and cantrack significant unexpected changes in behaviour and environment.

In one subpopulation, the running component appears to be inverselyrelated to the optic flow, consistent with a predictive coding framework.

In a different subpopulation, the running component appears to becorrelated with the optic flow, consistent with a sensory integrationframework.