tiling · 2020. 9. 23. · tiling •tiling by cones •tiling by retinal ganglion cells - midget...

Tiling

•Tiling by cones

•Tiling by retinal ganglion cells- midget vs. parasol (parvo/magno)- foveated sampling

I(X;R)�X

j

�j hrji

replacements

bw2

w1

x1x1

x2x2 f2f1

Figure 1: a. Schematic of the model (see text for description). The goal is to maximize informationtransfer between images x and the neural response r, subject to metabolic cost of firing spikes. b.Information about the stimulus is conveyed both by the arrangement of the filters and the steepnessof the neural nonlinearities. Top: two neurons encode two stimulus components (e.g. two pixels ofan image, x1 and x2) with linear filters (black lines) whose output is passed through scalar nonlinearfunctions (thick color lines; thin color lines show isoresponse contours at evenly spaced outputlevels). The steepness of the nonlinearities specifies the precision with which each projection isrepresented: regions of steep slope correspond to finer partitioning of the input space, reducing theuncertainty about the input. Bottom: joint encoding leads to binning of the input space according tothe isoresponse lines above. Grayscale shading indicates the level of uncertainty (entropy) in regionsof the input (lighter shades correspond to higher uncertainty). Efficient codes optimize this binning,subject to input distribution, noise levels, and metabolic costs on the outputs.

Parameter λj specifies the trade-off between information gained by firing more spikes, and the costof generating them. It is difficult to obtain a biologically valid estimate for this parameter, andultimately, the value of sensory information gained depends on the behavioral task and its context[26]. Alternatively, we can use λj as a Lagrange multiplier to enforce the constraint on the meanoutput of each neuron.

Our goal is to adjust both the filters and the nonlinearities of the neural population so as to maximizethe expectation of (3) under the joint distribution of inputs and outputs, p(x, r). We assume thefilters are unit norm (‖wj‖=1) to avoid an underdetermined model in which the nonlinearity scalesalong its input dimension to compensate for filter amplification. The nonlinearities fj are assumedto be monotonically increasing. We parameterized the slope of the nonlinearity gj =dfj/dyj usinga weighted sum of Gaussian kernels,

gj(yj |cjk, µjk,σj) =K∑

k=1

cjk exp

(

−(yj − µjk)2

2σ2j

)

, (4)

with coefficients cjk≥0. The number of kernelsK was chosen for sufficiently flexible nonlinearity(in our experimentsK = 500). We spaced µjk evenly over the range of yj and chose σj for smoothoverlap of adjacent kernels (kernel centers 2σj apart).

2.1 Computing mutual information

How can we compute the information transmitted by the nonlinear network of neurons? Mutualinformation can be expressed as the difference between two entropies, I(X ;R) = H(X)−H(X |R).The first term is the entropy of the data, which is constant (i.e. it does not depend on the model) andcan therefore be dropped from the objective function. The second term is the conditional differentialentropy and represents the uncertainty in the input after observing the neural response. It is computedby taking the expectation over output values H(X |R) = Er

[

−∫

p(x|r) ln p(x|r)dx]

. In general,computing the entropy of an arbitrary high dimensional distribution is not tractable. We make severalassumptions that allow us to approximate the posterior, compute its entropy, and maximize mutualinformation. The posterior is proportional to the product of the likelihood and the prior, p(x|r) ∝p(r|x)p(x); below we describe these two functions in detail.

3

1 16

16

1

ON−center

1 16

16

1

OFF−center

0

15

a b

c

Figure 2: In the presence of biologically realistic level of noise, the optimal filters are center-surround and contain both On-center and Off-center profiles; the optimal nonlinearities are hard-rectifying functions. a. The set of learned filters for 100 model neurons. b. In pixel coordinates,contours of On-center (Off-center) filters at 50% maximum (minimum) levels. c. The learned non-linearities for the first four model neurons, superimposed on distributions of filter outputs.

0

10

sp/sec

1 16 −3 0 30

10

sp/sec

0

12

24

sp/sec

1 11 −3 0 30

30

60

sp/sec

a b

Figure 3: a. A characterization of two retinal ganglion cells obtained with white noise stimulus[31]. We plot the estimated linear filters, horizontal slices through the filters, and mean output asa function of input (black line, shaded area shows one standard deviation of response). b. Forcomparison, we performed the same analysis on two model neurons. Note that the spatial scales ofmodel and data filters are different.

in the number of On-center neurons (bottom left panel). In this case, increasing the number ofneurons restored the balance of On- and Off-center filters (not shown). In the case of vanishinginput and output noise, we obtain localized oriented filters (top left panel), and the nonlinearities aresmoothly accelerating functions that map inputs to an exponential output distribution (not shown).These results are consistent with previous theoretical work showing that optimal nonlinearity in thelow noise regime maximizes the entropy of the output subject to response constraints [11, 7, 17].

How important is the choice of linear filters for efficient information transmission? We comparedthe performance of different filtersets across a range of firing rates (Fig. 5). For each simulation, were-optimized the nonlinearities, adjusting λj’s for desired mean rate, while holding the filters fixed.As a rough estimate of input entropyH(X), we used an upper bound – a Gaussian distribution withthe covariance of natural images. Our results show that when filters are mismatched to the noiselevels, performance is significantly degraded. At equivalent output rate, the “wrong” filters transmitapproximately 10 fewer bits; conversely, it takes about 50% more spikes to encode the same amountof information.

We also compared the coding efficiency of networks with variable number of neurons. First, wefixed the allotted population spike budget to 100 (per input), fixed the absolute output noise, and

6

Objective function:

Efficient coding model of retina(Karklin & Simoncelli 2012)

© 1999 Macmillan Magazines Ltd

!"##"$% #& '(#)$"

*++ !"#$%& ' ()* +,- ' .. /&0%$"%1 .,,, ' 22234567893:;<

:;=;78 5>>95854:9 ;? 6@4AB <;4;:C8;<56@: =@DC6 E5FC9F.,BGH I9:57F96C9 89=56@J9 9K:@656@;4 ;? L@??98946 :;49 :=5FF9F L9>94LF ;4 6C9=;:56@;4 ;? 6C9 E5FC3 " 89=569L @==7F@;4 @F 0892F698MF :;=;78FB 6C9>98:9>6@;4 ;? @889D7=58 >56:C9F ;? >5F69= :;=;78 2C@=9 J@92@4D>98@;L@: I=5:N 54L 2C@69 >566984F ;? C@DC F>56@5= ?89O794:A.BG.3P@<@=58=AB 89LQD8994 @F;=7<@4546 D856@4DF 2@6C F>56@5= ?89O794:@9F5I;J9 6C9 89F;=76@;4 =@<@6 =;;N =@N9 :C8;<56@: 54L =7<@4546 F>56@5=4;@F9GG3 "== ;? 6C9F9 >98:9>675= 988;8F 589 9K5<>=9F ;? 6C9 5=@5F@4D>8;L7:9L 2C94 6C9 6C899 :;49 F7I<;F5@:F F5<>=9 6C9 896@45= @<5D9@45L9O7569=A3 #C9A 589 5N@4 6; 6C9 988;8F 6C56 ;::78 @4 @<5D9F 65N942@6C L@D@65= :5<985F 6C56 C5J9 @4698=95J9L >@K9=F ;? L@??98946 F>9:685=F94F@6@[email protected] #C9 :=7<>@4D 6C56 89F7=6F ?8;< 9@6C98 6C9 854L;<;8 6C95DD89D569L 5FF@D4<946 ;? R 54L * :;49F 9K5:98I569F 6C9F9 988;8F30;6C ;? ;78 F7IS9:6F C5J9 896@45= >56:C9F ;? T 58:<@4 ;8 <;89

5:8;FF 6C56 :;465@4 ;4=A ;49 ;? 6C9 62; =;4D98U25J9=94D6CUF94F@6@J9:;49 :=5FF9F3 "=6C;7DC 6C9 9K@F694:9 ;? 6C9F9 >56:C9F @4L@:569F 6C566C9 68@:C8;<56 <5A F;<96@<9F <@FS7LD9 6C9 :;=;78 5>>95854:9 ;?6@4A ;IS9:6FB 6C9 >56:C9F 2@== I9 I949V:@5= @4 89:;J98@4D C@DCU?89O794:A =7<@454:9 >566984FB I9:57F9 :;86@:5= 4978;4F 6749L 6;C@DC F>56@5= ?89O794:A 589 <;89 =@N9=A 6; I9 ?9L IA :;46@D7;7F :;49F;? 6C9 F5<9 :=5FF3 #C@F <5A 9K>=5@4 6C9 ;IF98J56@;4 6C56B @4 F;<94;8<5= 68@:C8;<56FB 6C989 @F =@66=9 ;8 4; L@??9894:9 @4 89F;=76@;4 ?;8D856@4DF F994 2@6C ;4=A R :;49F ;8 ;4=A * :;49FB ;8 2C94 I;6C :;49:=5FF9F ;>98569 6;D96C98G+3 )4=A 2C94 ;49 :;49 :=5FF @F D8956=A74L98U89>89F9469LB 5F ;::78F @4 F;<9 C9698;WAD;7F :588@98F ?;8:;4D94@65= XU=@4N9L >8;654;>@5B @F 89F;=76@;4 :=958=A <9L@569L IA6C9 <;89 L94F9 F7I<;F5@:GY3#C9 =58D9 @4L@J@L75= L@??9894:9F @4 47<I98F 54L 58854D9<946 ;?

:;49 :=5FF9F 6C56 29 C5J9 ;IF98J9L @4L@:569 6C56 9J;=76@;4 C5F 4;6:;4J98D9L ;4 54 ;>6@<7< >8;>;86@;4 ;? R 54L * :;49F ?;8 6C9C7<54 9A93 ZF 6C@F I9:57F9 89LQD8994 :;=;78 J@F@;4 @F 5 89=56@J9=A492 ?956789 ;? J@F@;4 @4 ;=LU2;8=L >8@<569FGTBG[B ;8 L; 6C9 F656@F6@:F;? 456785= F:949FB ;>6@:5= I=788@4D 54L :=9J98 >;F6U89:9>6;8 >8;:9FUF@4D <5N9 RU 54L *U:;49 6;>;D85>CA 74@<>;86546 ?;8 J@F75=>98?;8<54:9\ #C9 @<5D@4D <96C;L L9F:8@I9L C989 5==;2F 7F 6;5LL89FF 6C9F9 O79F6@;4FB I9:57F9 @6 @F 4;2 >;FF@I=9 6; 5FF9FF J@F75=>98?;8<54:9 54L 6C9 :@8:7@68A ;? 6C9 896@45 @4 9A9F ?;8 2C@:C 6C968@:C8;<56@: <;F5@: @F N4;243 !

%9:9@J9L .[ ):6;I98] 5::9>69L G[ !;J9<I98 .,,^3

.3 _@==@5<FB 3̀ %3B P9N@D7:C@B !3B a55N9B _3B 085@458LB 3̀ a3 b c5:N98B )3 P3 @4 !"#$ %&'$()*+ *#%(",(-*&#). d9LF (5=I98DB "3 b *99B 03 03e ..QGG dc=947<B !92 1;8NB .,,.e3

G3 _@==@5<FB 3̀ %3B R5:*9;LB 3̀ Z3 "3 b a5AC;9B R3 c74:6569 F94F@6@J@6A ;? 6C9 I=79 F94F@6@J9 <9:C54@F<3/&+&#) 0(+. !"# .+T-Q.+-T d.,^.e3

+3 f78:@;B f3"3 (* 12.`@F68@I76@;4 54L<;8>C;=;DA ;? C7<54 :;49 >C;6;89:9>6;8F F65@49L2@6C 546@UI=79;>F@43 3. 4#$-. 5(6"#2. $"!# [.HQ[GY d.,,.e3

Y3 05I:;:NB a3_3 #C9 >;FF@I@=@6A ;? :;<>94F56@4D 5F68;4;<@:5= F99@4D3 %672. 8+*"#). 9#,. %1,&:. %&# GG,QG+[ d.,T+e3

T3 *@54DB g3B _@==@5<FB 3̀ %3 b R@==98B 3̀ #3 P7>984;8<5= J@F@;4 54L C@DCU89F;=76@;4 896@45= @<5D@4D6C8;7DC 5L5>6@J9 ;>6@:F 3. ;-*. 9#,. 8$. 8 "'# G^^YQG^,G d.,,-e3

[3 f5<>I9==B /3 _3 b %7FC6;4B _3 "3 a3 R95F789<946 ;? 6C9 F:;6;>@: >@D<946 @4 6C9 =@J@4D C7<54 9A933. %<=+&#2. >?#)@.A "$(# .+.Q.Y- d.,TTe3

-3 %7FC6;4B _3 "3 a3 b 05N98B a3 3̀ %9LhD8994 F94F@6@J@6A @4 4;8<5= J@F@;43 /&+&#) 0(+. '# -TQ^T d.,[Ye3^3 c;N;84AB g3B P<@6CB (3 f3 b_9F498BR3 @4 !"#$%&'$()*+ *# %(",(-*&#) d9LF (5=I98DB "3 b *99B 03 03e G+Q

+Y dc=947<B !92 1;8NB .,,.e3,3 f@:98;49B f3 R3 b !98D98B g3 *3 #C9 89=56@J9 47<I98F ;? =;4DU25J9=94D6CUF94F@6@J9 6; <@LL=9U

25J9=94D6CUF94F@6@J9 :;49F @4 6C9 C7<54 ?;J953 /&+&#) 0(+. !%# ..TQ.G^ d.,^,e3.H3 (@<5=B %3 *3 c3B c;N;84AB g3B P<@6CB (3 f3 b PC9J9==B P3 i3 /;J95= :;49 6C89FC;=LF3 /&+&#) 0(+. !)# [.Q-^

d.,^,e3..3 g5:;IFB j3 a3 b !9@6WB g3 @4 4#2#6" /&+&#) B(C,&(),&(+ (;=3 XZ d9L3 `87<B 03e .H-Q..G di=7298

":9L9<@:B !96C98=54LFB .,,+e3.G3 g5:;IFB j3 a3 b `99D54B g3 /3 P>9:685= F94F@6@J@6A ;? <5:5O79 <;4N9AF <95F789L 2@6C &%j E@:N98

>C;6;<968A3 /&+. 5(6"#+,&. "'# ,G.Q,G^ d.,,-e3.+3 0;2<5N98B g3 i3 b `58645==B a3 g3 "3 (@F75= >@D<946F ;? 8;LF 54L :;49F @4 5 C7<54 896@453 3. %<=+&#2.

>?#)@.A !)*# TH.QT.. d.,^He3.Y3 `58645==B a3 g3 "3B 0;2<5N98B g3 i3 bR;==;4B g3 3̀ a7<54 J@F75= >@D<946Fk <@:8;F>9:68;>C;6;<968@:

89F7=6F ?8;< 6C9 9A9F ;? F9J94 >98F;4F3 %"#,. 0. 9#,. ?#)@. D !!(# ..TQ.+H d.,^+e3.T3 15<5D7:C@B #3B R;67=FNAB "3 j3 b `99IB P3 P3 (@F75= >@D<946 D949 F687:6789 54L 9K>89FF@;4 @4 C7<54

896@4593 E6$. F#2. G()(*. %# ,^.Q,,H d.,,ê3.[3 a5DF68;<B P3 "3B !9@6WB g3 b !9@6WB R3 (58@56@;4 @4 :;49 >;>7=56@;4F ?;8 89LUD8994 :;=;8 J@F@;4

9K5<@49L IA 545=AF@F ;? <%!"3 5(6"#0(-#"* )# .,[+Q.,[- d.,,ê3.-3 R;==;4B g3 3̀ b 0;2<5N98B g3 i3 #C9 F>56@5= 58854D9<946 ;? :;49F @4 6C9 >8@<569 ?;J953 51*6"( $%(#

[--Q[-, d.,,Ge3.^3 c5:N98B )3 P3B _@==@5<FB 3̀ %3 b 094F@4D98B 3̀ j3 cC;6;89:9>6;8 6854F<@6654:9 @<5D@4D ;? 6C9 >8@<569

>C;6;89:9>6;8 <;F5@:3 3. 5(6"#+,&. "%# GGT.QGG[H d.,,[e3.,3 a;=<D894B /3 $I98 L94 /58I94F@4434#$-. 0()@. 4#)'". %("&#@. H)*("). 9,&. F(@. 4#-()<1'() "# ^HQ,^

d.^^Ye3GH3 i857FN;>?B g3 f;=;8 5>>95854:9 ;? F<5== F6@<7=@ 54L 6C9 F>56@5= L@F68@I76@;4 ;? :;=;8 89:9>6;8F3 3. ;-*.

9#,. 8$. &'# ..-. d.,[Ye3G.3 0892F698B 3̀ )4 6C9 74L7=56@;4F 9K:@69L @4 6C9 896@45 IA 6C9 5:6@;4 ;? =7<@4;7F >;@46F 54L =@49F3 ?#)@.

I@&)7. %<&2#+. F1'. 3. 9,&. "# .[,Q.-Y d.^+Ge3GG3 P9N@D7:C@B !3B _@==@5<FB 3̀ %3 b 085@458LB 3̀ a3 &?V:@94:A @4 L969:6@;4 ;? @F;=7<@4546 54L

@F;:C8;<56@: @4698?9894:9 ?8@4D9F3 3. ;-*. 9#,. 8$. 8. "(# G..^QG.++ d.,,+e3G+3 _@==@5<FB 3̀ %3 @4 8@J1),(+ &) %<#*#"(,(-*&#)k %"#,. 9=$-. !"#)*&("+ /&+612 9,&. .+TQ.Y^ d!56@;45=

":5L9<AB _5FC@4D6;4 `fB .,,He3GY3 R@A5C585B &3B c;N;84AB g3B P<@6CB (3 f3B 058;4B %3 b 058;4B &3 f;=;8 J@F@;4 @4 62; ;IF98J98F 2@6C C@DC=A

I@5F9L *_PhR_P :;49 856@;F3 /&+&#) 0(+. $*# [H.Q[.G d.,,ê3GT3 !56C54FB g3B #C;<5FB 3̀ b a;D49FFB 3̀ P3 R;=9:7=58 D9496@:F ;? C7<54 :;=;78 J@F@;4k 6C9 D949F

94:;L@4D I=79B D8994 54L 89L >@D<946F3 9,&(),( !$!# .,+QGHG d.,^[e3G[3 R;==;4B g3 3̀ ll#C;M FC9 N499=9L @4 6C56 >=5:9 2C989 6C9A D892333MMk #C9 7F9F 54L ;8@D@4F ;? >8@<569

:;=;78 J@F@;43 3. IK-. D&#2. "'%# G.Q+^ d.,^,e3G-3 `@DD=9B c3 g3 9*1*&+*&,12 8)12=+&+ #: 9-1*&12 %#&)* %1**(")+ d":5L9<@:B *;4L;4B .,^+e3

+,-./01234252.678_9 6C54N 3̀ 085@458LB 3̀ `5:9AB g3 g5:;IFB g3 *@54DB 3̀ R@==98 54L )3 c5:N98 ?;8 6C9@85FF@F654:93 _9 5:N4;2=9LD9 V454:@5= F7>>;86 ?8;< 6C9 /@DC6 ?;8 P@DC6 89F958:C L@J@F@;4 ;? c89J9460=@4L49FF "<98@:5 d6; "3%3e 54L 6C9 !56@;45= &A9 Z4F6@6769 54L %9F958:C 6; c89J946 0=@4L49FF d6;3̀%3_3e3

f;889F>;4L94:9 54L 89O79F6F ?;8 <5698@5=F FC;7=L I9 5LL89FF9L 6; "3%3 d9U<5@=k 58;;8L5m>;><5@=3;>637C39L7e3

!"#$%& ' !"#$%&'&(&$) *+,-# &. /0# /)*'0)&+,/*' '&1# +&",*'2 3($#4 -)##1 ,1%

)#% '&(&$)" )#5)#"#1/ /0# 64 7 ,1% 8 '&1#"4 )#"5#'/*9#(:2 () *) 6$;<#'/ =>?"

/#+5&),( ,1% 1,",( )#/*1,4 )#"5#'/*9#(:4 ,/ &1# %#-)## &. #''#1/)*'*/:2 +) 6$;<#'/

@A?" 1,",( )#/*1,4 ,/ &1# %#-)## &. #''#1/)*'*/:2 ># 5#).&)+#% , "/,/*"/*',( /#"/ .&)

),1%&+1#"" ,''&)%*1- /& B*--(#CD2 ># '&+5,)#% /0# %*"/)*;$/*&1 &. ,(( *1/#)'&1#

%*"/,1'#" &. /0# +#,"$)#% 7E'&1# ,)),: F*/0 GHH "*+$(,/*&1" %#)*9#% .)&+ /0#

",+# +&",*' *1 F0*'0 /0# ",+# 1$+;#) &. 7 '&1#" F#)# ),1%&+(: ,""*-1#%2

=>?" ,)),: F," 1& %*..#)#1/ .)&+ ),1%&+ ,/ #*/0#) (&',/*&12 @A?" ,)),: "0&F#%

"*-1*I',1/ '($+5*1- &. /0# %,/, J!! H!HGK ;$/4 ;#',$"# &. &5/*',( ;($)4 /0#

5&""*;*(*/: &. , ),1%&+ ,""*-1+#1/ &. 7 '&1#" ',11&/ ;# )$(#% &$/2 6',(# ;,)

)#5)#"#1/" L ,)'+*1 &. 9*"$,( ,1-(#2

Cones sample the light intensity distribution over the

joint domain of space and wavelength I(x, y, λ)

Human - cone spectral sensitivities

wavelength (nm)

Cones sample the light intensity distribution over the joint domain of space and wavelength I(x, y, λ)

x

λ

Tiling by retinal ganglion cells

Cone vs. retinal ganglion cell spacingas a function of eccentricity

ConesRGC’s

Smoothing and subsampling by retinal ganglion cells

Parvo- and Magno-cell dendritic field diameteras a function of eccentricity

Magno(parasol)

Parvo(midget)

Temporal information loss in the macaque early visual system

PLOS Biology | https://doi.org/10.1371/journal.pbio.3000570 January 23, 2020 5 / 31

(Horwitz, 2020)

(from parasol) (from midget)

Parvo and Magno cells encode complementaryaspects of spatio-temporal structure

(from Van Essen & Anderson 1995)Ocko, S., Lindsey, J., Ganguli, S., & Deny, S. The emergence of multiple retinal cell types through efficient coding of natural movies. NeurIPS 2018.

Foveated sampling

(Wayne Maddison)

Active vision in jumping spiders

(Bair & Olshausen, 1991)

Lewicki et al. Scene analysis in the natural environment

FIGURE 5 | Scene analysis in electroreception. The “electric image” ofthe external environment is determined by the conductive properties ofsurrounding objects. The electric field emanates from the electric organ inthe tail region (gray rectangle) and is sensed by the electroreceptive skinareas, using two electric “foveas” to actively search and inspect objects.Shown are the field distortions created by two different types of objects: aplant that conducts better than water, above (green) and a non-conductingstone, below (gray). (Redrawn from Heiligenberg, 1977).

copy of the EOD signal is sent to electrosensory areas of thebrain. Thus, it is possible for the animal to directly compare thesensed signal with that which was actually generated. An objectwith low or no capacitance, such as a non-living object, willleave the waveform shape unaffected. Most living objects how-ever, such as insect larvae, other fish, and plants possess compleximpedances, and so they will significantly alter the waveformshape, which behavioral studies show is detectable by the animal(von der Emde, 2006).

Due to the high conductivity of water, the range over whichthe electric fish can sense objects is only a few centimeters.Nevertheless, electroreception mediates a wide range of sceneanalysis behaviors important to the animal’s survival, which wedescribe here.

Object recognition in electric scenesThe mormyrid’s object recognition and discrimination abilitieshave been explored through behavioral studies (von der Emdeand Schwarz, 2002; von der Emde, 2004; von der Emde et al.,2010). By assessing performance on simple association tasks, ithas been shown that electric fish are capable of discriminatingthe shape of objects (e.g., cube vs. pyramid), even against com-plex and variable backgrounds. Doing so is non-trivial becausethe electric fields from multiple objects will superimpose andcreate a seemingly complex electric image on the electrorecep-tor array. Thus, the animal must solve a figure-ground problemsimilar to that in vision or audition, in which the sensory contri-butions of background or clutter must be discounted in order toproperly discern an object. Perhaps even more impressive is thefact that the animal can generalize to recognize different shapesindependent of their material properties (metal or plastic) or dis-tance. It can discriminate small from large objects, irrespective ofdistance. Thus, the animal is capable of extracting invariances inthe environment from the complex electroreceptor activities – i.e.,despite variations due to material properties or distance, it cannevertheless make correct judgments about the shape and size ofobjects.

Active perception during foragingWhen foraging for food, mormyrids utilize their two electric“foveas” in an active manner to search and inspect objects. Thetwo foveas are composed of a high density region of electrore-ceptors, one on the nasal region, and the other on the so-calledSchnauzenorgan (Bacelo et al., 2008). Unknown objects are firstapproached and inspected by the ventral nasal organ, and thenmore finely inspected by the Schnauzenorgan (von der Emde,2006). When foraging, the animal engages in a stereotypicalbehavior in which it bends its head down at 28◦ such thatthe nasal fovea is pointing forward or slightly upward, and itscans the Schnauzenorgan from side to side across the surfaceto search for prey. When a prey item is detected (presumably fromits capacitive properties) it is inspected by the Schnauzenorganbefore the fish sucks in its prey. Thus, the animal must cor-rectly interpret the highly dynamic patterns of activity on thesensory surface in accordance with this scanning movement inorder to properly detect and localize prey. This is an example ofan active process demanding the coordination of perception andaction.

Spatial navigationMormyrids are frequently seen swimming backward, and theyavoid obstacles with ease, finding their way through crevices inrocks (Lissmann, 1958). Presumably these abilities are mediatedby the electric sense, since the eyes, which are poorly developed,are at the front of the animal. They are also known to navi-gate at night in complete darkness (von der Emde, 2004). Thus,it would appear that electric fish can obtain a sufficient repre-sentation of 3D scene layout from the electric field in order toplan and execute maneuvers around objects. How accurate andwhat form this representation takes is not known, but it has beenshown through behavioral studies that they can judge the distanceto an object from the spatial pattern across the electroreceptor

Frontiers in Psychology | Perception Science April 2014 | Volume 5 | Article 199 | 12

Weakly electric fish

Low resolution

High resolution

Foveated sampling

Retinal ganglion cell spacing as a function of eccentricity

∆E ≈ .01(|E| + 1)

−100 −80 −60 −40 −20 0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1

Eccentricity (deg.)

sam

ple

spac

ing

(deg

.)

Letter size vs. eccentricity(Anstis, 1974)

Human eye movements during viewing of an image

Yarbus (1967)

!"#$ %#&&'($ &()* +"' +,$-$ %#$./$$'% ,0)1' #2 +",+ #+ #$ 2)+ ('3'+#+#1'4 5,." )& +"'67 8 97 ,.+$ #21):1'% #2 +',;*,-#2< #$ /2#=/'> ,2% $) ('=/#('$ 0)+" , %#&&'('2+ *)+)(3()<(,* ,2% 3('$/*,0:? , %#&&'('2+ 3,++'(2 )& '?' *)1'*'2+$ ,$ @'::4 !"/$ @' .,2'A3'.+ +) :',(2 2)+ )2:? ")@ '?' *)1'*'2+$ ,(' #21):1'% #2 $#2<:' ,.+$> 0/+ ")@ +"'$',.+$ B)#2 +)<'+"'(4 C -'? =/'$+#)2 "'(' #$ +) @",+ 'A+'2+ +"' '?'$ ,(' 3(),.+#1' ,2%:',% +"' *)+)( 3()<(,*> $''-#2< )/+ #2&)(*,+#)2 +",+ @#:: 0' ('=/#('% #2 +"' 2',(&/+/('> )( @"'+"'( +"'? ,(' '$$'2+#,::? (',.+#1'> 0'#2< .,::'% /3 0? +"' *)+)( 3()<(,*@"'2 $3'.#&#. #2&)(*,+#)2 #$ ('=/#('%4 D' @#:: $")@ +",+ 1#$/,: &#A,+#)2 %)'$ #2%''%3('.'%' *)+)( *,2#3/:,+#)2> 3/++#2< '?' *)1'*'2+$ #2 +"' 1,2</,(% )& ',." *)+)(,.+> (,+"'( +",2 ,$ ,%B/2.+$ +) #+4

!) )/( -2)@:'%<' +"'(' ",1' $) &,( 0''2 2) $+/%#'$ )& +"' '?' *)1'*'2+$ #21):1'%#2 ,.+#1#+#'$ )& %,#:? :#1#2<> #2 , 2,+/(,: $'++#2<4 !"#$ #$ 2) %)/0+ 0'.,/$'> /2+#: ('.'2+:?>*)$+ '?';*)1'*'2+ ('.)(%#2< %'1#.'$ @'(' 2)+ $/#+,0:' &)( /$' )/+$#%' +"' :,0)(,+)(?4E'(' @' ",1' '*3:)?'% , :#<"+@'#<"+ "',%;*)/2+'% '?' .,*'(, /$'% 3('1#)/$:? &)(%(#1#2< $+/%#'$ FG,2% ,2% G'' HII6J4 K+ 3()1#%'$ , 1#'@ )& +"' $.'2' ,"',%> @#+" , %)++",+ #2%#.,+'$ &)1',: %#('.+#)2> ,$ @':: ,$ +"' .))(%#2,+'$ )& '?' %#('.+#)2 (':,+#1' +)+"' "',% F&#</(' HJ4

!',;*,-#2<> ,+ :',$+ &)( +"' L(#+#$" 3)3/:,+#)2> #$ , 1'(? &,*#:#,(> )1'(:',(2'%,.+#1#+? #21):1#2< , $'=/'2.' )& ,.+#)2$ @"#." +?3#.,::? )../( #2 , 3,(+#./:,( )(%'(4M/." ,/+)*,+'% ,.+#1#+#'$> .)*3(#$#2< N$+'(')+?3#. $'=/'2.'$ )& ,.+#)2$O FM.",2- ,2%C0':$)2 HIPPJ> ,(' $)*'+#*'$ $/33)$'% 2)+ +) ('=/#(' &''%0,.- &)( +"'#( 'A'./+#)2FQ2%'(@))% ,2% 51'(,++ HIIRJ4 S2.' :',(2'%> +"'? .',$' +) ('=/#(' )2;:#2' *)2#+)(#2<)( $/3'(1#$)(? ,++'2+#)2 FT)(*,2 ,2% M",::#.' HIURV M",::#.' HIUUJ> ,2% (/2 %#('.+:?&()* , *'*)(? N$.(#3+O4 !"' 3(#2.#3,: .)2.:/$#)2 &()* +"' $+/%? 3('$'2+'% "'(' #$ +",+,:*)$+ '1'(? ,.+ #2 +"' +',;*,-#2< $'=/'2.' #$ </#%'% ,2% ."'.-'% 0? 1#$#)2> @#+"'?' *)1'*'2+$ /$/,::? 3('.'%#2< *)+)( ,.+#)2$4 !"/$ #2 , :#+'(,: $'2$' #+ #$ 2)+ +(/'+",+ +"'$' $/33)$'%:? ,/+)*,+'% ,.+$ ,(' /2%'( )3'2;:))3 .)2+():> ,2% #+ )3'2$ /3+"' =/'$+#)2 )& @",+ #$ *',2+ 0? N$/3'(1#$)(? ,++'2+#)2O> @"'2 +"' '?'$ ,(' *,2#&'$+:?,++'2%#2< +) +"' )0B'.+$ #21):1'% #2 ,/+)*,+'% 0'",1#)/($4

W(':#*#2,(? ,..)/2+$ )& +"#$ $+/%? ",1' ,33',('% ':$'@"'(' FX#2%:,? HIIUV G,2%'+ ,: HIIUV E,?")' ,2% G,2% HIIIJ4

F,J F0J

!"#$%& '( W(#2+$ &()* F,J +"' ,.+#1#+? 1#%')> ,2% F0J '?';*)1'*'2+ 1#%') )& +"' $,*' #2$+,2+>@"'2 +"' $@''+'2'( #$ %()33'% #2+) +"' */< FY4H6 )2 &#</(' YJ4 !"' "',%;*)/2+'% .,*'(, ,2%0,.-3,.- ,(' $")@2 #2 F,J4 K2 F0J +"' @"#+' %)+ #$ +"' %#('.+#)2 )& ('<,(% )& +"' &)1', F#2+) +"'*/<J4 !"' '?' .,2 0' $''2 #2 +"' 0)++)* +"#(% )& +"' &(,*'> @#+" , 0(#<"+ '::#3$' &#++#2< +"' #(#$4!"' ,2</:,( @#%+" )& /33'( 3,(+ )& +"' &(,*' #$ ,33()A#*,+':? Y9 %'<4 T)+' +",+ F0J #$ (#<"+ 8:'&+ ('1'($'% .)*3,('% @#+" F,J 0'.,/$' )& +"' *#(()( #2 +"' .,*'(, $?$+'*4

!"!# $ %&'() * $+'',+) - ./01+(

! "#$%&'()$*&#!"#$ "% $&' ()$*"+# $&($ ,(-' ./ ".0 1*2'# *+2"12' 2*#*"+3 *% 4' (0' +"0,(115 #*6&$'789' &(2' $" 1")($' ":;')$#3 )&(+6' $&'*0 /"#*$*"+#3 ,(+*/.1($' $&', *+ 2(0*".# 4(5#3(11 /0'#.,(:15 .+7'0 2*#.(1 )"+$0"18 9' &(2' 1*$$1' "0 +" )"+#)*".# -+"41'76' "% 4&'0'".0 '5'# (0' %*<($*+6 ($ (+5 *+#$(+$3 (+7 :')(.#' "% $&*# 4' $'+7 $" $&*+- "% $&' '5'#(# /(##*2' 0')'*2'0#3 0($&'0 1*-' )(,'0(#3 $(-*+6 ./ (+7 /(##*+6 "+ $&' *+%"0,($*"+0'=.*0'7 %"0 $&' /(0$*).1(0 $(#-8 >"4'2'03 #)0.$*+5 "% $&' '5'# "% #",'"+' '+6(6'7 *+ ()",/1'< ,"$"0 $(#- #&"4# $&($ $&*# *# +"$ $&' )(#'8 ?&' '5'# 7(0$ %0", "+' /1()' $"(+"$&'03 $4" "0 $&0'' $*,'# ( #')"+78 @+ $&*# #$.75 4' (#- $&' =.'#$*"+A (0' $&'#' '5',"2','+$# '##'+$*(115 0(+7",3 "0 (0' $&'5 *+$*,($'15 0'1($'7 $" $&' 0'=.*0','+$# "%$&' ,"$"0 $(#-B @% $&' 1($$'0 *# $0.'3 (0' %*<($*"+# 7*0')$'7 #/')*%*)(115 $" $&' /1()'#%0", 4&*)& *+%"0,($*"+ *# +''7'73 (+7 )(+ $&' '5'C,"2','+$ /($$'0+ $&.# :' $&".6&$"% (# (+ *+$'60(1 /(0$ "% $&' ,"$"0 /0"60(, *$#'1%B

@+ ()$*2*$*'# 4&'0' ,"2','+$# "% $&' '5'# &(2' :''+ ,.)& #$.7*'73 #.)& (# 0'(7*+6DEFG'6(+ HIIJK G(5+'0 HIILM3 ,.#*) 0'(7*+6 D9'(2'0 HINOK P(+7 (+7 Q.0+'(.< HIIRM3 (+7#$''0*+6 ( )(0 DP(+7 (+7 P'' HIINM3 $&' #$0($'65 "% $&' ").1","$"0 #5#$', *# $" -''/ $&')'+$0' "% 6(S' 2'05 )1"#' $" $&' /"*+$ ($ 4&*)& *+%"0,($*"+ *# '<$0()$'7T4*$&*+ ( %'41'$$'0# "0 +"$'# *+ $'<$ (+7 ,.#*) 0'(7*+63 (+7 4*$&*+ ( %'4 7'60''# "% $&' *+#*7' "% $&':'+7 D$&' $(+6'+$ /"*+$#M 4&'+ #$''0*+6 "+ ( 4*+7*+6 0"(78 @+ (+ (0$*%*)*(1 $(#- $&($*+2"12'7 $&' (##',:15 "% ( )"/5 "% ( /($$'0+ ,(7' "% )"1".0'7 :1")-#3 U(11(07 '$ (1 DHIIVM(+7 >(5&"' '$ (1 DHIIWM %".+7 $&($ '2'05 ()$*"+T)&""#*+6 ( :1")-3 )&')-*+6 *$# )"1".03%*+7*+6 *$# /0"/'0 /"#*$*"+T*+2"12'7 ( +'4 %*<($*"+3 4*$& '5' ,"2','+$# 6'+'0(115/0')'7*+6 ,"$"0 ()$# :5 ( %0()$*"+ "% ( #')"+78 X5' ,"2','+$# $&.# #'', $" :' =.*$'$*6&$15 )"./1'73 $',/"0(115 (+7 #/($*(1153 $" $&' ,"$"0 ()$*"+# "% $&' /(0$*).1(0 $(#-8

@+ $&*# /(/'0 4' (#- $" 4&($ '<$'+$ $&*# *# $0.' %"0 '2'057(5 ()$*2*$*'#3 (+74' &(2' )&"#'+ $" #$.75 $&' 0($&'0 (0)&'$5/(1 $(#- "% ,(-*+6 $'( DG.#$'7 '$ (1 HIILM8

!"# $%&#' %( )*'*%+ ,+- #.# /%)#/#+0' *+ 0"# 1%+0$%&%( ,10*)*0*#' %( -,*&. &*)*+2

!"#$"%&'()* +,,,* -(./0" 12* %34"5 +6++ 7 +612

3*1",#& 4,+-5 6#*& 3#++*#5 7#++*(#$ 89'0#-8/55"9 :")&#" ;(# <"/#(5$'")$" 3)= >3?(#3&(#@ (; A9%"#'0")&3. !5@$B(.(4@* 8$B((. (; C'(.(4'$3.8$'")$"5* D)'-"#5'&@ (; 8/55"9* C#'4B&() C<+ ,EF* DGH "I03'.J KLML>3)=N5/55"9L3$L/OP"$"'-"= Q K3@ +,,,* ') #"-'5"= ;(#0 , R/4/5& +,,,

+,-$%.)$/ ?&' (*, "% $&*# #$.75 4(# $" 7'$'0,*+' $&' /($$'0+ "% %*<($*"+# 7.0*+6 $&' /'0%"0,(+)'"% ( 4'11C1'(0+'7 $(#- *+ ( +($.0(1 #'$$*+6 D,(-*+6 $'(M3 (+7 $" )1(##*%5 $&' $5/'# "% ,"+*$"0*+6()$*"+ $&($ $&' '5'# /'0%"0,89' .#'7 ( &'(7C,".+$'7 '5'C,"2','+$ 2*7'" )(,'0(3 4&*)& /0"2*7'7( )"+$*+.".# 2*'4 "% $&' #)'+' (&'(73 4*$& ( 7"$ *+7*)($*+6 %"2'(1 7*0')$*"+ 4*$& (+ ()).0()5 "%(:".$ H 7'68 Y #')"+7 2*7'" )(,'0( 0')"07'7 $&' #.:;')$F# ()$*2*$*'# %0", ()0"## $&' 0"",8 ?&'2*7'"# 4'0' 1*+-'7 (+7 (+(15#'7 %0(,' :5 %0(,'8 Q"2'(1 7*0')$*"+ 4(# (14(5# )1"#' $" $&' ":;')$:'*+6 ,(+*/.1($'73 (+7 2'05 %'4 %*<($*"+# 4'0' *00'1'2(+$ $" $&' $(#-8 ?&' %*0#$ ":;')$C0'1($'7%*<($*"+ $5/*)(115 1'7 $&' %*0#$ *+7*)($*"+ "% ,(+*/.1($*"+ :5 J8LZ #3 (+7 2*#*"+ ,"2'7 $" $&' +'<$":;')$ (:".$ J8ZH # :'%"0' ,(+*/.1($*"+ "% $&' /0'2*".# ":;')$ 4(# )",/1'$'8 X()& ":;')$C0'1($'7 ()$$&($ 7*7 +"$ *+2"12' ( 4(*$*+6 /'0*"7 1(#$'7 (+ (2'0(6' "% O8O # (+7 *+2"12'7 (:".$ R %*<($*"+#8G".6&15 ( $&*07 "% (11 %*<($*"+# "+ ":;')$# )".17 :' 7'%*+*$'15 *7'+$*%*'7 4*$& "+' "% %".0 ,"+*$"0C*+6 %.+)$*"+#A !"#$%&'( ":;')$# .#'7 1($'0 *+ $&' /0")'##3 )&*+#%&'( $&' &(+7 "0 ":;')$ *+ $&' &(+7$" ( +'4 1")($*"+3 (,&)&'( $&' (//0"()& "% "+' ":;')$ $" (+"$&'0 D'6 -'$$1' (+7 1*7M3 (+7 #-+#.&'($&' #$($' "% #",' 2(0*(:1' D'6 4($'0 1'2'1M8 9' )"+)1.7' $&($ (1$&".6& $&' ()$*"+# "% $'(C,(-*+6(0' [(.$",($'7F (+7 /0")''7 4*$& 1*$$1' )"+#)*".# *+2"12','+$3 $&' '5'# )1"#'15 ,"+*$"0 '2'05 #$'/"% $&' /0")'##8 ?&*# $5/' "% .+)"+#)*".# ($$'+$*"+ ,.#$ :' ( )",,"+ /&'+",'+"+ *+ '2'057(5 1*%'8

STUJ+VL+VW2X%1,6Y

https://www.youtube.com/watch?v=mtYFNsRcxY4

$ %

�L�[

ıL

�L�\

Figure 1: A: Diagram of single kernel filter parameterized by a mean µi and variance �i B: First row

Examples from our variant of the cluttered MNIST dataset. Second row Examples with additionalrandom rescaling of the digit.

This factorization is shown in equation 3, where the kernel is defined as an isotropic gaussian. For82

each kernel filter, given a center µi and scalar variance �i, a two dimensional gaussian is defined over83

the input image as shown in Figure 1A.84

ki(m,n) = p(m;µi,x,�i)p(n;µi,y,�i) (3)

While this factored formulation reduces the space of possible transformations from input to output, it85

can still form many different mappings from an input U to output V . Figure 2B shows the possible86

windows which an input image can be mapped to an output V . The blue circles denote the central87

location of a particular kernel. Each kernel maps to one of the outputs Vi. The kernel filters in88

our model can be adjusted through two distinct mechanisms: control and training. control defines89

adjustments to the retinal sampling lattice as a whole and can include translation and rescaling of90

the entire lattice. Translational control can be considered analgous to the motor control signals91

which executes saccades of the eye in biology. In contrast, training defines structural adjustments to92

individual kernels which include its position in the lattice as well as its variance. These adjustments93

are only possible during training and are fixed afterwards. Training adjustments can be considered94

analagous to the layout of the retinal sampling lattice which is directed by evolutionary pressures in95

biology.96

3 Recurrent Neural Architecture for Attention97

We develop a recurrent model of overt attention inspired by Mnih et al. (2014). A sample input98

image U is reduced by a glimpse generator using equation 4 to create a output ‘glimpse’ Vt. We99

omit the sample index n to simplify notation. This glimpse Vt is processed by a fully-connected100

recurrent network frnn(). Equation 4-9 details the feedforward process of generating the kernel filter101

configurations which define the retinal sampling lattice for the next time point.102

3

Learning the glimpse window sampling array(Cheung, Weiss & Olshausen, 2017)

Example MNIST scenes

• Network is trained to correctly classify the digit in the scene.

• To do this it must find a digit and move its glimpse window to that location.

Object identity Control

Recurrent network

Glimpse

Scene

Evolution of the sampling array during training

Translation only(Dataset 1)

Translation only(Dataset 2)

Translation & zoom(Dataset 1)

Translation & zoom(Dataset 2)

Learned sampling arrays for different conditions

(Perry, Oehler & Cowey, 1984)Macaque Retina Model

Sam

plin

g In

terv

alEccentricity

Ill? V. A. Perry PI al.

cells is shown in Fig. Q(B). In this case the slopes of the regression lines of cell body size on the logarithm of cell density for the nasal (a = 43.4, h = -6.87) and the temporal cells (u = 53.6, h = 9.1) were not significantly different (r = 1.0, df‘ 54). Thus the differences in cell body size of PX cells in different parts of the retina is clearly related to the variation in ganglion cell density, although the differences in dendritic field size is not simply related to cell density alone. The correlation coefficient for the dendritic field size of both nasal and temporal Pa cells and the logarithm of the ganglion cell density was 0.97, i.e. most of the variance in dendritic field size can be attributed to differences in ganglion cell density.

As a check on the accuracy of our ganglion cell counts we used them to estimate the total number of ganglion cells in the macaque retina which can, in turn, be compared with published estimation of the number of axons in the optic nerve. To do this we plotted onto a scaled drawing of one Nissl-stained whole-mounted retina (670mm’ in area) the isodensity contours estimated from the counts along the

A

meridia. The isodensity contours were drawn as approximately ellipsoid but tapering in the nasal retina as has been shown in previous studies of peripheral ganglion cell counts of the primate retina.6’,“x The area between adjacent contours was multiplied by the mean ceil density of the two isodensity contours. This was repeated for the area between all pairs of contours and the totals for all areas were summed to give the total number of ganglion cells. This yielded an estimate of 1.4 x 10” ganglion cells in the retina, in good agreement with estimates of the number of axons in the optic nerve i.e. 1.5-1.8 x IOh, I.4 x I O' and 1.2-1.3 x 10”.40~so.” If we neglect the naso-temporal overlap, which is small in primates,h.h’ then approximately 60’1/ of the cells lie in the nasal retina and 40“;) in the temporal retina.

Our intention was to show which cell types project to the magno- and parvocellufar layers of the lateraf geniculate nucleus and to estimate the percentage of retinal ganglion cells at different eccentricities which

0 1 2 3 L 5 6 7 8 9 10 li 12 13 I‘ ECCENTRICITY fTH?

) .L--L.L-I__L. 0 1 2 3 L 5 6 7 8 9 10 I! 12 13 li ECCENTRICITY mm

Fig. 6(A) and (B)




A





Fig. 6(A) and (B)

Pα

Pβ

Comparison to primate retina

Eccentricity

(Perry, Oehler & Cowey, 1984)Macaque Retina




A





Fig. 6(A) and (B)




A





Fig. 6(A) and (B)

Pα

Pβ

Model

Rec

eptiv

e fie

ld d

iam

eter

Comparison to primate retina

A FOVEATED RETINA-LIKE SENSOR

USING CCD TECHNOLOGY

J. Van der Spiegel, G. Kreider Univ. of Pennsylvania, Dept. of Electrical Engineering

Philadelphia, PA 19104-6390

C. Claeys, I. Debusschere IMEC, Leuven, Belgium

G. Sandini University of Genova, DIST, Genova, Italy

P. Dario, F. Fantini Scuola Superiore S. Anna, Pisa, Italy

P. Bellutti, G. Soncini IRST, Trento, Italy

ABSTRACT

A CCD imager whose sampling structure is loosely modeled after the biological visual system is described. Its architecture and advantages over conventional cameras for pattern recognition are discussed. The sensor has embedded in its structure a logarithmic transformation that makes it size and rotation invariant. Simulations on real images using the actual sensor geometry have been performed to study the sensor performance for 2D pattern recognition and object tracking.

A CCD imager consisting of 30 concentric circles and 64 sensors per circle, whose pixel size increases linearly with eccentricity has been fabricated. The central part has a constant resolution with 102 photocells. The CCD is made in a three phase buried channel technology with triple poly and double metal layers. Preliminary results of the testing are given showing the validity of the design.

189

C. Mead et al. (eds.), Analog VLSI Implementation of Neural Systems© Kluwer Academic Publishers 1989

200

Figure 8: Photograph of the fovea, conslstmg of 102 photosensitive cells, and the first ten concentric circles.

Driving Electronics

One of the complications of this architecture is the relatively large amount of clocks and control signals to read out and synchronize the charge flow. Up to 18 different clocks are required. When the sensor has to be used as part of a moving platform for tracking purposes, it is important to minimize the number of wires and external interconnections. Also the dimensions and weight of the clock drivers should be small. For this reason an integrated clocking system has been developed that generates all the required clocks. It has been fabricated in a 2 !lm CMOS process. The chip is fully custom designed in order to reduce the amount of real estate and power dissipation as much as possible. The total chip area is less than 3 mm2. A photograph of the chip is given in Fig. 9 [27]. This chip will be mounted together with the CCD imager on a lightweight substrate and incorporated into the motor control platform. The chip is fully functional. Measured outputs of the controller chip is shown in Fig. 10.

A Foveated Image Sensor in Standard CMOS Technology

Robert Wodnicki, Gordon W. Roberts, Martin D. Levine Department of Electrical Engineering, McGill University,

Montrkal, Qukbec, CANADA, H3A 2A7

Abstract

We describe the design and implementation of a CMOS foveated image sensor for use in mobile robotic and machine vision applications. T h e sensor is biologically motivated and performs a spatial image transformation from Cartesian to log-polar coordinates. As opposed to tradi- tional approaches, the sensor benefits from a high degree of integration, minimal power consumption and ease of manufacture due to the use of a s tandard 1.2pm ASIC CMOS process. T h e prototype imager operates at 28 frames/sec when interfaced to a PC.

, j-1 X

Fovea image

ring i

Introduction

Foveation is a biologically motivated image transformation which has a t t racted the interest of researchers in the fields of computer vision and robotics. I ts principle advantages are the realization of a high degree of image compression as well as the property of scale and rotation invariance [2]. T o date various computational implemen- tations [3][6], as well as a fully custom CCD [a] image sensor have been proposed. While these approaches achieve adequate performance, they nevertheless suffer from the need for considerable support resources such as networks of DSP processors and digital frame-grabbers. These resources may be readily available in a laboratory environment for use with a tethered robot , however truly autonomous mobile systems will require foveated sensors which are extremely compact and energy efficient. Re- cent advances in VLSI technology have made poissible the implementation of image sensors using standard CMOS ASIC’s [4]. Such image sensors benefit from the integration of image sensing and image processing functions on the same die, yielding a vast reduction in power consumption and system mass. These savings make possible the realization of a completely self-contained foveated image sensor for use on mobile robots. We have designed, fabricated and tested such a device for use in a robot eye for an autonomous robot system [3]. In the present discus- sion we summarize key design issues and give iresults of the functioning sensor.

Foveation

The concept of foveation in machine vision stems from a detailed examination of the human visual pathway [ B ] . The human retina can be roughly divided into two distinct regions. The f ovea is a small area of very high,

J 0

Mapping template Periphery image

Figure 1: T h e foveated mapping

constant photoreceptor density located near the center of the retinal plane. Outside the fovea, in the region known as the periphery, visual acuity decreases as a function of radial distance from the center of the retina, due to spatial averaging of incident intensity performed over regions known as receptzve f ields.

Based on psychophysical experiments, researchers have characterized the image transformation performed by the visual pathway in mathematical terms. This nonlinear image transformation is known as the log-polar or foveated mapping. Fig. 1 illustrates how the mapping is performed. Image coordinates are mapped from the original image via a mapping template (a) to separate images for the fovea and the periphery (b). D a t a in the original image corresponding to the fovea undergoes a one-to-one mapping to the fovea image. Data corresponding to the periphery undergoes a many-to-one mapping in which all image values within a receptive field (RF) are averaged to produce a single value in the periphery image. RF’s in the periphery are distributed along rays of angular displacement , AQ. All RF’s on ring i have radial displacement given by,

15.5.1

IEEE 1995 CUSTOM INTEGRATED CIRCUITS CONFERENCE

357

0-7803-2584-2/95 $3.00 01995 IEEE

Figure 5: Photomicrograph of the CMOS foveated sensor

_j X

Fovea image

‘4

Original image

(4

Periphery image

(b)

Figure 6: Example of sensor performance. (a) original image with a cross marked A at the center of gaze. (b) sensor output . The detailed features of the central part of the original image are preserved in the fovea image, while i ts surroundings are mapped to the periphery image with a log-polar function as indicated in Fig. 1. Horizontal lines in the periphery image are due t o digital scanners in the external circuitry.

15.5.4 360

tiling · 2020. 9. 23. · tiling •tiling by cones •tiling by retinal ganglion cells - midget...

Documents