![Page 1: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/1.jpg)
On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)
Huicheng Zheng, Christophe Laurent and Grégoire Lefebvre
13th September 2006
Thanks to the MUSCLE Internal Fellowship (http://www.muscle-noe.org).
ICANN’06
![Page 2: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/2.jpg)
2
Outline• Introduction• Minimization of the ASSOM objective
function• Fast-learning methods
– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments• Conclusions
![Page 3: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/3.jpg)
3
Motivation of ASSOM
• Learning “invariance classes” with subspace learning and SOM [Kohonen. T., et al., 1997]– For example: spatial-translation invariance
rectangles
circles
triangles
……
![Page 4: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/4.jpg)
4
Applications of ASSOM• Invariant feature formation
[Kohonen, T., et al., 1997]• Speech processing
[Hase, H., et al., 1996]• Texture segmentation
[Ruiz del Solar, J., 1998]• Image retrieval
[De Ridder, D., et al., 2000]• Image classification
[Zhang, B., et al., 1999]
![Page 5: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/5.jpg)
5
ASSOM Modules Representing Subspaces
The module arrays in ASSOM
Rectangular topology
Hexagonal topology
1b 2b Hb
Q
x
1Tbx
2ˆ )(L jx
2Tbx Hbx
T
A module representing the subspace L(j)
c
i
j
ci
j
![Page 6: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/6.jpg)
6
Competition and Adaptation
• Repeatedly:– Competition: The winner
– Adaptation: For the winner and the modules i in its neighborhood
– Orthonormalize the basis vectors
2ˆmaxarg )(L j
jc x
)(')()( ),( ih
ic
ih t bxpb
xx
xxIxp
)(ˆ)()(),(
T)()(
i
thtt ic
ic
L
N×N matrix:)(' i
hb
)(ihb )(i
cp
![Page 7: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/7.jpg)
7
Transformation Invariance• Episodes correspond to signal subspaces.• Example:
– One episode, S, consists of 8 vectors. Each vector is translated in time with respect to the others.
![Page 8: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/8.jpg)
8
Episode Learning
• Episode winner
• Adaptation: for each sample x(s) in the episode X={x(s), s S} – Rotate the basis vectors
– Orthonormalize the basis vectors
Ss
jsc j
2)(ˆmaxarg )(L
x
)(')()( )),(( ih
ic
ih ts bxpb
![Page 9: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/9.jpg)
9
Deficiency of the Traditional Learning Rule
• Rotation operator pc(i)(x(s),t) is an N×N matrix.
– N: input vector dimension
• Approximately:NOP (number of operations) ∝ MN2
– M: subspace dimension
![Page 10: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/10.jpg)
10
Efforts in the Literature
• Adaptive Subspace Map (ASM) [De Ridder, D., et al., 2000]:– Drop topological ordering– Perform a batch-mode updating with PCA– Essentially not ASSOM.
• Replace the basis updating rule [McGlinchey, S.J., Fyfe, C., 1998]– NOP ∝ M2N
![Page 11: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/11.jpg)
11
Outline• Introduction• Minimization of the ASSOM objective
function• Fast-learning methods
– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments• Conclusions
![Page 12: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/12.jpg)
12
Minimization of the ASSOMObjective Function
XXx
xd)(
)(
)(~
2
2
)( )(
Ps
shE
i Ss
ic
iL
)()( ˆ~ii LL
xxx where:
(projection error)
P(X): probability density function of X
Solution: Stochastic gradient descent:
)('2
T)()(
)(
)()()()( i
hSs
ic
ih
s
sstht b
x
xxIb
)(t : Learning rate function
![Page 13: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/13.jpg)
13
Minimization of the ASSOMObjective Function
)(tWhen is small:
Ss
ic
Ss
ic
s
sstht
s
sstht 2
T)(
2
T)(
)(
)()()()(
)(
)()()()(
x
xxI
x
xxI
In practice, better stability has been observed by the modified form proposed in [Kohonen, T., et al., 1997]
Ss
ic
ic
ss
ssthtt
i )()(ˆ
)()()()()(
)(
T)()(
xx
xxIM
L
![Page 14: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/14.jpg)
14
Minimization of the ASSOMObjective Function
• corresponds to a modified objective function:)()( ticM
XXx
xd)(
)(
)(ˆ )()(m P
s
shE
i Ss
ic
iL
• Solution to Em:
Ss
ic
ic
ss
ssthtt
i )()(ˆ
)()()()()(
)(
T)()(
xx
xxIB
L
• When is small:)(t
)()( )()( tt ic
ic BM
![Page 15: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/15.jpg)
15
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
![Page 16: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/16.jpg)
16
Insight on the Basis Vector Rotation
• Recall: traditional learning
)()(ˆ
)()()()()),((
)(
)()(
ss
ssthtts
i
Ti
ci
cxx
xxIxp
L
)(')()( )),(( ih
ic
ih ts bxpb
![Page 17: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/17.jpg)
17
Insight on the Basis Vector Rotation
)(ihb
)(),()(,
)( stsihc
ih xb
)()(ˆ
)()()(),(
)(
)(')()(
,ss
sthtts
i
ih
Ti
cihc
xx
bx
L
)()(ˆ
)()()()(
)(
)(')()(')(
ss
sstht
i
ih
Ti
ci
hi
hxx
bxxbb
L
scalar
projection
• For fast computing, calculate first, then scale x(s) with to get
• NOP ∝ MN
• Referred to as FL-ASSOM (Fast-Learning ASSOM)
),()(, tsihc
Scalar
),()(, tsihc )(i
hb
![Page 18: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/18.jpg)
18
Insight on the Basis Vector Rotation
)(ihb
)(' ihb
)(),()(,
)( stsihc
ih xb
)(sx
![Page 19: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/19.jpg)
19
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
![Page 20: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/20.jpg)
20
Batch-mode Fast Learning(BFL-ASSOM)
• Motivation: Re-use the previously calculated during module competition.
)()(ˆ
)()()(),(
)(
)('T)()(
,ss
sthtts
i
ihi
cihc
xx
bx
L
• In the basic ASSOM, L(i) keeps changing with receiving of each component vector x(s). has to be re-calculated for each x(s).
)(ˆ )( siLx
)(ˆ )( siLx
![Page 21: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/21.jpg)
21
Batch-mode Rotation• Use the solution to the modified objective
function Em:
Ss
Ti
ci
css
ssthtt
i )()(ˆ
)()()()()(
)(
)()(
xx
xxIB
L
• Subspace remains the same for all the component vectors in the episode. We can now use calculated during module competition.
)(ˆ )( siLx
![Page 22: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/22.jpg)
22
Batch-mode Fast Learning
Ss
ihc
ih sts )(),()(
,)( xb
)()(ˆ
)()()(),(
)(
)(')()(
,ss
sthtts
i
ih
Ti
cihc
xx
bx
L
where ),()(, tsihc is a scalar defined by:
• Correction is a linear combination of component vectors x(s) in the episode.
• For each episode, one orthonormalization of basis vectors is enough.
)(ihb
![Page 23: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/23.jpg)
23
Outline
• Introduction
• Minimization of the ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
![Page 24: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/24.jpg)
24
Experimental Demonstration• Emergence of translation-invariant filters
– Episodes are drawn from a colored noise image
– Vectors in episodes are subject to translation
white noise image colored noise image
• Example episode (magnified):
![Page 25: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/25.jpg)
25
Resulted Filters
1b
2b
FL-ASSOM BFL-ASSOM
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
6
0 5 10 15 20 25 30(×10
3)
FL-ASSOM
BFL-ASSOM
e
t
Decrease of the average projection error e with learning step t:
![Page 26: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/26.jpg)
26
Timing Results
Times given in seconds for 1,000 training steps.
M: subspace dimension
N: input vector dimension
VU: Vector Updating time
WL: Whole Learning time
![Page 27: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/27.jpg)
27
Timing Results
0
200
400
600
800
1000
1200
50 100 200 400 N
VU(s)
ASSOM
FL-ASSOM
BFL-ASSOM
0
200
400
600
800
1000
1200
2 3 4 M
VU(s)
ASSOMFL-ASSOM
BFL-ASSOM
Change of vector updating time (VU) with input dimension N:
Change of vector updating time (VU) with subspace dimension M:
Vertical scales of FL-ASSOM and BFL-ASSOM have been magnified 10 times for clarity.
![Page 28: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/28.jpg)
28
Outline
• Introduction
• Minimization of ASSOM objective function
• Fast-learning methods– Insight on the basis vector rotation– Batch-mode basis vector updating
• Experiments
• Conclusions
![Page 29: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/29.jpg)
29
Conclusions
• The basic ASSOM algorithm corresponds to a modified objective function.
• Updating of basis vectors in the basic ASSOM correponds to a scaling of the component vectors in the input episode.
• In batch-mode updating, the correction to the basis vectors is a linear combination of component vectors in the input episode.
• Basis learning can be dramatically boosted with the previous understandings.
![Page 30: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/30.jpg)
30
References• De Ridder, D., et al., 2000: The adaptive subspace map for image
description and image database retrieval. SSPR&SPR 2000.
• Hase, H., et al., 1996: Speech signal processing using Adaptive Subspace SOM (ASSOM). Technical Report NC95-140, The Inst. of Electronics, Information and Communication Engineers, Tottori University, Koyama, Japan.
• Kohonen, T., et al., 1997: Self-Organized formation of various invariant-feature filters in the adaptive-subspace SOM. Neural Computation 9(6).
• McGlinchey, S. J., Fyfe, C., 1998: Fast formation of invariant feature maps. EUSIPCO’98.
• Ruiz del Solar, J., 1998: Texsom: texture segmentation using Self-Organizing Maps. Neurocomputing 21(1–3).
• Zhang, B., et al., 1999: Handwritten digit recognition by adaptive-subspace self-organizing map (ASSOM). IEEE Trans. on Neural Networks 10:4.
![Page 31: On the Basis Learning Rule of Adaptive-Subspace SOM (ASSOM)](https://reader031.vdocuments.us/reader031/viewer/2022013004/5681598b550346895dc6d283/html5/thumbnails/31.jpg)
31
Thanks and questions?