facial expression recognition using kcca with combining correlation kernels and kansei information
DESCRIPTION
Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information. Yo Horikawa Kagawa University , Japan. 1. Purpose of this study Apply kernel canonical correlation analysis (kCCA) with correlation kernels to facial expression recognition - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/1.jpg)
1
Facial Expression Recognition using KCCA with Combining Correlation
Kernels and Kansei Information
Yo Horikawa
Kagawa University , Japan
![Page 2: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/2.jpg)
2
1. Purpose of this study
Apply kernel canonical correlation analysis (kCCA) with correlation kernels to facial expression recognition
・ Combining multi-order correlation kernels
・ Use of Kansei information
![Page 3: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/3.jpg)
3
2. Facial expression recognition
Facial images → Expressions
Basic six expressions:
happiness, sadness, surprise, anger, disgust, fear
Happy Sad Surprised Angry Disgusted Fearful Neutral
![Page 4: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/4.jpg)
4
3. Kernel canonical correlation analysis (KCCA)
Pairs of feature vectors of sample objects: (xi, yi) (1 ≤ i ≤ n)
Canonical variates (u, v): projections with the maximum correlation between (implicit) nonlinear functions h(xi ) and h’(yi ).
u = wφ ・ h(x) v = wθ ・ h’(y) u
h(x)h’(y) v
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
![Page 5: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/5.jpg)
5
Canonical variates are calculated with the kernel function.
u = ∑i=1n fiφ(xi, x)
v = ∑i=1n giθ(yi, y)
Kernel functions: the inner products of implicit functions of x and y
φ(xi, xj) = h(xi ) ・ h(xj ) θ(yi, yj) = h’(yi ) ・ h’(yj )
f = t(f1, ∙∙∙, fn) and g = t(g1, ∙∙∙, gn): the eigenvectors of the generalized eigenvalue problem:
0 ΦΘ f Φ2+γxI 0 f = λ ΘΦ 0 g 0 Θ2+γyI g
Φ=Φij =φ(xi, xj) Θ=Θij =θ(yi, yj) (1 ≤ i, j ≤ n)
I: Identity matrix of n×n
![Page 6: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/6.jpg)
6
4. KCCA for classification problems and use of Kansei information
4.1 CCA for classification problems
Use an indicator vector (IV) as the second feature vector y.
y = (y1, ∙∙∙, ync) corresponding to x: yc = 1 if x belongs to class c yc = 0 otherwise
(nc: the number of classes) In the recognition of 6 basic facial expressions
(happiness, sadness, surprise, anger, disgust, fear, neutral)
y = ( 1 , 0 , 0 , 0 , 0 , 0 , 0 )
![Page 7: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/7.jpg)
7
Canonical variates ur (1 ≤ r ≤ nc-1) for a new object (x, ?) are calculated by
ur =∑i=1n friφ(xi, x) (1 ≤ r ≤ nc-1)
Standard classification methods are applied in the canonical variate space (u1, …, unc-1).
KCCA
IV
(1, 0, 0, 0, 0, 0, 0) y
x ImageCanonical variates
(u1, …, u6)
- 0.20
- 0.15
- 0.10
- 0.05
0.00
0.05
0.10
0.15
- 0.15 - 0.10 - 0.05 0.00 0.05 0.10 0.15
u1
u2
- 0.15
- 0.10
- 0.05
0.00
0.05
0.10
0.15
- 0.15 - 0.10 - 0.05 0.00 0.05 0.10 0.15
u3
u4
- 0.15
- 0.10
- 0.05
0.00
0.05
0.10
0.15
0.20
- 0.20 - 0.15 - 0.10 - 0.05 0.00 0.05 0.10 0.15 0.20
u5
u6
![Page 8: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/8.jpg)
8
4.2 Kansei information and its use in KCCA
Semantic ratings on the expressions of facial images by human
Ratings on the basic six expressions: (happiness, sadness, surprise, anger, disgust, fear)
( 5 , 2 , 3, 1, 0, 1 )
Semantic rating vector (SRV)
KCCACanonical variates
(u1, …, u5)
SRV
(4.39, 1.35, 2.29, 1.16, 1.23, 1.26) y
x Image
Classifiers
![Page 9: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/9.jpg)
9
5. Correlation kernel
Correlation kernel: an inner product of the autocorrelation functions of the feature vectors.
rxi(t1) =∫xi(t)xi(t+t1)dt rxj(t1) =∫xj(t)xj(t+t1)dt
φ(xi, xj) =∫rxi(t1) ・ rxj(t1) dt1
xi(t) xj(t)
-1.5
-1
-0.5
0
0.5
1
1.5
2
0 0.5 1 1.5 2 2.5 3 3.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
0 0.5 1 1.5 2 2.5 3 3.5
-1
-0.5
0
0.5
1
1.5
2
2.5
-1.5 -1 -0.5 0 0.5 1 1.5
0
0.2
0.4
0.6
0.8
1
1.2
-1.5 -1 -0.5 0 0.5 1 1.5
![Page 10: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/10.jpg)
10
The kth-order autocorrelation of data xi(t):
rxi(t1, t2, ∙∙∙ , tk-1) = ∫xi(t)xi(t+t1) ・・・ xi(t+tk-1)dt
The inner product between rxi and rxj is calculated with the k-th power of the 2nd-order cross-correlation function:
rxi・ rxj =∫{ccxi, xj(t1)}k dt1
ccxi, xj(t1) =∫xi(t)xj(t+t1)dt
The calculation of explicit values of the autocorrelations is avoided. → Higher-order autocorrelations are tractable with practical computational cost.
Linear correlation kernel:φ(xi(t), xj(t)) = rxi・ rxj
![Page 11: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/11.jpg)
11
Calculation of correlation kernels rxi・ rxj for 2-dimensional image data: x(l, m) (1≤ l ≤ L, 1≤ m ≤ M)
・ Calculate the cross-correlations between xi(l, m) and xj(l, m):
ccxi, xj(l1, m1) = ∑l=1L-l1∑m=1
M-m1 xi(l, m)xj(l+l1, m+m1)/(LM)
(1 ≤ l1 ≤ L1, 1 ≤ m1 ≤ M1)
・ Sum up the kth-power of the cross-correlations:
rxi・ rxj = ∑l1=0L1-1∑m1=0
M1-1 {ccxi, xj(l1, m1)}k /(L1M1)
![Page 12: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/12.jpg)
12
Modified correlation kernels Higher-order and odd-order correlation kernels are less performed.
Correlation kernel (C) : ∑l1, m1 ccxi, xj(l1, m1)k (14)
→ Modified correlation kernels with the kth root and absolute values
Lp norm kernel (P):
rxi∙rxj = sgn(ccxi, xj(l1, m1))|∑l1,m1{ccxi, xj(l1, m1)}k|1/k (15)
Absolute correlation kernel (A): rxi∙rxj = ∑l1, m1 |ccxi, xj(l1, m1)|k (16)
Absolute Lp norm kernel (AP): rxi∙rxj = |∑l1, m1{ccxi, xj(l1, m1)}k |1/k (17)
Absolute Lp norm absolute kernel (APA):
rxi∙rxj = |∑l1, m1|ccxi, xj(l1, m1)|k|1/k (18)
Max norm kernel (Max): rxi∙rxj = max l1, m1 ccxi, xj(l1, m1) (19)
Max norm absolute kernel (MaxA): rxi∙rxj = max l1, m1 |ccxi, xj(l1, m1)| (20)
![Page 13: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/13.jpg)
13
6. Combining correlation kernels
Multiple classifiers may give higher performance than a single classifiers.
Cartesian spaces of the canonical variates obtained with a set of the kernel functions
e.g.,
U = (u1, ···, unc-1), U’ = (u’1, ···, u’nc-1), U” = (u”1, ···, u”nc-1)
→ U U’ U” = (⊗ ⊗ u1, ···, unc-1, u’1, ···, u’nc-1, u”1, ···, u”nc-1)
Classifier 1 ‘7’
Classifier 2 ‘9’
Classifier 3 ‘7’
Classifier 4 ‘7’ ・・・
Classifier M ‘1’
Objects
Final decision ‘7’
![Page 14: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/14.jpg)
14
7. Facial expression recognition experiment
Object: JAFFE (Japanese female facial expression) database
213 facial images of 10 Japanese females
3 or 4 examples of each of 6 basic facial expressions
(happiness, sadness, surprise, anger, disgust, fear)
and a neutral face
8bit gray scale valued of 256×256 pixels
Happy Sad Surprised Angry Disgusted Fearful Neutral
Figure 1. Sample images in JAFFE database. From left to right: happiness, sadness, surprise, anger, disgust, fear, neutral.
![Page 15: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/15.jpg)
15
ALL 213 images in JAFFE database
![Page 16: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/16.jpg)
16
Center regions of 200×200 pixels are taken.
They are resized to 20×20 pixels with averaging of 10×10 pixels.
Preprocessing: linear normalization with the mean 0 and SD 1.0.
Happy Sad Surprised Angry Disgusted Fearful Neutral
Figure 2. Images of 20×20 real valued matrix data of Fig. 1 as the first feature x in kCCA.
![Page 17: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/17.jpg)
17
Semantic rating vectors (SRVs) in JAFFE database
Averages of semantic ratings on 6 expressions (happiness, sadness, surprise, anger, disgust, fear) on 5 point scales obtained from 60 Japanese females
(4.39, 1.35, 2.29, 1.16, 1.23, 1.26)
Happy
(HAP, SAD, SUR, ANG, DIS, FEA)
(4.77, 1.29, 2.45, 1.26, 1.23, 1.23)
(1.39, 3.97, 1.68, 2.19, 3.68, 3.61)
Images SRV
Sad
Happy
(2.87, 1.55, 4.68, 1.52, 1.52,1.65 )
Surprised
(HAP, SAD, SUR, ANG, DIS, FEA)
(1.55, 1.90, 2.10, 4.32, 3.90, 1.81)
(3.03, 2.16, 2.06, 1.94, 1.84, 1.87)
Images SRV
Neutral
Angry
![Page 18: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/18.jpg)
18
Experiment ( ) ⅠFacial expression recognition with 2 images for each expression per person Sample set: 2 images×7 expressions×10 persons
= 140 images Test set: the remaining 73 images
Experiment ( ) ⅡFacial expression recognition with a leave one-person out method Sample set: Images of 9 persons (about 190 images) Test set: Images of the remaining 1 person
(about 20 images)Averaging 10 tests
![Page 19: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/19.jpg)
19
The kernel function φ and the second feature vector y is shown with the set (φ, y) using the following symbols.
For the kernel function φ of image data Kth-order correlation kernel: Ck Kth-order Lp norm kernel: Pk Kth-order absolute correlation kernel: Ak Max norm kernel: Max etc.,
Total 44 kinds
For the second feature vector y Indicator vector: IV Impression vector: SRV
E.g., (C2, SRV) is the 2nd-order correlation kernel and the semantic rating vectors.
Classifiers in the canonical space: Nearest neighbor method
![Page 20: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/20.jpg)
20
8. Results of the experiment
Experiment ( ) ⅠCorrect classification rates (CCRs) with single classifiers
0.6
0.7
0.8
0.9
1.0
LG
0.1
G1
G10 C2
C3
C4
C5
C6
C7
C8
C9
C10 P2
P3
P4
P5
P6
P7
P8
P9
P10 A3
A5
A7
A9
AP3
AP5
AP7
AP9
APA3
APA5
APA7
APA9 M K S3
S4
S5
S6
S7
S8
S9
S10
Max
Kernel
CC
R
IVSRV
Figure 3. CCR with single kernel functions of 44 kinds in the experiment (I). IV: indicator vector, SRV: semantic rating vector.
Highest CCR (the most right-hand side) is obtained with Indicator vectors (IV), not with semantic rating vectors (SRV).
Highest (SRV) 89% Highest (IV) 94.5%
![Page 21: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/21.jpg)
21
Experiment No ofkernels
Kind of2nd vector Kernels of highest CCR CCR (%)
HighestCCR (%) inthe paststudies
IV (A5, I) 94.5SRV (C4, SRV) 89.0IV (C2. IV), (C5, IV) 95.9
IV + SRV (L, SRV), (Q3, SRV) 95.9IV (L, IV), (A9, IV), (S4, IV) 95.9
IV + SRV (C5, IV), (C10, IV), (C2, SRV) 97.3
(Ⅰ )
1
94.6 [5]2
3
Table 1( ). Highest CCR with single kernel functions and combining Ⅰtwo and three kernel functions in the experiment ( ).Ⅰ
Combining correlation kernels increases CCRs.
Highest CCR (97.3%) is superior to the past studies (94.6%)
It is obtained with not only indicator vectors (IV) but also semantic rating vectors (SRV).
![Page 22: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/22.jpg)
22
Experiment ( ) Ⅱ
Figure 4. Highest CCRs with kernel functions of 44 kinds in 10 tests in the experiment (II). Single kernel (a), combining two (b) and three (c) kernels.
IV: indicator vector, SRV: semantic rating vector.
(a) 1 kernel
0.40.50.60.70.80.91.0
1 2 3 4 5 6 7 8 9 10 Ave
Test No.
CC
R IVSRV
(b) 2 kernels
0.4
0.50.6
0.70.8
0.91.0
1 2 3 4 5 6 7 8 9 10 AveTest No.
CC
R IVIV + SRV
(c) 3 kernels
0.4
0.50.6
0.7
0.80.9
1.0
1 2 3 4 5 6 7 8 9 10 AveTest No.
CC
R IVIV + SRV
Highest average CCR (the most right-hand side in Fig. 4(c)) is again obtained with combining 3 kernels including semantic rating vectors (SRV).
![Page 23: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/23.jpg)
23
Table 1( ). Highest CCR with single kernel functions Ⅱand combining two and three kernel functions in the experiment ( ).Ⅱ
Experiment No ofkernels
Kind of2nd vector Kernels of highest CCR CCR (%)
HighestCCR (%) inthe paststudies
IV (P10, I) 51.1SRV (P10, SRV) 46.8IV (L, I), (P5, I) 63.3
IV + SRV (P5, I), (G1, SRV) 61.2IV (L, I), (C9, I), (M, I) 65.2
IV + SRV (A7, I), (M, I), (C9, SRV) 67.0
(Ⅱ )
1
(77.0 [6])2
3
Highest CCR (67.0%) is again obtained with combining 3 kernels including not only indicator vectors but also semantic rating vectors (SRV).
![Page 24: Facial Expression Recognition using KCCA with Combining Correlation Kernels and Kansei Information](https://reader036.vdocuments.us/reader036/viewer/2022081603/56814084550346895dac0a70/html5/thumbnails/24.jpg)
24
9. Conclusion
KCCA with multiple correlation kernels and Kansei information was applied to facial expression recognition through the experiment with JAFFE database.
High correct classification rates (CCRs) equivalent to the past studies were obtained with correlation kernel CCA without any feature extraction.
Combining multiple correlation kernels and Kansei information with the semantic rating vectors contributed to increase CCRs.