Student: Chao-Hong Meng

Advisor: Lin-Shan Lee

June 15, 2009

Phone Sequence sh iy hv ae

MFCC/PLP

Structural SVMHMM

Motivation

Baseline Model

Structural SVM

Experiments

Conclusion

Speech Recognition:

Modeling is hard

Traditional approach:

Acoustic Model

Language Model

Structural SVM:◦ A model can handle structural output

◦ Formulated by Joachims

◦ Directly model

As a preliminary research◦ is MFCC/PLP/Posterior

◦ is phone sequence

Motivation

Baseline Model

Structural SVM

Experiments

Conclusion

Baseline Model:◦ Monophone HMM

◦ Tandem

Difference is their inputs:◦ HMM: MFCC/PLP

◦ Tandem: Posterior

Motivation

Baseline Model

Structural SVM

Experiments

Conclusion

Define a compatibility function◦ is feature vector sequence

◦ is the correct phone sequence of

Goal is to find a such that

High to Low

We assumes that

is inner product

is combined feature function to encode the relationship of x and y

In this research, the transition count and emission count are considered

The output of :

◦ (transition count, emission count)

Settings:◦ 3 different phone labels

◦ 2-dim feature vectors

A training sample:

Calculate◦ Transition count matrix A

◦ Emission count matrix B:

A

B

C

A B C

A

B

C

z1

z2

z2z1

Concate the rows of A and B

The output of consists of◦ transition count

◦ emission count

Output label set (phone set) with size K:

Kronecker delta:

Define

◦ Only one element is not zero in

Tensor product:

Define as follows:

Emission Count

Transition Count

For different pairs of (x, y), there is a different value of

Recall we define compatibility function as follows

So we have a different preference to y for a given x

w to be estimated Obtain from training data

contains the information about transition and emission

w

Y

X

Purpose:

◦ Given a training sample , we want pairs with answer be the first.

◦ The gap between first ant the others be as large as possible

Margin:

Training sample 1

Training sample 2

Training sample 3

Maximise

The primal form of Structural SVM:

Where are slack variables, which could let margin be negative.

means the margin should be at least

Error tolerance weight

Motivation

Baseline Model

Structural SVM

Experiments

Conclusion

Corpus: TIMIT◦ Continuous English◦ Phone set: 48 phones

Frontend: ◦ MFCC/PLP + delta + delta delta (39 dim in total)◦ processed by CMS

HMM◦ 3 state HMM for each phone.◦ 32 Gaussian Mixture in each state.

Tandem◦ 1000 hidden nodes◦ Looking at 4 previous frames and 4 next frames and current frames◦ Reduce to 37 dimension with PCA

Training set Testing set

# of Sentences 3696 192

Structural SVM◦ Define as previously stated:

◦ The output dimension of

48 * 48 + 48 * 39 = 4176

Emission Matrix

Transition Matrix

Assume First Order Hypothesis

Could be Second Order

48 48

39 48

58.00%

60.00%

62.00%

64.00%

66.00%

68.00%

70.00%

72.00%

HMM

HMM Tandem

Best: 70.42%

Better

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

Primal C=10

Dual C=10

In theory, they should be the same

Solving Primal Form

Solving Dual Form

35.00%

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

C=1

C=10

C=100

C=1000

Slack Variable Weight

Accuracy increases with C

Input: PosteriorInput: MFCC/PLPBest: 71.75%

35.00%

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

C=1 C=10 C=100 C=1000

MFCC

PLP

MLP-MFCC

MLP-PLP

PCA-37-MLP-MFCC

PCA-37-MLP-PLP

Without dim-reduction is better than dim-reduction

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

HMM

SVM-struct

absolute1.33% improvement

71.75% 70.42%

Getting Worse

35.00%

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

First, C=1

Second, C=1

First, C=10

Second, C=10

C = 1

C = 10

Better Better

Motivation

Baseline Model

Structural SVM

Experiments

Conclusion

Structural SVM performs badly when input is MFCC/PLP

But Structural SVM can improve absolute 1% over Tandem when using Posterior Probability.