Text-to-speech synthesizer based on combination of composite wavelet andhidden Markov models

Nobukatsu Hojo1, Kota Yoshizato1, Hirokazu Kameoka1,2,Daisuke Saito1, Shigeki Sagayama1,∗

1Graduate School of Information Science and Technology, the University of Tokyo, Japan2 Communication Science Laboratories, NTT Corporation, Japan

{hojo, yoshizato, kameoka, dsaito, sagayama}@hil.t.u-tokyo.ac.jp

AbstractThis paper proposes a text-to-speech synthesis (TTS) sys-tem based on a combined model consisting of the Compos-ite Wavelet Model (CWM) and the Hidden Markov Model(HMM). Conventional HMM-based TTS systems using cep-stral features tend to produce over-smoothed spectra, whichoften result in muffled and buzzy synthesized speech. Thisis simply caused by the averaging of spectra associated witheach phoneme during the learning process. To avoid the over-smoothing of generated spectra, we consider it important to fo-cus on a different representation of the generative process ofspeech spectra. In particular, we choose to characterize speechspectra using the CWM, whose parameters correspond to thefrequency, gain and peakiness of each underlying formant. Thisidea is motivated by our expectation that the averaging of theseparameters would not lead directly to the over-smoothing ofspectra, as opposed to the cepstral representations. To describethe entire generative process of a sequence of speech spectra, wecombine the generative process of a formant trajectory using anHMM and the generative process of a speech spectrum usingthe CWM. A parameter learning algorithm for this combinedmodel is derived based on an auxiliary function approach. Weconfirmed through experiments that our speech synthesis sys-tem was able to generate speech spectra with clear peaks anddips, which resulted in natural-sounding synthetic speech.Index Terms: text-to-speech synthesis, hidden Markov model,composite wavelet model, formant, Gaussian mixture model,auxiliary function

1. IntroductionThis paper proposes a new model for text-to-speech synthe-sis (TTS). One promising approach for TTS involves methodsbased on statistical models. In this approach, the first step isto formulate a generative model of a sequence of speech fea-tures. The second step is to train the parameters of the assumedgenerative model given a set of training data in a speech cor-pus. The third step is to produce the most likely sequence ofspeech features given a text input and transform it into a speechsignal. With this approach, one key to success is that the as-sumed generative model reflects the nature of real speech well.To model the entire temporal evolution of speech features, ahidden Markov model (HMM) and its variants including the“trajectory HMM” have been introduced with notable success[1, 2, 3]. HMMs are roughly characterized by the structure ofthe state transition network (i.e., a transition matrix) and an out-put distribution assumption. In conventional HMM-based TTSsystems, a Gaussian mixture emission distribution of a cepstral

feature [1, 2] or a line spectrum pair (LSP) feature [4, 3] is typ-ically used as the output distribution of each state. The aim ofthis paper is to seek an alternative to the conventional speechfeature and the state output distribution. Namely, this paper isconcerned with formulating a new model for the generative pro-cess of speech spectra and combining it with an HMM.

The Gaussian distribution assumption of a cepstral featuredescribes probabilistic fluctuations of spectra in the power di-rection, while that of an LSP feature describes probabilisticfluctuations of spectral peaks in the frequency direction. Sincethe frequencies and powers of peaks in a spectral envelope cor-respond to the resonance frequencies and gains of the vocaltract, they both vary continuously over time according to thephysical movement of the vocal tract. In particular, resonancefrequencies and gains both vary significantly in the boundarybetween one phoneme and another. To achieve higher qualityspeech synthesis, we consider it important to describe a gen-erative model that takes account of the fluctuations of spec-tral peaks in both the frequency and power directions ratherthan a fluctuation in just one direction. Conventional HMM-based TTS systems using cepstral features tend to produce over-smoothed spectra, which often result in muffled and buzzy syn-thesized speech. This is simply caused by the averaging of ob-served log-spectra assigned to each state during the training pro-cess (Fig. 2 (a)). Although some attempts were made to empha-size the peaks and dips of generated spectra via post-processing[5], it is generally difficult to restore original peaks and dipsonce spectra are over-smoothed. By contrast, this paper pro-poses tackling this problem by introducing a different spectralrepresentation called the Composite Wavelet Model (CWM)[6, 7, 8] as an alternative to cepstral and LSP representations,on which basis we formulate a new generative model for TTS.

2. Generative model of spectral sequence2.1. Motivation

Taking the mean of cepstral features associated with a particu-lar phoneme amounts to taking the mean of the correspondinglog spectra. Since the resonance frequencies and gains of thevocal tract both vary continuously during the transition fromone phoneme to another, if we simply take the mean of the logspectra, the spectral peaks and dips will be be blurred and indis-tinct (Fig. 2 (a)). By contrast, if we can appropriately treat thefrequencies and powers of individual spectral peaks as speechfeatures, we expect that taking their means will not cause theblurring of spectra (Fig. 2 (b)). CWM approximates a speechspectral envelope from the sum of the Gaussian distributions,

8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, Spain

129

Figure 1: Spectral representation based on CWM

interpreted as a function of frequency (see Fig. 1 for a graphicalillustration). This means each Gaussian distribution functionroughly corresponds to a peak in a spectral envelope. CWMis thus convenient for describing both the frequency and powerfluctuations of spectral peaks.

Another notable feature of CWM is that its parameters canbe easily transformed into a signal. Since a Gaussian functionin the frequency domain corresponds to a Gabor function in thetime domain, CWM parameters can be directly converted intoa signal by superimposing the corresponding Gabor functions.CWM can thus be regarded as a speech synthesizer with an FIRfilter and its superiority to conventional systems with IIR filtershas already been shown in [6].

One straightforward way of incorporating CWM into anHMM-based TTS system would be to first perform CWM pa-rameter extraction on a frame-by-frame basis, and then trainthe parameters of an HMM by treating the extracted CWM pa-rameters as a sequence of feature vectors. However, our pre-liminary investigation revealed that this simple method did notwork well [9]. One reason for this is the common difficultyof formant tracking. Although formants and their time courses(formant trajectories) are clearly visible to the human eye in thespectrum and in the spectrogram, automatic formant extractionand tracking are far from trivial. Many formant extraction al-gorithms miss a formant that is present, insert a formant whenthere is none, or mislabel them (such as label F1 as F2, or F3

as F2), mostly as a result of incorrectly pruning the correct for-mant at the frame level. This is also the case for frame-by-frameCWM parameter extraction since it can be thought of as a sortof formant extraction. Fig. 3 shows an example of results ob-tained with frame-by-frame CWM analysis. As this exampleshows, the index of each Gaussian distribution function in theCWM at one particular frame is not always consistent with thatat another frame. This was problematic when training the HMMparameters since the mean of the emission distribution of eachstate is given as the average of the feature vectors assigned tothe same state. To train the HMM parameters appropriately, theindices of the Gaussian functions of the CWM assigned to thesame state must always be consistent. This implies the need forthe joint modeling of the CWM and HMM. Motivated by theabove discussion, we here describe the entire generative pro-cess of a sequence of speech spectra by combining the genera-tive process of a speech spectrum based on the CWM with thegenerative process of a CWM parameter trajectory based on anHMM.

2.2. Formulation

The CWM consists of parameters (roughly) corresponding tothe frequency and power of spectral peaks. Specifically, theCWM approximates a spectral envelope by employing a Gaus-sian mixture model (GMM) interpreted as a function of fre-

frequency

power

(a) Mean in cepstral domain

frequency

power

(b) Mean in CWM domain

Figure 2: Expectations of spectra taken in different domains.The dashed lines indicate the training samples and the solidlines indicate the mean spectra.

quency. Namely, the CWM fω,l is defined as

fω,l =

KX

k=1

wk,l√2πσk,l

exp

„

− (ω − µk,l)2

2σ2k,l

«

, (1)

where ω, l denotes the indices of frequency and time, respec-tively, and K is the number of mixture components of theGMM. See Fig. 1 for a graphical illustration. µk,l, σk,l and wk,l

are the mean, variance and weight of each Gaussian (when in-terpreted as a probability density function), and thus correspondto the frequency, peakiness and power of the peaks in a spectralenvelope, respectively.

Using the CWM representation given above as a basis, herewe describe the generative process of an entire sequence of ob-served spectra. We consider an HMM that generates a set con-sisting of µk,l, ρk,l := 1/σ2

k,l, and wk,l at time l (see Fig. 4).Each state of the HMM represents a label indicating linguis-tic information, which can be simply either a phoneme label asshown in Fig. 4 or a context label (as used in [10]). Given statesl at time l, we assume that the CWM parameters are generatedaccording to

P (µk,l|sl) = N (µk,l; mk,sl , η2k,sl

), (2)

P (ρk,l|sl) = Gamma(ρk,l; a(ρ)k,sl

, b(ρ)k,sl

), (3)

P (wk,l|sl) = Gamma(wk,l; a(w)k,sl

, b(w)k,sl

), (4)

where N (x; m, η2) denotes the normal distribution andGamma(x; a, b) the gamma distribution:

Gamma(x; a, b) = xa−1 exp(−x/b)

Γ(a) ba. (5)

These distribution families are chosen for simplifying thederivation of the parameter estimation algorithm described inthe next section. Given the following sequence of CWM pa-rameters, — = {µk,l}k,l, ȷ = {ρk,l}k,l, w = {wk,l}k,l, wefurther assume that a spectrum {yω,l}ω observed at time l isgenerated according to

P (yω,l|—, ȷ, w) = Poisson(yω,l; fω,l), (6)

where Poisson(x; λ) denotes the Poisson distribution:

Poisson(x; λ) =λxe−λ

x!. (7)

This assumption is also made for simplifying the derivation ofthe parameter estimation algorithm described in the next sec-tion. fω,l denotes the sequence of spectrum models defined by

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama

130

0 0.1 0.2 0.3 0.4 0.50

2

4

6

8

Time (s)

Fre

quen

cy (

kHz)

(a) The frame-independent CWM parameter extractionmethod [9]

0 0.1 0.2 0.3 0.4 0.50

2

4

6

8

Time (s)

Fre

quen

cy (

kHz)

(b) The proposed method

Figure 3: An example of the formant frequencies extractedusing (a) the frame-independent CWM parameter extractionmethod [9] and (b) the proposed method. The extracted for-mant frequencies are plotted with different colors according tothe indices of the Gaussian distribution functions in CWM.

Eq. 1. It should be noted that the maximization of the Poissonlikelihood with respect to fω,l amounts to optimally fitting fω,l

to yω,l by using the I-divergence as the fitting criterion [8]. Thenext section describes the parameter estimation algorithm of thegenerative model presented above.

3. Parameter estimation algorithmHere we describe the parameter learning algorithm given aset of training data. The spectral sequence of the train-ing data is considered to be an observed sequence and aset of unknown parameters of the present generative modelis denoted by Θ. Θ consists of the state sequence s ={sl}l, the parameters of the state emission distributions „ =

{mk,i, ηk,i, a(ρ)k,i , b

(ρ)k,i , a

(w)k,i , b

(w)k,i }k,i , and the CWM parame-

ters {—, ȷ, w}.Given a set of observed spectra Y = {yω,l}ω,l, we would

like to determine the estimate of Θ that maximizes the pos-terior density P (Θ|Y ) ∝ P (Y |Θ)P (Θ), or equivalently,log P (Y |Θ) + log P (Θ). Here, log P (Θ|Y ) is written as

log P (Θ|Y )c= log P (Y |Θ) + α log P (Θ), (8)

log P (Θ)c= log P (s) + log P (—|s, „)

+ log P (ȷ|s, „) + log P (w|s, „), (9)

������ ������

����

�����

����

����

⊹∱∻≬⊹∲∻≬≷∱∻≬≷∲∻≬ ≆∡∻≬

≙∡∻≬

�����

���� ∲

∶

∶

∴

⊹≫∻≬ ⊻ ≎ ∨≭≫∻≳≬ ∻ ⊺∲≫∻≳≬∩⊽≫∻≬ ⊻ ≇≡≭≭≡∨≡∨⊽∩≫∻≳≬ ∻ ≢∨⊽∩≫∻≳≬∩≷≫∻≬ ⊻ ≇≡≭≭≡∨≡∨≷∩≫∻≳≬ ∻ ≢∨≷∩≫∻≳≬∩∳

∷

∷

∵

≙∡∻≬ ⊻ ≐≯≩≳≳≯≮ ∨≆∡∻≬∩

≳≬

Figure 4: Illustration of the present HMM

where α is a regularization parameter that weighs the impor-tance of the log prior density relative to the log-likelihood. c

=denotes equality up to constant terms. Unfortunately, this op-timization problem is non-convex, and finding the global opti-mum is an intractable problem. However, we can employ anauxiliary function approach to find a local optimum [8].

As mentioned above, we notice from Eqs. (6) and (7) that− log P (Y |Θ) is equal up to constant terms to the sum of theI-divergence between yω,l and fω,l

I(Θ) :=X

ω,l

“

yω,l logyω,l

fω,l− yω,l + fω,l

”

. (10)

Hence, maximizing P (Θ|Y ) amounts to minimizing I(Θ) −α log P (Θ) with respect to Θ. By invoking Jensen’s inequalitybased on the concavity of the logarithm function, we obtain aninequality

I(Θ) =X

ω,l

“

yω,l logyω,l

fω,l− yω,l + fω,l

”

(11)

≤X

ω,l

n

yω,l log yω,l − yω,l

−X

k

“

λk,ω,l loggk,ω,l

λk,ω,l+ gk,ω,l

”o

, (12)

where

gk,ω,l =

r

ρk,l

2πexp

„

− ρk,l

2(ω − µk,l)

2

«

. (13)

By using J (Θ, λ) to denote the upper bound of I(Θ), i.e., theright-hand side of (12), equality holds if and only if

λk,ω,l =wk,lgk,ω,l

PKj=1 wj,lgj,ω,l

. (14)

Now, when λk,ω,l is given by Eq. (14) with an arbitrary Θ, theauxiliary function J (Θ, λ)−α log P (Θ) becomes equal to theobjective function I(Θ) − α log P (Θ). Then, the parameterthat decreases J (Θ, λ)−α log P (Θ) while keeping λ fixed willnecessarily decrease I(Θ)−α log P (Θ), since inequation (12)guarantees that the original objective function is always even

8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, Spain

131

smaller than the decreased J (Θ, λ) − α log P (Θ). Therefore,by repeating the update of λ by Eq. (14) and the update of Θthat decreases J (Θ, λ) − α log P (Θ), the objective functiondecreases monotonically and converges to a stationary point.

With fixed s and „, the auxiliary function J (Θ, λ) −α log P (Θ) is minimized with respect to the CWM parameters,—, ȷ, and w, under the following updates

µk,l =Dk,lη

2k,iρk,l + αmk,i

Ck,lη2k,iρk,l + α

, (15)

ρk,l =Ck,l + 2α(a

(ρ)k,l − 1)

Ck,lµ2k,l − 2Dk,lµk,l + Ek,l − 2α/b

(ρ)k,l

, (16)

wk,l =Ck,l + α(a

(w)k,l − 1)

1 + α/b(w)k,l

, (17)

where

Ck,l =X

ω

yω,lλk,ω,l, (18)

Dk,l =X

ω

ωyω,lλk,ω,l, (19)

Ek,l =X

ω

ω2yω,lλk,ω,l. (20)

With fixed —, ȷ, w and „, the state sequence minimizingJ (Θ, λ) − α log P (Θ) can be obtained by using the Viterbialgorithm. With fixed —, ȷ, w and s, the auxiliary function isminimized under the following updates.

As for {mk,i, ν2k,i}k,i,

mk,i =1

Ni

X

l∈Ti

µk,l (21)

ν2k,i =

1

Ni

X

l∈Ti

µ2k,l − (

1

Ni

X

l∈Ti

µk,l)2, (22)

where Ti = {l|sl = i}. As for {a(ρ)k,i , b

(ρ)k,i , a

(w)k,i }k,l, b

(w)k,i }k,l,

although the updates are not derived in a closed form, they areupdated as the root of the following equations

log a(ρ)k,i − ψ(a

(ρ)k,i) = log(

1

Ni

X

l∈Ti

ρk,l) − 1

Ni

X

l∈Ti

ρk,l

(23)

log a(w)k,i − ψ(a

(w)k,i ) = log(

1

Ni

X

l∈Ti

wk,l) − 1

Ni

X

l∈Ti

wk,l

(24)

b(ρ)k,i =

1Ni

P

l∈Tiρk,l

a(ρ)k,i

(25)

b(w)k,i =

1Ni

P

l∈Tiwk,l

a(w)k,i

, (26)

where ψ(a) denotes the digamma function

ψ(a) =∂Γ(a)

∂a/Γ(a). (27)

A spectrogram of sentence A01 from the ATR503 data set‘Arayurugenjitsuwo subete jibunnohohe nejimagetanoda’ andan example of the trajectory of the mean parameters of theCWM extracted from it are shown in Fig. 5.

Figure 5: Spectrogram of a natural voice and the trajectories ofextracted frequency parameters (a Japanese sentence of A01 inthe ATR-503 data set).

4. Speech Synthesis Experiment andEvaluation

4.1. Evaluation criterion

To evaluate the spectral distortion caused by the training pro-cess, we measured the spectral distortion between the synthe-sized speech and real speech.

Bark spectral distortion [11] is known to be an objectivemeasure for incorporating psychoacoustic responses. We usedBark spectral distortion and log spectral distortion to mea-sure the spectral divergence between the synthesized and realspeech. Using this criterion, we compared the present methodwith HTS-2.1 [10], a conventional HMM speech synthesis sys-tem with mel-cepstrum features and the global variance model[?].

The time alignment between the two spectral sequences wascomputed by using the dynamic time warping (DTW) algo-rithm. The average distortions on 53 synthesized speech sen-tences were compared and a statistical test was conducted.

4.2. Experimental Conditions

All the phoneme boundaries of the training data were given bythe phoneme labels of HTS [10]. This means that the state se-quences of the HMM are assumed to be known in the train-ing stage. The number of the Gaussian functions in the CWMwas set at 24. The initial CWM parameters were determinedby performing the method reported by [9] to estimate the CWMparameters from the entire spectral sequence corresponding toeach phoneme. We used an acoustic model of 1 state and a left-to-right HMM with no skip, as with HTS-2.1 [10]. We used“STRAIGHT” [12] to compute the spectral envelopes of thetraining data, with an analysis interval of 5 ms. We used 450uttered sentences for the training and 53 uttered sentences forthe synthesis and evaluation, respectively. All the speech sam-ples were uttered by a Japanese male speaker and recorded witha 16kHz sampling rate and a 16 bit quantization, which weretaken from the demo site of HTS [13].

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama

132

log spectral distortion cepstral distortion0.8

0.85

0.9

0.95

1

1.05

Nor

mal

ized

Dis

tort

ion

CWMmcep−gvmcep−nongv

Figure 6: Normalized squared distances between the log spec-tra and cepstra of real and synthesized speech by the proposedmethod (red) , the ordinary mel-cepstral method with gv (blue)and without gv (green).

0 2 4 6 8−10

−8

−6

−4

−2

0

2

Frequency(kHz)

log

pow

er

CWMmcepsample

Figure 7: An example of the synthesized spectral envelope cor-responding to the phoneme ’/e/’

4.3. Experimental Results

The measured spectral distortions are shown in Fig. 6. The dis-tances were normalized according to the values obtained withthe proposed and conventional methods. We confirmed througha statistical test that the Bark spectral distortion obtained withthe proposed method was significantly smaller than that ob-tained with the conventional method, while the two methodswere comparable in terms of the log spectrum distortion. Thisresult can be interpreted as showing that the proposed modelwas able to synthesize spectra properly especially in a low fre-quency region. Since the changes in the peak frequencies ofspectra are mainly seen in the low frequency region, this re-sult indicates that the proposed model is superior to the conven-tional model as regards modeling fluctuations in the frequencydirection. Fig. 7 shows an example of the synthesized spectralenvelope along with the spectral envelope of real speech cor-responding to the phoneme ’/e/’. In addition to the superiorityof the proposed model in terms of the Bark spectral distortion,the proposed method was able to restore spectral peaks and dipsmore clearly than the conventional model, especially in the 1.5to 6 kHz frequency range. In future we plan to conduct a sub-jective evaluation test to confirm whether the perceptual qualityof the synthesized speech has actually been improved.

5. ConclusionThis paper proposed a text-to-speech synthesis (TTS) sys-tem based on a combined model consisting of the Compos-ite Wavelet Model (CWM) and the Hidden Markov Model(HMM). To avoid the over-smoothing of spectra generated bythe conventional HMM-based TTS systems using cepstral fea-tures, we considered it important to focus on a different repre-sentation of the generative process of speech spectra. In particu-lar, we chose to characterize the speech spectra with the CWM,whose parameters correspond to the frequency, gain and peak-iness of each underlying formant. This idea was motivated byour expectation that averaging these parameters would not di-rectly cause the over-smoothing of the spectra, unlike the cep-stral representations. To describe the entire generative processof a sequence of speech spectra, we combined the generativeprocess of a formant trajectory using an HMM and the gener-ative process of a speech spectrum using the CWM. We devel-oped a parameter learning algorithm for this combined modelbased on an auxiliary function approach. We confirmed throughexperiments that our speech synthesis system was able to gen-erate speech spectra with clear peaks and dips, which resultedin natural-sounding synthetic speech.

6. References[1] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and

T. Kitamura, “Simultaneous modeling of spectrum, pitchand duration in HMM-based speech synthesis,” in Proc.of Eurospeech 1999, 1999, pp. 2347–2350.

[2] H. Zen, K. Tokuda, and T. Kitamura, “An introduction oftrajectory model into HMM-based speech synthesis,” inProc. of 5th ISCA Speech Synthesis Workshop, 2004, pp.191–196.

[3] Y. Qian, Z.-J. Yan, Y.-J. Wu, F. K. Soong, G. Zhang, andL. Wang, “An HMM trajectory tiling (HTT) approach tohigh quality TTS,” in Proc. of Interspeech 2010, 2010, pp.422–425.

[4] F. Itakura, “Line spectrum representation of linear predic-tor coefficients of speech signals,” J. Acoust. Soc. Am.,vol. 57, no. 1, p. S35, 1975.

[5] T. Toda and T. Tokuda, “A speech parameter generationalgorithm considering global variance for HMM-basedspeech synthesis,” IEICE Trans. Inf. & Syst., vol. E90-D,no. 5, pp. 816–824, 2007.

[6] T. Saikachi, K. Matsumoto, S. Sako, and S. Sagayama,“Discussion about speech analysis and synthesis by com-posite wavelet model,” in Proc. ASJ Spring Meeting,vol. 2, 2006, pp. 89–94, in Japanese.

[7] P. Zolfaghari and T. Robinson, “Formant analysis usingmixtures of Gaussians,” in Proc. of ICSLP ’96, vol. 2,1996, pp. 1229–1232.

[8] H. Kameoka, N. Ono, and S. Sagayama, “Speech spec-trum modeling for joint estimation of spectral envelopeand fundamental frequency,” IEEE Trans. Audio, Speech& Language Process., vol. 18, no. 6, pp. 1507–1516,2010.

[9] N. Hojo, K. Minami, D. Saito, H. Kameoka, andS. Sagayama, “HMM speech synthesis using speech anal-ysis based on composite wavelet model.” in Proc. ASJ Au-tumn Meeting, no. 2-7-7, 2013, in Japanese.

8th ISCA Speech Synthesis Workshop • August 31 – September 2, 2013 • Barcelona, Spain

133

[10] H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. W.Black, and K. Tokuda, “The HMM-based speech synthe-sis system version 2.0,” in Proc. Proc. of 6th ISCA SpeechSynthesis Workshop, vol. 6, 2007, pp. 294–299.

[11] S. Wan, A. Sekey, and A. Gersho, “An objective measurefor predicting subjective quality of speech coders,” IEEEJournal on Selected Areas in Communications, vol. 10,no. 5, pp. 819–829, 1992.

[12] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Re-structuring speech representations using a pitch-adaptivetime-freqency smoothing and an instantaneous-frequency-based f0 extraction,” Speech Communication, vol. 27, no.3–4, pp. 187–207, 1999.

[13] http://hts.sp.nitech.ac.jp/.

N. Hojo, K. Yoshizato, H. Kameoka, D. Saito, S. Sagayama

134