ee 225a digital signal processing supplementary material 1 ...ee225a/sp12/supplement.pdf · prof....

PROF. EDWARD A. LEE, DEPARTMENT OF EECS, UNIVERSITY OF CALIFORNIA, BERKELEY

EE 225A — DIGITAL SIGNAL PROCESSING — SUPPLEMENTARY MATERIAL 1

EE 225aDigital Signal ProcessingSupplementary Material

1. Allpass Sequences

A sequence is said to be allpass if it has a DTFT that satisfies

. (1)

Note that this is true iff

(2)

which in turn is true iff

(3)

which in turn is true iff

. (4)

For rational Z transforms, the latter relation implies that poles (and zeros) of are cancelled

by zeros (and poles) of . In other words, if has a pole at , then must

have have a zero at the same , in order for (4) to be possible. Note further that a zero of

at is a zero of at . Consequently, (4) implies that poles (and zeros)

ha n( )

Ha ejω( ) 1=

Ha ejω( )Ha* ejω( ) 1=

ha n( )*ha* n–( ) δ n( )=

Ha z( )Ha* 1

z*----

1=

Ha z( )

Ha* 1

z*----

Ha z( ) z c= Ha* 1

z*----

z c=

Ha* 1

z*----

z c= Ha z( ) z 1

c*-----=



have zeros (and poles) in conjugate-reciprocal locations, as shown below:

Intuitively, since the DTFT is the Z transform evaluated on the unit circle, the effect on the mag-nitude due to any pole will be canceled by the effect on the magnitude of the corresponding zero.Although allpass implies conjugate-reciprocal pole-zero pairs, the converse is not necessarily trueunless the appropriate constant multiplier is selected to get unity magnitude. Consider the Z trans-form of the form

. (5)

This has a pole at and a zero at 1/ , as shown in the above figure. It is easy to verify that it isallpass, so an allpass Z transform may consist of the product of any number of such terms. Anynon-unity multiplier on (5) would prevent this from being allpass, by the definition in (1).

Note further that a “pure delay” term for , which is obviously allpass, has poles atzero and zeros at infinity, so it automatically satisfies the allpass property. So does a “pure ad-

vance” term for .A transfer function of the form

(6)

is a product of a pure delay or advance term and terms of form (5), and hence is allpass. Conversely,in order to have pole-zero pairs at conjugate reciprocal locations, and unity magnitude on the unitcircle, any rational allpass Z transform must have form (6). This can also be written

(7)

where

. (8)

c

1/c∗

Ha z( ) z 1– c*–

1 cz 1––-------------------=

c c*

z M– M 0> MM

zM M 0>

Ha z( ) zM z N– a1*z N– 1+ ... aN

*+ + +

1 a1z 1– ... aNz N–+ + +-----------------------------------------------------------=

Ha z( ) A z( )

zN M– A∗ 1z∗-----

--------------------------------=

A z( ) 1 a1*z ... aN

* zN+ + +=



Evaluating this at it is easy to directly verify that (7) is allpass.Note that there is no need for the allpass filter to be causal for it to be stable. It can have poles atinfinity, with corresponding zeros at zero. It can also have poles anywhere outside the unit circle,as long as there is a corresponding zero at the reciprocal-conjugate location.

2. Minimum-Phase Sequences

Definition: The sequence with rational Z transform is minimum phase if it:

1. is stable,2. is causal, and3. has all zeros inside or on the unit circle.

Be careful with zeros at infinity, as introduced for example by an uncanceled term, which makea sequence non-minimum phase.The sequence is strictly minimum phase if it is minimum phase and also has no zeros on the unitcircle. An LTI system is said to be minimum phase if its impulse response is a minimum-phasesequence.Importance: A strictly minimum-phase system has a stable, causal, and minimum-phase inverse.This is easily seen by observing that the poles of the inverse are zeros of the system, and zeros ofthe inverse are poles of the system.

Fact: Any stable rational Z transform can be factored into a cascade of a minimum-phaseand an allpass Z transform,

. (9)

Proof is by construction: Locate the poles and zeros of . Assign all those inside the unit or onthe unit circle to . For each zero (or pole) outside the unit circle at location , assign to

a zero (or pole) at . Then assign a pole (or zero), to cancel this zero (or pole), at

z ej ω=

hm n( ) Hm z( )

z 1–

H z( )

H z( ) Ha z( )Hm z( )=

H z( )Hm z( ) c

Hm z( ) 1/c∗ 1/c∗



to , and a zero (or pole) at to make the filter allpass. In pictures,

Then select the constant multipliers so that is allpass. Notice in the above example that is not real (because the poles and zeros are not in complex-conjugate pairs), and furthermore it isnot causal (because if it is stable, the ROC is annular). Consequently, the allpass filter is also notcausal.

Key idea: The minimum-phase filter has an inverse that has the inverse magnitude frequen-

cy response of . If has no zeros on the unit circle, then will be strictly minimum

phase, so will be stable, causal, and minimum phase. Hence, the factorization in (9) allowsus to invert any magnitude frequency response, as long as there are no zeros on the unit circle.

3. Real-Valued Discrete-Time Fourier Transforms

The DTFT of a sequence is real iff

(10)

which is true iff is conjugate-symmetric,

. (11)

Taking Z transforms on both sides, we see that is real-valued iff

. (12)

Since is real, the phase modulo is zero, so is sometimes called a zero-phase se-

Ha z( ) c

H z( ) Ha z( )

=

Hm z( )

Ha z( ) h n( )

Hm1– z( )

H z( ) H z( ) Hm z( )

Hm1– z( )

H ej ω( ) h n( )

H ej ω( ) H∗ ej ω( )=

h n( )

h n( ) h∗ n–( )=

H ej ω( )

H z( ) H∗ 1z∗-----

=

H ej ω( ) π h n( )



quence. Note, however, that this terminology ignores the ambiguity of multiples of .

Equation (12) implies that if there is a zero at , then there is also a zero at . Simi-larly, if there is a pole at , then there is also a pole at , as shown in the followingdiagram:

Note that this not the same type of symmetry encountered for allpass sequences, which had pole-zero pairs at conjugate-reciprocal locations.

These symmetry relationships also apply to poles and zeros at zero and infinity. A factor for in a Z transform introduces zeros (or poles) at without matching zeros (or poles)

at , so a zero-phase Z transform cannot have such a term. Correspondingly, except for thetrivial case , a zero-phase sequence is neither causal nor anticausal. Any poles at zeromust be matched by an equal number of poles at infinity.Although a real-valued DTFT implies pole-pole and zero-zero pairs at conjugate-reciprocal loca-tions, note that the converse is not true unless the multiplicative constant is carefully selected.Consider the following example,

. (13)

This has a zero at and a zero at . It also has a pole at zero (due to the first factor) and

a pole at infinity (due to the second factor). The general form of this example can be written

(14)

Evaluating this on the unit circle we see that

, (15)which is not only real, but also non-negative. A related example,

(16)

is real, but not non-negative on the unit circle. A product or ratio of some number of terms of forms

π

z c= z 1/c∗=z d= z 1/d∗=

c

1

c*-----

d

1

d*-----

zM

M 0≠ M z 0=z ∞=

H z( ) 1=

H z( ) 1 cz 1––( ) 1 c*z–( )=

z c= z 1

c*-----=

H z( ) A z( )A* 1

z*----

=

H ejω( ) A e jω( )A* ejω( ) A e jω( )2

= =

H z( ) 1 cz 1––( )– 1 c*z–( )=



(13) and (16) will be real on the unit circle, but is not necessarily non-negative.

4. Linear-Phase Sequences

Suppose

(17)

where is real. Then

modulo , (18)

so is called a linear-phase sequence. Again, the terminology ignores the ambiguity of mul-tiples of . If is an integer, then

(19)so that

. (20)

This might take a two-sided finite sequence and make a causal version.

5. Positive Semi-Definite Sequences

The sequence with DTFT is said to be positive semi-definite (or non-negative defi-nite) if is real and

. (21)

As with any zero-phase sequence, a pole (or zero) at implies a pole (or zero) at . However,now we also have the stronger result that zeros or poles on the unit circle must be double.

Theorem: If the sequence has a real-valued DTFT and rational Z transform ,then zeros or poles on the unit circle must be double.

F e j ω( ) e jωτ– H ej ω( )=

H ej ω( )

F∠ ej ω( ) ωτ= π

f n( )π τ N=

F z( ) z N– H z( )=

f n( ) h n N–( )=

h n( )

h n( ) H ej ω( )H ej ω( )

H ej ω( ) 0≥

c 1/c*

h n( ) H ejω( ) H z( )



Since the proof is rather involved, it is worth giving an intuitive argument first. A single pole orzero on the unit circle causes a phase change of when the frequency response moves across it.Consequently, if the frequency response is positive on one side of the lone zero or pole, it will benegative on the other side. Showing this formally is a bit more painful.

Also note that it follows from this theorem that any positive semi-definite can be factored asin (14). The theorem, as stated, justifies that the poles and zeros can be separated as in (14). Fur-thermore, if we assign all the poles and zeros that are inside or on the unit circle to the first factorin (14), we see that we can write the factorization as

(22)

where is minimum phase. In the course of proving the above theorem, we will clear out anydoubts about the constant multipliers in the above factorization.

Proof: We have already shown that a real DTFT implies that for every pole or zero at there mustbe a corresponding pole or zero at the reciprocal conjugate location. This is not sufficient to showthat the zeros and poles on the unit circle are double however. To do this, factor the Z transform asfollows

(23)

where contains all the poles and zeros that are not on the unit circle, and is written in theform

. (24)

It is easy to verify that this is real-valued and non-negative on the unit circle. For simplicity, wenow assume has no poles on the unit circle (i.e. it is stable). This assumption is not necessary,but it simplifies the proof. In this case, the remaining factor can be written

(25)

where accounts for all the poles and zeros at zero and infinity not accounted for in (24),

π

H z( )

H z( ) Hm z( )Hm* 1

z*----

=

Hm z( )

c

H z( ) Hs z( )Hu z( )=

Hs z( )

Hs z( )1 ciz

1––( ) 1 ci*z–( )

i 1=

N

∏

1 diz1––( ) 1 di

*z–( )i 1=

M

∏---------------------------------------------------------=

H z( )

Hu z( ) CzL 1 aiz1––( )

i 1=

K

∏=

zL



accounts for all the zeros on the unit circle, and is some constant. Noting that each

term accounts for not just a zero at , but also a pole at , we might observe

that these poles need to be balanced with poles at infinity for the DTFT to be real. Since a factor gives us a pole at infinity and a zero at zero, we could conclude that it is necessary that .Since must be an integer, we could conclude immediately that must be even, so the numberof zeros on the unit circle must be even. This is true for any zero-phase sequence, regardless ofwhether it is positive semi-definite.

The above conclusion can be reinforced by observing that is real-valued iff (12) is satis-fied, or

(26)

It is clear from (24) that

(27)

so we need only ensure that

(28)

From (25), this requires that

. (29)

The polynomial on the left has powers of ranging from to . The one on the right haspowers of ranging from to . The only way these two ranges could be the same is if

, which means is even, as expected.Showing that the zeros must be double zeros is a bit more involved. First observe that it is necessarythat evaluated on the unit circle be real. Write this as

. (30)

Expanding each term in the product

ai ejθi= C

1 aiz1––( ) z ai= z 0=

zL K/2=

L K

H ej ω( )

H z( ) H s z( )Hu z( ) Hs* 1

z*----

Hu* 1

z*----

H 1

z*----

= = =

Hs z( ) H s* 1

z*----

=

Hu z( ) Hu* 1

z*----

=

Hu z( ) CzL 1 aiz1––( )

i 1=

K

∏ C*z L– 1 ai*z–( )

i 1=

K

∏ Hu* 1

z*----

= = =

z L L K–z L– L– K+

K 2L= K

Hu z( )

Hu ejω( ) CejωL 1 ejθie jω––

i 1=

2L

∏=



(31)

which leads to

(32)

where

. (33)

For this to be real, it is necessary and sufficient that

, (34)

for any real valued constant . For it to be real and non-negative, it is necessary and sufficient

(35)and that the zeros are double, so that we can write (32) as

(36)

where we have numbered the zeros so that . It should be clear that these conditions aresufficient. To show they are necessary, observe that if we have a zero that is not a double zero, thenthere will be some region that crosses only one zero. The term in

(32) will change signs in this region, so it cannot be non-negative on both sides of . Q.E.D.

Hu ejω( ) Ce jωL ej θi ω–( )/2

ej– θi ω–( )/2

ej θi ω–( ) /2

–

i 1=

2L

∏=

Hu ejω( ) Ce jS/2 1–( )L θi ω–( )/2( )sini 1=

2L

∏=

S θii 1=

2L

∑=

C Ae j– S/2=

A

C Ae j– S/2 1–( )L=

Hu ejω( ) CejS/2 1–( )L θi ω–( )/2( )sin[ ]2

i 1=

L

∏=

θi θ2L i–=

θi ε– θi ε+,( ) θi ω–( )/2( )sin

θi



6. Filtering Random Processes

For WSS random process and LTI system ,

Define the cross correlations

(37)

and observe that

(38)

which we can write

. (39)

Since the ZT of is , the cross power spectrum follows from this

. (40)

The cross correlation is different in the other order,

(41)

or

(42)

or

. (43)

The cross power spectrum in this direction is

x n( ) H z( )

H z( )x n( ) y n( )

rxy m( ) E x n( )y* n m–( )[ ]=

rxy m( ) h* k( )E x n( )x* n m– k–( )[ ]k∑ h* k( )rx m k+( )

k∑= =

rxy m( ) h* m–( )*rx m( )=

h* n–( ) H* 1

z*----

Rxy z( ) H* 1

z*----

R z( )=

ryx m( ) E y n( )x* n m–( )[ ]=

ryx m( ) h k( )E x n k–( )x* n m–( )[ ]k∑ h k( )rx m k–( )

k∑= =

ryx m( ) h m( )*rx m( )=



. (44)

It is easy to show that is WSS, and to compute its power spectrum:

(45)

(46)

so

(47)

So the output power spectrum is

. (48)

7. Innovations Process & Spectral Factorization

Definition:

Define the “inverse filter”:

Ryx z( ) H z( )Rx z( )=

y n( )

ry m( ) E h k( )x n k–( )k∑

h i( )x n m– i–( )i

∑ *

=

= h k( ) h* i( )rx m i k–+( )i

∑k∑ h k( )rxy m k–( )

k∑=

ry m( ) h m( )*h* m–( )*rx m( )=

Ry z( ) H z( )H* 1

z*----

Rx z( )=

v n( )x n( )H z( )Assuming is WSS and

has no poles or zeroson the unit circle.

x n( )Rx z( )

“Whitening filter” — Caus-al, monic, and strictly mini-mum phase.

“Innovations process” —

white, .Rv z( ) σv2=

x n( )v n( )G z( )



where . Since is white, samples are uncorrelated, and hence in some senseeach one brings new information about the random process . When such a representation ex-ists, where is stable and causal, then is called a linear process.

The power spectrum of can be written

. (49)

This is called the “canonical spectral factorization” of .

Knowing the whitening filter and the power of its output implies knowing the power spectrum, be-

cause it implies knowing , and

. (50)

The “standard normalization” results from constraining and to be monic, i.e.

and . (51)Since these are causal,

and . (52)

This defines the value unambiguously. For rational polynomial forms,

(53)

The key result: Given any rational power spectrum without poles or zeros on the unit circle, thereexists a canonical spectral factorization, and hence a monic, minimum-phase whitening filter, anda monic minimum-phase synthesis filter. This is a form of Wold’s theorem.

The proof is by construction. gets the poles and zeros inside the unit circle, leaving the ones

outside for .

G z( ) H 1– z( )= v n( )x n( )

G z( ) x n( )

x n( )

Rx z( ) σv2G z( )G* 1

z*----

=

Rx z( )

G z( ) H 1– z( )=

Rx ejω( ) σv2 G ejω( )

2=

G z( ) H z( )

g 0( ) 1= h 0( ) 1=

G ∞( ) 1= H ∞( ) 1=

σv2

H z( )1 β i z 1––( )∏1 αi z 1––( )∏

---------------------------------=

G z( )

G* 1/z*( )



E.g.:

is positive semi-definite, so zeros on the unit circle are double, and create no difficulties.

8. Power of the Innovations Process

Fact: the power of the innovations process equals the geometric mean of the power spectrum:

. (54)

This can be recognized as a geometric mean by considering the limiting case of

, (55)

where the sum becomes an integral.From spectral factorization we can write

. (56)

We need to show that the second term is zero, given that is monic and minimum phase.

Recall that the complex logarithm of a complex number is

G* 1

z*----

G z( )

Sx z( )

σv2 exp 1

2π------ ln Rx ω( )[ ]dω

π–

π

∫

=

xnn 1=

N

∏N1N---- xnln

n 1=

N

∑

exp=

12π------ ln Rx ω( )[ ]dω

π–

π

∫ ln σv2( ) 1

2π------ ln G ej ω( )

2dω

π–

π

∫+=

G z( )

a



(57)

for any integer . The principal value uses . This expression for the complex logarithm iseasy to understand from the form

. (58)We do not always need or want the principal value. It is easy to see that we can always pick appro-priate values of each of the logarithms (not necessarily the principal values) such that

. (59)Hence, whenever we write such a relation, we insist that compatible values of the multivalued log-arithms be used.Consider

. (60)

Note that has all poles and zeros inside or on the unit circle. Hence, it is analytic and non-zero outside the unit circle. This makes analytic outside the unit circle, which impliesthat there exists a causal sequence such that

. (61)

This sequence is called the complex cepstrum of . Although is not necessarily stable (itis unstable if has zeros on the unit circle), it is nonetheless bounded (proven below). Hence,

. (62)

Since is monic and causal, so . Furthermore, is the inverse DTFT

of evaluated at , so

. (63)

Note that the imaginary part of this integral is always zero for a real random process as longas we choose values of the multi-valued complex logarithm that are conjugate symmetric about

. We require such a choice.

ln a( ) l n a j a∠ 2kπ+( )+=

k k 0=

a a ej a∠ 2kπ+( )=

ln ab( ) ln a( ) ln b( )+=

ln G z( )G* 1

z*----

ln G z( )[ ] ln G* 1

z*----

+=

G z( )ln G z( )[ ]

c k( )

ln G z( )[ ] c k( )z k–

k 0=

∞

∑=

g n( ) c k( )G z( )

c 0( ) ln G ∞( )[ ]=

G z( ) G ∞( ) 1= c 0( ) 0= c 0( )

ln G ej ω( )[ ] n 0=

c 0( ) 12π------ ln G ej ω( )[ ]dω

π–

π

∫ 0= =

x n( )

ω 0=



Use a similar argument for the integral of , observing that is anticausal and

monic to get

. (64)

Combining these two zero-valued integrals,

. (65)

Hence,

(66)

and the desired result follows.

The only loose end is to prove that the complex cepstrum is bounded even when it is not sta-ble (i.e. when has zeros on the unit circle). We will show this only for rational polynomialpower spectra, for which can be written as

, (67)

where and . For this form,

(68)

where again we must use compatible values for the complex logarithm.

Consider each term . It can be written as a power series expansion

ln G* 1

z*----

G* 1

z*----

12π------ ln G* ej ω( )[ ]dω

π–

π

∫ 0=

12π------ ln G ej ω( )

2dω

π–

π

∫ 0=

12π------ ln Rx ω( )[ ]dω

π–

π

∫ ln σv2( )=

c n( )G z( )

G z( )

G z( )1 αi z 1––( )

i 1=

N

∏

1 βi z 1––( )i 1=

M

∏------------------------------------=

αj 1≤ βj 1<

ln G z( )( ) ln 1 αi z 1––( )i 1=

M

∑ ln 1 βi z 1––( )i 1=

N

∑–=

ln 1 αi z 1––( )



, (69)

valid for , or . From this expression we can recognize the inverse Z transform,

(70)

Note that although this is not absolutely summable if (and hence is not stable), it is

bounded for all . Furthermore,

(71)

is bounded for all .A similar argument can be applied to show that the anticausal sequence

. (72)

is also bounded.

ln 1 αi z 1––( )αiz

1–( )n

n--------------------

n 1=

∞

∑–=

αiz1– 1< z αi>

ZT 1– ln 1 αi z 1––( )[ ]

0; n 0≤

αin–

n---------; n 0>

=

αi 1=

n

ZT 1– ln G z( )( )[ ] c n( )

0; n 0≤

αin

n------

i 1=

M

∑–βi

n

n----- ;

i 1=

N

∑+ n 0>

= =

n

ZT1–

ln G* 1

z*----



9. Wiener Filters

Objective is to minimize the power in (MSE criterion).

Example applications:1. (try to minimize the noise)2. (try to predict )3. (try to predict )

Assumption:• and are jointly WSS ( is independent of ).

Constraint:• is usually, but not always, constrained to be causal.

10. Wiener-Hopf Equations

Minimize the mean-squared error (MSE):

. (73)

Expanding this,

(74)

LTI

filterf n( )

s n( )

signal

w n( )

noise

x n( ) y n( )

our design d n( )

e n( )

e n( )

d n( ) s n( )=d n( ) s n m+( )= s n( )d n( ) x n m+( )= x n( )

x n( ) d n( ) E x n( )d* n m–( )[ ] m

f n( )

ε E d n( ) y n( )– 2[ ]=

ε E d n( )d* n( )[ ]=



(75)

(76)

Hence

(77)

This is a real-valued quadratic function of the possibly complex variables for all . This sug-gests that by taking the partial derivative of with respect to each , and setting this to zero, wecan find that values of that minimize . However, this derivative in general does not exist. In

particular, does not exist anywhere. However, note that is a real-valued quadratic function

of the real and imaginary parts and of for all . Hence, we can take the partial de-

rivatives with respect to each of these real-valued variables and set those to zero. The values of that make the partial derivatives zero yield a minimum if the function is unimodal. We assumefor now that it is.Instead of setting the partial derivatives to zero, we can equivalently set the following complex gra-dients1 to zero,

(78)

and

1. Note that the complex gradient has other definitions, such as . This would serve equally

well for our optimization problem and is used in [S. M. Kay, Modern Spectral Estimation, Prentice-Hall, 1988] and [S. Haykin, Adaptive Filter Theory, 2nd Ed., Prentice-Hall, 1991]. Our definition is used in [C. W. Therrien, Discrete Random Signals and Statistical Signal Processing , Prentice-Hall, 1992] and [D. H. Brandwood, “A Complex Gradient Operator and its Application in Adaptive Array Theory,” IEE Proceedings, 130(1), pp. 11-16, February 1983]. The key advantage of our definition is that if the function being differentiated is an analytic

function of , then .

f * k( )E d n( )x* n k–( )[ ]k∑ f k( )E d* n( )x n k–( )[ ]

k∑––

f m( )f * k( )E x n m–( )x* n k–( )[ ]m∑

k∑+

ε rd 0( ) 2Re f * k( )rdx k( )k∑

– f m( )f * k( )rx k m–( )m∑

k∑+=

f i( ) iε f i( )

f i( ) ε

fddf*

ε

fR i( ) f I i( ) f i( ) i

f i( )ε

εf

∇fR∂∂ j

fI∂∂+

ε=

ff∂

∂ε εf

∇=

εf i( )∇ 12---

fR i( )∂∂ j

fI i( )∂∂–

ε 0= =



(79)

Equivalence follows from the observation that

(80)

and

. (81)

With our definition of the complex gradient, it is easy to verify that

, , and (82)

and

, , and . (83)

Using these we get

. (84)

It is easy to show that

(85)

(using the conjugate symmetry of ), so it is sufficient to set only one of the two gradients to

zero (either one). Setting we get the Wiener-Hopf Equations:

(86)

A solution to these equations will minimize . We can easily modify these equations to constrainthe filter to be causal by just modifying the limits on the summation,

εf

*i( )

∇ 12---

fR i( )∂∂ j

fI i( )∂∂+

ε 0= =

εf * i( )

∇ εf i( )∇+fR i( )∂

∂ε=

εf * i( )

∇ εf i( )∇– jfI i( )∂

∂ε=

f *f∇ 0= ff∇ 1= f f *

f∇ f *=

f *f *∇ 1= f

f *∇ 0= f f *f *∇ f=

εf * i( )

∇ r– dx i( ) f k( )rx i k–( )k∑+=

εf i( )∇ εf * i( )

∇( )*=

rx m( )

εf * i( )

∇ 0=

f k( )rx i k–( )k∑ rdx i( ) , for all i=

εf n( )



, for . (87)

11. Some Easy Examples:

1. Suppose . Then . The Wiener-Hopf equations are satisfied by

. (88)Interpretation:

2. Suppose , where is white and independent of (a very strange situation. Again , so the solution is the same.Interpretation:

Because is independent of the noise , there is nothing the filter can do to remove it. So the solution is the same as before.But a more realistic scenario would be for to be independent of .

3. Suppose is white. I.e. . Then the Wiener-Hopf equations are satisfied by

f k( )rx i k–( )k 0=

∞

∑ rdx i( )= i 0≥

d n( ) x n( )= rdx i( ) rx i( )=

f n( ) δ n( )=

x n( )

d n( )

e n( )

δ n( )

d n( ) s n( ) x n( ) w n( )–= = w n( ) x n( )rdx i( ) rx i( )=

LTI

filterf n( )

s n( )

signal

w n( )

noise

x n( ) y n( )

d n( )

e n( )

x n( ) w n( )w n( )

s n( )x n( ) rx k( ) σ2δ k( )=



, (89)

where is the unit step function (so this filter is causal). Knowing the joint statistics of and , we can find the MMSE filter that estimates from . The solution is

easy when is white. For other situations, we can use a whitening filter to construct a general solution.

12. Non-Causal Wiener Filter Solution

Equation (86), which has no causality constraint on the Wiener filter, can be written

. (90)

Taking Z transforms and solving for we get

. (91)

This is sometimes called the unrealizable Wiener filter, but we prefer to call it the non-causalWiener filter.

13. Causal Wiener Filtering, Rational Power Spectra

The causality constraint in (87) complicates things considerably. Conceptually break the filter intotwo parts:

f n( )rdx n( )

σ2--------------- u n( )=

u n( )x n( ) d n( ) d n( ) x n( )

x n( )

f i( )*rx i( ) rdx i( )=

F z( )

F z( )Rdx z( )Rx z( )---------------=

f ′ n( )s n( )

w n( )

x n( ) y n( )

d n( )

e n( )h n( )

v n( )

f n( )



where is the whitening filter for , is the innovations process, and is the Wien-er filter designed for the white input. I.e.

(92)

Notation:

. (93)

(I.e., the Z transform of the nonnegative side only). With this

. (94)

A more convenient form would express as a function of , which is assumed to beknown.

14. Derivation of the General Solution

Note that since

(95)

(96)

(97)

. (98)

This relation can be written

h n( ) x n( ) v n( ) f ′ n( )

f ′ n( )rdv n( )

σv2

--------------- u n( )=

Rdv z( )[ ]+ rdv k( )z k–

k 0=

∞

∑=

F ′ z( ) 1

σv2

------ Rdv z( )[ ]+=

F′ z( ) Rdx z( )

v n( ) h k( )x n k–( )k 0=

∞

∑=

rdv m( ) E d i( )v* i m–( )[ ]=

h* k( )E d i( )x* i m– k–( )[ ]k 0=

∞

∑=

h* k( )rdx m k+( )k 0=

∞

∑=



. (99)

Taking Z transforms on both sides,

(100)

or

. (101)

Hence, the overall Wiener filter solution is

(102)

15. A More Involved Example

Assume in the following that is white noise, or :

Assume further that is white noise , uncorrelated with . Inwords, we are given noisy observations of a first-order AR process, and need to design a Wienerfilter to remove the noise as much as possible.

rdv m–( ) h* m( )*rdx m–( )=

Rdv1z---

H* z*( )Rdx1z---

=

Rdv z( ) H* 1

z*----

Rdx z( )=

F z( ) H z( )F ′ z( ) 1

σv2

------H z( ) H* 1

z*----

Rdx z( )+

= =

s n( )

w n( )

x n( ) y n( )

d n( )

e n( )f n( )1

1 az 1––-------------------

u n( )

u n( ) Ru z( ) σu2=

w n( ) Rw z( ) σw2=( ) s n( ) d n( )=



16. Linear Equalization

Scenario:

The “channel” is assumed to be causal and stable. The “data” is assumed to be white

with power spectrum . Our task is to design the Wiener filter to minimize the power in

. The general solution is

(103)

where is the power of the innovations process for , and is the whitening filter. To

get a more specific result, observe that we can factor into an allpass and a minimum phasecomponent,

, (104)

where by definition we scale these so that

. (105)

The whitening filter (which by definition is monic) is

, (106)

where is the constant needed to make it monic. The innovations process is generated by the sys-tem:

Note that since , where is allpass, then

F z( )x n( ) y n( )

C z( )d n( ) e n( )

C z( ) d n( )

σd2 F z( )

e n( )

F z( ) 1

σv2

------ H z( ) H* 1

z*----

Rdx z( )+

=

σv2 x n( ) H z( )

C z( )

C z( ) Cm z( )Ca z( )=

Ca ejω( ) 1=

H z( ) KCm1– z( )=

K

H z( )x n( ) v n( )

C z( )d n( )

C z( )H z( ) Ca z( )K= Ca z( )



. (107)

Moreover

. (108)

Putting this all together, plugging (106), (107), and (108) into (103) we get

. (109)

If happens to be minimum phase, i.e. and , then intuitively we

expect , which is exactly what we get. If it is not minimum phase, however, then

we have the term . Note that the inverse Z transform of is . But since

causal, the non-negative time part of this consists of exactly one sample, . Hence

(110)

and we can write the final solution as

. (111)

As a specific example, suppose

(112)

where , so the system is stable, but , so the system is not minimum phase. In this case,

(113)

and

. (114)

To verify this, observe that the product equals , that evaluates to unity on the unit cir-

σv2 σd

2 K 2=

Rdx z( ) C* 1

z*----

Sd z( ) σd2Ca

* 1

z*----

Cm* 1

z*----

= =

F z( ) Cm1– z( ) Ca

* 1

z*----

+

=

C z( ) C z( ) Cm z( )= Ca z( ) 1=

F z( ) C 1– z( )=

Ca* 1

z*----

+

Ca* 1

z*----

ca* n–( )

Ca z( ) ca* 0( )δ n( )

Ca* 1

z*----

+

ca* 0( )=

F z( ) ca* 0( )Cm

1– z( )=

C z( ) 1 cz 1––

1 pz 1––-------------------=

p 1< c 1>

Cm z( ) c*– z 1–+

1 pz 1––-----------------------=

Ca z( ) 1 cz 1––

c*– z 1–+-----------------------=

C z( ) Ca z( )



cle, and that is minimum phase. To find we can use the final value theorem, evalu-

ating at (because it is anti-causal). This is

. (115)

Plugging this into (111) we get

. (116)

We see that instead of inverting the zero outside the unit circle, we invert its conjugate reciprocal.Furthermore, the closer gets to the unit circle, the closer the solution gets to the solution when

is minimum phase. Moreover, the further gets from the unit circle, the closer the Wienerfilter gets to zero. This latter result is intuitive because the filter is getting further from minimumphase, so the inverse of the minimum phase part is less useful.

17. Normal Equations

Let be a vector of dimension M, representing the taps of an FIR Wiener filter (i.e. the first Mvalues of the causal impulse response). For this filter to be optimal, the taps must satisfy the normalequations,

. (117)

Assuming that these equations result from minimizing a unimodal quadratic function, it followsthat there must be at least one solution for . This conclusion can be reinforced algebraically by

proving that is always in the range space of . We write this

, (118)

where is simply the set of vectors that can be written as for some .

Consider a vector in the nullspace of , meaning that , where is the zero vector.This is written

Cm z( ) ca* 0( )

Ca* 1

z*----

z 0=

Ca* 1

z*----

z 0=

1 c*z–c– z+

----------------z 0=

1c---–= =

F z( ) 1 pz 1––

c*– z 1–+----------------------- 1

c 2-------- 1 pz 1––

1 c*( )1–z

1–+

-------------------------------= =

cC z( ) c

f

Rxf* rdx*=

f

rdx* Rx

rdx* ℜ Rx( )∈

ℜ Rx( ) Rxp p

q Rx Rxq 0= 0



. (119)

For this vector,

. (120)

Letting , we can write

. (121)

Hence

(122)

or

. (123)

Noticing that this is the product of two complex scalars that are complex conjugates of one another,we can write it as

. (124)

From this we infer that with probability one. In words, with probability one, an out-come of the random vector will be orthogonal to any vector in the nullspace of .

Now, note that

(125)

so that

. (126)

But since with probability one, this inner product is zero, so is orthogonal to any

vector in the nullspace of . The following two facts from linear algebra let us infer directly

from this that is in the range space of . These facts use the property that :

and . (127)

q η Rx( )∈

qHRxq 0=

x n( ) x n( ) x n 1–( ) ... ,x n M– 1+( ), ,[ ]T=

Rx E x n( )xH n( )[ ]=

qHE x n( )xH n( )[ ]q 0=

E qHx n( )xH n( )q[ ] 0=

E qHx n( )2

[ ] 0=

qHx n( ) 0=x n( ) q Rx

rdx* E d* n( )x n( )[ ]=

qHrdx* E d* n( )qHx n( )[ ]=

qHx n( ) 0= rdx*

q Rx

rdx* Rx Rx Rx

H=

ℜ Rx( )⊥ η Rx( ) ℜ Rx( ) η Rx( )⊕ CM=



18. Geometric Series

for any (128)

To prove, just multiply both sides by .

Note that, taking the limit as ,

, for any . (129)

1. Dirac Delta Functions

The Dirac delta function is defined by the following two relations:

for all (130)and

. (131)

From this definition, we can get the following identity

(132)

valid for any real . To prove this, just observe that

for all (133)and

. (134)

The latter is obtained by a change of variables, where gets replaced by and hence get re-placed by .

am

m 0=

n

∑ 1 an 1+–1 a–

---------------------= n 0>

1 a–

n ∞→

am

m 0=

∞

∑ 11 a–------------= a 1<

δ t( )

δ t( ) 0= t 0≠

δ t( )dt∞–

∞

∫ 1=

aδ at( ) δ t( )=

a

aδ at( ) 0= t 0≠

aδ at( )dt∞–

∞

∫ 1=

t at dtadt



19. Working with Impulse Trains

The following identity relates an infinite sum of complex exponentials with an infinite Dirac deltatrain:

. (135)

It can be verified by observing that the left hand side is the DTFT of the signal , from thedefinition of the DTFT. So applying the inverse DTFT formula to the right hand side, we shouldobtain . Doing this,

. (136)

Note that the limits of the integral enclose exactly one of the impulses in the train (the one at). So applying the sifting rule, we get .

Another form of the same identity relates a delta train in the time domain with a sum of exponen-tials in the frequency domain:

(137)

We can verify this by changing variables in the previous identity. But for extra reinforcement, letus verify it independently by letting

(138)

and viewing the sum of exponentials as a Fourier series expansion for the periodic function ,which has period . Find the Fourier series coefficients using the formula

. (139)

Again, the integral encloses just one of the delta functions, so we apply the sifting rule. A more

e jωn–

n∑ 2π δ ω 2πk–( )

k∑=

x n( ) 1=

x n( ) 1=

x n( ) 12π------ 2π δ ω 2πk–( )ejωndω

k∑

π–

π

∫=

ω 0= x n( ) 1=

δ t kT–( )k∑ 1

T--- e

j 2πT------ mt

m∑=

x t( ) δ t kT–( )k∑=

x t( )T

X m( ) 1T--- x t( )e

j2πT------ mt–

dtT/2–

T/2

∫1T--- δ t kT–( )

k∑ e

j2πT------ mt–

dtT/2–

T/2

∫= =



formal way to simplify the above expression is to exchange the integral and summation and changevariables, letting , getting

(140)

We can recognize this as a sum of integrals with adjoining limits and simplify to

. (141)

Since the Fourier Series coefficients are constant, the Fourier Series representation of is aninfinite sum of exponentials with equal weight. From this representation, it is easy to see that the Fourier transform of , which can itself bewritten as a sum of exponentials, can also be expressed as a sum of Dirac delta functions,

(142)

The first of these expressions comes from the Fourier integral and the sifting rule of Dirac deltafunctions. The second follows from the first identity (after some manipulation), or can be derivedfrom knowledge that

(143)

(easily verified using the inverse Fourier integral) and the fact that is a linear combination ofsuch terms.

Substituting in place of we get the related identity

(144)

The first of these two forms,

, (145)

is easily recognized as the DTFT of

τ t kT–=

X m( ) 1T--- δ τ( )e

j 2πT------ m τ kT+( )–

dτkT T/2–

kT T /2+

∫k∑=

X m( ) 1T--- e

j2πT------ mkT 1

T---= =

x t( )

x t( )

FT δ t kT–( )k∑ e j– ωmT

m∑ 2π

T------ δ ω 2π

T------ m–

m∑= =

FT ej2π

T------ mt

2πδ ω 2πT------m–

=

x t( )

N T

e j– ωmN

m∑ 2π

N------ δ ω 2π

N------m–

m∑=

X e jω( ) e j– ωmN

m∑=



(146)

where is now the Kronecker delta function. (To verify this, just note that the DTFT of each

component is , from the DTFT definition). If we apply the inverse DTFT for-

mula to the second of the two forms for , after some non-trivial labor, we get the new iden-tity for Kronecker delta trains

(147)

Note that unlike the corresponding identity for Dirac delta trains, the sum of exponentials is finite.

To verify this more easily than applying the inverse DTFT formula, note that the finite sum of ex-ponentials is exactly the inverse DFT of , as is easily seen by comparing the above ex-pression to the inverse DFT formula. Hence, we can just verify that , where weuse the DFT of length . (Equivalently, we could view the above sum of exponentials as a discreteFourier series (DFS) and just verify the DFS coefficients.)

(148)

Exchanging summations, and changing variables using we get

(149)

The double summation is actually a sum of sums with adjoining limits, so

. (150)

Thus our identity is verified.A side benefit of this approach is that we have verified that

x n( ) δ n mN–( )m∑=

δ n( )

δ n mN–( ) e j– ωmN

X e jω( )

δ n mN–( )m∑ 1

N---- e

j2πN------ kn

k 0=

N 1–

∑=

X k( ) 1=DFT x n( )[ ] 1=

N

X k( ) DFT x n( )[ ] δ n mN–( )m∑ e

j– 2πN------ nk

n 0=

N 1–

∑= =

r n mN–=

X k( ) δ r( )ej– 2π

N------ r mN+( )k

r mN–=

mN N+– 1–

∑m∑=

X k( ) δ r( )ej– 2π

N------ r mN+( )k

r∑ e

j– 2πN------ Nk

2

1= = =



(151)

for a DFT of order . We can show similarly that

, (152)

again for a DFT of order .

We can now use identity (147) to find the DTFT of

. (153)

Applying the DTFT definition,

. (154)

Exchanging the order of summations and combining the exponentials,

. (155)

We can now use identity (135) to represent the inside summation as a delta train,

. (156)

Exchanging summations again, we can recognize this as a sum of sums with adjoining limits, get-ting the simpler expression

(157)

As a picture:

DFT δ n mN–( )m∑ 1=

N

DFT 1[ ] N δ m kN–( )k∑=

N

x n( ) δ n mN–( )m∑=

X e jω( ) 1N---- e

j 2πN------ kn

k 0=

N 1–

∑

e jωn–

n∑=

X e jω( ) 1N---- e

j ω 2πN------k–

n–

n∑

k 0=

N 1–

∑=

X e jω( ) 2πN------ δ ω 2π m k

N----+

–

m∑

k 0=

N 1–

∑=

DTFT δ n mN–( )m∑ 2π

N------ δ ω 2π k

N----–

k∑=



(158)The DTFT of a discrete-time impulse train is an impulse train in frequency, just as the FT of a con-tinuous impulse train is an impulse train in frequency.

2π2πN------

0

ω

ee 225a digital signal processing supplementary material 1 ...ee225a/sp12/supplement.pdf · prof....

Documents