symbolic dynamics of digital signal processing systemseprints.lincoln.ac.uk › 4008 › 2 ›...

1

Symbolic Dynamics of Digital Signal Processing Systems

Dr. Bingo Wing-Kuen Ling

School of Engineering, University of Lincoln.Brayford Pool, Lincoln, Lincolnshire, LN6

7TS, United Kingdom.Email: [email protected]

2

My research interestsSymbolic dynamics

Limit cycle, fractal and chaotic behaviors of digital filters with two’s complement arithmetic, sigma delta modulators and perceptron training algorithms

Optimization theorySemi-infinite programming, nonconvex optimization and nonsmooth optimization with applications to filter, filter bank, wavelet kernel and pulse designs

Time frequency analysisNonuniform filter banks, filter banks with block samplers to multimedia and biomedical signal processing

Control theoryImpulsive control, optimal control, fuzzy control and chaos control for HIV model and avian influenza model

3

Acknowledgements

Prof. Xinghuo Yu (Royal Melbourne of Institute of Technology)Dr. Joshua D. Reiss (Queen Mary University of London)Dr. Herbert Ho-Ching Iu (University of Western Australia)Dr. Hak-Keung Lam (King’s College London)

4

OutlineIntroductionDigital filters with two’s complement arithmeticSigma delta modulatorsPerceptron training algorithmsConclusionsQ&A Session

5

IntroductionDefinition

Symbolic dynamics is a kind of system dynamics which involves multi-level signals.

MotivationsMany practical signal processing systems, such as digital filters with two’s complement arithmetic, sigma delta modulators and perceptron training algorithms, are symbolic dynamical systems. They are found in almost everywhere.

6

IntroductionChallenges

Symbolic dynamical systems could lead to chaotic, fractals and divergent behaviors.Stability conditions are generally unknown.Admissibility conditions are generally unknown.

7

Digital filters with two’s complement arithmetic

What are digital filters?Digital filters are systems that are characterized in the frequency domain.

( ) ( )( )ωωω

UYH =

h(k)( )ku ( )ky

( ) ( )∑∀

−=k

kjekuU ωω ( ) ( )∑∀

−=k

kjekyY ωω( ) ( )∑∀

−=k

kjekhH ωω

8

Types of digital filtersLowpass filters

Allow a low frequency band of a signal to pass through and attenuate a high frequency band.


9

Types of digital filtersBandpass filters

Allow intermediate frequency bands of a signal to pass through and attenuate both low and high frequency bands.


10

Types of digital filtersHighpass filters

Allow a high frequency band of a signal to pass through and attenuate a low frequency band.


11

Types of digital filtersBand reject filters

Allow both low and high frequency bands of a signal to pass through and attenuate intermediate frequency bands.


12

Types of digital filtersNotch filters

Allow almost all frequency components to pass through but attenuate particular frequencies.


13

Types of digital filtersOscillators

Allow particular frequency components to pass through and attenuate almost all frequency components.


14

Types of digital filtersAllpass filters

Allow all frequency components to pass through.

0 1 2 3 40

0.5

1

1.5

2

Frequency ω

Mag

nitu

de re

spon

se |H

( ω)|

(g)


15


Hardware schematic

z-1

z-1

a

b

Accumulator f(•)( )ku ( )ky

16

When no input is present, the filter can be described by the following nonlinear state space difference equation:

( ) ( )( ) ( )( )

( )( ) ( )( ) ⎟

⎟⎠

⎞⎜⎜⎝

⎛⎥⎦

⎤⎢⎣

⎡⋅+⋅

=

=⎥⎦

⎤⎢⎣

⎡++

≡+

kxakxbfkx

kkxkx

k

21

2

2

1 F11

1 xxformDirect


17

where f(•) is the nonlinear function associated with the two’s complement arithmetic

1 3-1-3

1

-1

v

f(v) linesStraight

overflow No OverflowOverflow


18

where

and m is the minimum integer satisfying

( )( ) ( ){ }

( ) { }mmks

xxxxIkxkx

ab

,,1,0,1,,

11,11:,

10

21212

2

1

LL −−∈

<≤−<≤−≡∈⎥⎦

⎤⎢⎣

⎡

⎥⎦

⎤⎢⎣

⎡=A

( ) ( )( ) ( )kskxkx

k ⋅⎥⎦

⎤⎢⎣

⎡+⎥

⎦

⎤⎢⎣

⎡⋅=+

20

12

1Ax

1212 21 +⋅≤⋅+⋅≤−⋅− mxaxbm


19

is called symbolic sequences.( )ks

( ) ( ) ( )( ) ( ) ( )( ) ( ) ( ) 311

110131

21

21

21

<⋅+⋅≤⇒−=<⋅+⋅≤−⇒=−<⋅+⋅≤−⇒=

kxakxbkskxakxbkskxakxbks


20

For example: ( ) [ ] 5.0 and 1,6135.06135.00 =−=−= abTx( ) ( ) ( )( ) ( ) ( )

( ) ( ) ( ) ( )( ) ( ) ( )( ) ( ) ( )( ) ( ) ( ) ( )( ) ( ) ( ) ( )

( )LL

M

M

11001000

9161.019 and 1180839.118189311.018 and 1170689.11717

0166184.016160157597.01515

9982.015 and 1140018.11414

011534.011009203.000

221

221

21

21

221

21

21

−−=

==⇒−=⋅+⋅−=−=⇒=⋅+⋅

=⇒=⋅+⋅=⇒−=⋅+⋅

−=−=⇒=⋅+⋅

=⇒−=⋅+⋅=⇒=⋅+⋅

s

xsxaxbxsxaxb

sxaxbsxaxb

xsxaxb

sxaxbsxaxb

valuecomplement sTwo'


21

Stability analysis of the corresponding linear system

Eigenvalues of A

242 baa ⋅+±

=λ


22

R1 R3

R5

R2

R4

-1

2-2

1

a

b

R3R1


23

R5 :Magnitudes of both eigenvalues <1.⇒The corresponding linear system is stable.

R1 and R3 : Either one of the magnitudes of the eigenvalues < 1.⇒The corresponding linear system is unstable.R2 and R4 :Magnitudes of both eigenvalues are greater than 1.⇒The corresponding linear system is unstable.


24

Effects of different initial conditions when the eigenvalues of system matrix are on the unit circle

Type I trajectoryThere is a single rotated and translated ellipse in the phase portrait.

( ) ⎥⎦

⎤⎢⎣

⎡−

=−==612.0

612.00,1,5.0 xba


25


26

Type II trajectoryThere are some rotated and translated ellipses in the phase portrait.

( ) ⎥⎦

⎤⎢⎣

⎡−

=−==616.0

616.00,1,5.0 xba


27This corresponds to the limit cycle behavior.


28

Type III trajectoryThere is a fractal pattern on the phase portrait.

( ) ⎥⎦

⎤⎢⎣

⎡−

=−==6135.0

6135.00,1,5.0 xba


29Conclusion: Very sensitive to initial conditions


30

Step response of second order digital filters with two’s complement arithmetic

The system can be represented as:

where and⇒DC offset on symbolic sequences.

( ) ( )( ) ( ) ( )( )

( )( ) ( ) ( )kskukxkx

kukxakxbfkx

k

⋅⎥⎦

⎤⎢⎣

⎡+⋅+⎥

⎦

⎤⎢⎣

⎡⋅=

⎟⎟⎠

⎞⎜⎜⎝

⎛⎥⎦

⎤⎢⎣

⎡+⋅+⋅

=+

20

1

2

1

21

2

BA

x

⎥⎦

⎤⎢⎣

⎡=

10

B⎥⎦

⎤⎢⎣

⎡=

ab10

A


31

For ( ) ⎥⎦

⎤⎢⎣

⎡==−=−=

3.03.0

0,1,1,5.1 xcba


32

For 1 and 1,5.1 =−=−= cba

s(k)=0

s(k)=1

s(k)=-1

s(k)=-2


33

For ( ) ( )L,0,1,1,0,7.07.0

0,1,1,5.1 −−=⎥⎦

⎤⎢⎣

⎡−−

==−=−= scba x


34

For 1 and 1,5.1 =−=−= cba


35

For ( ) ⎥⎦

⎤⎢⎣

⎡−−

==−=−=99.099.0

0,1,1,5.1 xcba


36

For 1 and 1,5.1 =−=−= cba


37

Sinusoidal response of second order digital filters with two’s complement arithmetic


38


39


Autonomous case when the eigenvalues of A are inside the unit circle

40

–1 –0.8 –0.6 –0.4 –0.2 0 0.2 0.4 0.6 0.8 1–1

–0.8

–0.6

–0.4

–0.2

0

0.2

0.4

0.6

0.8

1

x1

x2


41

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

x1

x2


42

Autonomous case for third order digital filters with two’s complement arithmetic realized in cascade form

System block diagram:

( )kx3

Delay

Delay

Delay

c

( )•f

( )kx2

( )kx1b

a

( )•f( )ky1

( )ky


43


44


45

Autonomous case for third-order digital filters with two’s complement arithmetic realized in parallel form

System block diagram:

( )kx3

Delay

Delay

Delay

c

( )•f

( )kx2

( )kx1b

a

( )•f

( )ky1

( )ky2

( )ky


46


47


48

Admissibility of second order digital filters with two’s complement arithmetic

Admissible set of periodic symbolic sequences is defined as a set of periodic symbolic sequences such that there exists an initial condition that produces the symbolic sequences.

sx(0)

Admissible set of periodic symbolic sequences

Set of initial conditions


49

For example, whens Admissible

,0,1,5.0 =−== cbaM1 ( )L,0=s2 ( )L,1,1−=s ( )L,1,1 −=s

3 ( )L,1,0,0=s ( )L,0,1,0=s( )L,0,0,1=s( )L,1,0,0 −=s ( )L,0,1,0 −=s

( )L,0,0,1−=s

s admissibleNot ( )L,1=s ( )L,1−=s( )L,0,1=s ( )L,1,0=s( )L,0,1−=s ( )L,1,0 −=s( )L,1,1,0=s ( )L,1,1,0 −=s( )L,1,1,0 −=s( )L,1,0,1=s ( )L,1,0,1 −=s

( )L,1,1,0 −−=s

( )L,0,1,1 −=s ( )L,0,1,1 −=sL15 foundNot L


50

A periodic sequence with period is admissible if and only if for

s M

( )( )1

sin2

sin

12

cos,mod1

1

0 <⋅⎟⎠⎞

⎜⎝⎛ ⋅

⎟⎠

⎞⎜⎝

⎛⋅⎟⎠⎞

⎜⎝⎛ −−⋅+

≤−∑−

=

θθ

θ

M

jMMjisM

j

1,,1,0 −= Mi L


51

What is sigma delta modulators?Sigma delta modulators are devices implementing sigma delta modulations and are widely used in analogue-to-digital conversions.The input signals are first oversampled to obtain the inputs of the sigma delta modulators. The loop filters are to separate the quantization noises and the input signals so that very high signal-to-noise ratios could be achieved at very coarse quantization schemes.

Sigma delta modulators

52

Sigma delta modulatorsBlock diagram

( )( )

( )( )zFzF

zUzS

+=

1

F(z) Qu(k) y(k) s(k)

( )( ) ( )zFzNzS

+=

11

53

( ) ( ) ( )[ ]Tkykyk 12 −−≡x

( ) ( ) ( )( ) ( )( )( ) ( ) ( )( )( )2211cos2

21cos2−−−−−−−

+−−−=⇒kyQkykyQku

kykykyθ

θ

( ) 21

21

cos21cos2F −−

−−

+−−

=zz

zzzθ

θ

( ) ( ) ( )( ) ( )( )( ) ( ) ( )( )( )2211cos2

21cos2−−−−−−−=

−+−−⇒kyQkykyQku

kykykyθ

θ

( ) ( ) ( )[ ]Tkukuk 12 −−≡u

( ) ( )( ) ( )( )[ ]TkyQkyQk 12 −−≡s

Sigma delta modulatorsNonlinear state space dynamical model:

54

There are only finite numbers of possibilities of s(k). Hence, s(k) can be viewed as symbol and s(k) is called a symbolic sequence.

⎥⎦

⎤⎢⎣

⎡−

≡θcos21

10A ⎥

⎦

⎤⎢⎣

⎡−

≡θcos21

00B

( ) ( ) ( ) ( )( )kkkk suBAxx −+=+1

( )⎩⎨⎧−

≥≡

otherwise101 y

yQ


55-1 -0.5 0 0.5 1 1.5 2

-1

-0.5

0

0.5

1

1.5

2

x1

x 2

Second order marginally stable bandpasssigma delta modulators


56


57

Second order strictly stable bandpass sigma delta modulators

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

x 104

-1

-0.5

0

0.5

1

1.5

2

Clock cycle

x 1(k)


58


59


60

Sigma delta modulatorsGeneral second order bandpass sigma delta modulators

61

Admissibility of periodic sequence

⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢

⎣

⎡

≡

−−

−−

−

−−

102

21

110

0121

MM

MM

M

MM

DDDDD

DDDDDDD

D

LL

OM

MOOOM

O

L

( ) ( ) ( ) ( )[ ]TMsMsss 1,1,0,0 2121 −−≡ Lζ

( ) ( ) ⎥⎦

⎤⎢⎣

⎡++

≡θθ

θθ1sin1sin

sinsinjcjd

jcjdjD


62

Σ be the admissible set of periodic output sequences with period M

( ) ( )( )11 ,, −−−−≡ MMdiag AIAIK L

[ ]Tυυτ ,, L≡

( ) ( ) ( )( )1,,1,0 −≡ Mssss L

( ) ( )⎥⎦

⎤⎢⎣

⎡ ++≡ ∑∑

=

−

=

M

j

M

jjucdjucd

1

1

0sin

sin,sin

sinθ

θθ

θυ

( )( ){ }ζτDζKQs =+=Σ :


63

What is a perceptron training algorithm?A perceptron is a single neuron that employs a single bit quantization function as its activation function.

where

Perceptron training algorithms

( )kx1

( )kxd

( )kw1

( )kwd

M

( )kw0

Q ( )ky

( ) ( ) ( )( )kkQky T xw=

( )⎩⎨⎧

<−≥

≡0101

zz

zQ

( ) ( ) ( )[ ]Td kxkxk ,,,1 1 L≡x( ) ( ) ( ) ( )[ ]Td kwkwkwk ,,, 10 L≡w

64

What is a perceptron training algorithm?The weights of a perceptron is usually found by the perceptron training algorithm.

where t(k) is the desired output corresponding to x(k).If w(k) converges, then the steady state value of w(k) could be employed as the weights of the perceptron.

( ) ( ) ( ) ( ) ( )kkyktkk xww2

1 −+=+


65

Example 1Iteration index k wold(k) x(k) t(k) y(k) wnew(k)

0 [0 0 0]T [1 5 1]T -1 1 [-1 -5 -1]T

1 [-1 -5 -1]T [1 2 1]T -1 -1 [-1 -5 -1]T

2 [-1 -5 -1]T [1 1 1]T 1 -1 [0 -4 0]T

3 [0 -4 0]T [1 3 3]T 1 -1 [1 -1 3]T

4 [1 -1 3]T [1 4 2]T -1 1 [0 -5 1]T

5 [0 -5 1]T [1 2 3]T 1 -1 [1 -3 4]T

6 [1 -3 4]T [1 5 1]T -1 -1 [1 -3 4]T

7 [1 -3 4]T [1 2 1]T -1 -1 [1 -3 4]T

8 [1 -3 4]T [1 1 1]T 1 1 [1 -3 4]T

9 [1 -3 4]T [1 3 3]T 1 1 [1 -3 4]T

10 [1 -3 4]T [1 4 2]T -1 -1 [1 -3 4]T

11 [1 -3 4]T [1 2 3]T 1 1 [1 -3 4]T

Converge

( ) ( ) ( ) ( ) ( )kkyktkk xww2

1 −+=+


66

Example 1

0 2 4 60

1

2

3

4

x1

x 2

LinearlySeparable


67

Example 2Iteration index k wold(k) x(k) t(k) y(k) wnew(k)

0 [0 0 0]T [1 0 0]T -1 1 [-1 0 0]T

1 [-1 0 0]T [1 0 1]T 1 -1 [0 0 1]T

2 [0 0 1]T [1 1 0]T 1 1 [0 0 1]T

3 [0 0 1]T [1 1 1]T -1 1 [-1 -1 0]T

4 [-1 -1 0]T [1 0 0]T -1 -1 [-1 -1 0]T

5 [-1 -1 0]T [1 0 1]T 1 -1 [0 -1 1]T

6 [0 -1 1]T [1 1 0]T 1 -1 [1 0 1]T

7 [1 0 1]T [1 1 1]T -1 1 [0 -1 0]T

8 [0 -1 0]T [1 0 0]T -1 1 [-1 -1 0]T

9 [-1 -1 0]T [1 0 1]T 1 -1 [0 -1 1]T

10 [0 -1 1]T [1 1 0]T 1 -1 [1 0 1]T

11 [1 0 1]T [1 1 1]T -1 1 [0 -1 0]T

Limit Cycle

( ) ( ) ( ) ( ) ( )kkyktkk xww2

1 −+=+


68

Example 2

- 1 0 1 2- 1

0

1

2

x 1

x 2

NonlinearlySeparable


69


-1 -0.5 0 0.5 1-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

x1

Class 1Class -1

0 10 20 30 40-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Time index k

w0(k

)

0 10 20 30 401

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

2.8

3

Time index k

w1(k

)

0 10 20 30 40-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Time index k

w2(k

)

Example 3

70

Perceptron training algorithmsExample 4

The set of training feature vectorsDesirable outputInitial weightResult:Set of weights of the perceptron:

Invariant set of weights of the perceptron:Invariant map:Note: is not bijective because

⎪⎭

⎪⎬

⎫

⎪⎩

⎪⎨

⎧

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

11

1,

11

1,

111

,111

{ }1,1,1,1 −−( ) [ ]T1,1,10 −−−=w

( ) [ ]Tk 0,2,0=w Zk∈∀

[ ]{ }T1,1,1 −−−=℘[ ]{ } [ ]{ }TTF 1,1,11,1,1: −−−→−−−ℑ

33:~ ℜ→ℜℑF ( ) ( ) ( )kkk T xwx >2

Zk∈∀

⎪⎭

⎪⎬

⎫

⎪⎩

⎪⎨

⎧

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−−

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

02

0,

111

,000

,111

71


-1 -0.5 0 0.5 1-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

x1

x 2

Class 1Class -1

0 10 20 30 40-1

-0.9

-0.8

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

Time index k

w0(k

)

0 10 20 30 40-2

-1.8

-1.6

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

Time index k

w1(k

)

0 10 20 30 40-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Time index k

w2(k

)

72

Perceptron training algorithmsExample 5

The set of training feature vectorsDesirable outputInitial weightInvariant set of the weights of the perceptron consists of three hyperplanes.Note: is not bijective because

{ }1,1,1,1 −−

33:~ ℜ→ℜℑF

( ) ( ) ( )kkk T xwx >2Zk∈∃

⎪⎭

⎪⎬

⎫

⎪⎩

⎪⎨

⎧

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

− 0668.11139.01

,1364.0

1832.21

,5883.0

7258.01

,1867.0

1746.01

( ) [ ]T2133.0,7923.0,10 −−=w

73


0 1 2 3-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

x1

x 2

Class 1Class -1

-2-1.5

-1-0.5

0

-1

0

1

2

3

4-1.5

-1

-0.5

0

0.5

1

1.5

w0

w1

w2

74

Many digital signal processing systems are symbolic dynamical systems.These symbolic dynamical systems could exhibit fractal, chaotic and divergent behaviors.By investigating the properties of these symbolic dynamical systems, unwanted behaviors could be avoided.

Conclusions

75

Q&A Session

Thank you!

Let me think…

Bingo

symbolic dynamics of digital signal processing systemseprints.lincoln.ac.uk › 4008 › 2 ›...

Documents