ee645: independent component analysis elliot taniguchi advisor: dr. kuh may 16, 2003

EE645: Independent Component Analysis

Elliot Taniguchi

Advisor: Dr. Kuh

May 16, 2003

Presentation Overview

ICA Motivation Mathematical Formulation Fast ICA Algorithm Applications

Noise Separation and Feature Extraction Digital Watermarking

Motivation for ICA

Cocktail Party Problem Suppose you are in a crowded room with

many people. How do you understand what any one person is saying?

Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only

Cocktail Party Problem??

Cocktail Party Problem

ICA Separation Algorithm Separation of Speech

Signals Humans can separate

multiple signals with only two ears/sensors

ICA needs as many ears/sensors as message signals

Here we assume he has four ears!

Recovered Messages

Mathematical Formulation Overview

ICA Definition ICA Assumptions

Independent Signals Non-Gaussian

ICA Limitations Scaling Permutations No. of Sensors

ICA Definition

Mixed Signals in Matrix Notation

n

i

is1

iasAx

Variable Definitions

Signalt Independen i ~

Matrix ngMultiplexi ~

Signal dMultiplexe~

this

A

xj

ICA Solution

Signal Separation

Find using the ICA Algorithm

ssIsA)(W

s)(AWxWs

ˆ

W

ICA Block Diagram (2 Signals)

Signal #1 Signal #2

MultiplexedSignal #2

MultiplexedSignal #1

a11

a12 a21

a22

ICA Assumption #1: Independence Probability Density Definition

Expected Value Definition

functions are ,

)}({)}({)}()({

21

22112211

hh

yhEyhEyhyhE

spdf' are ,

)()()(

21

22112,1

pp

ypypyyp

ICA Assumption #2: Non-Gaussian Property of Gaussian signals

Addition of two independent Gaussian random variables is another single Gaussian random variable.

Information Lost! Kurtosis Function

Special Case: kurt(N) = 0

224 }){(3}{)( yEyEykurt

Limitation #1: Scaling

ICA maximizes independence between signals.

)}ˆ({*)}ˆ({

)}ˆ(*)ˆ({)}ˆ(*)ˆ({

222111

221121222111

XhEXhE

XhXhEXhXhE

Assuming theSeparated Signalsare Independent

Scaling Factorfor Signal #1

Scaling Factorfor Signal #2

Limitation #2: Signal Permutations

The mixing matrix and independent components are unknown.

sAsPPA

sPPAsIAsAx1

1

*)*(*)*(

*)*(***

sPssIsPP

sPIPsPAWP

sPAWPsAWx

1

11

1

***)*(

*))*(*(*)*)*(*(

*)*(*)*(**ˆ

Limitation #3: No. of Sensors

Sensor Requirement The number of separated signals cannot be

larger than the number of inputs. Current research is being done to reduce this

constraint.

ICA Separation Technique

Central Limit Theorem If two random (non-Gaussian) signals added,

the resulting signal will be more Gaussian than the original two random signals

ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity

Results in separating the two signals

Fast ICA Algorithm Overview

“Fixed-Point” Algorithm Implementation

Fast ICA algorithm Extensions

Algorithm Speed & Performance Currently the fastest Most Commonly Used

Fast ICA Algorithm

1. Choose a random initial weight vector.

2. Let,

3. Let,

4. Repeat until converges.

www /wxwxwxw TT )}(g{)}g({ EE

w

Fast ICA Extensions

Preprocessing Normalize mean to zero Pre-Whitening

Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)

Noise Separation Example

Separation of Noise Impulsive Noise Additive White Gaussian Noise

Implementation Two Sensor Setup Fast ICA Algorithm

Noise Generation

AWGN Gaussian R.V.

Impulsive Noise Poisson R.V. Gaussian R.V.

)(*),( PmNI

),( mN

Physical Setup

Receiver #1Receiver #2

ImpulsiveNoise

GaussianNoise

Noise Separation Example

0 100 200 300 400 500 600 700 800 900 1000-30

-20

-10

0

10

20

30Noise Separation Simulation

P(

= 1

)*N

(m =

0,

= ,

5)

0 100 200 300 400 500 600 700 800 900 1000-3

-2

-1

0

1

2

3

N(m

= 0

, =

1)

Noise Separation Example

0 100 200 300 400 500 600 700 800 900 1000-30

-20

-10

0

10

20

30Noise Separation Simulation

Mul

tiple

xed

Sig

nal #

1

0 100 200 300 400 500 600 700 800 900 1000-30

-20

-10

0

10

20

30

Mul

tiple

xed

Sig

nal #

2

Noise Separation Example

0 100 200 300 400 500 600 700 800 900 1000-20

-10

0

10

20Noise Separation Simulation

Sep

arat

ed S

igna

l #1

0 100 200 300 400 500 600 700 800 900 1000-3

-2

-1

0

1

2

3

Sep

arat

ed S

igna

l #2

Noise Separation Performance

0 100 200 300 400 500 600 700 800 900 1000-40

-20

0

20

40

Orig

inal

Sig

nal #

1

Noise Separation Simulation

0 100 200 300 400 500 600 700 800 900 1000-40

-20

0

20

40

Nor

mal

ized

IC

A S

igna

l #1

0 100 200 300 400 500 600 700 800 900 1000-0.1

-0.05

0

0.05

0.1

Err

or :

Sig

nal #

1 MSE = 0.00051558

Noise Separation Performance

0 100 200 300 400 500 600 700 800 900 1000-4

-2

0

2

4

Orig

inal

Sig

nal #

2

Noise Separation Simulation

0 100 200 300 400 500 600 700 800 900 1000-4

-2

0

2

4

Nor

mal

ized

IC

A S

igna

l #2

0 100 200 300 400 500 600 700 800 900 1000-0.2

-0.1

0

0.1

0.2

Err

or :

Sig

nal #

2 MSE = 7.3829e-005

Noise Separation & Feature Extraction

ICA performs well in Blind Source Separation ICA for Feature Extraction

Reduce Complexity of the Neural Network Train only on the appropriate signal

Detection and Estimation of Hidden Signals

Noise Detection and Estimation: Method #1

FastICASensor #3

Sensor #4

Sensor #2

Sensor #1

Feature #3

Feature #4

Feature #2

Feature #1

N.N.(BP,RBF,SVM)

Noise Detection and Estimation: Method #2

FastICASensor #3

Sensor #4

Sensor #2

Sensor #1

Feature #3

Feature #4

Feature #2

Feature #1

N.N.(BP,RBF,SVM)

DecisionDevice

Noise Detection and Estimation:Conclusion Additional Preprocessing

Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network

Statistical Moments Signal Samples

Possible Neural Networks Back Propagation SVM Radial Basis Functions

Problem Need to train Neural Network in Matlab

Digital Watermarking of Music

Motivation Popularity of Digital Storage Devices

Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information?

Leaving digital signatures of its artist

Essential Properties for Watermarking Undetectable Irremovable Resilient

Detection & Estimation of Watermarks Detection of Watermark

Authenticate copyrighted music Estimation of Watermark

Authenticate copyrighted music Information on artist, producer, etc.

Watermarking Model

Process Mix the original musical data with watermark Keep watermark Power relatively low

Ensure high quality of the watermarked music Watermark is better hidden

hidden is watermark that theensure to :Note

WatermarkEmbedded

Data MusicalCopyright

Key Encryption

Data Musical dWatermarke

12 11

2221

1211

aa

aa

aa

Popular Digital Formats

Wave Format Mp3 Format

Bit Rate 1411 [kbps] 128 [kbps]

Channels 2 2

Audio Sample Rate

44 [kHz] 44 [kHz]

Detection of Watermark

Watermarking Detection Algorithm1. Use the ICA model to randomly mix the watermark

and music file.

2. Save the watermarked music in the popular *.wav format

3. Read the saved *.wave file. Separate the watermark and the music file.

4. Identify the watermark using statistical methods (mean, std, etc.)

Performance Statistic Correlation Coefficient (Absolute Value)

Detection Performance

3.85 3.852 3.854 3.856 3.858 3.86 3.862 3.864 3.866 3.868 3.87

x 10-6

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Signal-to-Noise Ratio

Cor

rela

tion

Coe

ffic

ient

Performance of ICA Watermarking: 16 Bit, 44100 Hz

3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861

x 10-6

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Signal-to-Noise RatioC

orre

latio

n C

oeff

icie

nt

Performance of ICA Watermarking: 8 Bit, 44100 Hz

Estimation of Watermark

Watermarking Estimation Algorithm1. Use the ICA model to randomly mix the watermark

and music file.2. Save the watermarked music in the popular *.wav

format3. Read the saved *.wave file. Separate the watermark

and the music file.4. Identify the watermark using statistical methods

(mean, std, etc.)5. Digitize the watermark signal.

Performance Statistic Bit Error Rate

Estimation Performance

3.85 3.855 3.86 3.865

x 10-6

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Performance of ICA Watermarking: 16 Bit, 44100 Hz

Signal-to-Noise Ratio

Bit

Err

or R

ate

3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861 3.862 3.863

x 10-6

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Signal-to-Noise RatioB

it E

rror

Rat

e

Performance of ICA Watermarking: 8 Bit, 44100 Hz

Is the Music Content Preserved?

“Hero” – Mariah CareyOriginal Wave file from CD

Hero.wav29 [s]

“Hero” – Mariah CareyWatermarked Wave fileHeroWatermarked.wav

29 [s]

Resilience of Proposed Watermark

Resiliency Previous Simulations show that wav format is

resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3

compression and decompression?

Mp3 Compression/Decompression

Actual Mp3 compression program used CDex Version 1.40 Release

Mp3 (lossy) Compression Down Sampling Filter banks

Mp3 Decompression Up Sampling Reconstruction Filter

Detection and Estimation Performance

Bad News Correlation Coefficients 0 MSE 0.5

Possible Problems with Mp3 Compression Down Sampling

Watermark information is lost Quantization Noise

Watermark information absorbs into the quantization noise

Lossy Compression 11:1 Compression Rate

How to Improve its Resilience?

Alternative Approaches Synchronization of the music data

Time shift in Mp3 compression? Storing watermark in certain frequencies

(where less quantization occurs) Error Coding

Hamming Reed-Solomon

Conclusion

Wave to Wave Format Very good performance (even for 8-bit wave

files) SNR is very low. Music Integrity is excellent

Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3

Compression Algorithms

References

[1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.

[2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000.

[3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. http://www.cis.hut.fi/~aapo/.

[4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. April 1999.

[5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. http://www.cis.hut.fi/projects/ica/. April 1999.

[6] Introduction to Blind Source Separation. http://www.cnl.salk.edu/~tewon/Blind/. [7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th

International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.

[8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998.[9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th

International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.