Widrow-Hoff Learning

Outline

• 1 Introduction• 2 ADALINE Network• 3 Mean Square Error• 4 LMS Algorithm• 5 Analysis of Converge• 6 Adaptive Filtering

Introduction

• In 1960, Bernard Widrow and his doctoral student Marcian Hoff introduced the ADALINE (ADAptive LInear NEuron)network and LMS(Least Mean Square) algorithm.

Perceptron Network

• Figure: a=hardlim(Wp+b)

ADALINE Network

• Figure: a=purelin(Wp+b)=Wp+b

Single ADALINE

decision boundary

Mean Square Error

Mean Square Error(conti.)

Mean Square Error(conti.)

Error analysis

𝐹 (𝐱 )=𝑐+𝐝𝑇 𝐱+𝟏𝟐𝐱𝑻 𝐀𝐱

Error analysis(conti.)

d = -2h and A = 2R

= 0

definite

Example 1

Example 1(conti.)

Example 1(conti.)

Approximate Steepest Descent

Approximate Gradient

Approximate Gradient(conti.)

Approximate Gradient(conti.)

LMS Algorithm

LMS Algorithm (conti.)

Example 2

Example 2(conti.)

, W(0)=

Example 2(conti.)

Example 2(conti.)

Example 2(conti.)

Analysis of Convergence

Analysis of Convergence(conti.)

Analysis of Convergence(conti.)

Example 3

Perceptron rule V.S. LMS algorithm

Perceptron rule V.S. LMS algorithm(conti.)

Perceptron rule V.S. LMS algorithm(conti.)

Perceptron rule V.S. LMS algorithm(conti.)

Adaptive Filtering

Tapped Delay Line

Adaptive Filter

Adaptive Noise Cancellation