AdaptiveLinearFilteringUsingInteriorPoint
OptimizationTechniques
Lecturer:TomLuo
StanfordUniversityEE392oZ.Q.Luo
Overview
A.InteriorPointLeastSquares(IPLS)Filtering
–IntroductiontoIPLS
–RecursiveupdateofIPLS
–Convergence/transientanalysisofIPLS
B.Applications
–Systemidentification
–Beamforming
–ChannelequalizationinaCDMAforwardlink
1
StanfordUniversityEE392oZ.Q.Luo
InteriorPointOptimizationforOptimalLinearFiltering
•Adiscrete-timelinearsystemcanbedescribedby
yi=xTiw∗+vi,i=1,2,...
•Usinginputoutputpairsxi,yithelinearleast-squaresproblemisthentoestimateafilterwthatminimizesthemean-squarederror
Fn(w)=1
n
n∑
i=1
(
yi−xTiw
)
2
=1
nyTnyn−2w
Tnpxy(n)+w
TRxx(n)w,(1)
whereyn=[y1,y2,...,yn]T,pxy(n)=
1n
∑
ni=1xiyi,Rxx(n)=
1n
∑
ni=1xix
Ti.
Note:pxy(n)andRxx(n)arebothrecursivelyupdatablewithper-samplecomplexity
ofO(M2).
•Theoptimumlinearfilterthensatisfies∇Fn(w)=0,orRxx(n)w−pxy(n)=0.
2
StanfordUniversityEE392oZ.Q.Luo
OneMotivation:TransientConvergence
•TheRLSalgorithmestimates(seee.g.,SayedandKailath’96)
wrlsn:=
[
δ
nI+Rxx(n)
]
−1
pxy(n),
whereδnIisaregularizationtermtoimproveconditioning.
•Problem:Theregularizationdependsentirelyontheconstantδ.If,forexample,
–SNRisunderestimated=⇒slowerasymptoticconvergence
–SNRisoverestimated=⇒badtransientbehaviour
•Remedy:
wn:=[αnI+Rxx(n)]−1pxy(n),αnadjustedadaptively
3
StanfordUniversityEE392oZ.Q.Luo
TheAnalyticCenterApproach
•Formulateaconvexfeasibilityproblemateachiteration.wisafeasiblefilteronlyifit
iscontainedin
Ωn=w∈RM|Fn(w)≤τn,‖w‖
2≤R
2,(2)
1stconstraint:minimizethemean-squarederrorFn(w).
2ndconstraint:makeΩnaboundedregion.
•TheanalyticcenterwanofΩnistheminimizerof
φn(w)=−log(τn−Fn(w))−log(R2−‖w‖
2)
whichcanbefoundbysolving∇φn(w)=0
∇Fn
sn(wan)+
2wan
tn(wan)=0,thereforew
an=
(
sn(wan)
tn(wan)
I+Rxx(n)
)
−1
pxy(n).
wheresn(w):=τn−Fn(w)andtn(w):=R2−‖w
an‖
2.
4
StanfordUniversityEE392oZ.Q.Luo
Definitionofτn
Weareconcernedwiththebehaviourofαn=sn/tn.
1.Thegoalistomakeαn∼‖∇Fn(w)‖.
∇Fn(w)=2(Rxx(n)w−pxy(n))
Thus,if
•‖∇Fn(w)‖islarge=⇒needαnlargeforregularization.
•‖∇Fn(w)‖issmall=⇒needonlyasmallαn.
2.Define
sn:=βR√2‖∇Fn(w
an−1)‖
R/√2:anormalizationconstantrequiredtoshowasymptoticconvergence
β:alsorequiredtoshowconvergence
3.Thedefinitionofτnfollowssimplyfromτn=Fn(wan−1)+sn
5
StanfordUniversityEE392oZ.Q.Luo
AnExample
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)
w(2)
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)w
(2)
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)
w(2)
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)
w(2)
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)
w(2)
−100−80−60−40−20020406080100−100
−80
−60
−40
−20
0
20
40
60
80
100
w(1)
w(2)
6
StanfordUniversityEE392oZ.Q.Luo
AsymptoticConvergenceAnalysis
Condition1.(BoundedAutocorrelationmatrix)Thereexistn0>0,σ1>0,σ2>0
suchthat
σ1I≤1
n
n∑
i=1
xixTi≤σ2I,∀n≥n0.
Condition2.(BoundedOutputs)Thereexistsafixedρysuchthatforalln>n0there
holds1
n
n∑
i=1
y2i≤ρy.
TheleftinequalityinCondition1isknownasweakpersistentexcitationcondition.
Theorem1.Letthesequenceofestimateswn,n=1,2,3,...begeneratedbythe
IPLSalgorithm.Then
‖∇Fn(wn)‖=2‖Rxx(n)wn−pxy(n)‖=O(1/n).
7
StanfordUniversityEE392oZ.Q.Luo
TransientConvergenceAnalysis
Condition1.(BoundedAutocorrelationmatrix)Thereexistn0>0,σ1>0,σ2>0
suchthat
σ1I≤1
n
n∑
i=1
xixTi≤σ2I,∀n≥n0.
Condition2.(NoNoise)Thesystemisfreefrommeasurementnoise,i.e.,
yi=xTiw∗,i=1,2,....
Weassumethatthedatahasnostatisticalfluctuations.Convergencethenimplies
thephasingoutofeffectsofinitializationandthusisdictatedentirelybythetransient
behaviourofthealgorithm.
Theorem2.Letthesequenceofestimateswn,n≥MbegeneratedbytheIPLS
algorithm.Iftheobservationsarefreeofnoise,then
‖wn−w∗‖=O(R−1)‖wn−1−w∗‖.
8
StanfordUniversityEE392oZ.Q.Luo
TransientAnalysis–asimpleexample
Example.yi=w∗xi+vi,withxi=w∗=1,i=1,2,...
•RLSAssumingnostatisticalaveraging(e.g.,nonoise),
Rxx(n)=E(xxT)=1,pxy(n)=E(xy)=1,∀n
TheRLSestimatorthenreducestowrlsn=(1+δ/n)−1
.
•IPLSNow,Fn(w)=(w−1)2,and∇Fn(w)=2(w−1).Evaluatingτn,the
condition∇φn(wan)=0fortheanalyticcenterbecomes,
2(wan−1)
βR√2|wa
n−1−1|+(wan−1−1)2−(wa
n−1)2+
2wan
R2−(wan)2=0
whichimplies
|wan−1|=O(R−1
)|wan−1−1|,
i.e.,exponentialdecayofthetransienterror.
9
StanfordUniversityEE392oZ.Q.Luo
DirectComparisonofRLS,IPLS
PropertyRLSIPLS
AsymptoticConvergenceO(1/n)O(1/n)
ComputationalComplexityO(M2)O(M
2.2)
TransientConvergenceO(1/n)O(R−n)
RobustnesstoInitializationnoyes
Additionalconstraintsreq.newalgorithmeasilyaccommodated
Numerical
Stability
(inconstrained
case)
problemsoccur
whenλissmall
(λ:forgetting
factor)
stableevenat
smallvaluesof
λ,andincases
oflimited
precision
calculationsSlidingwindowimplementationcanbeaccommodatedeasilyaccommodated
10
StanfordUniversityEE392oZ.Q.Luo
TheInteriorPointLeastSquares(IPLS)algorithm1.Wedon’tneedtheexactanalyticcenterofΩn,anapproximatecenterissufficient.
2.SuchanapproximatecenterisfoundbytakingjustasingleNewtoniterationinthe
minimizationofφn(w).
wn:=wn−1−(∇2φn(wn−1))−1
∇φn(wn−1),(3)
Tocompute(3)weneed
∇φn(w)=∇Fn
sn(w)+
2w
tn(w),(4)
∇2φn(w)=
(∇Fn)T∇Fn
s2n(w)
+∇2Fn
sn(w)+4ww
T
t2n(w)
+2I
tn(w)(5)
where∇Fn=−2pxy(n)+2Rxx(n)wand∇2Fn=2Rxx(n).
3.TocomputetheNewtondirection,anO(M2.2)recursiveupdateprocedurehasbeen
devised(usingtheworkofPowell,1997).
11
StanfordUniversityEE392oZ.Q.Luo
InteriorPointLeastSquares(IPLS)Algorithm
Step1:Initialization.Letβ,Rbegiven.Setw0=0,pxy(0)=0,Rxx(0)=0,
∇F0(0)=0.
Step2:Updating.Forn≥1,acquirenewdataxn,yn.Thenrecursivelyupdate
pxy(n)=n−1n
pxy(n−1)+1
nxnyn,Rxx(n)=
n−1n
Rxx(n−1)+1
nxnx
Tn.
Update
•∇2Fn(wn−1)and∇Fn(wn−1)
•sn(wn−1)andtn(wn−1)
•(∇2φn(wn−1))−1
∇φn(wn−1)usingtheupdateprocedure(orusing(4),(5))
Step3:Recentering.ThenewcenterofΩnisobtainedbytakingjustoneNewton
iterationstartingatwn−1:
wn:=wn−1−(∇2φn(wn−1))−1
∇φn(wn−1)
Setn:=n+1,andreturntoStep2.
12
StanfordUniversityEE392oZ.Q.Luo
Summary
Contributions
•providedanewlookat(recursive)adaptivefiltering•firstapplicationofinteriorpointoptimizationtoadynamicproblem
FeaturesofIPLS
•convergesasymptoticallyattherateO(1/n)•exhibitsfasttransientconvergence,andisrobusttoinitialization
•easilyaccommodatesadditionallinearorconvexquadraticconstraints,andis
numericallystable
•O(M2.2)complexity
13
StanfordUniversityEE392oZ.Q.Luo
Application:SystemIdentification
•PerformanceMeasureεip(n)=‖wn−w∗‖2andεrls(n)=‖w
rlsn−w∗‖
2
•Sources(i)WhiteGaussiannoise,(ii)WhiteGaussiannoisefilteredthrough
H(z)=1+2z−1
+3z−2
(1−1.1314z−1+0.64z−2)(1+0.9z−1).
•SNRs(i)SNR1=40dB,(ii)SNR2=10dB
•NominalParameterSettings
RLSλ=1,δ=10−4
IPLSβ=2,R=1000
•Experiment1w∈R20,w(i)∈[−1,+1],500independentMonteCarlo
trials
•Experiment2ComparingslidingwindowversionsofRLS(Liu&He’95)andIPLS:
w∈R10,Tl=15
14
StanfordUniversityEE392oZ.Q.Luo
SystemIdentification:Experiment1
20406080100120140160180200
10−2
10−1
100
Iteration k
IPAFRLS
2040608010012014016018020010
−2
10−1
100
Iteration k
IPAFRLS
(a)SNR=40dB,Source:WhiteGaussianNoise(b)SNR=40dB,Source:Correlated
20406080100120140160180200
100
101
Iteration k
IPAFRLS
20406080100120140160180200
100
101
102
Iteration k
IPAFRLS
(c)SNR=10dB,Source:WhiteGaussianNoise(d)SNR=10dB,Source:Correlated
15
StanfordUniversityEE392oZ.Q.Luo
SystemIdentification:Experiment2
05010015020025030035040010
−2
10−1
100
101
Iteration k
SW−RLSSW−IPAF
Figure1:Comparisonofsliding-windowversionsofIPLSandRLSwhenchannel
characteristicschangeabruptly(atiteration100).
16
StanfordUniversityEE392oZ.Q.Luo
Application:MinimumVarianceBeamforming
AdaptiveFilteringAlgorithm
ArrayAdjustableWeights
Steering Vectors
Target
Interferer
Output
Σ
Signal
y(n)
Sensor
17
StanfordUniversityEE392oZ.Q.Luo
MinimumVarianceBeamforming
Byadaptivelyadjustingthetapweightshi(n)thebeamformermust
1.SteeringCapability:protectthetargetsignal
cH(θ)h(n)=1,∀n,θ=θ1,θ2,...
cH(θ)=[1,e−jθ
,...,e−j(M−1)θ],
where
M:numberoftapweightshi,
θi:ElectricalAngledeterminedbythedirectionofthetargetiwithrespecttothe
firstsensor
2.Minimizetheeffectsoftheinterferers
i.e.,minimizetheOutputPowerE(|y|2)ofthebeamformer
ThisbeamformingproblemcanbecastintheframeworkofaConstrainedAdaptive
EstimationProblem.
18
StanfordUniversityEE392oZ.Q.Luo
Beamforming:ConstrainedAdaptiveEstimation
minimizeFn:=1
n
n∑
i=1
λn−i|d(i)−x
Tih|
2,
subjecttoCTh=f,h∈R
M,
(6)
λ:forgettingfactor
d(·):desiredresponse
xTi:vectorinputsequence
h:vectoroftapweights
C,f:definethelinearconstraintsonh
•IntheMinimumVarianceBeamformingproblemthereferencesignald(·)iszero.Duringtheadaptionprocessweassumethatnotargetispresent.
•TherowsofCcorrespondtosteeringvectorconstraints.
19
StanfordUniversityEE392oZ.Q.Luo
Beamforming:NumericalSimulation
Input:(interferenceat0.3,0.325and0.7)
x(n)=sin(0.3nπ)+sin(0.325nπ)+sin(0.7nπ)+b(n).
b(n):whiteGaussiannoiseat40dB.
Constraints:(desiredresponseatfreq.0.2and0.5)
CT=
1cos(0.2π)...cos((M−1)0.2π)1cos(0.5π)...cos((M−1)0.5π)0sin(0.2π)...sin((M−1)0.2π)0sin(0.5π)...sin((M−1)0.5π)
,f=
1
1
0
0
20
StanfordUniversityEE392oZ.Q.Luo
Beamforming:NumericalSimulation
00.10.20.30.40.50.60.70.80.91−80
−60
−40
−20
0
20
40
Normalized frequency
abs H(f) [dB
]
LCMV LC−FLS IPM1 Optimum Filter
0500100015002000250030003500400010
−2
10−1
100
101
102
Time (Iteration)
Norm
[ hopt − h(n) ]
LCMV LC−FLSIPM1
Figure2:(a)Freq.responseatIteration4000,(b)Mean-squarederrorinh(n)
LCMVµ=0.1
LCFLSλ=0.99,Eo=0.1
IPLSλ=0.99,ε=0.01,R=100,β=2
Precision4digitsforLCFLSandIPLS
21
StanfordUniversityEE392oZ.Q.Luo
Application:ChannelEqualizationinaCDMADownlink
LC (z)
LC (z)
LC (z)
1
2
K
Σ
u (k)
u (k)
u (k)
1
2
K
H (z)Σ
Channel
noise
chiprate
sampler
SignalReceived
Code Filters
y(k)
Figure3:Discrete-timemodelofCDMAdownlink
22
StanfordUniversityEE392oZ.Q.Luo
Equalizer/DecoderStructure
L FFF
FBF
ΣReceived signal
code matched filter
y(k)u (k) 1^
C (z ) 1-1r(k)x(k)
x(k)
Figure4:CodeMatchedFilter–ChiprateDFE
23
StanfordUniversityEE392oZ.Q.Luo
CDMADownlink:SystemDescription
SourcesQPSKwithuniformprobabilitiesforeachsymbol(i.i.d.)
FadingChannelLOScomponentis5dBhigherthan2multipathcomponents,fading
ratefD=0.005,delayspread:≤6Tc
02040608010012014016018020010
−2
10−1
100
101
Symbol Index
Individual path amplitudes
LOS PathFirst multi−pathSecond Multi−Path
StaticChannelFadingChannelsampledatarandominstant.
24
StanfordUniversityEE392oZ.Q.Luo
Experiment1:StaticChannel,Singleuser
810121416182010
−5
10−4
10−3
10−2
10−1
100
SNR
BE
R
SQRT−RLSIPLS
81012141618200.4
0.5
0.6
0.7
0.8
0.9
1
SNR
Prob(packet arrival)
SQRT−RLSIPLS
MessageSignalN=200,NT=10,CL=16,Users=1
Algorithmsλ=1.0,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
25
StanfordUniversityEE392oZ.Q.Luo
Experiment1:StaticChannel,4Users
810121416182022242610
−4
10−3
10−2
10−1
100
SNR
BE
R
SQRT−RLSIPLS
81012141618202224260.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SNR
Prob(packet arrival)
SQRT−RLSIPLS
MessageSignalN=200,NT=10,CL=16,Users=4
Algorithmsλ=1.0,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
26
StanfordUniversityEE392oZ.Q.Luo
Experiment1:DependenceonTrainingLength
246810121416182010
−5
10−4
10−3
10−2
10−1
100
NT
BE
R
SQRT−RLSIPLS
24681012141618200
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
NT
Prob(packet arrival)
SQRT−RLSIPLS
MessageSignalN=200,SNR=12dB,CL=16,Users=1
Algorithmsλ=1.0,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
27
StanfordUniversityEE392oZ.Q.Luo
Experiment2:Time-VaryingChannel,Singleuser
101214161820222410
−4
10−3
10−2
10−1
100
SNR
BE
R
SQRT−RLS, NT=2SQRT−RLS, NT=10IPLS, NT=2IPLS, NT=10
10121416182022240
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SNR
Prob(packet arrival)
SQRT−RLS, NT=2SQRT−RLS, NT=10IPLS, NT=2IPLS, NT=10
MessageSignalN=200,NT=2/10,CL=16,Users=1
Algorithmsλ=0.85,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
28
StanfordUniversityEE392oZ.Q.Luo
Experiment2:Time-VaryingChannel,4users
101214161820222426283010
−3
10−2
10−1
100
SNR
BE
R
SQRT−RLS, NT=10IPLS, NT=10
10121416182022242628300.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SNR
Prob(packet arrival)
SQRT−RLS, NT=10IPLS, NT=10
MessageSignalN=200,NT=2/10,CL=16,Users=4
Algorithmsλ=0.85,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
29
StanfordUniversityEE392oZ.Q.Luo
Experiment2:DependenceonTrainingLength
246810121416182010
−3
10−2
10−1
100
NT
BE
R
SQRT−RLSIPLS
24681012141618200.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
NT
Prob(packet arrival)
SQRT−RLSIPLS
MessageSignalN=200,SNR=16dB,CL=16,Users=1
Algorithmsλ=0.85,δ=R=104,β=2
EqualizerMff=14,Mfb=2,delay=1,pfr=10−2
30
StanfordUniversityEE392oZ.Q.Luo
Conclusions
•TransientconvergenceofIPLSisO(1/Rn),n≥M
•SuperiortransientconvergencetoRLSevenwhenn<M
•GainofusingIPLSovertheRLSalgorithmcanrangefrom5-6dBtowellover10dB
31