lecture 3: regression analysis & model fitting · lecture 3: regression analysis & model...

Lecture3:

Regressionanalysis&modelfitting

ShaneElipotTheRosenstielSchoolofMarineandAtmosphericScience,

UniversityofMiami

Createdwith{Remark.js}using{Markdown}+{MathJax}

Loading[MathJax]/jax/output/HTML-CSS/jax.js

http://remarkjs.com/

https://daringfireball.net/projects/markdown/

https://www.mathjax.org/

References

[1]Bendat,J.S.,&Piersol,A.G.(2011).Randomdata:analysisandmeasurementprocedures(Vol.729).JohnWiley&Sons.

[2]Thomson,R.E.,&Emery,W.J.(2014).Dataanalysismethodsinphysicaloceanography.Newnes.dx.doi.org/10.1016/B978-0-12-387782-6.00003-X

[3]Press,W.H.etal.(2007).Numericalrecipes3rdedition:Theartofscientificcomputing.Cambridgeuniversitypress.

[4]vonStorch,H.andZwiers,F.W.(1999).StatisticalAnalysisinClimateResearch,CambridgeUniversityPress

[5]Rawlings,J.,Pantula,S.G.,andDickey,D.A.(1998)AppliedRegressionAnalysis,Aresearchtool,secondedition,Springer

[6]Wunsch,C.(2006).Discreteinverseandstateestimationproblems:withgeophysicalfluidapplications.CambridgeUniversityPress.

[7]Fan,J.andGijbels,I.(1996),LocalPolynomialModellingandItsApplications,CRCPress

http://dx.doi.org/10.1016/B978-0-12-387782-6.00003-X

Lecture3:Outline1. Introduction2. Linearregression3. Polynomialinterpolation4. LocalPolynomialmodeling5. Anoteonnonlinearmodeling

1.Introduction

Introduction

Regressionanalysisconsistsinusingmathematicalexpressions(thatismodeling,ormodellingintheU.K.)todescribetosomeextentthebehaviorofarandomvariable(r.v.)ofinterest.Thisvariableiscalledadependentvariable.Thevariablesthatarethoughttoprovideinformationaboutthedependentvariableandareincorporatedinthemodelarecalledindependentvariables.Themodelsusedinregressionanalysistypicallyinvolveunknownconstants,calledparameters,whicharetobeestimatedfromthedata.

Introduction

Regressionanalysisconsistsinusingmathematicalexpressions(thatismodeling,ormodellingintheU.K.)todescribetosomeextentthebehaviorofarandomvariable(r.v.)ofinterest.Thisvariableiscalledadependentvariable.Thevariablesthatarethoughttoprovideinformationaboutthedependentvariableandareincorporatedinthemodelarecalledindependentvariables.Themodelsusedinregressionanalysistypicallyinvolveunknownconstants,calledparameters,whicharetobeestimatedfromthedata.

Themathematicalcomplexityofthemodel,andthedegreetowhichitisrealistic,dependonhowmuchisknownabouttheprocessandthepurposeoftheregressionanalysis(andtheabilityandknowledgeofthescientist).

Mostregressionmodelsthatwewillencounterarelinearintheirparameters.Iftheyarenotlinear,theycanoftenbelinearized.

Criticalthinkingshouldbeemployed,asanymodelcanbefittedto(orregressedagainst)anydata.

Example

DailyatmosphericCO measuredatMaunaLoainHawaiiatanaltitudeof3400m.DatafromDr.PieterTans,NOAA/ESRL(www.esrl.noaa.gov/gmd/ccgg/trends/)andDr.RalphKeeling,ScrippsInstitutionofOceanography(scrippsco2.ucsd.edu).

2

http://www.esrl.noaa.gov/gmd/ccgg/trends/

http://scrippsco2.ucsd.edu/

Example

Determiningthelineartrendofthistimeseriesisanexampleoflinearregression.Furthermodelingcouldincludeestimatingtheseasonalcycleofthetimeseriesetc.

2.Linearregression

Simplelinearregression

Wearegoingtoreviewthesimplestlinearmodelinvolvingoneindependentvariable andonedependentvariable .Inparallelwewillalsopresenttheequationsforamoregeneralmodelrelatingto dependentvariables .Matlabusesnotationthatressemblethematrixformulasforthegeneral(multivariate)linearmodel.

x yy

p , ,… ,x1 x2 xp


Wearegoingtoreviewthesimplestlinearmodelinvolvingoneindependentvariable andonedependentvariable .Inparallelwewillalsopresenttheequationsforamoregeneralmodelrelatingto dependentvariables .Matlabusesnotationthatressemblethematrixformulasforthegeneral(multivariate)linearmodel.

Asanexample,wewillseethattheleastsquaressolutionofthelinearmodelis

whichinMatlabcanbewritten

B = (X'*X)^-1*X'*Y;

butisbettercodedas

B = (X'*X)\X'*Y;

x yy

p , ,… ,x1 x2 xp

= ( X Yβ XT )−1XT


Previously,whenexaminingasetofobservations of ,weassumedthattheexpectation,ortruemean,wasconstant,i.e.

.

Yi y

E[ ] =Yi μy



.

Weknowconsiderthecasewhenthemeanisafunctionofanothervariable,asanexampletime.Linearregressionscanconsistinestimatingthetrendandtheseasonalcycleofyourtimeseries.

Yi y

E[ ] =Yi μy

InthecaseoftheCO record,themeanisclearlynotaconstant,increasingeveryyear,butalsooscillatingwithineachyear.



.

Weknowconsiderthecasewhenthemeanisafunctionofanothervariable,asanexampletime.Linearregressionscanconsistinestimatingthetrendandtheseasonalcycleofyourtimeseries.

Yi y

E[ ] =Yi μy

2

Thismodelisapplicableasanexampleforestimatingalineartrendofatimeseries,oranylinearrelationshipbetweentwor.vs.


Thesimplestmodelisthatthetruemeanorexpectationofchangesataconstantrateasthevalueof decreasesorincreases:

where and aretheparameterstoestimate.

yx

E[ ] = + , i = 1,…,nYi β0 β1Xi

β0 β1


Theobservationsofthedependentvariable arelookedatasindividualrealizationsofther.vs. withpopulationsmeans .Thedeviationof from istakenintoaccountbyincorporatingarandomerror inthelinearmodel

yYi E[ ]Yi

Yi E[ ]Yi

ϵi

= + +Yi β0 β1Xi ϵi

The areassumednormallyindependentidenticallydistributed(i.i.d.)r.vs.as .Sinceand areconstant, .

Incontrast,theobservedvaluesof aresupposedtobefreeoferrors,treatedasconstants.


Theobservationsofthedependentvariable arelookedatasindividualrealizationsofther.vs. withpopulationsmeans .Thedeviationof from istakenintoaccountbyincorporatingarandomerror inthelinearmodel

yYi E[ ]Yi

Yi E[ ]Yi

ϵi


ϵi

∼ N (0,σ) ,β0 β1Xi ∼ N (E[ ],σ)Yi Yi

X

Generallinearmodel(ormultipleregression)

Thegenerallinearmodelwith independentvariablesforobservation is

Thereare parameterstoestimate:aconstant andfactors .Inmatrixnotation,for observations,weobtainthelinearsystem

pi

= + + +⋯+ +Yi β0 β1Xi1 β2Xi2 βpXip ϵi

= p+ 1p′ ( )β0 p,… ,β1 βp n

Y = Xβ+ ϵ

⎡

⎣⎢⎢⎢⎢Y1

Y2

⋮Yn

⎤

⎦⎥⎥⎥⎥

(n× 1)

=

⎡

⎣⎢⎢⎢⎢⎢11

⋮1

X11

X21

⋮Xn1

X12

X22

⋮Xn2

⋯⋯

⋯

X1p

X2p

⋮Xnp

⎤

⎦⎥⎥⎥⎥⎥

(n× )p′

+

⎡

⎣⎢⎢⎢⎢β0

β1

⋮βp

⎤

⎦⎥⎥⎥⎥

( × 1)p′

⎡

⎣⎢⎢⎢⎢ϵ1

ϵ2

⋮ϵn

⎤

⎦⎥⎥⎥⎥

(n× 1)

Generallinearmodel

Eachelement isapartialregressioncoefficientthatquantifiesthechangeinthedependentvariable perunitchangeintheindependentvariable ,assumingallotherindependentvariablesareheldconstant.

β =

⎡

⎣⎢⎢⎢⎢β0

β1

⋮βp

⎤

⎦⎥⎥⎥⎥

βj

Yi

Xij

Simplelinearmodel

Ifthe were andthemodelwereabsolutelytrue,anytwopairsofobservations wouldbeenoughtosolveforthetwounkownparameters and .


ϵi 0( , )Xi Yi

β0 β1

Simplelinearmodel

Ifthe were andthemodelwereabsolutelytrue,anytwopairsofobservations wouldbeenoughtosolveforthetwounkownparameters and .

Yet,becauseoferrors,anothermethodisused,calledleastsquaresestimation,whichgivesasolution,orestimate thatleadstothesmallestpossiblesumofsquareddeviationsoftheobservations fromtheestimates oftheirtruemeans .


ϵi 0( , )Xi Yi

β0 β1

( , )β0 β1

Yi E[ ]Yi E[ ]Yi

Simplelinearmodel:LSsolution

Let providetheestimateofthetruemean

suchthatthesumofsquaresofdeviationsfromthemean

isminimized.

iscalledthe -thobservedresidual.

( , )β0 β1

= + ≡E[ ]Yi β0 β1Xi Yi

SS(Res) = ( − =∑i=1

n

Yi Yi )2 e2i

ei i

Generallinearmodel

Forthegeneralmodel,inmatrixnotation,

andtheresidualsarefoundinthe vector

andthesumofsquaresofresidualsis

whichisaminimumbecauseof .

Howtofind ?

≡ XY β

(n× 1)

e = Y− = Y−XY β

SS(Res) = e = (Y− (Y− ) = (Y−X (Y−X )eT Y)T Y β)T β

β

β

Simplelinearmodel

Themethodtofindthevalues thatminimize isclassic.Youtakethederivativesof withrespecttoeachofthe parametersandequatetheresultstozero.Youobtainasystemof equationswith unkowns.Forthesimplelinearmodelyouobtainthenormalequations

whichsolutionis

( , )β0 β1 SS(Res)SS(Res)

p+ 1p+ 1 p+ 1

nβ0

β0 ∑i

Xi

+

+

β1 ∑i

Xi

β1 ∑i

X2i

=

=

∑i

Yi

∑i

XiYi

β1

β0

=

=

= =( − )( − )∑i Xi X

¯ ¯¯Yi Y

¯ ¯¯

( −∑i Xi X¯ ¯¯ )2

sxy

sxx

sxy

s2x

−Y¯ ¯¯

β1X¯ ¯¯

Simplelinearmodel

Thepredictedvaluesfromthesolutionofthelinearmodelare

canbeinterpretedasbeingboththeestimateofthepopulationmeanof foragivenvalueof ,andthepredictedvaluevalueofforafuturevalueof whichis .

= + =Yi β0 β1Xi E[ ]Yi

Yi

y x yx Xi

Generallinearmodel

Forthemultipleregressionmodel,thenormalequationsareobtained

andtheleastsquares(LS)solutionis

Thepredictedvaluesof are

with calledtheprojectionmatrix.Thislastexpressionsshowsthattheestimated arelinearfunctionofalltheobservedvalues .

= 0 → X = Y∂SS(Res)

∂βXT β XT

= ( X Yβ XT )−1XT

y

Y ==

X = X( X Yβ XT )−1XT

PY

P = X( XXT )−1XT

Yi

Yi

Simplelinearmodel

Theobservationsof cannowbewrittenasthesumoftheestimatedpopulationmeanforagivenvalueof andaresidual

Thesumofthesquaresoftheobservationsare

sinceitcanbeshownthat .

Thesumofthesquaresoftheobservationsisthesumofthesquares"accountedfor"bythemodelplusthesumofthesquaresof"unaccountedfor".

yx

= +Yi Yi ei

∑i

Y 2i =

=

( +∑i

Yi ei)2

+∑i

Yi

2 ∑i

e2i

2 = 0∑i Yi

2ei

Simplelinearmodel

Using ,thedecompositionofthesumofthesquarescanbeusedasfollows

= (1/n)Y¯ ¯¯ ∑i Yi

− n∑i

Y 2i Y

¯ ¯¯ 2

( −∑i

Yi Y¯ ¯¯ )2

=

=

− n∑i

Yi

2Y¯ ¯¯ 2

( −β12∑

i

Xi X¯ ¯¯ )2

+

+

∑i

e2i

∑i

e2i

Simplelinearmodel


Whatdoesthissay?[Uptoafactor ]

= (1/n)Y¯ ¯¯ ∑i Yi

− n∑i

Y 2i Y

¯ ¯¯ 2

( −∑i

Yi Y¯ ¯¯ )2

=

=

− n∑i

Yi

2Y¯ ¯¯ 2

( −β12∑

i

Xi X¯ ¯¯ )2

+

+

∑i

e2i

∑i

e2i

1/(n− 1)

Simplelinearmodel


Whatdoesthissay?[Uptoafactor ]

Itapproximatelysaysthat:

"Thetotalvariancefromobservations"="variancefromtheregression"+"varianceoftheresiduals"

Inthemodel ,theregressionpartis .

iscalledtheregressioncoefficient.

= (1/n)Y¯ ¯¯ ∑i Yi

− n∑i

Y 2i Y

¯ ¯¯ 2

( −∑i

Yi Y¯ ¯¯ )2

=

=

− n∑i

Yi

2Y¯ ¯¯ 2

( −β12∑

i

Xi X¯ ¯¯ )2

+

+

∑i

e2i

∑i

e2i

1/(n− 1)

= +Yi β1Xi β0 β1Xi

β1

Coefficientofdetermination

Fromthelinearmodel,weareinterestedinaquantitycalledthecoefficientofdetermination

Forthesimple(univariate)linearmodel,

isthusthesquareofthePearson'scorrelationcoefficientbetween and .

= = 1 −R2( − −∑i Yi Y

¯ ¯¯ )2 ∑i e2i

( −∑i Yi Y¯ ¯¯ )2

∑i e2i

( −∑i Yi Y¯ ¯¯ )2

R2 =

=

=( −β1

2∑i Xi X

¯ ¯¯ )2

( −∑i Yi Y¯ ¯¯ )2

( )sxy

s2x

2s2x

s2y

= =s2xy

s2xs2y

⎛⎝⎜

sxy

s2xs2y

− −−−√⎞⎠⎟2

r2xy

R2

x y

Simplelinearmodel

Atraditionalinterpretationof isthatitisameasureofthefractionofvarianceofthedependentvariable explainedbytheindependentvariable .

Thisiswhythe(squareofthe)Pearsoncorrelationcoefficientisveryquicklyinterpretedasbeingameasureoftheamountofvarianceexplainedbetweentwovariables.

Asanexampleif youwilloftenreadsometinglike"xisabletoexplain49%ofthevarianceofy".(Since )

R2

R2

yx

= 0.7rxy= 0.490.72

Simplelinearmodel:uncertainties

Inthemodel whereweassumedthat,wedidnotknowthevariance .= + +Yi β0 β1Xi ϵi

∼ N (0,σ)ϵi σ2


Inthemodel whereweassumedthat,wedidnotknowthevariance .Anunbiased

estimateof isgivenbytheresidualmeansquare:

This"mean"valueisobtainedbydividingthe bythenumberofdegreesoffreedomfortheresidualswhichisthenumberofdatapoints minusthenumberofparametersofthemodel

.

= + +Yi β0 β1Xi ϵi∼ N (0,σ)ϵi σ2

σ2

= ≡σ2

s2∑i e

2i

n− (p+ 1)

SS(Res)

(n)(p+ 1)


AnumberofformulasforthevarianceoftheestimatescanbederivedandusedforcalculatingCIs:

Var[ ]β1

Var[ ]β0

Var[ ]Yi

Var[ ]Y0

=

=

=

=

s2

( −∑i Xi X¯ ¯¯ )2

[ + ]1n

X¯ ¯¯ 2

( −∑i Xi X¯ ¯¯ )2

s2

[ + ]1n

( −Xi X¯ ¯¯ )2

( −∑i Xi X¯ ¯¯ )2

s2

[1 + + ]1n

( −X0 X¯ ¯¯ )2

( −∑i Xi X¯ ¯¯ )2

s2


AnumberofformulasforthevarianceoftheestimatescanbederivedandusedforcalculatingCIs:

Var[ ]β1

Var[ ]β0

Var[ ]Yi

Var[ ]Y0

=

=

=

=

s2

( −∑i Xi X¯ ¯¯ )2

[ + ]1n

X¯ ¯¯ 2

( −∑i Xi X¯ ¯¯ )2

s2

[ + ]1n

( −Xi X¯ ¯¯ )2

( −∑i Xi X¯ ¯¯ )2

s2

[1 + + ]1n

( −X0 X¯ ¯¯ )2

( −∑i Xi X¯ ¯¯ )2

s2

: , = , , ∼ t(0,n− p−H0 β 1,0 Yi m1,0,Yi

−β1 m1

Var[ ]β1

− −−−−−√−β0 m0

Var[ ]β0

− −−−−−√−Yi mYi

Var[ ]Yi

− −−−−−√

Generallinearmodel:uncertainties

Forthegenerallinearmodel,theformulasare

P

Var[ ]β

Var[ ]Y

Var[ ]Y0

Var[e]=σ

2s2

=

=

=

=

=

=

X( XXT )−1XT

( XXT )−1σ2

Pσ2

[I+ ( X ]X0 XT )−1X0T σ2

(I−P)σ2

e/(n− p− 1)eT

Generallinearmodel:uncertainties

Forthegenerallinearmodel,theformulasare

Bewarethattheexpressionabovesarematrices.Asanexampleforthesimplelinearmodelforwhich :

thisimpliesthattheparameterestimatescovary.

P

Var[ ]β

Var[ ]Y

Var[ ]Y0

Var[e]=σ

2s2

=

=

=

=

=

=

X( XXT )−1XT

( XXT )−1σ2

Pσ2

[I+ ( X ]X0 XT )−1X0T σ2

(I−P)σ2

e/(n− p− 1)eT

p+ 1 = 2

Var[ ] = [ ]βVar( )β0

Cov( , )β1 β0

Cov( , )β0 β1

Var( )β1

NotehowtheCIsfortheprediction forfuturevalues

of arelargerthantheCIsforthepredictionofthemeanof

.Thevarianceofthepredictionisthevarianceofestimatingthemeanplusthevarianceofthequantityestimated.

TheCIsarethesmallestfor.

Inthiscase,IgenerateddataandIhadprescribed


Y0X0 x

Yi

=X0 X¯ ¯¯

= 0, = 0.8, = 0.04β0 β1 σ2

Linearmodelbyleastsquares

Themethodofleastsquarestofindasolutiontothegenerallinearmodelisappropriatewhenfourassumptionsarevalid:(1)therandomerrors arenormallydistributed,(2)independent,(3)withzeromeanandconstantvariance ,and(4)the areobservationsofthe independentvariablesmeasuredwithouterrors.


ϵiσ2 Xij

p



Ifwerelyonalargenumber ofdata,thenormalassumptionmaybeinvokedbecauseoftheCLT.Otherwise,MaximumLikelihoodmethodscanbeused.AsanexampleseeElipotetal.(2016).


ϵiσ2 Xij

p

n

http://dx.doi.org/10.1002/2016JC011716




Whenthedependentvariableobservationsarenormallydistributedbutdonothavethesamevariances,orerrors,themethodofweightedleastsquarescanbeimplemented.


ϵiσ2 Xij

p

n

http://dx.doi.org/10.1002/2016JC011716




Whenthedependentvariableobservationsarenormallydistributedbutdonothavethesamevariances,orerrors,themethodofweightedleastsquarescanbeimplemented.

Whentheindependentvariablesareactuallynotindependent(becausetheyaremaybecorrelated),themethodofgeneralleastsquarescanbeimplemented.Seereferences[5]and[6].


ϵiσ2 Xij

p

n

http://dx.doi.org/10.1002/2016JC011716

Linearmodelbyweightedleastsquares

Let'sassumethatthevarianceofeach (andthusofeach )iswhere isaconstant.Asanexample,someobservationsmay

havebetteraccuracythanothers.

ϵi Yi

a2i σ2 σ


Wecanconsiderthefollowingrescaledmodel,dividingby :

or

Becausethevarianceofthe is ,thevarianceofthebecomes .Wecannowuseleastsquarestoregress onthe

.

ai

= + + +⋯+ +Yi

ai

1ai

β0 β1Xi1

aiβ2

Xi2

aiβp

Xip

ai

ϵiai

= + + +⋯+ +Y ∗i X∗

i0β0 β1X∗i1 β2X

∗i2 βpX

∗ip ϵ∗i

ϵi a2i σ2 ϵ∗i

σ2 Y ∗i

X∗ij


Theprinciplehereistoassigntheleastamountofweighttotheobservationswiththelargestvariance,orerror.Theweightingmatrixis

Considerthegenerallinearmodelequationleft-multipliedby

whichcanberewrittenas

with ,etc.

W =

⎡

⎣⎢⎢⎢⎢⎢1/a10

⋮0

01/a2

⋮0

⋯0

⋱⋯

0

⋮1/an

⎤

⎦⎥⎥⎥⎥⎥

W

WY = WXβ+Wϵ

= β+Y∗ X∗ ϵ∗

= WYY∗


Theweightedleastsquaresolutionis

with

β

Var[ ]β

Var[ ]YVar[e]

=

=

==

( X YX′V−1 )−1X′V−1

( XX′V−1 )−1σ2

X( XX′V−1 )−1X′σ2

[V−X( X ]X′V−1 )−1X′ σ2

= W =V−1 W′

⎡

⎣⎢⎢⎢⎢⎢a21

0

⋮0

0

a22

⋮0

⋯0

⋱⋯

0

⋮a2n

⎤

⎦⎥⎥⎥⎥⎥


Theweightedleastsquaresolutionis

with

Infact,theweightingmatrixcanhavewhatevercoefficientyouwant!Hereitisaspecialcasethatsimplifiestheformofthesolution.Seesection2.4ofreference[6].

β

Var[ ]β

Var[ ]YVar[e]

=

=

==

( X YX′V−1 )−1X′V−1

( XX′V−1 )−1σ2

X( XX′V−1 )−1X′σ2

[V−X( X ]X′V−1 )−1X′ σ2

= W =V−1 W′

⎡

⎣⎢⎢⎢⎢⎢a21

0

⋮0

0

a22

⋮0

⋯0

⋱⋯

0

⋮a2n

⎤

⎦⎥⎥⎥⎥⎥

3.Polynomialinterpolation

Polynomialfitting

Fittingapolynomialfunctionofanindependentvariable toadependentvariable isalinearregressionproblemwhichconsistsinestimatingthecoefficientsofthepolynomial

xy

y = + x+ +…+ +…β0 β1 β2x2 βkx

k

Polynomialfitting


Wewillreviewtwogeneralcases.Thefirstcaseisglobalpolynomialfittingwhereyouarefittingapolynomialfunctionthatexactlyestimateyourdata,maybepiecewise,inseparateintervals.Thispolynomialisofmaximumorderofthenumberofobservationsminusoneandiscalledaninterpolatingpolynomial.

xy

y = + x+ +…+ +…β0 β1 β2x2 βkx

k

Polynomialfitting



Thesecondcaseiscalledlocalpolynomialestimationwhenyouarefittingapolynomialinthevicinity,thatiswithinawindow,ofagivenvalueof .Thispolynomialofarbitraryorderapproximateyourdatalocallyandthesolutionistypicallyobtainedbyweightedleastsquares.

xy

y = + x+ +…+ +…β0 β1 β2x2 βkx

k

x

Polynomialfitting



Thesecondcaseiscalledlocalpolynomialestimationwhenyouarefittingapolynomialinthevicinity,thatiswithinawindow,ofagivenvalueof .Thispolynomialofarbitraryorderapproximateyourdatalocallyandthesolutionistypicallyobtainedbyweightedleastsquares.

Thesetwotypesofmethodscanbeusedingeneraltoprocessyourdatatoeitherinterpolateorgridyourdata.

xy

y = + x+ +…+ +…β0 β1 β2x2 βkx

k

x

Interpolatingpolynomial

Assumeyouhave pairsofobservations andwouldliketointerpolate forgivenvalueof .

Thereexistsaninterpolatingpolynomialoforder givenbythefollowingLagrangeformula

whichpassesthroughyourdatapoints,i.e.

N ( , )Xi Yi

y x

N − 1

(x) =PN−1 ∑k=1

N⎛⎝⎜⎜ ∏

j=1j≠k

N x−Xj

−Xk Xj

⎞⎠⎟⎟ Yk

( ) =PN−1 Xi Yi


ExamplewithN = 5

Polynomialfitting

Alternativelyyoucanuseleastsquarestofitapolynomialofanyorderequaltoorlessthan withthemodelN − 1

= + + +…Yi β0 β1Xi β2X2i βN−1X

N−1i


Bewarethatinterpolatingpolynomialcanquicklygenerateverylargeoscillations!

Sameexampleasbeforeexceptthattheoriginaldatapointwasmovedto .

= 2Xi

2.9

Piecewiselinearinterpolation

Apiecewiselinearinterpolation,orsimplylinearinterpolationconsistincalculatingtheinterpolatingpolynomialoforder1overaninterval ,i.ewith points.TheLagrangeformulagives

whichcanberearrangedtogivethelinearinterpolant

InMatlabitisimplementedby

yi = interp1(x,y,xi);

[ , ]Xk Xk+1 N = 2

(x) = ( ) + ( )P1x−Xk+1

−Xk Xk+1Yk

x− Xk

−Xk+1 XkYk+1

(x) = + (x− )L1 Yk Xk−Yk+1 Yk

−Xk+1 Xk

Piecewiselinearinterpolation:errors

TheLagrangeformulagivesyouaneasywaytoestimatetheinterpolationerror.If and aretheerrorsoruncertaintiesfor

and ,sinceδk δk+1

Yk Yk+1

(x) = +P1 akYk ak+1Yk+1

δ (x) =P1 +a2kδ2k

a2k+1δ

2k+1

− −−−−−−−−−−−−√

Hermitepolynomialinterpolation

TheissueoflargeoscillationsininterpolatingcanbereinedinbyusingHermitepolynomialswhichsatisfyadditionalconditionsonitsderivativesatthedatapoints,

where istobespecified. isthe -thderivativeof .

( ) = , ( ) =Pn Xk Yk P(1)n Xk dk

dk P(ν)n ν Pn

Hermitepolynomialinterpolation

TheissueoflargeoscillationsininterpolatingcanbereinedinbyusingHermitepolynomialswhichsatisfyadditionalconditionsonitsderivativesatthedatapoints,

where istobespecified. isthe -thderivativeof .

ApopularHermitepolynomialistheshape-preservingpiecewisecubicHermiteinterpolatingpolynomialorshape-preservingpchip,implementedinMatlabby

yi = pchip(x,y,xi);

( ) = , ( ) =Pn Xk Yk P(1)n Xk dk

dk P(ν)n ν Pn

pchipexample

Apchippolynomialiscubic(order3)anditsderivatives ,orslopes,ateachdatapointarezeroortheharmonicmeansofconsecutiveslopes:

dk

= + with =1dk

1δk

1δk−1

δk−Yk+1 Yk

−Xk+1 Xk

Splineinterpolation

Anotherpopularinterpolationmethodusescubicsplineswhicharepiecewisecubicinterpolatingpolynomialswithconstraintsonthesecondderivativetobeacontinuous.ItisimplementedinMatlabas

yi = spline(x,y,xi);

Cubicinterpolation

Anothermethodusingpiecewisepolynomialsoforder3iscalledcubicconvolutionandisdescribedindetailinKeys1981.ThismethodisaccessibleinoneorhigherdimensionsinMatlabas

yi = interpn(x,y,xi,'cubic');

http://dx.doi.org/10.1109/TASSP.1981.1163711

Somecomments

Interpolatingpolynomialsandsplinesaregreatsetsoftoolthatallowyoutoquicklyinterpolateyourdata.Splinesarenotnecessarilypolynomialoforder3andcanbeofgreaterorder.Thereexistsaverylargebodyoflitteraturedealingwithsplines.

Somecomments


Wehavedealtsofarwithmethodsofinterpolationinonedimensionbutthesecanbeeasilyexpandedintwoormoredimensions,notablythelinearandcubicmethods.

Somecomments


Wehavedealtsofarwithmethodsofinterpolationinonedimensionbutthesecanbeeasilyexpandedintwoormoredimensions,notablythelinearandcubicmethods.

Polynomialinterpolationimpliesthatyouareexactlyrecoveringyourdata,i.e .Thisimpliesthatyourdataareeffectivelyerrorfree.Wenowrelaxthisconditionandreviewsomeprinciplesoflocalpolynomialmodeling.

P ( ) =Xi Yi

4.LocalPolynomialModeling

Polynomialbyleastsquares

Wesawearlierthatwecanuseleastsquarestofitapolynomialofanyorderequaltoorlessthanyour datapoints:N − 1

= + + +…Yi β0 β1Xi β2X2i βN−1X

N−1i

Localpolynomialfitting

Onceagainweattempttoestimatethevalueofadependentvariablegivenavalueoftheindependentvalue .Herewefollowclosely

reference[7].Theideaistoestimateanarbitraryfunctionanditsderivativenoted withthemodel

where areobservationsand isthevarianceof at.

y xm(x)

(x), (x),… , (x)m(1) m(1) m(p)

= m( ) + σ( )Yi Xi Xi

( , )Xi Yi σ( )Xi Yi

x = Xi


Onceagainweattempttoestimatethevalueofadependentvariablegivenavalueoftheindependentvalue .Herewefollowclosely

reference[7].Theideaistoestimateanarbitraryfunctionanditsderivativenoted withthemodel

where areobservationsand isthevarianceof at.

Thefunction isapproximatedlocallybyapolynomialoforderbyconsideringaTaylorexpansionintheneighborhoodof as

y xm(x)

(x), (x),… , (x)m(1) m(1) m(p)

= m( ) + σ( )Yi Xi Xi

( , )Xi Yi σ( )Xi Yi

x = Xi

m(x)p x0

m(x) ≈ m( )x0

=

+ ( )(x− ) + (x− +…+ (x−m′ x0 x0( )m′′ x0

2!x0)2

m(p)

p!+ (x− ) + (x− +…+ (x−β0 β1 x0 β2 x0)2 βp x0)p


Thefunction ismodeledlocallyas

and .Theestimatesof ofthispolynomialareobtainedforeachlocationofinterest byleastsquaresfitting,minimizingthefollowingexpression

where

iscalledakernelfunction,actingoverahalf-bandwidth .

m(x)

m(x) = (x− .∑j=0

p

βj x0)j

( ) = j!m(j) x0 βj βj

x0

{ − ( − ( − )∑i=1

n

Yi ∑j=0

p

βj Xi x0)j}2Kh Xi x0

(x) = K ( )Kh

1h

x

h

K h


Inthisexample,theunknownfunctiongiving isestimatedat usinganorderonepolynomial,usingdatapointswithintheorangewindow.

E[ ] = m( )Yi Xi

X = x0


Forfittingapolynomialtoyourdata,anumberofaspectsneedtobeconsidered,allcoveredinmanydetailsasanexampleinreference[7]:

1. Whichorderpolynomialdoyouneed?Areyoutryingtoestimatethevalueofyourunknownfunctiononly,orareyoutryingtoestimatethe -thderivativeaswell?Inthiscase,itisrecommendedthat beanoddnumber.

2. Whatbandwidth doyouneed?Itwilldependsonthedensityofyourdata,aswellastheorderofthechosenpolynomial.Thechoiceofthebandwidthisacompromisebetweenbiasandvarianceofyourestimate.Sinceyouaretryingtoestimateparametersbyleastsquaresyoushouldhaveatleastthatnumberofpointsinyourwindow.

3. Whatshapeshouldthekernelfunctionhave?Shoulditbeuniform?Gaussian?Quadratic?AquadratickernelcalledtheEpanechnikovkernelisoftenrecommended

(seepracticalthisafternoon!)

νp− ν

h

p+ 1


Thisfigureshowsanexampleoffittingaknownfunctionembeddedinnoisewithaknownvariance.Itshowstheimpactofthebandwidthandpolynomialorderonthebiasandvarianceoftheestimates.


Asimplersmootherconsistsinestimatingthefunction asapolynomialoforder .TheequivalentiscalledtheNadaraya-Watsonkernelestimatordefinedas

ThetypicalkernelfunctionsusedaretheGaussiankernel

andthesymmetricBetafamily

where isacomplicatedfunctionofnointeresthere.

m(x)0

(x) ≡mh

( − x)∑j=1

n

Kh Xj Yj

( − x)∑j=1

n

Kh Xj

K(z) = ( exp(− /2)2π−−

√ )−1 z2

K(z) = (1 − , γ = 0, 1,… ,1

B(1/2,γ+ 1)t2)γ+

B(z,w)

5.Anoteonnonlinearfitting

Nonlinearfitting

Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?

Nonlinearfitting

Whatdoesonedowhenthefunctionyouaretryingtofittoyourdataisnonlinearinyourparameter?Asanexample,youexpectthatasinusoidfunctionisagoodmodeltodescribethedependencyofyourdependentvariable ontheindependentvariable ,i.e.

where istheamplitudeand isthephase.

y x

y(x) = a cos(x+ ϕ)

a ϕ

Nonlinearfitting


where istheamplitudeand isthephase.Inthiscaseyou'reinluckbecauseyoucanusetrigonometricidentitiesandwrite

y x

y(x) = a cos(x+ ϕ)

a ϕ

y(x) = a cos(ϕ) cos(x) − a sin(ϕ) sin(x)

Nonlinearfitting


where istheamplitudeand isthephase.Inthiscaseyou'reinluckbecauseyoucanusetrigonometricidentitiesandwrite

Youhavelinearizedyourproblem,andyouarenowfacedwithamultiplelinearregressionproblem,estimating and

asafunctionofobservationsof and (seepracticalthisafternoon).

y x

y(x) = a cos(x+ ϕ)

a ϕ

y(x) = a cos(ϕ) cos(x) − a sin(ϕ) sin(x)

a cos(ϕ)a sin(ϕ) y, cos(x) sin(x)

Nonlinearfitting

Whatifyoureallycannotlinearizeyourproblem?

Nonlinearfitting

Whatifyoureallycannotlinearizeyourproblem?Asanexample,itisoftenusefultomodelthelaggedcorrelationfunction ,asinBealetal.2015

ρ(τ)

http://dx.doi.org/10.1175/JPO-D-14-0119.1

Nonlinearfitting

Inthisparticularcase,weassumedthatthelaggedcorrelationfunctionforthealong-shorecomponentofvelocity,asafunctionofseparationdistance(lag)wasgivenby

Thegoalisheretofitthedataforthevalueoftheparameter ,aspatiallengthscale.Weapplythesameprincipleofminimization,tryingtofindthevalue minimizing

(r) = cos( )ρh e−(r/rh)2 πr

2rh

rh

rh

SS(Res) = ( ( ) −∑i=1

N

ρh ri ρi)2

Nonlinearfitting

Inthisparticularcase,weassumedthatthelaggedcorrelationfunctionforthealong-shorecomponentofvelocity,asafunctionofseparationdistance(lag)wasgivenby

Thegoalisheretofitthedataforthevalueoftheparameter ,aspatiallengthscale.Weapplythesameprincipleofminimization,tryingtofindthevalue minimizing

Sincetheproblemcannotbeputinlinearform,theleastsquaremethodisnotavailable.Insteadyoumustrelyonnonlinearoptimizationroutines.Asanexample,Matlabcanapplycommonalgorithmsbythefunction .

(r) = cos( )ρh e−(r/rh)2 πr

2rh

rh

rh

SS(Res) = ( ( ) −∑i=1

N

ρh ri ρi)2

fminsearch

Practicalsession

Pleasedownloaddataatthefollowinglink:

PleasedownloadtheMatlabcodeatthefollowinglink:

MakesureyouhaveinstalledandtestedthefreejLabMatlabtoolboxfromJonathanLillyatwww.jmlilly.net/jmlsoft.html

https://www.jmlilly.net/jmlsoft.html

lecture 3: regression analysis & model fitting · lecture 3: regression analysis & model...

Documents