dart tutorial sec’on 4: how should observaons impact an ...single observed variable, single...

TheNa'onalCenterforAtmosphericResearchissponsoredbytheNa'onalScienceFounda'on.Anyopinions,findingsandconclusionsorrecommenda'onsexpressedinthispublica'onarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNa'onalScienceFounda'on.

©UCAR

DARTTutorialSec'on4:Howshouldobserva'onsimpactanunobserved

statevariable?Mul'variateassimila'on.

Singleobservedvariable,singleunobservedvariable.

Sofar,haveknownobserva'onlikelihoodforsinglevariable.Now,supposepriorhasanaddi'onalvariable.Willexaminehowensemblemethodsupdateaddi'onalvariable.Basicmethodgeneralizestoanynumberofaddi'onalvariables.MethodsrelatedtoKalmanfilterinsomesense,butnotdonehere.

DARTTutorialSec'on4:Slide2

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Whatshouldhappentotheunobservedvariable?

DARTTutorialSec'on4:Slide3

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Updateobservedvariablewithoneofthepreviousmethods.

DARTTutorialSec'on4:Slide4

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Updateobservedvariablewithoneofthepreviousmethods.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

DARTTutorialSec'on4:Slide5

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Updateobservedvariablewithoneofthepreviousmethods.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

DARTTutorialSec'on4:Slide6

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Computeincrementsforpriorensemblemembersofobservedvariable.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide7

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Computeincrementsforpriorensemblemembersofobservedvariable.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide8

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Computeincrementsforpriorensemblemembersofobservedvariable.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide9

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Computeincrementsforpriorensemblemembersofobservedvariable.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide10

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Onevariableisobserved.Computeincrementsforpriorensemblemembersofobservedvariable.

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide11

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Usingonlyincrementsguaranteesthatifobserva'onhadnoimpactonobservedvariable,theunobservedvariableisunchanged.Highlydesirable!

33.5

44.5

5

Unob

s.

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide12

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Assumethatallweknowisthepriorjointdistribu'on.Howshouldtheunobservedvariablebeimpacted?1stchoice:leastsquaresEquivalenttolinearregression.Sameasassumingbinormalprior.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide13

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Howshouldtheunobservedvariablebeimpacted?1stchoice:leastsquaresBeginbyfindingleastsquaresfit.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide14

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Next,regresstheobservedvariableincrementsontoincrementsfortheunobservedvariable.Equivalenttofirstfindingimageofincrementinjointspace.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide15

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Next,regresstheobservedvariableincrementsontoincrementsfortheunobservedvariable.Equivalenttofirstfindingimageofincrementinjointspace.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide16

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Next,regresstheobservedvariableincrementsontoincrementsfortheunobservedvariable.Equivalenttofirstfindingimageofincrementinjointspace.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide17

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Next,regresstheobservedvariableincrementsontoincrementsfortheunobservedvariable.Equivalenttofirstfindingimageofincrementinjointspace.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide18

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Next,regresstheobservedvariableincrementsontoincrementsfortheunobservedvariable.Equivalenttofirstfindingimageofincrementinjointspace.

DARTTutorialSec'on4:Slide19

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Regression:Equivalenttofirstfindingimageofincrementinjointspace.Thenprojec'ngfromjointspaceontounobservedpriors.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide20

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Regression:Equivalenttofirstfindingimageofincrementinjointspace.Thenprojec'ngfromjointspaceontounobservedpriors.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide21

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Regression:Equivalenttofirstfindingimageofincrementinjointspace.Thenprojec'ngfromjointspaceontounobservedpriors.3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide22

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Regression:Equivalenttofirstfindingimageofincrementinjointspace.Thenprojec'ngfromjointspaceontounobservedpriors.

DARTTutorialSec'on4:Slide23

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Havejointpriordistribu'onoftwovariables.Regression:Equivalenttofirstfindingimageofincrementinjointspace.Thenprojec'ngfromjointspaceontounobservedpriors.3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Observed Variable

Increments

DARTTutorialSec'on4:Slide24

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Nowhaveanupdated(posterior)ensemblefortheunobservedvariable.FiangGaussiansshowsthatmeanandvariancehavechanged.Otherfeaturesofthepriordistribu'onmayalsohavechanged.3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

−2 0 2 4Obs.

DARTTutorialSec'on4:Slide25

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Nowhaveanupdated(posterior)ensemblefortheunobservedvariable.FiangGaussiansshowsthatmeanandvariancehavechanged.Otherfeaturesofthepriordistribu'onmayalsohavechanged.3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

Prior State Fit

−2 0 2 4Obs.

DARTTutorialSec'on4:Slide26

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

Nowhaveanupdated(posterior)ensemblefortheunobservedvariable.FiangGaussiansshowsthatmeanandvariancehavechanged.Otherfeaturesofthepriordistribu'onmayalsohavechanged.3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

Prior State Fit

Posterior Fit

−2 0 2 4Obs.

DARTTutorialSec'on4:Slide27

Ensemblefilters:Upda'ngaddi'onalpriorstatevariables

CRITICALPOINT:Sinceimpactonunobservedvariableissimplyalinearregression,candothisINDEPENDENTLYforanynumberofunobservedvariables!Couldalsodomanyatonceusingmatrixalgebraasintradi'onalKalmanFilter.

3

3.5

4

4.5

5

Unob

serv

ed S

tate

Var

iabl

e

Prior State Fit

Posterior Fit

−2 0 2 4Obs.Anderson,J.L.,2003:

Alocalleastsquaresframeworkforensemblefiltering.Mon.Wea.Rev.,131,634-642

DARTTutorialSec'on4:Slide28

Mul'variateassimila'onwithDART:

Thebasicregressioncodeistrivial:However,DARTadvancedop'onscanobscurethecode.

Seeassimila/on_code/modules/assimila/on/assim_tools_mod.f90subroutine update_from_obs_inc

Togenerateoutputfromamul'variateLorenz_63experiment(thevalueofcutoffispresumedtobelarge–setinSec'on3):

cd models/lorenz_63/work; ./filter

NowdoMatlabdiagnos'cs(seesec'on1).

•  Doesmul'variatedobeker?•  Besuretorecordtheerrorvaluesforcomparison.•  Canyouiden'fyanyobviousperformancedifferences?

DARTTutorialSec'on4:Slide29

Mul'variateassimila'oninLorenz63:

Whathappensifnotallstatevariablesareobserved?

1.  Tryobservingonlyxandy(ignorezobserva'onsfromabove).Inmodels/lorenz_63/workeditinput.nml

&filter_nml … async = 0, adv_ens_command = "./advance_model.csh”,

obs_sequence_in_name = "obs_seq.out”, obs_sequence_out_name = "obs_seq.final”, …

Execute./filtertoproducenewassimila'on.

Lookattheerrorsta's'csand'meserieswithMatlab.

Recordtheerrorandspreadvaluesandcomparetounivariatecase.

DARTTutorialSec'on4:Slide30

Mul'variateassimila'oninLorenz63:

Whathappensifnotallstatevariablesareobserved?

2.  Tryobservingonlyx(ignorey and zobserva'onsfromabove).Inmodels/lorenz_63/workeditinput.nml

&filter_nml … obs_sequence_in_name = "obs_seq.out.xy” …

Execute./filtertoproducenewassimila'on.

Lookattheerrorsta's'csand'meserieswithMatlab.

Recordtheerrorandspreadvaluesandcomparetounivariatecase.

Whatwouldhappenedifwemadethisintoaunivariateassimila'on?&assim_tools_nml filter_kind = 1 cutoff = 1000000.0

DARTTutorialSec'on4:Slide31

Mul'variateassimila'oninLorenz63:

Whathappensifnotallstatevariablesareobserved?

3.  Tryobservingonlyz(ignorex and yobserva'onsfromabove).Inmodels/lorenz_63/workeditinput.nml

&filter_nml … obs_sequence_in_name = "obs_seq.out.x” …&assim_tools_nml … cutoff = 0.00001

Execute./filtertoproducenewassimila'on;lookattheerrorsta's'csand'meserieswithMatlab.

Recordtheerrorandspreadvaluesandcomparetounivariatecase.Dynamicsforxandyaresymmetric;zcanNOTdis'nguishthem.Howdowewantfiltertohandlethis?Doesitdowhatwewantinthiscase?

DARTTutorialSec'on4:Slide32

1.   FilteringForaOneVariableSystem2.   TheDARTDirectoryTree3.   DARTRun>meControlandDocumenta>on4.   Howshouldobserva>onsofastatevariableimpactanunobservedstatevariable?

Mul>variateassimila>on.5.   ComprehensiveFilteringTheory:Non-Iden>tyObserva>onsandtheJointPhaseSpace6.   OtherUpdatesforAnObservedVariable7.   SomeAddi>onalLow-OrderModels8.   DealingwithSamplingError9.   MoreonDealingwithError;Infla>on10.   RegressionandNonlinearEffects11.   Crea>ngDARTExecutables12.   Adap>veInfla>on13.   HierarchicalGroupFiltersandLocaliza>on14.   QualityControl15.   DARTExperiments:ControlandDesign16.   Diagnos>cOutput17.   Crea>ngObserva>onSequences18.   LostinPhaseSpace:TheChallengeofNotKnowingtheTruth19.   DART-CompliantModelsandMakingModelsCompliant20.   ModelParameterEs>ma>on21.   Observa>onTypesandObservingSystemDesign22.   ParallelAlgorithmImplementa>on23.  Loca'onmoduledesign(notavailable)24.  Fixedlagsmoother(notavailable)25.   Asimple1Dadvec>onmodel:TracerDataAssimila>on

DARTTutorialIndextoSec'ons

DARTTutorialSec'on4:Slide33