DynamicEconomicsQuantitative Methods andApplicationsJero meAdda andRussell CooperTheMIT PressCambridge,MassachusettsLondon,EnglandContents1 Overview 1I Theory2 Theoryof Dynamic Programming 72.1 Overview 72.2 Indirect Utility 72.2.1 Consumers 72.2.2 Firms 82.3 Dynamic Optimization:ACake-EatingExample 92.3.1 DirectAttack 102.3.2 Dynamic Programming Approach 122.4 SomeExtensions oftheCake-Eating Problem 162.4.1 InniteHorizon 162.4.2 TasteShocks 202.4.3 DiscreteChoice 222.5 GeneralFormulation 242.5.1 Nonstochastic Case 242.5.2 Stochastic DynamicProgramming 292.6 Conclusion 313 NumericalAnalysis 333.1 Overview 333.2 Stochastic Cake-EatingProblem 343.2.1 ValueFunctionIterations 343.2.2 Policy Function Iterations 403.2.3 Projection Methods 413.3 Stochastic DiscreteCake-Eating Problem 463.3.1 ValueFunctionIterations 473.4 ExtensionsandConclusion 503.4.1 Larger StateSpaces 503.5 Appendix: AdditionalNumericalTools 523.5.1 InterpolationMethods 523.5.2 NumericalIntegration 553.5.3 HowtoSimulatetheModel 594 Econometrics 614.1 Overview 614.2 SomeIllustrative Examples 614.2.1 CoinFlipping 614.2.2 Supplyand DemandRevisited 744.3 EstimationMethods and AsymptoticProperties 794.3.1 Generalized Methodof Moments 804.3.2 Maximum Likelihood 834.3.3 Simulation-Based Methods 854.4 Conclusion 97II Applications5 Stochastic Growth 1035.1 Overview 1035.2 Nonstochastic GrowthModel 1035.2.1 AnExample 1055.2.2 NumericalAnalysis 1075.3 StochasticGrowthModel 1115.3.1 Environment 1125.3.2 Bellmans Equation 1135.3.3 SolutionMethods 1155.3.4 Decentralization 1205.4 AStochasticGrowth Modelwith EndogenousLaborSupply 1225.4.1 PlannersDynamic ProgrammingProblem 1225.4.2 NumericalAnalysis 1245.5 ConfrontingtheData 1255.5.1 Moments 1265.5.2 GMM 1285.5.3 Indirect Inference 1305.5.4 Maximum LikelihoodEstimation 131viii Contents5.6 SomeExtensions 1325.6.1 TechnologicalComplementarities 1335.6.2 MultipleSectors 1345.6.3 TasteShocks 1365.6.4 Taxes 1365.7 Conclusion 1386 Consumption 1396.1 OverviewandMotivation 1396.2 Two-PeriodProblem 1396.2.1 BasicProblem 1406.2.2 Stochastic Income 1436.2.3 PortfolioChoice 1456.2.4 Borrowing Restrictions 1466.3 InniteHorizon Formulation: TheoryandEmpiricalEvidence 1476.3.1 BellmansEquationfor theInniteHorizonProblem 1476.3.2 Stochastic Income 1486.3.3 Stochastic Returns: PortfolioChoice 1506.3.4 Endogenous Labor Supply 1536.3.5 Borrowing Constraints 1566.3.6 Consumption over theLifeCycle 1606.4 Conclusion 1647 DurableConsumption 1657.1 Motivation 1657.2 Permanent IncomeHypothesisModel ofDurableExpenditures 1667.2.1 Theory 1667.2.2 Estimationofa Quadratic UtilitySpecication 1687.2.3 QuadraticAdjustment Costs 1697.3 NonconvexAdjustment Costs 1717.3.1 GeneralSetting 1727.3.2 IrreversibilityandDurablePurchases 1737.3.3 ADynamic DiscreteChoiceModel 1758 Investment 1878.1 OverviewandMotivation 1878.2 GeneralProblem 188Contents ix8.3 NoAdjustment Costs 1898.4 Convex AdjustmentCosts 1918.4.1 QTheory: Models 1928.4.2 QTheory: Evidence 1938.4.3 EulerEquationEstimation 1988.4.4 BorrowingRestrictions 2018.5 Nonconvex Adjustment: Theory 2028.5.1 Nonconvex Adjustment Costs 2038.5.2 Irreversibility 2088.6 Estimation of a Rich Model of Adjustment Costs 2098.6.1 GeneralModel 2098.6.2 Maximum LikelihoodEstimation 2128.7 Conclusion 2139 Dynamicsof EmploymentAdjustment 2159.1 Motivation 2159.2 GeneralModelof DynamicLaborDemand 2169.3 Quadratic Adjustment Costs 2179.4 RicherModels ofAdjustment 2249.4.1 PiecewiseLinear Adjustment Costs 2249.4.2 Nonconvex Adjustment Costs 2269.4.3 Asymmetries 2289.5 TheGapApproach 2299.5.1 Partial Adjustment Model 2309.5.2 Measuring theTargetandthe Gap 2319.6 Estimation of a Rich Model of Adjustment Costs 2359.7 Conclusion 23810 FutureDevelopments 24110.1 Overviewand Motivation 24110.2 PriceSetting 24110.2.1 OptimizationProblem 24210.2.2 EvidenceonMagazinePrices 24410.2.3 AggregateImplications 24510.3 OptimalInventoryPolicy 24810.3.1 InventoriesandtheProduction-SmoothingModel 24810.3.2 Prices and InventoryAdjustment 25210.4 CapitalandLabor 254x Contents10.5 TechnologicalComplementarities:EquilibriumAnalysis 25510.6 SearchModels 25710.6.1 ASimpleLaborSearchModel 25710.6.2 Estimationofthe Labor SearchModel 25910.6.3 Extensions 26010.7 Conclusion 263Bibliography 265Index 275Contents xi1OverviewInthisbookwestudyarichset of appliedproblemsineconomicsthat emphasize the dynamic aspects of economic decisions. Althoughourultimategoalsaretheapplications,weprovidesomebasictech-niquesbeforetacklingthedetailsof specicdynamicoptimizationproblems. Thiswayweareabletopresent andintegratekeytoolssuchasdynamicprogramming, numerical techniques, andsimula-tionbasedeconometricmethods. Weutilizethesetoolsinavarietyofapplicationsinbothmacroeconomicsandmicroeconomics. Over-all,thisapproachallowsustoestimatestructuralparametersandtoanalyzetheeffectsof economicpolicy.Theapproachwepursuetostudyingeconomicdynamicsisstruc-tural. Asresearcherswehavefrequentlyfoundourselvesinferringunderlyingparametersthat represent tastes, technology, andotherprimitives fromobservations of individual households and rmsaswellasfromeconomicaggregates.Whensuchinferencesaresuc-cessful, we can then test competing hypotheses about economicbehaviorandevaluate theeffects ofpolicy experiments.Toappreciatethebenetsofthisapproach, considerthefollowingpolicyexperiment. Inrecent years anumber of Europeangovern-ments haveinstitutedpolicies of subsidizingthescrappingof oldcarsandthepurchaseofnewcars. Whataretheexpectedeffectsofthese policiesonthecar industryandongovernmentrevenues?At some level this question seems easy if a researcher knows thedemandfunctionforcars. Butofcoursethatdemandfunctionis, atbest, elusive. Furtherthedemandfunctionestimatedinonepolicyregimeisunlikelytobeveryinformativeforanovel policyexperi-ment,suchasthisexampleof car scrapping subsidies.An alternative approach is to build and estimate a model ofhousehold dynamic choice of car ownership. Once the parameters ofthis model are estimated, thenvarious policyexperiments canbeevaluated.1This approachseems considerablymore difcult thanjustestimatingademandfunction, andofcoursethisisthecase. Itrequires the specicationandsolutionof a dynamic optimizationproblem and then the estimation of the parameters. But, as we argueinthisbook, thismethodology is feasibleandyieldsexcitingresults.Theintegrationof dynamicoptimizationwithparameter estima-tionisat theheart of ourapproach. Wedevelopthisideabyorga-nizing thebookin twoparts.Part I provides areviewof the formal theoryof dynamic opti-mization. This is atool usedinmanyareas of economics, includ-ingmacroeconomics, industrial organization, labor economics, andinternational economics. Asinprevious contributions tothestudyof dynamicoptimization, suchasbySargent (1987) andbyStokeyandLucas(1989), ourpresentationstartswiththeformal theoryofdynamicprogramming. Becauseof thelargenumber of other con-tributions in this area, our presentation in chapter 2 relies on existingtheoremsontheexistenceofsolutionstoavarietyofdynamicpro-gramming problems.Inchapter3wepresentthenumerical toolsnecessarytoconducta structural estimation of the theoretical dynamic models. Thesenumerical tools serve bothtocomplement the theoryinteachingstudentsaboutdynamicprogrammingandtoenablearesearchertoevaluatethequantitativeimplicationsof thetheory. Inour experi-ence theprocess of writingcomputer code tosolvedynamic pro-grammingproblemshasprovedtobeauseful devicefor teachingbasic concepts ofthis approach.Theeconometrictechniquesofchapter4providethelinkbetweenthedynamicprogrammingproblemanddata. Theemphasis is onthemappingfromparameters of thedynamicprogrammingprob-lemtoobservations. Forexample, avectorofparametersisusedtonumericallysolveadynamicprogrammingproblemthatisusedtosimulatemoments. Anoptimizationroutinethenselectsavectorofparameterstobringthesesimulatedmomentsclosetothemomentsobservedin thedata.Part II is devotedto the application of dynamic programmingtospecicareasof economicssuchasthestudyof businesscycles,consumption, andinvestment behavior. The presentation of each1. This exercise is described in some detail in the chapter on consumer durables in thisbook.2 Chapter1applicationinchapters 5 through10 contains four elements: pre-sentationof theoptimizationproblemas adynamicprogrammingproblem, characterization of the optimal policy functions, estimationofthe parameters, andpolicy evaluation using thismodel.While the applications might be characterized as macroeconomics,themethodologyisvaluableinotherareasofeconomicresearch, intermsofboththetopicsandthetechniques. Theseapplicationsuti-lizematerial frommanyotherpartsofeconomics. Forexample, theanalysis of thestochasticgrowthmodel includes taxationandourdiscussion of factor adjustment at the plant level is certainly relevantto researchers inlabor andindustrial organization. Moreover weenvisionthese techniques tobe useful inanyproblemwhere theresearcherisapplyingdynamicoptimizationtothedata. Thechap-ters contain referencesto otherapplicationsof these techniques.What is newabout our presentationis theuseof anintegratedapproach to the empirical implementation of dynamic optimiza-tion models. Previous texts have provided a mathematical basisfordynamicprogramming,butthosepresentationsgenerallydonotcontainanyquantitativeapplications.Other textspresenttheunder-lying econometric theory but generally without specic applications.Ourapproachdoesboth, andthusaimstolinktheoryandapplica-tion asillustrated inthechapters ofpart II.Our motivation for writing this book should be clear. Fromthe perspective of understanding dynamic programming, explicitempirical applications complement the underlyingtheoryof opti-mization. Fromtheperspectiveofappliedmacroeconomics, explicitdynamic optimizationproblems, posedas dynamic programmingproblems, provide needed structure for estimation and policyevaluation.Since the book is intended to teach empirical applications ofdynamic programming problems, we have created a Web site for thepresentationof computer codes(MATLABandGAUSS) aswell asdatasetsuseful fortheapplications. Thismaterial shouldappeal toreaderswishingtosupplement thepresentationinpart II, andwehopethatWebsitewillbecomeaforumforfurtherdevelopmentofcodes.Our writingthis bookhas benetedfromjoint workwithJoaoEjarque, John Haltiwanger, Alok Johri, and Jonathan Willis. Wethanktheseco-authorsfortheirgeneroussharingofideasandcom-putercodeaswellastheircommentsonthenaldraft.ThanksalsoOverview 3gotoVictor Aguirregabiria, YanBai, JoyceCooper, DeanCorbae,Zvi Eckstein, SimonGilchrist, HangKang, PeterKlenow, SamKor-tum, ValerieLechene, NicolaPavoni, AldoRustichini, andMarcosVeraforcommentsonvariouspartsofthebook.Wealsoappreciatethe comments of outside reviewers and the editorial staff at The MITPress. Finally, we are grateful toour manymasters anddoctoralstudentsat Tel AvivUniversity, Universityof Texasat Austin, theIDEI at the Universite de Toulouse, the NAKEPhDprograminHolland, theUniversityof Haifa, theUniversityof Minnesota, andUniversityCollegeLondonfortheirnumerouscommentsandsug-gestionsduringthepreparation of this book.4 Chapter1ITheory2TheoryofDynamicProgramming2.1 OverviewThemathematical theoryof dynamic programmingas ameans ofsolvingdynamicoptimizationproblemsdatestotheearlycontribu-tions of Bellman(1957) andBertsekas (1976). For economists, thecontributions of Sargent (1987) and Stokey andLucas(1989) provideavaluablebridgetothisliterature.2.2 IndirectUtilityIntuitively, the approachof dynamic programmingcanbe under-stoodbyrecallingthethemeofindirectutilityfrombasicstaticcon-sumer theoryor areducedformprot functiongeneratedbytheoptimizationof arm. Thesereducedformrepresentationsof pay-offssummarizeinformationabouttheoptimizedvalueofthechoiceproblemsfacedbyhouseholdsandrms. Aswewillsee, thetheoryofdynamicprogramming takesthisinsight to adynamic context.2.2.1 ConsumersConsumer choicetheoryfocusesonhouseholds that solveVI; p maxcucsubject to pc I;where c is a vector of consumption goods, p is a vector of prices andIisincome.1Therst-order condition isgiven by1. Assumethat thereareJ commoditiesinthiseconomy. Thispresentationassumesthat youunderstandthe conditions under whichthis optimizationproblemhas asolution and whenthat solution can be characterized byrst-order conditions.ujcpj l for j 1; 2; . . . ; J;wherelis themultiplier onthebudget constraint andujc isthemarginalutilityfrom goodj.Here VI; p is an indirect utility function. It is the maximized levelof utilityfromthecurrent state I; p. Someoneinthisstatecanbepredictedtoattainthislevel of utility. Onedoesnot needtoknowwhatthatpersonwilldowithhisincome;itisenoughtoknowthathewill actoptimally. Thisisverypowerful logicandunderliestheideabehind thedynamicprogrammingmodelsstudied below.Toillustrate, what happens if wegivetheconsumer abit moreincome?WelfaregoesupbyVII; p > 0. Cantheresearcherpredictwhat will happenwithalittle more income? Not reallysince theoptimizing consumer isindifferent with respect to howthisisspent:ujcpj VII; p for all j:It is inthis sensethat theindirect utilityfunctionsummarizes thevalue of the households optimization problemand allows us todetermine the marginal value of income without knowing moreabout consumptionfunctions.Isthisall weneedtoknowabout householdbehavior?No, thistheoryisstatic. Itignoressavings, spendingondurablegoods, anduncertaintyoverthefuture. Theseareall important componentsinthe household optimization problem. Wewillreturn to these in laterchapters on the dynamic behavior of households. The point here wassimplytorecall akeyobject fromoptimizationtheory: theindirectutilityfunction.2.2.2 FirmsSupposethat armmust choosehowmanyworkers tohireat awageofwgivenitsstockofcapitalkandproductpricep. Thusthermmust solvePw; p; k maxlpf l; k wl:Alabordemandfunctionresultsthat dependson w; p; k. AswithVI; p, Pw; p; k summarizes the value of the rmgiven factor8 Chapter2prices,theproductpricep,andthestockofcapitalk.Boththeexi-bleandxedfactorscanbe vectors.Think of Pw; p; k as anindirect prot function. It completelysummarizesthevalueoftheoptimizationproblemofthermgivenw; p; k.Aswiththehouseholdsproblem,givenPw; p; k,wecandirectlycomputethemarginal valueof allowingthermsomeadditionalcapital asPkw; p; k pfkl; k without knowinghowthermwilladjust itslaborinput inresponsetotheadditionalcapital.But, isthisall thereistoknowaboutthermsbehavior?Surelynot, for wehavenot speciedwhere k comes from. Sothermsproblemis essentially dynamic, though the demandfor some ofitsinputscanbetakenasastaticoptimizationproblem. Theseareimportant themes in the theory of factor demand, and we will returntothem inour rmapplications.2.3 DynamicOptimization:ACake-EatingExampleHerewewill lookataverysimpledynamicoptimizationproblem.Webeginwithanitehorizonandthendiscussextensionstotheinnitehorizon.2Supposethat youarepresentedwithacakeof sizeW1. At eachpoint of time, t 1; 2; 3; . . . ; T, you can eat some of the cake but mustsavetherest. Let ctbeyour consumptioninperiodt, andlet uctrepresenttheowofutilityfromthisconsumption.Theutilityfunc-tion is not indexed by time: preferences are stationary. We canassume that u is real valued, differentiable, strictly increasing, andstrictlyconcave.Furtherwecanassumelimc!0u0c !y.Wecouldrepresent your lifetimeutility byXTt1bt1uct;where0 ab a1 andbiscalledthediscountfactor.Fornow, weassumethat thecakedoesnot depreciate(spoil) orgrow.Hencetheevolutionof thecakeovertimeisgovernedbyWt1 Wt ct2:12. Foraverycompletetreatmentofthenitehorizonproblemwithuncertainty, seeBertsekas (1976).Theory ofDynamic Programming 9for t 1; 2; . . . ; T. Howwouldyoundthe optimal pathof con-sumption, fctgT1 ?32.3.1 DirectAttackOne approach is to solve the constrained optimization problemdirectly. ThisiscalledthesequenceproblembyStokeyandLucas(1989). Consider theproblem ofmaxfctgT1 ; fWtgT12XTt1bt1uct 2:2subject to the transition equation (2.1), which holds for t 1; 2; 3; . . . ;T. Alsotherearenonnegativityconstraintsonconsumingthecakegiven byctb0 andWtb0. For this problem,W1is given.Alternatively, the owconstraints imposed by (2.1) for each tcouldbecombined,yieldingXTt1ct WT1 W1: 2:3The nonnegativityconstraints are simpler: ctb0for t 1; 2; . . . ; TandWT1b0. Fornow, wewill workwiththesingleresourcecon-straint. Thisisawell-behavedproblemastheobjectiveisconcaveandcontinuousandtheconstraintsetiscompact.Sothereisasolu-tion tothisproblem.4Lettinglbethemultiplieron(2.3), therst-orderconditionsaregiven bybt1u0ct l for t 1; 2; . . . ; Tandl f;wherefisthemultiplieronthenonnegativityconstraint onWT1.The nonnegativity constraints on ctb0 are ignored, as we canassume that the marginal utilityof consumptionbecomes inniteas consumption approacheszerowithinany period.3. Throughout, thenotation fxtgT1is usedtodenethesequence x1; x2; . . . ; xT forsome variablex.4. ThiscomesfromtheWeierstrasstheorem. SeeBertsekas(1976, app. B) orStokeyand Lucas(1989, ch.3) for a discussion.10 Chapter2Combiningtheequations, weobtainanexpressionthatlinkscon-sumption acrossanytwoperiods:u0ct bu0ct1: 2:4Thisisanecessaryconditionofoptimalityforanyt:ifitisviolated,theagent candobetterbyadjustingctandct1. Frequently(2.4) isreferredtoasaEulerequation.Tounderstandthiscondition, supposethat youhaveaproposed(candidate) solution for this problemgiven byfct gT1 , fWt gT12.EssentiallytheEulerequationsaysthat themarginal utilitycost ofreducingconsumptionbye inperiodt equals themarginal utilitygainfromconsumingtheextraeof cakeinthenext period, whichisdiscountedbyb. IftheEulerequationholds, thenitisimpossibletoincreaseutilitybymovingconsumptionacrossadjacent periodsgivenacandidatesolution.Itshouldbeclearthoughthatthisconditionmaynotbesufcient:it does not cover deviations that last more thanone period. Forexample, couldutilitybeincreasedbyreducingconsumptionbyeinperiodt savingthecake for twoperiods andthenincreasingconsumptioninperiodt 2?Clearly, thisisnotcoveredbyasingleEuler equation. However, by combining the Euler equation that holdacrossperiodt andt 1withthatwhichholdsforperiodst 1andt 2, we can see that such a deviation will not increase utility. This issimply becausethe combinationof Euler equations impliesthatu0ct b2u0ct2so that the two-period deviation from the candidate solution will notincreaseutility.Aslongastheproblemisnite, thefact that theEulerequationholdsacrossall adjacent periods impliesthat anynitedeviationsfromacandidatesolutionthatsatisestheEulerequationswill notincreaseutility.Isthisenough?Not quite. Imagineacandidatesolutionthat sat-isesalloftheEulerequationsbuthasthepropertythatWT> cTsothatthereiscakeleftover.Thisisclearlyaninefcientplan:satisfy-ingtheEulerequationsisnecessarybut not sufcient. Theoptimalsolution will satisfy the Euler equationfor eachperioduntil theagent consumestheentirecake.Formally, thisinvolvesshowingthatthenonnegativityconstraintonWT1mustbind.Infact,thisconstraintisbindinginthesolutionTheory ofDynamic Programming 11above: l f > 0. This nonnegativity constraint serves two importantpurposes. First, inthe absence of a constraint that WT1b0, theagentwouldclearlywanttosetWT1 y. Thisisclearlynotfea-sible. Second, thefact that theconstraint isbindingintheoptimalsolution guaranteesthat cakedoesnotremain after periodT.In effect the problemis pinned down by an initial condition(W1is given) andbya terminal condition(WT1 0). The set of(T 1) Euler equations and(2.3) thendetermine the time pathofconsumption.Let thesolutiontothisproblembedenotedbyVTW1, whereTisthehorizonoftheproblemandW1istheinitial sizeof thecake.VTW1 represents the maximal utility ow from aT-period problemgivenasizeW1cake. Fromnowon, wecall thisavaluefunction.This is completely analogous to the indirect utility functions ex-pressedfor thehousehold and therm.As in those problems, a slight increase in the size of the cake leadstoanincreaseinlifetimeutilityequal tothemarginal utilityinanyperiod. Thatis,V0TW1 l bt1u0ct; t 1; 2; . . . ; T:It doesnt matter whentheextracakeis eatengiventhat thecon-sumer is acting optimally. This is analogous to the point raisedaboveabouttheeffectonutilityofanincreaseinincomeinthecon-sumer choiceproblem with multiplegoods.2.3.2 DynamicProgrammingApproachSupposethat wechangethe problemslightly: weaddaperiod0andgiveaninitial cakeof sizeW0. Oneapproachtodeterminingtheoptimalsolutionofthisaugmentedproblemistogobacktothesequenceproblemandresolveitusingthislongerhorizonandnewconstraint. But, havingdoneall ofthehardworkwiththeTperiodproblem,it wouldbenice not to haveto doit again.FiniteHorizonProblemThe dynamic programming approachprovides a means of doingso. Itessentiallyconvertsa(arbitrary)T-periodproblemintoatwo-periodproblemwiththeappropriaterewritingoftheobjectivefunc-tion. This wayit uses thevaluefunctionobtainedfromsolvingashorter horizon problem.12 Chapter2Byaddingaperiod0toouroriginalproblem,wecantakeadvan-tageof theinformationprovidedinVTW1, thesolutionof theT-period problem given W1from (2.2). Given W0, consider the problemofmaxc0uc0 bVTW1; 2:5whereW1 W0 c0; W0given:In this formulation the choice of consumption in period 0 determinesthesizeof thecakethat will beavailablestartinginperiod1, W1.Now,insteadofchoosingasequenceofconsumptionlevels,wejustndc0. Oncec0andthusW1aredetermined, thevalueoftheprob-lemfromthen on is given by VTW1. This function completelysummarizes optimal behavior fromperiod1onward. For thepur-poses of the dynamic programming problem, it does not matter howthecakewillbeconsumedaftertheinitialperiod. Allthatisimpor-tant isthat theagent will beactingoptimallyandthusgeneratingutilitygivenbyVTW1. Thisistheprincipleofoptimality, duetoRichard Bellman, at work. Withthis knowledge, anoptimal decisioncanbemaderegardingconsumptioninperiod0.Notethat therst-order condition(assumingthat VTW1isdif-ferentiable) isgiven byu0c0 bV0TW1so that the marginal gain fromreducing consumption a little inperiod0issummarizedbythederivativeofthevaluefunction. Asnoted intheearlierdiscussionof theT-periodsequenceproblem,V0TW1 u0c1 btu0ct1fort 1; 2; . . . ; T 1. Usingthesetwo conditionstogether yieldsu0ct bu0ct1fort 0; 1; 2; . . . ; T 1, a familiarnecessaryconditionforanoptimalsolution.SincetheEulerconditionsfortheotherperiodsunderliethecre-ation of the value function, one might suspect that the solution to theT 1problemusingthisdynamicprogrammingapproachisidenti-Theory ofDynamic Programming 13cal tothat of thesequenceapproach.5This is clearlytruefor thisproblem: theset ofrst-order conditionsfor thetwoproblemsareidentical, andthus, giventhestrict concavityof theucfunctions,thesolutionswill beidenticalas well.Theapparenteaseofthisapproach, however, maybemisleading.Wewereabletomaketheproblemlooksimplebypretendingthatwe actually know VTW1. Of course, the way we could solve for thisis byeither tacklingthe sequence problemdirectlyor buildingitrecursively, startingfrom aninitialsingle-periodproblem.On this recursive approach, we could start with the single-period problem implying V1W1. We would then solve (2.5) to buildV2W1. Giventhis function, we couldmove toa solutionof theT 3problemandproceediteratively, using(2.5) tobuildVTW1foranyT.ExampleWe illustrate the construction of the value function in a specicexample. Assumeuc lnc. Supposethat T 1. ThenV1W1 lnW1.ForT 2,therst-order condition from(2.2) is1c1bc2;andtheresourceconstraint isW1 c1 c2:Working with these twoconditions,wehavec1 W11 band c2 bW11 b:Fromthis,wecansolvefor thevalueof thetwo-periodproblem:V2W1 lnc1 b lnc2 A2 B2lnW1; 2:6whereA2andB2areconstantsassociatedwiththetwo-periodprob-lem. Theseconstantsaregiven byA2 ln11 b b lnb1 b ; B2 1 b:5. By the sequence approach, we mean solving the problem using the direct approachoutlined inthe previoussection.14 Chapter2Importantly, (2.6)doesnotincludethemaxoperatoraswearesub-stitutingtheoptimaldecisionsintheconstructionofthevaluefunc-tion,V2W1.Usingthisfunction, theT 3 problemcanthenbewrittenasV3W1 maxW2lnW1 W2 bV2W2;wherethechoicevariableisthestateinthesubsequentperiod. Therst-order condition is1c1 bV02W2:Using(2.6)evaluatedatacakeofsizeW2, wecansolveforV02W2implying:1c1 bB2W2bc2:Here c2the consumptionlevel inthe secondperiodof the three-periodproblemandthusisthesameasthelevelofconsumptionintherst periodof thetwo-periodproblem. Further weknowfromthetwo-period problemthat1c2bc3:Thisplustheresourceconstraintallowsustoconstructthesolutionofthe three-periodproblem:c1 W11 b b2 ; c2 bW11 b b2 ; c3 b2W11 b b2 :SubstitutingintoV3W1 yieldsV3W1 A3 B3lnW1;whereA3 ln11 b b2 b lnb1 b b2 b2lnb21 b b2 ;B3 1 b b2:Thissolution canbe veriedfrom adirectattack on thethree-periodproblemusing(2.2) and(2.3).Theory ofDynamic Programming 152.4 SomeExtensionsoftheCake-EatingProblemHerewegobeyondtheT-periodproblemtoillustratesomewaystousethedynamicprogrammingframework.Thisisintendedasanoverview, andthedetailsoftheassertions, andsoforth,willbepro-videdbelow.2.4.1 InniteHorizonBasicStructureSupposethat forthecake-eatingproblem, weallowthehorizontogotoinnity. Asbefore, onecanconsidersolvingtheinnitehori-zon sequenceproblem given bymaxfctgy1; fWtgy2Xyt1btuctalongwiththetransitionequationofWt1 Wt ctfor t 1; 2; . . . :Inspecifying thisas adynamic programming problem, we writeVW maxc A 0; Wuc bVW c for all W:Hereucisagaintheutilityfromconsumingcunitsinthecurrentperiod. VW is thevalueof theinnitehorizonproblemstartingwithacakeofsizeW.Sointhegivenperiod,theagentchoosescur-rent consumption and thus reduces the size of the cake to W0 W c,as in the transition equation. We use variables with primes to denotefuturevalues. Thevalueof startingthenext periodwithacakeofthat size is then given by VW c, which is discounted at rate b < 1.For this problem, the statevariableis the size of the cake (W)given at the start of any period. The state completely summarizes allinformationfromthe past that is neededfor the forward-lookingoptimizationproblem. The control variableis the variable that isbeing chosen. In this case it is the level of consumption in the currentperiodc. Notethat cliesinacompact set. Thedependenceof thestate tomorrowon thestatetodayandthecontroltoday, given byW0 W c;iscalled thetransitionequation.16 Chapter2Alternatively, we can specify the problemso that instead ofchoosingtodaysconsumption wechoosetomorrowsstate:VW maxW0A 0; WuW W0 bVW0 for all W: 2:7Eitherspecicationyieldsthesameresult. Butchoosingtomorrowsstate often makes the algebra a bit easier, so we will work with (2.7).Thisexpressionisknownasafunctionalequation,anditisoftencalledaBellmanequationafter RichardBellman, oneof theorigi-nators of dynamic programming. Note that the unknown in theBellmanequationis thevaluefunctionitself: theideais tondafunction VW that satises this condition for all W. Unlike the nitehorizonproblem, thereis noterminal periodtousetoderive thevaluefunction. Ineffect, thexedpoint restrictionof havingVWonbothsidesof (2.7) will provideuswithameansof solvingthefunctionalequation.Note too that time itself does not enter into Bellmans equation: wecanexpress all relations without anindicationof time. This is theessenceof stationarity.6Infact wewill ultimatelyusethestation-arityof the problemtomake arguments about the existence of avaluefunctionsatisfyingthe functionalequation.Anal veryimportant propertyof thisproblemisthat all infor-mationabout thepast that bearsoncurrentandfuturedecisionsissummarizedbyW, thesizeof thecakeat thestart of theperiod.Whether the cake is of this size because we initially have a large cakeandcaneatalotofitorasmallcakeandarefrugaleatersisnot rel-evant. All that mattersisthat wehaveacakeof agivensize. Thispropertypartlyreectsthefactthatthepreferencesoftheagentdonot dependonpast consumption. If this werethe case, we couldamendtheproblem to allowthis possibility.Thenext part of this chapter addresses thequestionof whetherthereexistsavaluefunctionthatsatises(2.7). Fornowweassumethat asolution existssothat wecan exploreitsproperties.Therst-orderconditionfortheoptimizationproblemin(2.7)canbewritten asu0c bV0W0:6. As you may already know, stationarity is vital in econometrics as well. Thusmakingassumptions of stationarityineconomic theoryhaveanatural counterpartinempirical studies. Insomecaseswewill havetomodifyoptimizationproblemstoensure stationarity.Theory ofDynamic Programming 17Thismaylooksimple, butwhatisthederivativeofthevaluefunc-tion?It is particularlyhardtoanswer this, sincewedonot knowVW. However, wecanusethefactthatVWsatises(2.7)forallWto calculate V0. Assuming that this value function is differentiable,wehaveV0W u0c;a result we have seen before. Since this holds for all W, it will hold inthefollowing period,yieldingV0W0 u0c0:Substitution leadsto thefamilar Euler equation:u0c bu0c0:Thesolutiontothecake-eatingproblemwill satisfythis necessaryconditionfor allW.The link from the level of consumption and next periods cake (thecontrolsfromthedifferentformulations)tothesizeofthecake(thestate) isgivenby thepolicyfunction:c fW; W0 jW 1W fW:SubstitutingthesevaluesintotheEulerequationreducestheprob-lemtothesepolicyfunctions alone:u0fW bu0fW fW for all W:The policy functions above are important in applied research,for theyprovidethemappingfromthestatetoactions. Whenele-mentsofthestateaswell astheactionareobservable, thesepolicyfunctions will provide the means for estimating the underlyingparameters.AnExampleIngeneral, it isnot actuallypossibletondclosedformsolutionsfor the value function and the resulting policy functions. So we try tocharacterizecertainpropertiesof thesolution, andfor somecases,wesolvetheseproblemsnumerically.Nevertheless, asindicatedbyouranalysisofnitehorizonprob-lems, therearesomespecicationsoftheutilityfunctionthatallowustondaclosedformsolutiontothevaluefunction. Suppose, as18 Chapter2above, that uc lnc. Fromtheresults of theT-periodproblem,we might conjecture that the solution to the functional equation takestheformofVW A B lnW for all W:Bythis expressionwehavereducedthedimensionalityof theun-known function VW to two parameters, Aand B. But can wendvaluesfor AandBsuchthat VWwill satisfythefunctionalequation?Let ussupposethat wecan. For thesetwovaluesthefunctionalequationbecomesA B lnW maxW0lnW W0 bA B lnW0 for all W:2:8After somealgebra, therst-order conditionbecomesW0 jW bB1 bBW:Usingthisin (2.8) resultsinA B lnW lnW1 bB b A B lnbBW1 bB for all W:CollectingthetermsthatmultiplylnWandusingtherequirementthat thefunctionalequation holdsforallW, we ndthatB 11 bisrequiredforasolution.Afterthis,theexpressioncanalsobeusedtosolveforA. Thuswehaveveriedthatourguessisasolutiontothe functional equation. We knowthat because we cansolve forA; B suchthat thefunctional equationholds for all Wusingtheoptimal consumptionand savingsdecision rules.Withthissolution, weknowthatc W1 b; W0 bW:Thistellsusthattheoptimal policyistosaveaconstantfractionofthecakeand eattheremaining fraction.The solutiontoBcanbe estimatedfromthe solutiontothe T-periodhorizon problems whereTheory ofDynamic Programming 19BT XTt1bt1:Clearly, B limT!yBT. Wewill beexploitingtheideaofusingthevalue function to solve the innite horizon problem as it is related tothelimit of thenitesolutions inmuchof our numericalanalysis.Belowaresomeexercises that providesomefurther elements tothisbasicstructure.Bothbeginwithnitehorizonformulationsandthen progressto theinnitehorizon problems.exercise2.1 Utilityinperiodt is givenbyuct; ct1. Solve aT-period problemusing these preferences. Interpret the rst-orderconditions. HowwouldyouformulatetheBellmanequationfortheinnitehorizonversion ofthis problem?exercise 2.2 Thetransition equation is modiedsothatWt1 rWt ct;where r > 0 represents a return fromholding cake inventories.SolvetheT-periodproblemwiththisstoragetechnology. Interpretthe rst-order conditions. Howwouldyouformulate the Bellmanequationfortheinnitehorizonversionof thisproblem?Doesthesizeofr matter inthis discussion?Explain.2.4.2 TasteShocksAconvenient featureof thedynamicprogrammingproblemistheeasewithwhichuncertaintycanbeintroduced.7Forthecake-eatingproblem, the natural source of uncertainty has to do with the agentsappetite. Inothersettingswewill focusonothersourcesof uncer-taintyhavingtodowiththe productivityof labor or the endow-ments ofhouseholds.Toallowforvariationsofappetite, supposethatutilityovercon-sumptionisgivenbyeuc;where e is a randomvariable whose properties we will describebelow. Thefunctionucisagainassumedtobestrictlyincreasing7. Tobe careful, here we are addingshocks that take values ina nite andthuscountable set. See the discussion in Bertsekas (1976, sec. 2.1) for an introduction to thecomplexities ofthe problem with more generalstatements ofuncertainty.20 Chapter2andstrictlyconcave. Otherwise, the problemis the original cake-eatingproblem with an initialcakeofsizeW.Inproblems withstochastic elements, it is critical tobe preciseabout the timingof events. Does the optimizingagent knowthecurrent shockswhenmakingadecision?For thisanalysis, assumethattheagentknowsthevalueofthetasteshockwhenmakingcur-rent decisions but does not knowthefuturevalues of this shock.Thus the agent must use expectations of future values of e whendeciding how much cake to eat today: it may be optimal to consumeless today (save more) in anticipation of a high realization ofe inthefuture.For simplicity, assume that the taste shock takes on only twovalues: e A feh; elgwitheh> el> 0. Further wecanassumethat thetasteshockfollowsarst-orderMarkovprocess,8whichmeansthattheprobabilitythat aparticular realizationof e occurs inthecur-rent perioddepends onlythe value of e attainedinthe previousperiod.9Fornotation, letpijdenotetheprobabilitythatthevalueofegoesfromstateiinthecurrentperiodtostate jinthenextperiod.For example,plhisdened fromplh1Probe0 eh j e el;where e0refers to the future value of e. Clearly, pih pil 1 fori h; l. Let Pbe a 2 2 matrix with a typical element pijthatsummarizestheinformationabouttheprobabilityofmovingacrossstates.Thismatrixislogicallycalled atransitionmatrix.Withthisnotationandstructure, wecanturnagaintothecake-eatingproblem. Weneedtocarefullydenethestateofthesystemfor the optimizing agent. In the nonstochastic problem, the state wassimplythesizeof thecake. This providedall theinformationtheagent neededtomakeachoice. Whentasteshocksareintroduced,theagent needstotakethisfactor intoaccount aswell. Weknowthatthetasteshocksprovideinformationaboutcurrentpayoffsand,throughthePmatrix, areinformativeaboutthefuturevalueofthetasteshockaswell.108. FormoredetailsonMarkovchains, wereferthereadertoLjungqvistandSargent(2000).9. Theevolutioncanalsodependonthecontrolofthepreviousperiod.Notetoothatbyappropriaterewritingofthestatespace,richerspecicationsofuncertaintycanbeencompassed.10. Thisisapointthatwereturntobelowinourdiscussionofthecapitalaccumula-tionproblem.Theory ofDynamic Programming 21Formally theBellmanequationiswrittenVW; e maxW0euW W0 bEe0j eVW0; e0 for all W; e;whereW0 W c as before. Note thatthe conditional expectation isdenotedherebyEe0j eVW0; e0which, givenP, issomethingwecancompute.11Therst-order conditionfor thisproblem is given byeu0W W0 bEe0j eV1W0; e0 for all W; e:Using the functional equation to solve for the marginal value of cake,wend thateu0W W0 bEe0j ee0u0W0 W00: 2:9This, of course, isthestochastic Euler equationfor thisproblem.Theoptimalpolicyfunction is given byW0 jW; e:TheEuler equationcanberewritteninthese termsaseu0W jW; e bEe0j ee0u0jW; e jjW; e; e0:Thepropertiesofthepolicyfunctioncanthenbededucedfromthiscondition. Clearly, bothe0andc0dependontherealizedvalueof esothattheexpectationontherightsideof(2.9)cannotbesplitintotwo separatepieces.2.4.3 DiscreteChoiceToillustratetheexibilityof thedynamicprogrammingapproach,we buildonthis stochastic problem. Suppose that the cake mustbeeateninoneperiod.Perhapsweshouldthinkofthisasthewine-drinkingproblem, recognizingthat once agoodbottle of wine isopened, itmustbeconsumed. Furtherwecanmodifythetransitionequationtoallowthecaketogrow(depreciate) at rater.The cake consumption example becomes then a dynamic, sto-chasticdiscretechoiceproblem. Thisispartofafamilyofproblemscalledoptimalstoppingproblems.12Thecommonelementinall of11. Throughoutwe denote the conditionalexpectationofe0givene asEe0 j e.12. EcksteinandWolpin(1989) provideanextensivediscussionsof theformulationand estimation ofthese problemsinthe contextof labor applications.22 Chapter2these problems is the emphasis on the timing of a single event: whento eat the cake, when to take a job, when to stop school, when to stoprevisingachapter, andsoon. Infact, formanyof theseproblems,thesechoicesarenotonceinalifetimeevents,sowewillbelookingat problemsevenricher thanthoseof theoptimalstoppingvariety.Let VEW; e andVNW; e be the values of eatingsize Wcakenow(E)andwaiting(N), respectively, giventhecurrenttasteshocke A feh; elg. ThenVEW; e euWandVNW bEe0j eVrW; e0;whereVW; e maxVEW; e; VNW; e for all W; e:Tounderstandthisbetter, thetermeuWisthedirect utilityowfrom eating the cake. Once the cake is eaten, the problem has ended.SoVEW; e isjustaone-periodreturn.Iftheagentwaits,thenthereisnocakeconsumptioninthecurrentperiod,andinthenextperiodthecakeisofsize rW. Astastesarestochastic, theagentchoosingto wait must take expectations of the future taste shock, e0. The agenthasanoptioninthenextperiodofeatingthecakeorwaitingsomemore. Hencethevalueof havingthecakeinanystateisgivenbyVW; e, whichis the value attainedbymaximizingover the twooptions of eating or waiting. The cost of delaying the choice isdeterminedbythe discount factor b while the gains todelayareassociatedwiththegrowthofthecake, parameterizedbyr. Furthertherealizedvalueofe willsurelyinuencetherelativevalueofcon-suming thecake immediately.If r a1, thenthecakedoesntgrow. Inthiscasethereisnogainfromdelaywhene eh. If theagent delays, thenutilityinthenextperiodwillhavetobelowerduetodiscounting, andwithprobabil-ity phl, the taste shock will switch from low to high. So waiting to eatthecakeinthefuturewillnotbedesirable.HenceVW; eh VEW; eh ehuW for all W:Inthelowestate, mattersaremorecomplex. If bandraresuf-ciently close to 1, then there is not a large cost to delay. Further, if plh isTheory ofDynamic Programming 23sufciently close to 1, then it is likely that tastes will switch from lowtohigh. Thusit willbe optimalnot to eatthe cake instate W; el.13Herearesomeadditionalexercises.exercise2.3 Suppose that r 1. For a givenb, showthat thereexistsacriticallevelofplh, denotedbyplhsuchthatifplh> plh, thentheoptimal solutionisfortheagent towait whene elandtoeatthecakewhenehisrealized.exercise2.4 Whenr > 1, the problemis more difcult. Supposethattherearenovariationsintastes: eh el 1. Inthiscasethereisatrade-offbetweenthevalueofwaiting(asthecakegrows)andthecost of delay fromdiscounting.Suppose that r > 1anduc c1g=1 g. What is the solutiontotheoptimal stoppingproblemwhenbr1g< 1?What happensifbr1g> 1? What happenswhenuncertainty isadded?2.5 GeneralFormulationBuildingontheintuitiongainedfromthecake-eatingproblem, wenowconsideramoreformalabstracttreatmentofthedynamicpro-grammingapproach.14We beginwitha presentationof the non-stochastic problem and thenadduncertainty to theformulation.2.5.1 NonstochasticCaseConsidertheinnitehorizonoptimizationproblemofanagentwithapayoff functionforperiodt givenby~ ssst; ct.Therstargumentofthepayofffunctionistermedthestatevector st. Asnotedabove,this represents aset of variables that inuences theagents returnwithinthe period, but byassumption, these variables are outsideof the agents control within period t. The state variables evolve overtimeinamannerthatmaybeinuencedbythecontrolvector ct,thesecondargument ofthepayoff function. Theconnectionbetweenthestatevariablesover timeisgivenby thetransitionequation:13. In the following chapter on the numerical approach to dynamic programming, westudythiscase inconsiderable detail.14. Thissectionisintended to be self-contained and thus repeats some ofthe materialfromtheearlier examples. Our presentationis bydesignnot as formal as saythatprovided in Bertsekas (1976) or Stokey and Lucas (1989). The reader interested in moremathematicalrigor is urged toreview those texts and their many references.24 Chapter2st1 tst; ct:So, giventhecurrent stateandthecurrent control, thestatevectorfor thesubsequent periodisdetermined.Note that the state vector has a very important property: itcompletelysummarizesall of theinformationfromthepast that isneededtomakeaforward-lookingdecision. Whilepreferencesandthe transition equation are certainly dependent on the past, thisdependenceis representedbyst: other variables fromthepast donot affect current payoffs or constraints andthus cannot inuencecurrentdecisions. Thismayseemrestrictivebutitisnot: thevectorstmayinclude manyvariables sothat the dependence of currentchoices onthepastcanbequiterich.While the state vector is effectively determined by preferences andthetransitionequation,theresearcherhassomelatitudeinchoosingthecontrolvector.Thatis,theremaybemultiplewaysofrepresent-ingthesameproblemwithalternativespecicationsof thecontrolvariables.We assume that c A Cand s A S. In some cases the control isrestrictedtobeinasubset of Cthat depends onthestatevector:c A Cs. Further we assume that~ sss; c is bounded for s; c A S C.15Forthecake-eatingproblemdescribedabove, thestateofthesys-temwasthesizeof thecurrent cake Wtandthecontrol variablewas the level of consumption in period t, ct. The transition equationdescribingtheevolutionof thecakewasgivenbyWt1 Wt ct:Clearly, the evolutionof the cake is governedby the amount ofcurrentconsumption. Anequivalentrepresentation, asexpressedin(2.7),istoconsiderthefuturesizeofthecakeasthecontrolvariableand thentosimplywritecurrent consumptionasWt1 Wt.There are two nal properties of the agents dynamic optimizationproblemworthspecifying: stationarityanddiscounting. Notethatneitherthepayoffnorthetransitionequationsdependexplicitlyontime. True the problemis dynamic, but time per se is not of theessence. Inagivenstatetheoptimal choiceoftheagentwill bethesameregardless of when heoptimizes. Stationarityis important15. Ensuringthattheproblemisboundedisanissueinsomeeconomicapplications,such as the growth model. Often these problems are dealt with by bounding the sets CandS.Theory ofDynamic Programming 25bothfortheanalysisoftheoptimizationproblemandforempiricalimplementationofinnitehorizonproblems. Infact, becauseofsta-tionarity, we candispense withtimesubscripts as the problemiscompletely summarizedbythecurrentvaluesof thestatevariables.Theagentspreferencesarealsodependent ontherateat whichthe future is discounted. Let b denote the discount factor and assumethat 0 < b< 1. Thenwecanrepresent theagentspayoffsover theinnitehorizonasXtyt0bt~ ssst; ct: 2:10Oneapproachtooptimizationistomaximize(2.10) throughthechoiceof fctg fort 0; 1; 2; . . . givens0andsubjecttothetransitionequation.LetVs0 betheoptimizedvalueofthisproblemgiventheinitialstate.Alternatively, onecanadoptthedynamicprogramapproachandconsiderthe followingequation, calledBellmansequation:Vs maxc A Cs~ sss; c bVs0 for all s A S; 2:11wheres0 ts; c. Heretimesubscriptsareeliminated, reectingthestationarityoftheproblem. Instead, currentvariablesareunprimedwhilefutureonesaredenotedbya prime.AsinStokey and Lucas(1989),the problemcanbeformulatedasVs maxs0A Gsss; s0 bVs0 for all s A S: 2:12Thisisamorecompactformulation, andwewill useitforourpre-sentation.16Nonetheless, thepresentations inBertsekas (1976) andSargent (1987) follow(2.11). Assume that S isaconvex subsetof n_c *Loop over all consumptionlevels *c=c_L+(c_H-c_L)/n_c*(i_c-1)i_y=1EnextV=0 *Initialize the next valueto zero *do until i_y>n_y *Loop over all possiblerealizations of the futureendowment *nextX=R*(X[i_s]-c)+Y[i_y] *Next period amount ofcake *nextV=V(nextX) *Here we use interpolationto find the next valuefunction *EnextV=EnextV+nextV*Pi[i_y] *Store the expected futurevalue using the transitionmatrix *i_y=i_y+1endo *End of loop overendowment *aux[i_c]=u(c)+beta*EnextV *Stores the value of a givenconsumption level *i_c=i_c+1endo *End of loop overconsumption *newV[i_s,i_y]=max(aux) *Take the max over allconsumption levels *i_s=i_s+1endo *End of loop over size ofcake *V=newV *Update the new valuefunction *Figure3.1Stochastic cake-eating problem38 Chapter3Once the value functioniterationpiece of the programis com-pleted, thevaluefunctioncanbeusedtondthepolicyfunction,c cX. This is done by collecting all the optimal consumptionvalues cicfor everyvalue of Xis. Here again, we onlyknowthefunctioncX at the points of the grid. We canuse interpolatingmethodstoevaluatethepolicyfunctionat otherpoints. Thevaluefunctionandthepolicyfunctionaredisplayedingures3.2and3.3for particular valuesoftheparameters.As discussedabove, approximating the value functionandthepolicyrulesbyanitestatespacerequiresalargenumberofpointsonthis space (nshas tobe big). These numerical calculations areoftenextremelytime-consuming. Sowecanreducethenumber ofpointsonthegrid, whilekeepingasatisfactoryaccuracy, byusinginterpolations onthis grid. WhenwehaveevaluatedthefunctionvjRXis cic yi, i L; H, weusethenearestvalueonthegridtoapproximateRXis cic yi. Withasmall numberofpointsonthegrid, this canbeaverycrudeapproximation. Theaccuracyof thecomputationcanbeincreasedbyinterpolatingthefunctionvj: (seethe appendixfor more details). The interpolationis basedonthevaluesinV.Figure3.2Value function,stochasticcake-eatingproblemNumericalAnalysis 393.2.2 PolicyFunctionIterationsThevaluefunctioniterationmethodcanberather slow, asit con-vergesatarateb.ResearchershavedevisedothermethodsthatcanbefastertocomputethesolutiontotheBellmanequationinanin-nitehorizon. Thepolicyfunctioniteration, alsoknownasHowardsimprovementalgorithm,isoneofthese.WereferthereadertoJudd(1998) or Ljungqvist andSargent (2000) for moredetails.Thismethodstartswithaguessofthepolicyfunction,inourcasec0X. This policy function is then used to evaluate the value of usingthisrule forever:V0X uc0X b

iL; HpiV0RX c0X yi for all X:This policy evaluation step requires solving a systemof linearequations,giventhatwehaveapproximatedRX c0X yibyanXonourgrid.Nextwedoapolicyimprovementsteptocomputec1X:Figure3.3Policy function,stochasticcake-eatingproblem40 Chapter3c1X argmaxcuc b

iL; HpiV0RX c yi_ _for all X:Given this newrule, the iterations are continued to nd V1 ;c2 ; . . . ; cj1 until jcj1X cjXj is small enough. Theconver-gencerateismuchfasterthanthevaluefunctioniterationmethod.However, solving the policy evaluation step cansometimes bequitetime-consuming,especiallywhenthestatespaceislarge.Onceagain, the computationtime canbe much reduced if theinitial guessc0X iscloseto thetrue policy rulecX.3.2.3 ProjectionMethodsThesemethodscomputedirectlythepolicyfunctionwithout calcu-latingthevaluefunctions.Theyusetherst-orderconditions(Eulerequation)tobackoutthepolicyrules.Thecontinuouscakeproblemsatisesthe rst-order Eulerequationu0ct bREtu0ct1if the desired consumption level is less than the total resourcesX W y. Ifthereisacornersolution, thentheoptimal consump-tionlevel iscX X. Takingintoaccount thecorner solution, wecan rewritetheEulerequation asu0ct maxu0Xt; bREtu0ct1:We knowthat by the iid assumption, the problemhas onlyone state variable X, sothe consumptionfunctioncanbe writtenc cX. Asweconsider thestationarysolution, wedropthesub-script t inthenext equation. TheEulerequationcanthenberefor-mulated asu0cX maxu0X; bREy0 u0cRX cX y0 0 3:4orFcX 0: 3:5Thegoalistondanapproximation^ ccXofcX, forwhich(3.5)isapproximately satised. The problem is thus reduced to nd the zeroofF, whereFisanoperatoroverfunctionspaces. Thiscanbedonewithaminimizingalgorithm. Therearetwoissuestoresolve. First,NumericalAnalysis 41weneedtondagoodapproximationof cX. Second, wehavetodene ametric to evaluatethet oftheapproximation.SolvingforthePolicyRuleLet fpiXgbeabaseof thespaceof continuousfunctions, andletC fcig bea setof parameters. Wecan approximatecX by^ ccX; C

ni1cipiX:There is an innite number of bases to chose from. A simple one is toconsider polynomialsinXsothat ^ ccX; C c0 c1X c2X2 :Althoughthischoiceisintuitive, itisnotusuallythebestchoice. Inthefunctionspacethisbaseisnotanorthogonalbase,whichmeansthatsomeelements tend to becollinear.Orthogonal bases will yield more efcient and precise results.3Thechosenbaseshouldbecomputationallysimple. Itselementsshouldlooklike thefunctiontoapproximate, sothat thefunctioncXcanbeapproximatedwithasmall number of basefunctions. Anyknowledge of the shape of the policy function will be to a great help.If,forinstance,thispolicyfunctionhasakink,amethodbasedonlyonaseriesofpolynomialswill haveahardtimettingit. Itwouldrequire a large number of powers of the state variable to comesomewhereclosetothesolution.Havingchosenamethodtoapproximatethepolicyrule, wenowhave tobe more precise about what bringingF^ ccX; C close tozeromeans.Tobemorespecic,weneedtodenesomeoperatorsonthe space of continuous functions. For anyweightingfunctiongx, theinner product of twointegrablefunctions f1and f2onaspaceA is denedashf1;f2i _Af1x f2xgx dx: 3:6Twofunctions f1andf2aresaidtobeorthogonal, conditional onaweightingfunctiongx, ifhf1;f2i 0. Theweightingfunctionindi-cateswheretheresearcherwantstheapproximationtobegood.Weareusingtheoperatorh: ; :iandtheweightingfunctiontoconstructa metric to evaluate how close F^ ccX; C is to zero. This will be done3. Popular orthogonalbasesare Chebyshev, Legendre, orHermite polynomials.42 Chapter3bysolvingforC such thathF^ ccX; C;f Xi 0;where f Xissomeknownfunction.Wenextreviewthreemethodsthat differ in theirchoicefor thisfunctionf X.First, asimplechoicefor f XisF^ ccX; Citself. ThisdenestheleastsquaremetricasminChF^ ccX; C; F^ ccX; Ci:By the collocationmethod, detailedlater inthis section, we canchoose tondC asminChF^ ccX; C; dX Xii; i 1; . . . ; n;wheredX Xi isthemasspointfunctionatpointXi,meaningthatdX 1if X XianddX 0elsewhere. AnotherpossibilityistodeneminChF^ ccX; C; piXi; i 1; . . . ; n;wherepiXisabaseofthefunctionspace.ThisiscalledtheGaler-kin method. An application of this method can be seen below, wherethebaseistaken tobetentfunctions.Figure 3.4displays asegment of the computer code that calcu-latestheresidual functionF^ ccX; Cwhentheconsumptionruleisapproximatedbyasecond-orderpolynomial. Thiscanthenbeusedin oneof theproposedmethods.CollocationMethodsJudd (1992) presents in some detail this method applied to thegrowthmodel. ThefunctioncXisapproximatedusingChebyshevpolynomials. These polynomials are dened on the interval 0; 1 andtaketheformpiX cosi arccosX; X A 0; 1; i 0; 1; 2; . . . :For i 0, this polynomial is aconstant. For i 1, the polynomialis equal toX. As these polynomials areonlydenedonthe 0; 1interval, one can usually scale the state variables appropriately.4The4. ThepolynomialsarealsodenedrecursivelybypiX 2Xpi1X pi2X, i b2,withp00 1 andpX; 1 X.NumericalAnalysis 43policyfunction can then beexpressed as^ ccX; C

ni1cipiX:Next themethodndsC, which minimizeshF^ ccX; C; dX Xii; i 1; . . . ; n;whered is themass point function. HencethemethodrequiresthatF^ ccX; CiszeroatsomeparticularpointsXiandnotoverthewhole range XL; XH. The method is more efcient if these points arechosen to be the zeros of the base elementspiX, here Xi cosp=2i.This methodisreferredtoasanorthogonalcollocationmethod.C isprocedure c(x)cc=psi_0+psi_1*x+psi_2*x*xreturn(cc)endprocedure*Here we define anapproximation for theconsumption function basedon a second-orderpolynomial *i_s=1do until i_s>n_s *Loop over all sizes of thetotal amount of cake *utoday=U0(c(X[i_s])) *Marginal utility ofconsuming *ucorner=U0(X[i_s]) *Marginal utility if cornersolution *EnextU=0i_y=1*Initialize expected futuremarginal utility *do until i_y>n_y *Loop over all possiblerealizations of the futureendowment *nextX=R(X[i_s]-c(X[i_s]))+Y[i_y]*Next amount of cake *nextU=U0(c(nextX)) *Next marginal utility ofconsumption *EnextU=EnextU+nextU*Pi[i_y] *Here we compute the expectedfuture marginal utility ofconsumption using thetransition matrix Pi *i_y=i_y+1endo *End of loop over endowment *F[i_s]=utoday-max(ucorner,beta*EnextU)i_s=i_s+1endo *End of loop over size ofcake *Figure3.4Stochastic cake-eating problem, projectionmethod44 Chapter3thesolutiontoa system of nonlinear equations:F^ ccXi; C 0; i 1; . . . ; n:Thismethodisgoodatapproximatingpolicyfunctionsthatarerel-atively smooth. A drawback is that the Chebyshev polynomials tendtodisplay oscillationsat higher orders. TheresultingpolicyfunctioncX willalsotendtouctuate.Thereisnoparticularruleforchoos-ingn, the highest order of the Chebyshevpolynomial. Obviouslythehighernis, thebetterwillbetheapproximation, butthiscomesat thecost of increasedcomputation.FiniteElementMethodsMcGrattan(1996)illustratestheniteelementmethodwiththesto-chasticgrowthmodel (seealsoReddy1993foranin-depthdiscus-sionof niteelements).Tostart, thestatevariableXis discretizedover agrid fXisgnsis1.Theniteelementmethodis basedonthefollowing functions:pisX X Xis1Xis Xis1ifX A Xis1; Xis;Xis1 XXis1 XisifX A Xis; Xis1;0 elsewhere:___Figure3.5Basisfunctions, nite elementmethodNumericalAnalysis 45The function pisX is a simple functionin 0; 1, as illustratedingure3.5. It is infact asimplelinear interpolation(andanordertwospline; seetheappendixformoreonthesetechniques). Ontheinterval Xis; Xis1, thefunction^ ccXisequal totheweightedsumof pisX andpis1X. Heretheresidual functionsatiseshF^ ccX; C; piXi 0; i 1; . . . ; n:Equivalently, wecouldchoosea constantweighting function:_X0pisXF^ ccX dX 0; is 1; . . . ; ns:This gives asystemwithnsequations andnsunknowns, fcisgnsis1.This nonlinear systemcanbesolvedtondtheweights fcisg. Tosolvethesystem, theintegral canbecomputednumericallyusingnumerical techniques; see the appendix. As in the collocationmethod, the choice of nsis the result of a trade-off between increasedprecisionanda highercomputationalburden.3.3 StochasticDiscreteCake-EatingProblemWe present here another example of a dynamic programming model.It differs from the one presented in section 3.2 in two ways. First, thedecisionof theagent isnot continuous(howmuchtoeat) but dis-crete (eat or wait). Second, the problem has two state variables as theexogenousshockisseriallycorrelated.Theagent isendowedwithacakeof sizeW. Ineachperiodtheagent has todecidewhether or not toeat theentirecake. Evenifnoteaten,thecakeshrinksbyafactorreachperiod.Theagentalsoexperiencestasteshocks,possiblyseriallycorrelated,andwhichfol-lowanautoregressiveprocessoforderone. Theagentobservesthecurrent taste shock at the beginning of the period, before the decisiontoeatthecakeistaken. However,thefutureshocksareunobservedby the agent, introducing a stochastic element into the problem.Althoughthecakeisshrinking, theagentmightdecidetopostponetheconsumptiondecisionuntil aperiodwithabetterrealizationofthetasteshock.Theprogram of theagentcanbewrittenintheformVW; e maxeuW; bEe0j eVrW; e0; 3:7whereVW; eistheintertemporal valueofacakeofsizeWcondi-tional of the realizatione of the tasteshock. Here Ee0 denotes the46 Chapter3expectationwithrespect tothe future shocke, conditional onthevalue of e. The policy function is a function dW; e that takes a valueof zero if the agent decides to wait or one if the cake is eaten. We canalsodenea thresholdeW such thatdW; e 1 ife > eW;dW; e 0 otherwise._As insection3.2 the problemcanbe solvedbyvalue functioniterations. However, theproblemisdiscrete, sowecannot usetheprojection technique as the decision rule is not a smooth function butastepfunction.3.3.1 ValueFunctionIterationsAsbefore,wehavetodene,rst,thefunctionalformfortheutilityfunction,andweneedtodiscretizethestatespace.Wewillconsiderr < 1, sothe cake shrinks withtime andWis naturallyboundedbetweenW, theinitial sizeand0. Inthiscasethesizeof thecaketakesonlyvaluesequal tortW, t b0. HenceCS friWgisajudi-ciouschoiceforthestatespace. Contrarytoanequallyspacedgrid,this choice ensures that we do not needto interpolate the valuefunctionoutsideof thegridpoints.Next,weneedtodiscretizethesecondstatevariable,e.Theshockissupposedtocomefromacontinuousdistribution, andit followsanautoregressiveprocessof order one. WediscretizeeinI pointsfeigIi1followingatechniquepresentedbyTauchen(1986)andsum-marizedintheappendix. InfactweapproximateanautoregressiveprocessbyaMarkovchain.Themethoddeterminestheoptimaldis-cretepoints feigIi1andthetransitionmatrixpij Probet eij et1 ejsuchthattheMarkovchainmimicstheAR(1)process.Ofcourse,theapproximationisonlygood if I isbig enough.InthecasewhereI 2, wehavetodeterminetwogridpointseLandeH. TheprobabilitythatashockeLisfollowedbyashockeHisdenotedbypLH. Theprobabilityof transitions canbestackedinatransitionmatrix:p pLLpLHpHLpHH_ _with the constraints that the probability of reaching either a low or ahigh state next period is equal to one:pLL pLH 1 andpHL pHH NumericalAnalysis 471.ForagivensizeofthecakeWis risWandagivenshockej, j LorH, itiseasytocomputethersttermejurisW. Tocomputethesecondtermweneedtocalculatetheexpectedvalueoftomorrowscake. Givenaguessforthevaluefunctionofnextperiod, v: ; :, theexpectedvalue isEe0 j ejvris1W pjLvris1W; eL pjHvris1W; eH:TherecursionisstartedbackwardwithaninitialguessforV: ; :.ForagivenstateofthecakeWisandagivenshockej,thenewvaluefunctioniscalculatedfromequation(3.7).Theiterationsarestoppedwhentwosuccessivevaluefunctionsarecloseenough.Innumericalcomputing thevaluefunctionis stored as amatrixV of sizenW ne,wherenWandnearethenumberofpointsonthegridforWande.At eachiterationthematrixisupdatedwiththenewguessforthevaluefunction.Figure3.6givesanexampleofacomputercodethatobtains thevaluefunctionvj1W; e giventhe valuevjW; e.The waywe have computedthe grid, the next periodvalue issimpletocomputeasit isgivenbyVis 1; :. Thisruleisvalidifi_s=2do until i_s>n_s *Loop over all sizes of thecake *i_e=1do until i_e>2 *Loop over all possiblerealizations of the taste shock*ueat=u(W[i_s],e[i_e]) *Utility of doing the eating now*nextV1=V[i_s-1,1] *Next period value if low tasteshock *nextV2=V[i_s-1,2] *Next period value if high tasteshock *EnextV=nextV1*p[i_e,1]+nextV2*p[i_e,2]newV[i_s,i_e]=max(ueat,beta*EnextV)*Take the max between eating nowor waiting *i_e=i_e+1endo *End of loop over taste shock *i_s=i_s+1endo *End of loop over size of cake *V=newV *Update the new value function *Figure3.6Discrete cake-eatingproblem48 Chapter3is> 1.ComputingV1; :ismoreofaproblem.Onewayistouseanextrapolationmethodtoapproximatethevalues, giventheknowl-edgeofVis; :,is> 1.Figure 3.7 showsthe valuefunctionfor particular parameters. Theutility function is taken to beuc; e lnec, and lne is supposed tofollow an AR(1) process with mean zero, autocorrelation re 0:5 andwithanunconditional variance of 0.2. We have discretizede intofour grid points.Figure3.8showsthedecisionrule, andthefunctioneW. Thisthreshold wascomputed asthesolution of:uW; eW bEe0 j eVrW; e0;which is the value of the taste shock that makes the agent indifferentbetweenwaitingandeating,giventhesize ofthecakeW.We return later in this book to examples of discrete choice models.In particular, we refer the readers to the models presented in sections8.5 and7.3.3.Figure3.7Value function,discrete cake-eating problemNumericalAnalysis 493.4 ExtensionsandConclusionInthischapterwereviewedthecommontechniquesusedtosolvethedynamicprogrammingproblemsofchapter2. Weappliedthesetechniquestobothdeterministicandstochasticproblems, tocontin-uousanddiscretechoicemodels. Thesemethodscanbeappliedaswell tomorecomplicated problems.3.4.1 LargerStateSpacesThe two examples we have studied in sections 3.2 and 3.3 have smallstatespaces. Inempiricalapplicationsthestatespaceoftenneedstobe much larger if the model is to confront real data. For instance, theendowmentshocksmight beseriallycorrelatedortheinterestrateRmight alsobea stochastic andpersistentprocess.Forthevaluefunctioniterationmethod, thismeansthat thesuc-cessive value functions have tobe stackedina multidimensionalmatrix. Also the value functionhas to be interpolatedinseveraldimensions.ThetechniquesintheappendixcanbeextendedtodealFigure3.8Decision rule,discrete cake-eating problem50 Chapter3withthis problem. However, the value functioniterationmethodquicklyencountersthecurseofdimensionality.Ifeverystatevari-ableisdiscretizedintonsgridpoints, thevaluefunctionhastobeevaluatedbyNnspoints, whereNisthenumberof statevariables.This demands anincreasingamount of computer memoryandsoslowsdownthecomputation. Asolutiontothisproblemistoeval-uatethevaluefunctionforasubset ofthepointsinthestatespaceandthentointerpolatethevaluefunctionelsewhere. Thissolutionwasimplemented byKeaneandWolpin (1994).Projection methods are better at handling larger state spaces.Suppose that the problemis characterized by Nstate variablesfX1; . . . ; XNg. Theapproximated policy function can bewrittenas^ ccX1; . . . ; XN

Nj1

njij1cjijpijXj:Theproblem isthencharacterizedby auxiliaryparameters fcji g.exercise 3.1 Suppose that uc c1g=1 g. Construct the code tosolve for the stochastic cake-eating problem using the value functioniterationmethod.Plotthepolicyfunctionasafunctionofthesizeofthecakeandthestochasticendowment forg f0:5; 1; 2g. Comparethelevel andslopeof thepolicyfunctionsfordifferent valuesof g.Howdoyou interpret theresults?exercise3.2 Consider, again, thediscretecake-eatingproblemofsection3.3. Construct thecodetosolvefor this problem, withiidtaste shocks, using uc lnc, eL 0:8, eH 1:2, pL 0:3, andpH 0:7.Mapthedecision ruleas afunction ofthe sizeof thecake.exercise 3.3 Consider an extension of the discrete cake-eatingproblemof section 3.3. The agent can nowchoose among threeactions: eat the cake, store it in fridge 1 or in fridge 2. In fridge 1, thecake shrinks by a factor r: W0 rW. In fridge 2, the cake diminish byaxedamount: W0 W k. Theprogramof theagent is charac-terized asVW; e maxVEatW; e; VFridge 1W; e; VFridge 2W; ewithVEatW; e euW;VFridge 1W; e bEe0 VrW; e0;VFridge 2W; e bEe0 VW k; e0:___NumericalAnalysis 51Construct the code to solve for this problem, using uc lnc,eL 0:8,eH 1:2,pL 0:5,andpH 0:5.Whenwilltheagentswitchfrom onefridgetotheother?exercise3.4 Considerthestochasticcake-eatingproblem.Supposethat the discount rate b is afunctionof the amount of cake con-sumed: b Fb1 b2c,whereb1andb2areknownparametersandF is thenormal cumulativedistributionfunction. Construct thecodetosolveforthisnewproblemusingvaluefunctioniterations.Suppose g 2, b1 1:65, pL pH 0:5, yL 0:8, yH 1:2, andb2 1. Plotthepolicyrulec cX. Comparetheresultwiththatof thecasewherethediscount rateisindependent of thequantityconsumed. Howwouldyouinterpretthefactthatthediscountratedependsontheamount of cakeconsumed?3.5 Appendix:AdditionalNumericalToolsInthis appendixweprovidesomeuseful numerical tools that areoftenusedinsolvingdynamicproblems. Wepresent interpolationmethods, numerical integrationmethods, as well as a methodtoapproximateseriallycorrelatedprocessesbyaMarkovprocess. Thelast subsectionis devotedtosimulations.3.5.1 InterpolationMethodsWebrieyreviewthreesimpleinterpolationmethods. For furtherreadings, see,for instance, Presset al. (1986)or Judd(1996).Whensolvingthevaluefunctionorthepolicyfunction, weoftenhavetocalculatethevalueof thesefunctionsoutsideof thepointsofthegrid. Thisrequiresonetobeabletointerpolatethefunction.Using a good interpolation method can also save computer time andspacesincefewer gridpointsareneededtoapproximatethefunc-tions. Let us denote f x thefunctiontoapproximate. Weassumethat we know thisfunctionat a number of gridpoints xi,i 1; . . . ; I.We denote by fi f xi the values of the function at these gridpoints. Weareinterestedinndinganapproximatefunction^ff xsuchthat^ff x Ff x, basedontheobservations fxi;fig. Wepresentthreedifferent methodsanduseasanexamplethefunction f x x sinx.Figure3.9 displaystheresultsfor allthemethods.52 Chapter3LeastSquaresInterpolationAnatural waytoapproximate f is touseaneconometrictech-nique, such as OLS, to estimate the function^ff :. The rst step is toassumeafunctionalformfor^ff . Forinstance, wecanapproximate fwitha polynomialinx such as^ff x a0 a1x aNxN; N< I:Byregressing fionxi, wecaneasilyrecover theparameters an. Inpractice, thismethodisoftennot verygood, unlessthefunction fis well behaved. Higher-order polynomials tendto uctuate andcanoccasionallygiveanextremelypoort. Thisisparticularlytruewhenthefunctionisextrapolatedoutsideof thegridpoints, whenx > xIorx < x1. Theleastsquaremethodisaglobal approximationmethod.Assuch,thet can beonaveragesatisfactorybutmediocrealmost everywhere.Thiscan beseen in theexampleingure3.9.LinearInterpolationThismethodtsthefunction f withpiecewiselinear functionsontheintervals xi1; xi. Foranyvalueof xin xi1; xi, anapproxima-Figure3.9ApproximationmethodsNumericalAnalysis 53tion^ff x of f x canbefoundas^ff x fi1 fi fi1xi xi1x xi1:Aner gridwill give a better approximationof f x. Whenx isgreater thanxI, usingthis rulecanleadtonumerical problems astheexpressionabovemaynotbeaccurate.Notethattheapproxima-tion function^ff is continuous but not differentiable at the grid points.Thiscanbeanundesirablefeatureasthisnondifferentiabilitycanbetranslated to thevaluefunctionor thepolicy function.Thelinearinterpolationmethodcanbeextendedformultivariatefunctions. For instance, we can approximate the function f x; ygivendata on fxi; yj;fijg. Denote dx x xi=xi1 xi anddy y yi=yi1 yi.Theapproximationcan bewritten as^ff x; y dxdy fi1; j1 1 dx dy fi; j1 dx1 dy fi1; j 1 dx1 dy fi; j:Theformulacan beextendedtohigher dimensions aswell.SplineMethodsThis methodextends the linear interpolationby tting piecewisepolynomials while ensuring that the resulting approximate func-tion^ff isbothcontinuousanddifferentiableatthegridpointsxi.Werestrict ourself tocubicsplinesfor simplicity, but theliteratureonsplinesisverylarge(e.g.,seeDeBoor1978).Theapproximatefunc-tion isexpressedas^ffix fi aix xi1 bix xi12 cix xi13; x A xi1; xi:Hereforeachpointonthegrid,wehavetodeterminethreeparam-etersfai; bi; cig, so in total there is 3 I parameters to compute.However, imposingthecontinuityofthefunctionandofitsderiva-tiveupto thesecond order reducesthe number of coefcients:^f fix ^ffi1x;^ff 0i x ^ff 0i1x;^ff 00i x ^ff 00i1x:It isalsocommonpracticetoapply^ff 001 x1 ^ff 00I xI 0. Withtheseconstraints, the number of coefcients to compute is down to I. Some54 Chapter3algebra givesai fi fi1xi xi1 bixi xi1 cixi xi12; i 1; . . . ; I;ci bi1 bi3xi xi1; i 1; . . . ; I 1;cI bI3xI xI1;ai 2bixi xi1 3cixi xi12 ai1:___Solvingthis systemof equationleads toexpressions for the coef-cients fai; bi; cig. Figure 3.9shows that the cubic spline is averygoodapproximationtothe functionf .3.5.2 NumericalIntegrationNumerical integrationis oftenrequiredindynamic programmingproblems tosolvefor theexpectedvaluefunctionor tointegrateout anunobservedstate variable. For instance, solvingthe Bell-manequation(3.3)requiresonetocalculateEvX0 _vX0 dFX0,where F: is the cumulative density of the next period cash-on-handX.Ineconometricapplicationssomeimportantstatevariablesmightnotbeobserved.Forthisreasononemayneedtocomputethedeci-sionrule unconditional of this state variable. For instance, inthestochasticcake-eatingproblemof section3.2, if Xisnot observed,one could compute c _cX dFX, which is the unconditionalmean of consumption, and match it with observed consumption. Wepresentthreemethodsthatcanbeusedwhennumericalintegrationis needed.QuadratureMethodsThereare anumber of quadrature methods. We brieydetail theGauss-Legendremethod(moredetailedinformationcanbefoundinPresset al.1986). The integralof a functionf isapproximatedas_11f x dx Fw1 f x1 wnf xn; 3:8wherewiandxiarenweightsandnodestobedetermined.Integra-tionover adifferent domaincanbeeasilyhandledbyoperatingaNumericalAnalysis 55changeof theintegrationvariable. Theweights andthenodes arecomputed such that (3.8) is exactly satised for polynomials ofdegree2n 1or less. Forinstance, if n 2, denote fix xi1, i 1; . . . ; 4. Theweightsandnodes satisfyw1 f1x1 w2 f1x2 _11f1x dx;w1 f2x1 w2 f2x2 _11f2x dx;w1 f3x1 w2 f3x2 _11f3x dx;w1 f4x1 w2 f4x2 _11f4x dx:This is asystemoffour equations withfour unknowns. Thesolu-tionsarew1 w2 1andx2 x1 0:578. For larger valuesof n,thecomputationissimilar.Byincreasingthenumber ofnodesn,theprecision increases. Noticethat thenodesarenotnecessarily equallyspaced.Theweightsandthevalueofthenodesarepublishedintheliteraturefor commonlyusedvaluesofn.ApproximatinganAutoregressiveProcesswithaMarkovChainInthisdiscussionwefollowTauchen(1986)andTauchenandHus-sey(1991)andshowhowtoapproximateanautoregressiveprocessoforderonebyarst-orderMarkovprocess. Thiswaywecansim-plifythecomputationofexpectedvaluesinthevaluefunctionitera-tion framework.Toreturntothe value functioninthe cake-eatingproblem, weneed tocalculatetheexpectedvaluegivene:VW; e maxeuW; Ee0j eVrW; e0:The calculation of an integral at each iteration is cumbersome. So wediscretizetheprocesset, intoNpointsei, i 1; . . . ; N. NowwecanreplacetheexpectedvaluebyVW; ei max euW;

Nj1pi; jVrW; ej____; i 1; . . . ; N:56 Chapter3As inthe quadrature method, the methodinvolves ndingnodesejandweightspi; j. Aswewill seebelow, theeiandthepi; jcanbecomputed priortotheiterations.Supposethat etfollowsanAR(1) process, withanunconditionalmeanm andan autocorrelationr:et m1 r ret1 ut; 3:9whereutisanormallydistributedshockwithvariances2. Todis-cretize this process, we needtodetermine three different objects.First, weneedtodiscretizetheprocessetintoNintervals. Second,we need to compute the conditional mean of etwithin each intervals,whichwe denote by zi, i; . . . ; N. Third, we needto compute theprobabilityof transitionbetweenanyof theseintervals, pi; j. Figure3.10 shows the plot of the distribution ofe and the cut-off points eiaswellas theconditionalmeanszi.Westartbydiscretizingthereal lineintoNintervals, denedbythe limits e1; . . . ; eN1. As the process etis unbounded, e1 yandeN1 y. The intervals are constructedsuchthat ethas anequal probabilityof 1=Noffallingintothem. Giventhenormalityassumption,thecut-off points feigN1i1aredened asFei1 mse_ _Fei mse_ _ 1N ; i 1; . . . ; N; 3:10where F is the cumulativeof thenormal densityandseis thestandard deviation of e equal to s=1 r_. Working recursively, wegetFigure3.10Example of discretization,N 3NumericalAnalysis 57ei seF1i 1N_ _ m:Now that we have dened the intervals, we want to nd the averagevalueofe withinagiveninterval.Wedenotethisvaluebyzi,whichiscomputedasthemeanofetconditional onet A ei; ei1:zi Eetjet A ei; ei1 sefei m=se fei1 m=seFei1 m=se Fei m=se m:From(3.10),weknowthattheexpressionsimpliestozi Nsefei mse_ _ fei1 mse_ _ _ _ m:Next wedenethetransition probability aspi; j Pet A ej; ej1 j et1 A ei; ei1pi; j N2psep_ei1eieum2=2s2e _Fej1 m1 r rus_ _Fej m1 r rus_ __du:The computation of pi; jrequires the computation of a nontrivialintegral. Thiscanbedonenumerically. Notethat if r 0, meaninge is an iid process, theexpression aboveissimplypi; j 1N :WecannowdeneaMarkovprocessztthatwill mimicanauto-regressive process of order one, as dened in (3.9).zttakes its valuesin fzigNi1andthetransition betweenperiodt andt 1 isdenedasPzt zjjzt1 zi pi; j:ByincreasingN, thediscretizationbecomes ner andtheMarkovprocessgets closer totherealautoregressiveprocess.Example For N=3,r 0:5,m 0,ands 1,wehavez1 1:26; z2 0; z3 1:26;and58 Chapter3p __0:55 0:31 0:140:31 0:38 0:310:14 0:31 0:55__:3.5.3 HowtoSimulatetheModelOncethevaluefunctioniscomputed, theestimationortheevalua-tionof themodel oftenrequires thesimulationof thebehavior oftheagentthroughtime. Ifthemodelisstochastic, therststepistogenerateaseries for theshocks, for t 1; . . . ; T. Thenwegofromperiodtoperiodandusethepolicyfunctiontondouttheoptimalchoice for this period. We also update the state variable and proceedtonext period.HowtoProgramaMarkovProcessTheMarkovprocessischaracterizedbygridpoints, fzigandbyatransitionmatrixp, withelements pij Probyt zj=yt1 zi. Westart inperiod1. Theprocess ztis initializedat, say, zi. Next, wehavetoassignavalueforz2.Tothisend,usingtherandomgenera-tor ofthecomputer,wedrawauniformvariableu in 0; 1.Thestateinperiod 2, j, is dened ast=1oldind=1 *Variable to keep track ofstate in period t-1 *y[t]=z[oldind] *Initialize first period *do until t>T *Loop over all time periods *u=uniform(0,1) *Generate a uniform randomvariable *sum=0 *Will contain the cumulativesum of pi *ind=1 *Index over all possiblevalues for process *do until u