Lecture 1Mathematics for dynamic programing I

ZENG MING

March 30, 2016

1 / 25

Course Outline

I Math for Dynamic Programming I

I Math for Dynamic Programming II

I Stability of dynamic system

I Search and matching, a little stochastic dynamic programming

Main reference book:Recursive methods in economic dynamics by Stokey and Lucas(SL)Solutions manual by Irigoyen and Rossi-Hansberg(IRH)

2 / 25

Roadmap

I Infinite horizon economic problem and solution by sequential method

I Recursive formulation and solution by dynamic programming

I Algorithm, theoretical foundation and mathematics

I Theorem of Maximum

3 / 25

Problem: Deterministic Model of Optimal GrowthA representative agent with utility function

∑∞t=0 β

tU(ct), a representative firm withproduction function yt = F (kt). Agent owns the firm. Each period to accumulatecapital, agent will invest i . The objective of agent is to maximize her utility functiongiven feasibility constraint(given k0 > 0):

max{kt+1}∞t=0

∞∑t=0

βtU(ct)

s.t kt+1 = (1− δ)kt + it

ct + it = yt = F (kt)

k0 > 0

We make the following assumptions:i) F (0) = 0,F ′(k) > 0, limk→0 F

′(k) =∞, limk→∞ F ′(k) = 0ii) U is bounded, continuously differentiable, strictly increasing and strictly concave,with limc→0 U′(c) =∞If we define f (k) = F (k) + (1− δ)k, the equivalent planning problem:

max{kt+1}∞t=0

∞∑t=0

βtU(f (kt)− kt+1) (1)

s.t 0 ≤ kt+1 ≤ f (kt) (2)

k0 > 0 (3)

4 / 25

Sequential Method

Starting from the case when the planning horizon is finite numberT , our objective is to find sequence(may not be unique) {kt+1}Tt=0

which solves the problem (1) from the feasible set, that is, the setof sequences {kt+1}Tt=0 which satisfy (2)-(3). Fortunately, finitehorizon makes the feasible set a convex subset of RT+1. On theother hand, by assumption, objective function in (1) is continuousand strictly concave. Then Kuhn-Tucker conditions apply so thatlocal optimum is the global optimum, the solution to problem(1)-(3) is unique.Confirming the existence and uniqueness of solutions, we thenproceed to solve it. Since U ′(0) =∞, then agent will always wantto consume some quantities at any period between t = 1 tot = T . However, agent doesn’t care the allocation from T + 1onward since there is no benefit for her to keep kT+1 positive.Then we know that kT+1 = 0. This also means that (2) will neverbind except for kT+1.

5 / 25

Sequential Method(cont’d)Solution to the problem is obtained from Unconstrained Optimization::

βf ′(kt)U′[f (kt)− kt+1] = U ′[f (kt−1)− kt ] t = 1, 2 . . .T

kT+1 = 0

given k0.To solve for {kt+1}Tt=0, we need parametric form for U and f . Then wecan get the result by solving a second order difference equation with twoboundary conditions(details see Exercise 2.2 on SL, Page 12). Forexample, if we assume U(c) = ln(c) and f (k) = kα where 0 < α < 1,then the result is:

kt+1 = αβ1− (αβ)T−t

1− (αβ)T−t+1kαt t = 0, 1 · · ·T

In case of infinite horizon->normally by taking the limit of solutions fromfinite horizon problem, that is, kt+1 = αβkα

t , though additionaltheoretical arguments must be made so that “limT→∞ ” and “max” canbe exchanged.

6 / 25

A recursive formulationSuppose problem (1) has been solved for all possible values of k0, then we can define a function v by taking v(k0)to be the value of maximized objective function (1) for each k0(normally called value function). Under suchdefinition, v(k1) gives the value of maximized objective function of planning problem:

max∞∑t=1

βt−1U(f (kt )− kt+1)

s.t 0 ≤ kt+1 ≤ f (kt )

In the end, problem (1)-(3) can be written as:

v(k0) = maxc0,k1

[U(c0) + βv(k1)]

s.t.c0 + k1 ≤ f (k0)

k0 > 0

This is an equation of function since the unknown is v(·). We treat k0 as some predetermined state. Then thestarting date doesn’t really matter, we can omit time subscript and obtain Bellman equation:

v(k) = maxc,y

[U(c) + βv(y)] (4)

s.t.c + y ≤ f (k) (5)

We call the maximizer and the optimum(which are both functions) policy function and value functionReplace into c, we can obtain a functional equation of v :

v(k) = max0≤y≤f (k)

{U(f (k)− y(k)) + βv(y(k))} (6)

Computationally very appealing to cast macroeconomic model into the “recursive” form like (6)(not easy forcomplex model) so that we can use theory of dynamic programing to accelerate the model computation.

7 / 25

Computational method-Value function iteration

Objectives of computation: Solve for value function v and solve for policyfunction y = g(k)Iteration Algorithm:Start from taking an initial guess (bounded and continuous) v0, find v1 and yby solving

v1(k) = max0≤y≤f (k)

[U(f (k)− y) + βv0(y)] (7)

If ||v1 − v0|| < ε, return v0, otherwise use v1 as a new initial guess and solve forv2 and y from (5). Repeat this process until convergence and return thesolution v∗ and associated policy function g∗(k).

8 / 25

A simple computing example

Functional form:u(c) = ln c , y = Akα.1. Start from v0(k) = 0, choose c to maximize ln(c) + β × 0 subject toc + k = Akα. We obtain c = Akα and thus v1(k) = lnA + α ln k.2. Given v1, choose c to maximize ln(c) + β(lnA + α ln k) subject toc + k = Akα. We obtain c = Akα

1+αβ and thus

v2(k) = ln A1+αβ + β lnA + αβ ln αβA

1+αβ + α(1 + αβ) ln k.3. Try another few steps and by induction, we find the limiting policyfunction and value function:

c = (1− αβ)Akα

k = αβAkα

v(k) = (1− β)−1{ln[A(1− αβ)] +αβ

1− αβln(Aβα)]}+]

α

1− αβln k

9 / 25

An alternative approach-Howard’s policy improvementAlgorithm:

I Start from picking a feasible policy, y = g0(k) for each k, compute the lifetimeutility associated with using forever that policy w0(k)Exercise: Show that

wo(k) = U[f (k)− g0(k)] + βw0[g0(k)]

Solution: Lifetime utility given this policy rule and predetermined state k is:

w0(k) =∞∑t=0

βtu[f (kt)− g0(kt)] = U[f (k)− g0(k)] + β∞∑t=1

βt−1U[f (kt)− g0(kt)]

= U[f (k)− g0(k)] + βw0(g0(k))

I Generate a new policy y = g1(k) for each k that maximizes

w1(k) = maxy

[U(f (k)− y) + βw0(y)]

where I denote lifetime utility given k and policy g1(k) as w1(k)

I Iterate until convergence(lifetime utility can’t be significantly improved)

Remark: We call this “improvement” algorithm since after each iteration, we havenondecreasing lifetime utility wt(k) ≥ wt−1(k).

10 / 25

Example recap

Consider previous example, but now let’s use Howard’s method.1. Guess the initial policy function kt+1 = h0Ak

αt , then form

J0(k0) =∑∞

t=0 βt ln(Akα

t − h0Akαt ).

2. Next choose a new policy h1 by maximizingln(Akα − h1Ak

α) + βJ0(h1Akα), then

forming:J1(k0) =∑∞

t=0 βt ln(Akα

t − h1Akαt ).

3. Iterate until hj have converged.

11 / 25

An introspection of algorithm

I For sequential method we have Kuhn-Tucker condition to ensure theuniqueness of solution.

I But for our recursive method, up to now we only assume thesolution exists, but does it really exist? If so, is it unique so that ouriteration algorithms make sense?

I That’s why we need to introduce a theory of dynamic programming

For further theoretical analysis, we rewrite right hand side of (6) as:

(T [v ])(k) = max0≤y≤f (k)

[U(f (k)− y) + βv(y)]

where we actually define a new mapping T from some set of functions Cto C itself. Under this notation, solving for value function v is actuallyequivalent to finding the fixed point of mapping T . Intuitively, byapplying the fixed point theorem, it’s possible to guarantee the existenceand uniqueness property of solution to (6). However, it’s necessary todiscuss some technical conditions which let the theorem hold in dynamicprogramming problem.

12 / 25

Mathematical Preliminary-Banach SpaceSome definitions:

I A Real Vector Space X is a set of vectors together with two operations,addition and scalar multiplication, such that for any two vectors x , y ∈ X :i) x + y ∈ X and ii) for any real number α, β ∈ R, αx ∈ X . And the twooperations obey the following laws:a) x + y = y + x .b)(x + y) + z = x + (y + z). c)α(x + y) = αx +αy d) (α+ β)x = αx + βxe)(αβ)x = α(βx) f)x + 0 = x g) 0x = 0 h) 1x = x

I A Metric Space is a set S with metric (a distance measure)ρ : S × S → R such that for all x , y , z ∈ S : a) ρ(x , y) ≥ 0 where equalityholds iff x = y . b) ρ(x , y) = ρ(y , x). c) ρ(x , z) ≤ ρ(x , y) + ρ(y , z)

I A Normed Vector Space is a vector space S together with norm || · ||:S → R such that for all x , y ∈ S and α ∈ R: a) ||x || ≥ 0 with equality iffx = 0. b) ||αx || = |α|||x ||. c) ||x + y || ≤ ||x ||+ ||y ||

I A sequence {xn}∞n=0 in S is a Cauchy Sequence if for each ε > 0, thereexists N(ε) such that ρ(xn, xm) < ε for all n,m ≥ N(ε)

I A metric space (S , ρ) is Complete if every Cauchy sequence in Sconverges to an element in S .

I A complete normed vector space is called Banach Space

Remark: Normed vector space is a metric space, norm is a special metric13 / 25

Exercises 1.1

I Show that the following are vector spaces:a) any finite dimensional Euclidean space Rl

b) set of all infinite sequences (x0, x1, · · · )c) set of all continuous functions on interval [a, b]

I Show that the following are metric spaces:a) Set of integers N with metric ρ(x , y) = |x − y |b) Set of integers N with metric ρ(x , y) = 0 if x = y and ρ(x , y) = 1 if x 6= yc) Set of continuous, nondecreasing functions on [a, b] withρ(x , y) = maxt∈[a,b] |x(t)− y(t)|.

I Show that the following are normed vector spaces:

a) Rl with ||x || =√∑l

i=1 x2i (L2-norm)

b) Rl with ||x || = maxi |xi | (L∞-norm)

c) Rl with ||x || =∑l

i=1 |xi | (L1-norm)d) Set of continuous functions on [a, b] with ||x || = supa≤t≤b |x(t)| (Call thisspace C [a, b])

I Show that the last 7 spaces are all complete spaces

14 / 25

Completeness of C (X )

Theorem 1.1 Let X ⊂ Rl and let C (X ) be the set of boundedcontinuous functions f : X → R with norm definedas:||f || = supx∈X |f (x)|. Then C (X ) is a complete normed vectorspace.Sketch of proof: We have shown that C (X ) is a normed vectorspace, then it only requires to prove the completeness. That is, weseek to prove: for a Cauchy sequence {fn}, there exists f ∈ C (X )such that for any ε > 0, ∃ Nε so that ||fn − f || ≤ ε for all n > Nε.First, prove that limn→∞ fn exists and denote it by f , then show fis continuous and bounded. Boundedness of f is from theboundedness of fn and the very close distance between f and fnwhen n is large. Continuity is proved by following the definition ofcontinuous function.Remark: If X is compact, then we don’t need to impose theassumption of bounded function.

15 / 25

Mathematical Preliminary-Set basics

I Open set:Let X be a metric space. For any x ∈ X and ε > 0, defineε−neighborhood of x as: Nε(x) = {y ∈ X : ρ(x , y) < ε}. A subset S ofX is an open set if for each x ∈ S , there exists an ε > 0 such thatNε(x) ⊂ S

I Closed set:A subset S of X is a closed set if X − S is open in X

I Bounded set:A subset S of X is bounded if there exists an ε > 0 suchthat S ⊂ Nε(x) for some x ∈ S

I Compact set(not correct but fine for our study):A subset S of X iscompact iff it’s closed and bounded.

Remarks:

I If (S , ρ) is a complete metric space, S ′ a closed subset of S , then (S ′, ρ)is also a complete metric space

16 / 25

The contraction mapping theorem(CMT)Definition 1.2[Contraction Mapping] Let (S , ρ) be a metric space and T : S → S be afunction mapping S into itself. T is a contraction mapping with modulus β if for someβ ∈ (0, 1), ρ(Tx ,Ty) ≤ βρ(x , y) for all x , y ∈ STheorem 1.3[Contraction Mapping Theorem] If (S , ρ) is a complete metric space andT : S → S is a contraction mapping with modulus β, thena) T has exactly one fixed point v in Sb) for any v0 ∈ S , ρ(T nv0, v) ≤ βnρ(v0, v) for n = 0, 1, 2...

Proof:We want to show the existence and uniqueness of v ∈ S such that Tv = v .Define mappings {T n} where T 0x = x and T n = T (T n−1x). Choose v0 ∈ S anddefine {vn}∞n=0 by vn+1 = Tvn so that vn = T nv0. Since T is a contraction mapping,we have ρ(v2, v1) = ρ(Tv1,Tv0) ≤ βρ(v1, v0). By induction,ρ(vn+1, vn) ≤ βnρ(v1, v0). Thus from triangle inequality, for any m > n,

ρ(vm, vn) ≤ ρ(vm, vm−1) + · · ·+ ρ(vn+1, vn) ≤ βn

1−β ρ(v1, v0). Thus {vn} is a Cauchy

sequence, and its limit v ∈ S because S is complete.Next we show Tv = v , since for all n and v0 ∈ S ,ρ(Tv , v) ≤ ρ(Tv ,T nv0) + ρ(v ,T nv0) ≤ βρ(v ,T n−1v0) + ρ(v ,T nv0)→ 0 as n→∞,then ρ(Tv , v) = 0 and thus Tv = v .Finally we show that there is no other function v ∈ S satisfying Tv = v . Proving bycontradiction, if v 6= v , denote their distance by a, then0 < a = ρ(v , v) = ρ(Tv ,Tv) ≤ βρ(v , v) = βa. Since β < 1, then we must havea = 0, a contradiction.This theorem will be very useful to verify the existence of solutions to your dynamicprogramming problem

17 / 25

Further implication and exercises

In some cases, we may want to use CMT multiple times. The first time toshow the existence of solution, and the further usage of CMT to characterizemore properties of fixed point within a smaller range. The following twocorollaries set up foundations for doing this.Corollary 1.4:Let (S , ρ) be a complete metric space and let T : S → S be acontraction mapping with fixed point v ∈ S . If S ′ is a closed subset of S andT (S ′) ⊂ S ′, then v ∈ S ′. If in addition T (S ′) ⊂ S ′′ ⊂ S ′, then also v ∈ S ′′

Sketch of proof:for v0 ∈ S ′, {T nv0} is a sequence in S ′. Closed S ′ ensure thelimit point in S ′ itselfCorollary 1.5[N-stage contraction theorem]:Let (S , ρ) be a complete metricspace and let T : S → S . Suppose that for some integer N, TN : S → S is acontraction mapping with modulus β, then:a) T has exactly one fixed point in Sb) for any v0 ∈ S , ρ((TN)kv0, v) ≤ βkρ(v0, v)Sketch of proof:ρ(Tv , v) = ρ(T (TNv), (TNv)) = ρ(Tv , v) ≤ βρ(Tv , v), whichimplies ρ(Tv , v) = 0

18 / 25

Exercise 1.2Next exercise shows that how you can use CMT to prove existence and uniqueness ofa solution to a differential equation.But first let’s give a quick introduction todifferential equation.Differential equation treats a function as its unknown and it includes the derivatives ofthis unknown function. A class of ordinary differential equation(ODE) can be writtenx ′(t) = f (x , t).Consider the differential equation: x ′(t) = x + t. It’s easily verified thatx = Cet − t − 1 is the solution, where C is any constant. To make more sense, anadditional restriction needs to be imposed. We call this condition the boundarycondition. For example, we can impose x(0) = 0, which implies C = 1Exercise:Consider an ODE x ′(t) = f (x(t)), t ≥ 0, with boundary condition x(0) = c.Assume that f : R→ R is continuous and satisfies the Lipschitz condition, that is, forsome B > 0, |f (a)− f (b)| ≤ B|a− b| for all a, b ∈ R. For any M > 0, considerC [0,M], the space of continuous functions on [0,M] with sup norm. We have shownthat this space is complete.a) Define operator T as:

(Tv)(t) = c +

∫ t

0f [v(z)]dz 0 ≤ t ≤ M

Show that T maps C [0, t] onto itself.b) Show that for some τ > 0, T is a contraction on C [0, τ ]c) Show that the unique fixed point of T on C [0, τ ] is a differentiable function, andhence it’s the unique solution on [0, τ ] to the given ODE.

19 / 25

Blackwell’s Conditions

A useful theorem to verify a contraction mapping:Theorem 1.6[Blackwell’s sufficient conditions for a contraction] Let X ⊂ Rl and letB(X ) be a space of bounded functions f : X → R with the sup norm. LetT : B(X )→ B(X ) be an operator satisfying:a)(monotonicity) For f , g ∈ B(X ) and f (x) ≤ g(x) for all x ∈ X , we have(Tf )(x) ≤ (Tg)(x) for all x ∈ Xb)(discounting) There exists some β ∈ (0, 1) such that

[T (f + a)](x) ≤ (Tf )(x) + βa

for all f ∈ B(X ), a ≥ 0 and x ∈ X . Then T is a contraction with modulus βProof For any f , g ∈ B(X ), f ≤ g + ||f − g ||, then properties a) and b) imply thatTf ≤ T (g + +||f − g ||) ≤ Tg + β||f − g ||. Similarly, Tg ≤ Tf + β||f − g ||. Then wehave ||Tf − Tg || ≤ β||f − g ||.

Example: In our previous example, T is defined as(Tv)(k) = max0≤y≤f (k){U[f (k)− y ] + βv(y)}. Through Blackwell’s conditions, wecan show that T is a contraction mapping.For monotonicity, assume v and ω are two functions such that v ≤ ω, then maximizedvalue Tv ≤ Tω. For discounting, sinceT (v + a)(k) = max0≤y≤f (k){U[f (k)− y ] + β(v(y) + a)} = Tv(k) + βa.

20 / 25

Theorem of Maximum(TOM)

We want to have some conditions to ensure that the solution to optimizationproblem is continuous with respect to parameters as well as predeterminedstates. This is both theoretically and numerically important.Denote feasible set as Γ(x) given beginning-of-period state variable x , then ourobject of interest can be written as:

supy

f (x , y)

y ∈ Γ(x)

Let X ⊂ Rl , Y ⊂ Rm, and let f : X × Y → R be a single-valued function, letΓ : X → Y be a nonempty correspondence. Practically, f (x , ·) is continuousgiven x , and Γ(x) is nonempty and compact. Then we can define:

h(x) = maxy∈Γ(x)

f (x , y) (8)

G(x) = {y ∈ Γ(x) : f (x , y) = h(x)}. (9)

We try to seek restrictions on Γ(x) and f such that h and G are bothcontinuous. This is achived by Theorem of Maximum. However, beforestating the theorem, we need to introduce several mathematical preliminaries.

21 / 25

Some definitions and TOM

I Correspondence Γ : X → Y is a relation that assigns a set Γ(x) ∈ Y to eachx ∈ X

I A correspondence Γ : X → Y is lower hemi-continuous(l.h.c) at x if Γ(x) isnonempty and if for every y ∈ Γ(x) and every sequence xn → x , there existsN ≥ 1 and a sequence {yn}∞n=N such that yn → y and yn ∈ Γ(xn) for all n ≥ N.

I A compact-valued correspondence Γ : X → Y is upper hemi-continuous(u.h.c)at x if Γ(x) is nonempty and if for every sequence {xn} whose limit is x and forevery sequence {yn} such that yn ∈ Γ(xn) for all n, there exists a convergentsubsequence of {yn} whose limit point y is in Γ(x)

I A correspondence Γ : X → Y is continuous at x ∈ X if it’s both l.h.c and u.h.c.A correspondence Γ : X → Y is continuous if it’s continuous at every pointx ∈ X

I The graph of correspondence Γ : X → Y is defined as:A = {(x , y) ∈ X × Y : y ∈ Γ(x)}

Remark: It’s sometimes useful to deal with the graph instead of Γ(x). For example, if

A is closed, and for any bounded X ⊂ X , Γ(X ) is bounded, then Γ is compact-valuedand u.h.cTheorem 1.7[Theorem of Maximum] Let X ⊂ Rl and Y ⊂ Rm, let f : X × Y → R bea continuous function and let Γ : X → Y be a compact valued and continuouscorrespondence. Then function h : X → R defined in (8) is continuous, andcorrespondence G : X → Y defined in (9) is nonempty, compact valued and u.h.c

22 / 25

Exercises 1.3

I Γ is single-valued and u.h.c, then it’s continuous

I Γ is single-valued and l.h.c, then it’s continuous

I Let Γ : X → Y be a compact-valued u.h.c correspondence with graph A.If X is compact, then A is compact.(Use the fact that a metric space iscompact if it has Bolzano-Weierstrass property, that is, every sequencehas a convergent subsequence)

I Let Γ : R+ → R+ be defined by Γ(x) = [0, x ], then Γ is continuous

I Let f : Rl+ → R+ be a continuous function, define correspondenceΓ : Rl+ → R+ by Γ(x) = [0, f (x)]. Show that Γ is continuous

I Let fi : Rl+ × Rm → R+, i = 1, 2 . . . l be continuous functions. DefineΓ : Rl+ × Rm → Rl+ by:

Γ(x , z) = {y ∈ Rl+ : 0 ≤ yi ≤ fi (xi , z), i = 1, . . . l ,

l∑i=1

x i ≤ x}

Then Γ is continuous

23 / 25

What does TOM not say? Some Examples

I Let X = R and let Γ(x) = Y = [−1, 1], for all x ∈ X . Definef : X × Y → R by f (x , y) = xy 2. Then G(x) is u.h.c but not l.h.c atx = 0

I Let X = R and let Γ(x) = [0, 4] for all x . Definef (x , y) = max{2− (y − 1)2, x + 1− (y − 2)2}. Then G(x) is u.h.c butnot l.h.c at ?

I Let X = R+, Γ(x) = {y ∈ R : −x ≤ y ≤ x} and f (x , y) = cos(y). ThenG(x) is u.h.c but not l.h.c at?

24 / 25

Foundation for value function iterationIf Γ is convex-valued and f is strictly concave. Then G is single-valued andthus continuous. Here we have two theorems related to the property of G .Lemma 1.8 Let X ⊂ Rl and Y ⊂ Rm. Assume that the correspondenceΓ : X → Y is nonempty, compact and convex-valued, and continuous. Let A bethe graph of Γ. Assume function f : A→ R is continuous and f (x , ·) is strictlyconcave for each x ∈ X . Define function G : X → Y by

G(x) = arg maxy∈Γ(x)

f (x , y)

Then for each ε > 0 and x ∈ X , there exists δx > 0 such that

y ∈ Γ(x), |f [x ,G(x)]− f (x , y)| < δx ⇒ ||G(x)− y || < ε

If X is compact, then δ > 0 can be chosen independently of x .Theorem 1.9 Let X ,Y , Γ,A be defined as before. Let {fn} be a sequence ofcontinuous functions on A. Assume fn(x , ·) is strictly concave given each n andeach x . Assume further that fn → f uniformly. Define function Gn and G by:

Gn(x) = arg maxy∈Γ(x)

fn(x , y)

G(x) = arg maxy∈Γ(x)

f (x , y)

Then Gn → G pointwise. If X is compact-valued, then Gn → G uniformly.25 / 25

Lecture 2Mathematics for dynamic programing II

ZENG MING

March 30, 2016

1 / 19

Introduction

The problem (SP) we are interested in:

sup{xt+1}∞t=0

∞∑t=0

βtF (xt , xt+1)

s.t.xt+1 ∈ Γ(xt)

x0 ∈ X

As shown before, we have a functional equation(FE) formulation ofabove problem:

v(x) = supy∈Γ(x)

[F (x , y) + βv(y)], x ∈ X

What is the relationship between those two solutions? If they areequivalent, we only need to study the recursive formulation, suchas the property of value function and policy function.

2 / 19

Roadmap

I Principle of optimality: Solution to FE, evaluated at x0, givesthe value of supremum in SP when the initial state is x0 andthat a sequence {xt+1}∞t=0 attains the supremum in SP iff itsatisfies

v(xt) = F (xt , xt+1) + βv(xt+1)

I The converse direction is partially true since we need toimpose a boundedness condition

I After the relation between value function of FE andsupremum of SP, we discuss the relation between policy rulesof those two problems

I Under some more specific cases, we study properties of valuefunction and policy rules

I Envelope Theorem

I Problem I-Recursive Preference

I Problem II-Investment with convex cost3 / 19

Notations

Let X be the set of possible values for state variable x , Γ : X → Xbe the correspondence describing the feasibility constraints,A = {(x , y) ∈ X × X : y ∈ Γ(x)} be the graph of Γ.We first impose conditions under which SP is well defined, that is,the feasible set is nonempty and the objective function is welldefined for every point in the feasible setDefine any sequences {xt}∞t=0 ⊂ X a plan. Define set of feasibleplans as:

Π(x0) = {{xt}∞t=0 : xt+1 ∈ Γ(xt)}

given x0 ∈ XAssumption 2.1 Γ(x) is nonempty for all x ∈ XAssumption 2.2 For all x0 ∈ X and x ∈ Π(x0),limn→∞

∑nt=0 β

tF (xt , xt+1) exists(can be infinity)

4 / 19

Notation(cont’d)

If those two assumptions hold, then feasible set Π(x0) is nonemptyfor all x0 ∈ X and the objective function in SP is well defined forevery plan in the feasible set. Let u(x) =

∑∞t=0 β

tF (xt , xt+1), wethen define the supremum function v∗ : X → R by:

v∗(x0) = supx∈Π(x0)

u(x)

Then v∗ is the unique function satisfying:i)If |v∗(x0)| <∞, then v∗(x0) ≥ u(x) for all x ∈ Π(x0). And forany ε > 0, there exists x ∈ Π(x0) such that v∗(x0) ≤ u(x) + ε forsome y ∈ Γ(x0)ii)If v∗(x0) = +∞, then there exists a sequence {xk}∞k=1 in Π(x0)such that limk→∞ u(xk) = +∞iii)If v∗(x0) = −∞, then u(x) = −∞ for any x ∈ Π(x0)

5 / 19

Principle of OptimalityWe say v∗ satisfies FE if three conditions hold:i)If |v∗(x0)| <∞, then v∗(x0) ≥ F (x0, y) + βv∗(y) for all y ∈ Π(x0). And for anyε > 0, there exists y ∈ Π(x0) such that v∗(x0) ≤ F (x0, y) + βv∗(y) + εii)If v∗(x0) = +∞, then there exists a sequence {yk}∞k=1 in Γ(x0) such thatlimk→∞[F (x0, yk ) + βv∗(yk )] = +∞iii)If v∗(x0) = −∞, then F (x0, y) + βv∗(y) = −∞ for any y ∈ Γ(x0)Lemma 2.1 Let X ,F , β satisfy assumption 2.2, then for any x0 ∈ X andx = (x0, x1, . . .) ∈ Π(x0),

u(x) = F (x0, x1) + βu(x ′)

where x ′ = (x1, x2, . . .)Idea of proof: Noticelimn→∞

∑nt=0 β

tF (xt , xt+1) = F (x0, x1) + β limn→∞∑n

t=0 βtF (xt+1, xt+2)

Theorem 2.2[Principle of Optimality:Solution to SP is solution to FE]Let X , Γ,F , β satisfy assumption 2.1 and 2.2, then function v∗ satisfies FEIdea of proof: Verify conditions above.Remark: Given initial state, no improvement can be made in terms of lifetimeutility(logic of SP)Whatever the predetermined state is, remaining decisions must be optimal under thispredetermined state(logic of FE).You see that those two logics are not identical, though they are the same undertwo-period setting. Then theorem 2.2 establishes assumptions such that the lifetimeoptimal choice will also be sequentially optimal.

6 / 19

Does solution to FE also satisfy SP?

The principle of optimality only gives us the necessary conditionwhich we can focus on FE instead of SP. However, the sufficientcondition doesn’t hold in some cases since FE may have othersolutions not satisfy SP.Example:Consider the SP problem:

max{(ct ,xt+1)}∞t=0

∞∑t=0

βtct

s.t.0 ≤ ct ≤ xt − βxt+1

Agent can borrow at the interest rate 1β − 1 and doesn’t have any

borrowing constraint. Denote the wealth at time t as xt , then nextperiod wealth will be xt+1 = (1 + 1

β − 1)(xt − ct). Since there areno borrowing constraint, agent can actually borrow as much as shewants, consumption is then unbounded and it’s obvious that wehave v∗(x) = +∞.

7 / 19

Cont’d

Now write the problem in recursive form:

v(x) = supy∈Γ(x)

[x − βy + βv(y)], x ∈ X

where Γ(x) = (−∞, β−1x ]Notice v∗ =∞ satisfies this equation. However, function v(x) = xalso satisfy FE. But this solution doesn’t satisfy SP.We need to impose more conditions such that the converse is true.Theorem 2.3[Converse:Solution to FE is solution to SP]Let X , Γ,F , β satisfy assumption 2.1 and 2.2, if solution of FE vsatisfies: limn→∞ β

nv(xn) = 0, for all (x0, x1, . . .) ∈ Π(x0), thenv = v∗

8 / 19

Optimal plan and Optimal policy

After discussing the value function, let’s characterize relationship betweenoptimal feasible plans(SP) and optimal feasible policy(FE). The followingtheorem first establish the principle of optimality, then it shows that optimalplan is also an optimal policyTheorem 2.4[Principle of optimality-optimizer]Let X , Γ,F , β satisfy assumption 1 and 2, let x∗ = (x0, x

∗1 , . . .) ∈ Π(x0) be an

optimal plan for initial state x0, then:

v∗(x∗t ) = F (x∗t , x∗t+1) + βv∗(x∗t+1) (1)

The converse also holds:Theorem 2.5[Converse]Let X , Γ,F , β satisfy assumption 1 and 2, let x∗ = (x0, x

∗1 , . . .) ∈ Π(x0) be a

feasible plan for initial state x0, satisfying (1), with limt→∞ supβtv∗(x∗t ) ≤ 0,then x∗ attains the supremum in (SP) for initial state x0

9 / 19

Properties of Value function-When F is boundedAfter showing equivalence between SP and FE to a large extent, we can safely focuson FE.We study functional equation of the form:

v(x) = maxy∈Γ(x)

[F (x , y) + βv(y)] (2)

Denote A = {(x , y) ∈ X × X : y ∈ Γ(x)} be the graph of Γ.Assumption 2.3 X is a convex subset of Rl , correspondence Γ : X → X is nonempty,compact-valued and continuousAssumption 2.4 The function F is bounded and continuous, 0 < β < 1Under those two assumptions, assumption 2.1 and 2.2. will automatically be satisfied.Suppose B is a bound for |F (x , y)|, then the supremum function v∗ satisfies|v∗(x)| ≤ B

1−β . Then our objective is to search for a solution to (2) in the space of

bounded and continuous functions C(X ) with sup norm ||f || = supx∈X |f (x)|. Given asolution v to (2), we can define the policy correspondence G : X → X by:

G(x) = {y ∈ Γ(x) : v(x) = F (x , y) + βv(y)} (3)

Also we define the operator T on C(X ) by:

(Tf )(x) = maxy∈Γ(x)

[F (x , y) + βf (y)]

10 / 19

Cont’d

The following theorem shows that value function exists and isunique, also policy correspondence is compact-valued.Theorem 2.6[Existence and Uniqueness of value function]Let X , Γ,F , β satisfy Assumption 2.3 and 2.4, and let C (X ) be thespace of bounded continuous function f with sup norm. Thenoperator T has a unique fixed point v ∈ C (X ), and for allv0 ∈ C (X ):

||T nv0 − v || ≤ βn||v0 − v ||

Moreover, given v , the optimal policy correspondence G : X → Xdefined by (3) is compact-valued and u.h.c

11 / 19

Monotonicity and Convexity

To characterize v and G more sharply, we impose more restrictions on F and Γ.Assumption 2.5 For each y , F (·, y) is strictly increasing in each of its first l argument.Assumption 2.6 Γ is monotone in the sense that x ≤ x ′ implies Γ(x) ⊂ Γ(x ′)Theorem 2.7[Monotonicity]Let X , Γ,F , β satisfy Assumption 2.3-2.6, and let v be the unique solution to (2).Then v is strictly increasingAssumption 2.7 F is strictly concave, that is:

F [θ(x , y) + (1− θ)(x ′, y ′)] ≥ θF (x , y) + (1− θ)F (x ′, y ′)

and the inequality is strict if x 6= x ′

Assumption 2.8 Γ is convex in the sense that for any 0 ≤ θ ≤ l and x , x ′ ∈ X ,

y ∈ Γ(x), y ′ ∈ Γ(x ′)

implies θy + (1− θ)y ′ ∈ Γ[θx + (1− θ)x ′]Theorem 2.8[Convexity] Let X , Γ,F , β satisfy Assumption 1.3-1.4 and 1.7-1.8, and letv be the solution to (2) and G satisfies (3). Then v is strictly concave and G is acontinuous, single-valued function.

12 / 19

First order condition(FOC) under recursive form

Even though computationally it’s useful to work with FE instead of SP. Qualitatively,this may not be the case if we don’t have some “tractable conditions” to catch theeconomic sense of our model(such as Euler equation obtained by solving SP). Thefollowing discussion allows us to do this under recursive formulation, that is, we canobtain exactly the same Euler equation as in SP by combining the first order conditionand the envelope theoremSuppose we work with FE formulation:

v(x) = max0≤y≤f (x)

{U[f (x)− y ] + βv(y)}

If v is differentiable, we can actually obtain policy functions implicitly by the followingFOC(differentiate both side w.r.t y):

U′[f (x)− g(x)]− βv ′[g(x)] = 0

It’s clear that differentiability of U, f , v are quite important for such classical methodto work. We can assume properties for U and f , nonetheless assumptions about v willbe problematic. Also the equation is not useful unless we know v ′(x). Out next task isto first describe conditions under which v is differentiable, then to show how to obtainv ′.

13 / 19

Envelope TheoremTheorem 2.9[Benveniste and Scheinkman]Let X ⊂ Rl a convex subset, V : X → R be concave, let x0 ∈ intXand D be a neighborhood of x0. If there exists a concave anddifferentiable function W : D → R with W (x0) = V (x0) andW (x) ≤ V (x) for all x ∈ D, then V is differentiable at x0 andVi (x0) = Wi (x0)Assumption 2.9 F is continuously differentiable on the interior ofATheorem 2.10[Differentiability of value function-Envelopetheorem]Let X , Γ,F , β satisfy Assumption 2.3-2.4 and 2.7-2.9, and let v bethe solution to (2) and G satisfies (3). Moreover, assume F iscontinuously differentiable on intA. If x0 ∈ intX andg(x0) ∈ intΓ(x0), then v is continuously differentiable at x0, withderivative given by:

vi (x0) = Fi [x0, g(x0)], i = 1, · · · l

14 / 19

Cont’dRemark: The envelope theorem is about the change of value functionas we change the parameters or predetermined states, but not controlvariables. This is quite useful since we can then ignore max and directlycalculate the derivative v ′(·)Example:Again consider the FE equation:

v(kt) = max0≤kt+1≤f (kt)

{U[f (kt)− kt+1] + βv(kt+1)}

We already show that the first order condition:

U ′[f (kt)− kt+1]− βv ′[kt+1] = 0

To calculate v ′(·), we apply the envelope theorem:

v ′(kt) = U ′[f (kt)− kt+1]f ′(kt)

In the end, we obtain the Euler equation:

U ′[f (kt)− kt+1] = βU ′[f (kt+1 − kt+2)]f ′(kt+1)

which coincides with Euler equation obtained in Lecture 1 usingsequential method.

15 / 19

Problem I-Recursive Preference

Let L be the space of sequences c = (c0, c1, . . .) with ct ∈ Rl+ that are bounded in the

norm ||c||L = supt ||ct ||E , where || · ||E is the L2-norm on Rl . For anyc = (c0, c1, . . .) ∈ L, define:

1c = (c1, c2, . . .) ∈ L

cn = (c0, . . . , cn, 0, . . .) ∈ L

We have so far been dealing with preference u : L→ R of the formu(c) =

∑∞t=0 β

tU(ct), where U is continuous and bounded.a) Show that function with the above form u : L→ R is bounded and continuous inthe norm || · ||Lb) Let S be the vector space of all bounded continuous functions u : L→ R with norm||u|| = supc∈L |u(c)|. From the exercise in lecture 1, S is complete. Is it true thatlimn→∞ |u(c)− u(cn)| = 0 for all u ∈ S?c) For any β ∈ (0, 1) and U : Rl

+ → R, we can define an operator T : S → S byTu = U(c0) + βu(1c). Define u ∈ S as:

u(c) =∞∑t=0

βtU(ct)

show that u is the unique fixed point of T

16 / 19

Recursive Preference(cont’d)In addition to form of u, a much larger class of utility functions on L can be defined inthe following way.Let W : Rl

+ × R+ → R+ be a continuous function with properties:

(W1): W (0, 0) = 0(W2): for any z ∈ R+, W (·, z) : Rl

+ → R+ is bounded

(W3): for some β ∈ (0, 1), |W (x , z)−W (x , z ′)| ≤ β|z − z ′| , all x ∈ Rl+ and

z, z ′ ∈ R+

We then call function W as an aggregator function. W (x , z) can be interpreted as thelifetime utility if we consume x today and consume z in the rest of life.Two more properties of W will be useful:

(W4): W is increasing(W5): W is concave

d) Let W satisfy (W1)-(W3), define the operator Tw : S → S by(Twu)(c) = W [c0, u(1c)]. Show that Tw has a unique fixed point uw ∈ S and:

|uw (c)− uw (cn)| ≤ βn||uw || c ∈ L

Moreover, show that if (W4) holds then uw is increasing, and if (W5) holds then uw isconcavee) Assume W is continuously differentiable, obtain an expression for the marginal rateof substitution between ci,t and cj,t+k

17 / 19

Problem II-Investment with convex costsA firm has production technology z = f (k). Assume f is continuously differentiable,strictly increasing and strictly concave and that:

f (0) = 0

limk→0

f ′(k) = +∞

limk→∞

f ′(k) = 0

Assume p is the output price, q is the price of capital, r is the interest rate, setβ = 1

1+r.

Assume that capital must be purchased one period in advance and depreciates at rateδ ∈ (0, 1). Then firm’s problem given k0 is:

max{kt+1}∞t=0

∞∑t=0

βt{pf (kt)− q[kt+1 − (1− δ)kt ]}

Consider the recursive form:

v(k) = maxy{pf (k)− q[y − (1− δ)k] + βv(y)}

18 / 19

Investment(cont’d)

a) Find the exact solutions for v and for policy g . Derive the Euler equationb) What is the economic interpretation of the absence of a nonnegativityconstraint on gross investment? How our results will be altered if there arelower and upper bounds on gross investment, such as(1− δ)kt ≤ kt+1 ≤ (1− δ)kt + a, a > 0?c) Suppose now the total cost of investment is convex function c(·), where c isstrictly increasing, strictly convex and differentiable, with c(0) = 0. Then thefirm’s problem is:

max{kt+1}∞t=0

∞∑t=0

βt{pf (kt)− c[kt+1 − (1− δ)kt ]}

Write this in recursive form, and characterize the value function v and policyfunction g . Derive Euler equation.

19 / 19

Lecture 3Deterministic Dynamics

ZENG MING

March 30, 2016

1 / 23

Introduction

We have studied intensively the functional equation(FE) formulation:

v(x) = maxy∈Γ(x)

{F (x , y) + βv(y)}

We have estabilished:

I Existence of unique solution to FE

I Qualitative property of value function and policy function

I First order and envelope conditions of FE:

Fy (x , g(x)) + βv ′(g(x)) = 0

v ′(x) = Fx(x , g(x))

Before solving the model numerically, is it possible that we can characterize thebehavior of solutions? Are we able to understand more about the qualitativefacts of solution besides properties mentioned before, especially solution pathdynamics?

2 / 23

Roadmap

I Solution dynamics-a simple example

I An introduction to Difference Equation

I Linear System Dynamics

I Linear Quadratic Control

I Pertubation and Blanchard Kahn’s method

3 / 23

One Dimensional ExamplesRecall that one sector model of optimal growth leads to the functional equation:

v(k) = max0≤y≤f (k)

{U[f (k)− y ] + βv(y)} (1)

Assume f and U are strictly increasing, strictly concave and continuouslydifferentiable, with:

f (0) = 0, limk→0

f ′(k) = +∞

limk→∞

f ′(k) = 0, limc→0

U ′(c) = +∞

Those assumptions will imply a unique positive value k such that f (k) = k.Moreover, this is the maximum sustainable capital stock. Suppose kt > k, thenkt+1 ≤ f (kt) < kt . We then only consider interval [0, k]. From previouslectures, we have the following properties:

I (1) has a unique bounded continuous solution v on [0, k], such v isstrictly increasing and strictly concave.

I The maximum in (1) is attained at a unique policy g(k), where g is acontinuous function

I Given k0 ∈ [0, k], sequence {kt} defined by kt+1 = g(kt) uniquely solves(1)

4 / 23

Stationary Points

Before exploring the dynamics of policy function g , first let’s characterize thestationary points such that g(kt) = kt .

I Trivial stationary point: g(0) = 0 (economic system starts from k0 = 0)

I A more interesting stationary point is found by considering a strictlypositive k0. We write first order and envelope conditions for (1):

U ′[f (k)− g(k)] = βv ′[g(k)] (2)

v ′(k) = U ′[f (k)− g(k)]f ′(k) (3)

which gives a k∗ > 0 such that βf ′(k∗) = 1.

I The strict concavity of v implies: [v ′(k)− v ′(g(k))][k − g(k)] ≤ 0, orequivalently [f ′(k)− 1/β][k − g(k)] ≤ 0, with equality iff g(k) = k

Exercise 3.1: Using (2), show that g is strictly increasingWe summarize the result below:Result 3.2 Let f ,U, β satisfy assumptions above, and let g be the policyfunction for (1). Then g has two stationary points, k = 0 and k∗ = f −1′(1/β).And for any k0 ∈ (0, k], the sequence {kt}∞t=0 defined by kt+1 = g(kt)converges monotonically to k∗

5 / 23

Example(cont’d)

Note that the system dynamics is driven by:

U ′[f (k)− g(k)] = βv ′[g(k)]

v ′(k) = U ′[f (k)− g(k)]f ′(k)

we can write themas:U ′[f (kt)− kt+1] = βU ′[f (kt+1)− kt+2]f ′(kt+1), which is asecond-order nonlinear difference equation(Note that from SP, wecan also get the same Euler equation). To analyze more generaland complicated system dynamics, we need to have a nice theoryof difference equations and dynamic system. Here we will focus onthe linear system. As we will show later, even working withnonlinear model, its linear approximation is often quite useful forqualitative analysis .

6 / 23

Difference EquationsSuppose we are interested in the model:

yt = φyt−1 + wt

This is called a linear first order difference equation, which is anequation relating a variable yt to its previous value.Normally it’s solved by recursive substitution and it’s easy to show:

yt = φty0 + φt−1w1 + φt−2w2 + · · ·+ φwt−1 + wt

Note that above equality makes it easy to calculate the effect ofchanging w1 on yt(impulse response or dynamic multiplier):∂yt∂w1

= φt−1. More generally, we have:∂yt+j

∂wt= φj , that is, the

impulse response doesn’t depend on the t but instead depend onthe length of time separating the disturbance to wt and theobserved value of output (yt+j).Remark: Whether |φ| is larger or smaller than 1 is quite crucial forthe stability of system.

7 / 23

Difference Equations(cont’d)

More generally, we have the p-th order difference equation:

yt = φ1yt−1 + φ2yt−2 + · · ·+ φpyt−p + wt

Define ξt = [yt , yt−1, · · · yt−p+1]′, then p-th order differenceequation can be written as a first order equation:

ξt = F ξt−1 + vt

where F =

φ1 φ2 φ3 · · · φp−1 φp1 0 0 · · · 0 00 1 0 · · · 0 0· · · · · · · ·· · · · · · · ·0 0 0 · · · 1 0

, vt =

wt

0··0

8 / 23

Linear SystemsWe now consider linear model and analyze its stability.The systems of linear difference equations are of the form:

xt+1 = a0 + Axt (4)

where xt ∈ Rn.What motivates the use of linear system? Linear system corresponds to theequilibrium conditions in the economy when F in functional equation isquadratic and Γ(x) = Rn, in this case the optimal policy function g is linear.Also in general, one can approximate nonlinear model by a linear model.Assume I − A is nonsingular, denote x = (I − A)−1a0 the unique stationarypoint, then the deviation zt = xt − x satisfies: zt+1 = Azt .The dynamics of policy function are now determined by At , t = 0, 1 . . ..Since any square matrix can be written in Jordan form, we represent A by:

A = B−1ΛB (5)

Theorem 3.1[Stability of Linear System] Suppose I − A is nonsingular, and letx = (I − A)−1a0. Then limt→∞ xt = x for every sequence satisfying (4), iff thecharacteristic roots of A are all less than one in absolute valueThen the solution can be written as:

xt = x + B−1ΛtB(x0 − x)

9 / 23

A less restricted case

If some of the characteristic roots of A have absolute value greater or equal toone, then global stability is not guaranteed since now the convergence dependson the initial value x0. We want to characterize the set of x0 such that theconvergence is established.Theorem 3.2[When some roots have absolute value larger than one] SupposeI − A is nonsingular, and let x = (I − A)−1a0. Let B be a nonsingular matrixand Λ a Jordan matrix satisfying (5), and suppose that the first m diagonalelements of Λ are less than one in absolute value, the last n −m are greater orequal to one. Let {xt} be a sequence satisfying (4), then limt→∞ xt = x iff x0

satisfies:

x0 = x + B−1ω0

where w0i = 0, i = m + 1, . . . nRemark: As we will see later, this theorem is closely connected to Blanchardand Kahn’s method of solving rational expectation models

10 / 23

Nonlinear case

The stability theory of linear system has counterparts for nonlinear systems.Let X ⊂ Rn, let h : X → X where xt+1 = h(xt). Let x be a stationary point ofh. The idea is to find a linear approximation to h at x , so that the solution tothe resulting linear system is close to the solution to the nonlinear problem.Theorem 3.3[Nonlinear counterparts-Local result] Let x be a stationary pointof xt+1 = h(xt) and suppose h is continously differentiable in a neighborhood Nof x . Let A = [hi

j(x)] be the Jacobian matrix of first derivative of h:(h1, . . . hn)evaluated at x , assume I − A is nonsigular. Then if n characteristic roots of Aare all less than one in absolute value, and there exists a neighborhood U ⊂ Nsuch that if {xt} is a solution to xt+1 = h(xt) with xo ∈ U, then limt→∞ xt = x

11 / 23

Some results about SPOur previous discussion mainly focus on the FE formulation of economicmodel. Nevertheless, since Euler equation plays a central role in the analysis ofsolution dynamics. We wish to directly work with SP formulation.

sup{xt+1}∞t=0

∞∑t=0

βtF (xt , xt+1)

s.t.xt+1 ∈ Γ(xt)

We know that Lagrangian method will help us obtain the necessary conditionof optimal plan. Another approach is the variational approach. The idea is thata feasible variation on the optimal plan {x∗t+1}∞t=0 at one date can’t lead to animprovement. The necessary conditions can be solved from:

maxy

[F (x∗t , y) + βF (y , x∗t+2)]

s.t.y ∈ Γ(x∗t ), x∗t+2 ∈ Γ(y)

Thus we have:

Fy (x∗t , x∗t+1) + βFx(x∗t+1, x

∗t+2) = 0 (6)

This is a second order difference equations, which requires two boundaryconditions to identify the unique solution.

12 / 23

Transversality condition

We already have the initial condition x0 as the first boundarycondition. The other condition is proposed at the “infinity”:

limt→∞

βtFx(x∗t , x∗t+1)x∗t = 0 (7)

This condition has the following interpretation:Fx represents themarginal returns from increases in the predetermined states(suchas capital price in our growth model when xt is the current capitalstock kt), then this condition basically restricts the present valueof “capital stock” at t goes to zero as t goes to infinity.We then have the following theorem;Theorem 3.4[Sufficiency of the Euler and transversality conditions]Let X ⊂ Rl

+ and F satisfy assumptions 2.3-2.5,2.7 and 2.9. Thenthe sequence {x∗t+1}∞t=0 with x∗t+1 ∈ intΓ(x∗t ) is optimal for SPgiven x0 if it satisfies (6) and (7)

13 / 23

Euler Equations

In many cases, equilibrium conditions of economic model in the end is anonlinear dynamic system. As mentioned before, we only have a limited theoryof nonlinear system. Fortunately, there are some cases when we obtain thelinear system.From our previous discussion, we know that sufficient conditions for an interiorsolution to sequential problem are Euler equations and transversality condition:

Fy (xt , xt+1) + βFx(xt+1, xt+2) = 0 (8)

limt→∞

βtFx(xt , xt+1)xt = 0 (9)

For further analysis, we impose the following condition:There exists a point x ∈ X and neighborhood U of x such that for everyx0 ∈ U, there exists a sequence {xt}∞t=0 satisfying (8) and with xt → x(notethat (9) is automatically satisfied with convergent series)We will first consider the case when F is quadratic so that Euler equations arelinear(linear quadratic control). We then give some results for more general F .

14 / 23

Quadratic FConsider a special case when F is quadratic, such that Euler equations are linear.Then we can study the stability from our discussion of linear system.Obviously we have the formula for first order derivative:

Fx (x , y) = Fx + Fxxx + Fxyy

Fy (x , y) = Fy + Fyxx + Fyyy

Then Euler equation (8) becomes:

Fy + βFx + Fyxxt + (Fyy + βFxx )xt+1 + βFxyxt+2 = 0 (10)

Define zt = xt − x then (10) can be written as:

β−1F−1xy Fyxzt + β−1F−1

xy (Fyy + βFxx )zt+1 + zt+2 = 0

Convert the second order equation to first order equation:[zt+2

zt+1

]= Zt+1 = AZt =

[J KI 0

] [zt+1

zt

](11)

where J = −β−1F−1xy (Fyy + βFxx ), K = −β−1F−1

xy F′xy

Exercise 3.3: Nonsigularity of (I − A) and A are guaranteed by nonsigularity of Fxy

and (Fyx + Fyy + βFxx + βFxy )

15 / 23

Global stability of linear quadratic model

Lemma 3.5 If we have nonsigularity above and if λ is a characteristic root ofA, then (βλ)−1 is also a characteristic root.

A is a 2l × 2l matrix, then we have l roots greater than or equal to β−1/2,which is larger than 1, while other l roots whose norm are smaller than β−1/2.In other word, no more than l roots are smaller than one in absolute value.Moreover, if we have exactly l roots smaller than one, then linear quadraticmodel is globally stable and we obtain the following result:Theorem 3.6Let F : R2l → R be a strictly concave, quadratic function. Let Γ(x) = R,0 < β < 1. Assume Fxy and (Fyx + Fyy + βFxx + βFxy ) are nonsigular, let x bethe unique stationary point, assume A has l characteristic roots less than onein absolute value. Then for every x0 ∈ Rl , there exists a unique solution {xt} toSP, this sequence satisfies (10) and limt→∞ xt = xExercise 3.4: Show that g(x) = x − B−1

21 B22(x − x) is the policy functionRemark: Convergence speed is determined by the value of the largestcharacteristic root that is smaller than one in absolute value.

16 / 23

A more general F

Similar to theorem 3.3, if we choose other form of return functionF , we still have the stability property.Theorem 3.7 Assume (X ,Λ,A,F , β) satisfy assumption2.3-2.4,2.7-2.9, and let x be an interior stationary point of (1).Assume that F is twice continuously differentiable in aneighborhood N of (x , x). Let Fxx ,Fxy ,Fyy be the matrices ofsecond order derivative of F evaluated at (x , x). And assume Fxyand (Fyx + Fyy + βFxx + βFxy ) are nonsigular. Define the matrix Aas in (11), assume A has l characteristic roots less than one inabsolute value. Then for any x0 sufficiently close to x , the uniquesolution {xt} to SP satisfies limxt→∞ xt = x

17 / 23

ExampleEuler equation is:

U′[f (kt)− kt+1] = βf ′(kt+1)U′[f (kt+1)− kt+2]

and steady state is k∗ such that βf ′(k∗) = 1. Taking first order approximation aroundk∗, we find that:

−β−1U′′(kt − k∗) + [(1 + β−1)U′′ + βf ′′U′](kt+1 − k∗)− U′′(kt+2 − k∗) = 0

then we write the linear approximation in the form of (7):[zt+2

zt+1

]=

[1 + β−1 + f ′′/f ′

U′′/U′ −β−1

I 0

][zt+1

zt

]The characteristic polynomial for A is:

λ2 − [1 + β−1 +f ′′/f ′

U′′/U′]λ+ β−1 = 0

It’s easy to verify that one root is less than one(call it λ1) and the other exceeds β−1

We find the diagonalizing matrix B to be:[b11 −λ1b11

−λ−11 b22 b22

]Thus in the neighborhood of k∗, optimal paths for capital behave approximately askt+1 = k∗ + λ1(kt − k∗).

18 / 23

Pertubation method

The key idea of above example is to linearly approximate the Eulerequation. This is actually known as Pertubation method, which isone of the most popular numerical method for solvingmacroeconomic models. Computationally, pertubation is mucheasier to implement compared to value function iteration(DYNAREwill do everything once we type in the model). Moreover, valuefunction iteration requires the problem to be casted in recursiveform, which is tough for many models. Lastly, VFI will mostly beuseful to solve social planner’s problem. However, it coincides withthe model solution only if the welfare theorem holds. In distortedeconomy, this is generally not true.In the subsequent analysis,

19 / 23

Example:Ramsey’s optimal growth model

Consider the model equilibrium conditions:

yt = Atkαt

kt+1 = (1− δ)kt + it

Et(β(ct+1

ct)−γ(αAt+1k

α−1t+1 + 1− δ)) = 1

ct + it = yt

logAt+1 = ρ logAt + εt+1

Exercise:Loglinearize the above equilibrium conditions, from which we obtain:

yt+1 − at+1 − αkt+1 = 0

kt+1 − δk kt − δi it = 0

θ1cEt ct+1 + θaEt at+1 + θkEt kt+1 + θ2c cc = 0

yt+1 − γc ct+1 − γi it+1 = 0

at+1 − ρat = εt+1

20 / 23

Blanchard & Kahn method

Notice that pertubation method starts from equilibrium conditionsof the model, which generally can be written as:

Γ(Etzt+1, zt , vt+1) = 0 (12)

Blanchard and Kahn(1980) propose the solving method of themodel: [

x1,t+1

Et(x2,t+1)

]= A

[x1,t

x2,t

]+ Eft (13)

where the model variables have been divided into n1 × 1 vector ofendogenous predetermined variables x1t and n2 × 1 vector ofendogenous nonpredetermined variables x2t , ft contains exogenousvariables

21 / 23

Steps of BKI Identify a subset of variables that determine the model. Here we use {ct , kt , at}I Write the model in form (13). Here we have:[

kt+1

Et(ct+1)

]= A

[ktct

]+ Eat (14)

I Jordan decompose A = B−1ΛB, where the diagonal of Λ consists of eigenvaluesof A, are ordered in increasing absolute value from left to right. Write Λ as:

Λ =

[Λ1 00 Λ2

]where eigenvalues of Λ1 lie on or within the unit circle, while those of Λ2 lieoutside the unit circle.

I Partition B and E accordingly:

B =

[B11 B12

B21 B22

]E =

[E1

E2

]I Assume solution exists and is unique, write (14) as:[

kt+1

Et(ct+1)

]= B−1ΛB

[ktct

]+ Eat

22 / 23

BK cont’d

I Multiply both side by B:[˜kt+1

Et(˜ct+1)

]=

[Λ1 00 Λ2

] [ ˜kt˜ct

]+ [

D1

D2]at (15)

Note that after this transformation, nonpredetermined variables dependonly on the unstable eigenvalues of A contained in Λ2

I We then derive solution for the nonpredetermined variables by performinga forward iteration on the lower partion of (15):˜ct = Λ−1

2 Et(˜ct+1)− Λ−12 D2at Exercise: Show that

˜ct = −∑∞

i=0 Λ−i+12 D2Et(at+i ), then show that

ct = −B−122 B21kt − B−1

22

∑∞i=0 Λ−i+1

2 D2Et(at+i )

I In the end we have ct = −B−122 B21kt − B−1

22 Λ−12 (I − ρΛ−1

2 D2)−1at , andthen we can solve for nonexplosive portion kt+1 using (14)

Remark:If the number of explosive eigenvalues is equal to the number pfnonpredetermined variables, the system is said to be saddle-path stable and aunique solution exists. If number of explosive eigenvalues exceeds the numberof nonpredetermined state, then no solution exists

23 / 23

Lecture 4Search and Matching

Stochastic Dynamic Programming

ZENG MING

March 30, 2016

1 / 19

Roadmap

I Stochastic dynamic programming

I A model of search

I A model of matching

2 / 19

Environment

To avoid measure theory, here we focus on stochastic variable thattake finitely number of values

I Introduce stochastic random variable zt ∈ Z = {z1, . . . zN}, afinite set

I Define return function F (xt , xt+1, zt) where xt ∈ X . We callxt as the state variable and xt+1 as control variable.

I Feasible set now depends on zt , xt+1 ∈ G (xt , zt)

I zt follows a first-order Markov chain:current value of zt onlydepends on zt−1:Pr[zt = zj |z0, . . . zt−1] = Pr[zt = zj |zt−1]

I Represent this Markov Chain as:Pr[zt = zj |zt−1 = zj ′ ] = qjj ′

where∑N

j=1 qjj ′ = 1

3 / 19

Example:Optimal Growth Problem

Objective:

maxE0

∞∑t=0

βtu(ct)

s.t.kt+1 = f (kt , zt) + (1− δ)kt − ct

At time when ct is chosen, zt has been realized, so ct may dependon the entire history of zt , we write the history as z t = (z0, . . . zt),then denote ct = c(z t).Let xt = kt , thenxt+1 = kt+1 = f (kt , zt) + (1− δ)kt − c(z t) = k(z t), feasibilityrequires k(z t) ≤ f (k(z t−1), zt) + (1− δ)k(z t−1) for all z t−1 andzt ∈ Z

4 / 19

Cont’d

To describe policy function, we expect it to be the formkt+1 = π(kt , zt). And we write SP in FE formulation:

V (k , z) = supy∈[0,f (k,z)+(1−δ)k]

{u(f (k, z) + (1− δ)k − y) + βE [V (y , z ′)|z ]}

This FE having solution means that there exists a feasible policyfunction y = π(k , z) that achieves the value V (k , z) starting fromk and z :

V (k, z) = u(f (k , z) + (1− δ)k − π(k , z) + βE [V (π(k , z), z ′)|z ]

5 / 19

Equivalence between SP and FE

I Sequence Problem:

V ∗(x0, z0) = sup{x(z t)}∞t=−1

E0

∞∑t=0

βtU(x(z t−1), x(z t), zt)

s.t.x(z t) ∈ G (x(z t−1), zt)

I Functional Equation:

V (x , z) = supy∈G(x,z)

{U(x , y , z) + βE [V (y , z ′)|z ]}

s.t.x ∈ X , z ∈ Z

Denote set of feasible plans starting with x(t) and z(t):

Φ(x(t), z(t)) = {{x [z s ]}∞s=t−1 : x [z s ] ∈ G (x [z s−1], z(s))s = t − 1, t, t + 1 · · · }

We are then interested in 1) Solution V (x , z) to FE coincides withsolution V ∗(x , z) to SP; 2) set of optimal policy Π(x , z) ⊂ Φ(x , z) alsogenerates an optimal feasible plan for SP.

6 / 19

Principle of Optimality

Assumption 4.1: G (x , z) is nonempty for all x ∈ X and z ∈ Z . For allx(0) ∈ X , z(0) ∈ Z , x ∈ Φ(x(0), z(0)) andlimn→∞ E0

∑nt=0 β

tU(x(z t−1), x(z t), zt) existsAssumption 4.2: X is compact subset of RK , G is nonempty,compact-valued and continuous. Moreover, letXG = {(x , y , z) ∈ X × X × Z : y ∈ G (x , z)} and U to be continuous.Theorem 4.1:Suppose Assumption 4.1-4.2 hold. For x(0) ∈ X andz(0) ∈ Z , let x∗ ∈ Φ(x(0), z(0)) be a feasible plan that attainsV ∗(x(0), z(0)) of SP, then we have:

V ∗(x∗[z t−1], z(t)) = U(x∗[z t−1], x∗[z t ], z(t)) + βE [V ∗(x∗(z t), z(t + 1))|z(t)]

7 / 19

Properties of value functionAssumption 4.3: U is strictly concaveAssumption 4.4: U is strictly increasing in its first K arguments and Gis monotonicAssumption 4.5: U is continuously differentiable w.r.t x in the interiorof its domain XG

Theorem 4.2:[Concavity of Value function] V is strictly concave in x foreach z ∈ Z. Moreover the optimal plan can be expressed asx∗[z t ] = π(x∗(t), z(t)) where the policy function π(x , z) is continuous inx for each z ∈ ZTheorem 4.3:[Monotonicity of Value function w.r.t x ] For each z ∈ Z,V is strictly increasing in xTheorem 4.4:[Monotonicity of Value function w.r.t z ] For each x ∈ X ,V is strictly increasing in zTheorem 4.5:[Differentiability of Value function] Let x ′ ∈ IntX ,π(x ′, z) ∈ IntG (x ′, z) at z ∈ Z, then V (x , z) is continuouslydifferentiable at (x ′, z), with derivative given by:

DxV (x ′, z) = DxU(x ′, π(x ′, z), z)

8 / 19

Stochastic Euler Equation

Under some assumptions such that value function is differentiable,necessary conditions for an interior optimal plan:

DyU(x , y∗, z) + βE [Dx(y∗, z ′)|z ] = 0

Using envelope theorem we have:

DxV (x , z) = DxU(x , y∗, z)

The equality holds conditional on the realization of z , thus no“expectation” symbol.Those two equalities together imply stochastic Euler equation:

DyU(x , y∗, z) + βE [DxU(y∗, )|z ] = 0

9 / 19

Search Models

Given empirical facts of wages, employment and unemployment,what could be the micro-founded model the reconcile thoseregularities? We are interested in those models since they will makeclear the mechanism behind the data, also, policy makers want touse those models to assess the consequences of policy changes.Traditional analysis of frictionless market can’t easily explainseveral empirical facts, for example, why do unemployed workerssometimes choose to remain unemployed? How can wesimultaneously have unemployed workers and unfilled vacancies?etc...Search theory has provided a rigorous yet tractable framework thatcan be used to address those questions. In this framework, frictionsare important, and there is no centralized market for transaction.Both elements are important in classical supply demand framework.We introduce some examples of search theory here since dynamicprogramming is widely used in this framework.

10 / 19

McCall’s model of intertemporal job searchConsider an individual searching for a job, he seeks to maximizeE∑∞

t=0 βtxt . Income xt = ω if employed at wage ω and

xt = b > 0 if unemployed. An unemployed individual samples(i.i.d) one offer each period from a known distribution F (ω), hecan reject it and remain unemployed, or take it to work.For now, assuming that if a job is accepted, the worker keeps itforever. We can write the Bellman equation of being employed orunemployed :

W (ω) = ω + βW (ω)

U = b + β

∫ ∞0

max{U,W (ω)}dF (ω)

Or in a compact form:

v(ω) = max{ ω

1− β, b + β

∫v(w ′)dF (w ′)} (1)

11 / 19

McCall’s model(cont’d)

Notice that W (ω) is strictly increasing. We then define thereservation wage ωR such that W (ωR) = U.Exercise:1.Show that

ωR = T (ωR) = (1− β)b + β

∫ ∞0

max{ω, ωR}dF (ω)

2.Show that T is a contraction mapping.3. Show that we can write ωR as:

ωR = b +β

1− β

∫ ∞ωR

[ω − ωR ]dF (ω) (2)

12 / 19

Properties of reservation wage

Assume wage distribution has the support with upper bound B,then define h(ω) = β

1−β∫ Bω [ω′ − ω]dF (ω′).

Exercise: Show that h has the following property:

I h(0) = β1−βEω, h(B) = 0

I h′ < 0, h′′ > 0

Write (1) as:

ωR = b + β(E (ω)− b) + β

∫ ωR

0F (ω)dω (3)

Define the function g(s) =∫ s

0 F (p)dp.Exercise: Show that g(0) = 0, g(s) ≥ 0, g ′(s) > 0, g ′′(s) > 0, fors > 0

13 / 19

A variation-FiringNow consider a modification that each period the worker faces probability α ofbeing fired. This probability is independent of tenure. The worker receives afixed wage ω until she is fired. A worker who is fired becomes unemployed forone period before drawing a new wage, that is, a previously employed workerwho is fired at the beginning of a period can’t draw a new wage offer at thatperiod, but have to be unemployed for one period.Let v(ω) be the expected present value of a previously unemployed worker whohas offer ω in hand, then we have:

v(ω) = b + β

∫v(w ′)dF (w ′) = b + βEv

Exercise: 1. Show that the Bellman equation is:

v(ω) = max{ω + β(1− α)v(ω) + βα[b + βEv ], b + βE [v ]}

2. Show that the following is the solution:

v(ω) =ω + βα[b + βEv ]

1− β(1− α)ω ≥ ωR

v(ω) = b + βE [v ] ω ≤ ωR

where ωR solves:ω+βα[c+βEv ]1−β(1−α)

= b + βE [v ]

14 / 19

Waiting time

We often want to know the probability distribution of the waitingtime until a job offer is accepted. Let N be the random variable“length of time until a successful offer is made”, for example,N = 1 means the first job offer is accepted. Let λ be theprobability that a job offer is rejected.Exercise: 1.Show that λ =

∫ ωR

0 dF (ω)2. Show that P(N = 2) = (1− λ)λ and more generallyP(N = j) = (1− λ)λj−1

3. Show that the mean waiting time N is given by: (1− λ)−1

15 / 19

Continuous time formulation

Suppose the length of a period is ∆. Denote β = 11+r∆ , and

assume that the worker gets a wage offer with probability α∆ ineach period. Then we have the Bellman equation:

W (ω) = ∆ω +1

1 + r∆W (ω)

U = ∆b +α∆

1 + r∆

∫ ∞0

max{U,W (ω)}dF (ω) +1− α∆

1 + r∆U

Exercise:Show that when ∆→ 0, above two equations reduce to:

rW (ω) = ω (4)

rU = b + α

∫ ∞0

max{W (ω)− U, 0}dF (ω) (5)

Since r has the “frequency” unit, then rU can be regarded as theflow value of unemployment

16 / 19

Matching model

Market comes with both sides. Search model describes the behavior of worker.However, for general equilibrium purposes, it’s necessary to model the behaviorof the other side-the firm, and then put those two together(matching model).Two basic issues: how workers and firms meet, and how wages are determined.Suppose there are v vacancies posted by firms looking for jobs, u unemployedworkers looking for jobs. Assume the flow of contacts between firms andworkers is given by matching technology m = m(u, v) Assuming all workers andfirms are the same. Then arrival rates for unemployed workers and firms aregiven by:

αw =m(u, v)

u

αe =m(u, v)

v

Assume m is continuous, nonnegative, increasing in both arguments andconcave, with m(u, 0) = m(0, v) = 0 and χm(u, v) = m(χu, χv)Exercise: Show that αw and αe depend only on the ratio v

u, referred to as a

measure of market tightness

17 / 19

Wage determination

After successful matching, it will produce a flow of output y

I Worker gets a wage ω, his expected lifetime utility is W (ω).

I Firm earns expected discounted profit J(π), where π = y − ωOtherwise, worker gets U and the firm gets V .We are now interested in the case when W (ω) > U, J(π) > V ,how ω can be determined?Generalized Nash bargaining solution:

ω ∈ arg max[W (ω)− U]θ[J(y − ω)− V ]1−θ (6)

Exercise: Show that the solution satisfies:

θ[J(π)− V ]W ′(ω) = (1− θ)[W (ω)− U]J ′(π) (7)

18 / 19

What Nash solution is about?

From (4), we can similarly obtain the value function of the firmand the worker(assume the match dies exogenously with rate λ):

rJ = π + λ[V − J(π)]

rW = ω + λ[U −W (ω)]

And from (7), we have:

W (ω) = U + θ[J(π)− V −W (ω)− U] = U + θS

19 / 19