calculating principal eigenfunctions of non-negative...

Calculating principal eigenfunctions ofnon-negative integral kernels:

particle approximations and applications

Nikolas Kantas (Imperial)

COSMOS WORKSHOP PARIS, FEB. 2-5, 201616

joint work with Nick Whiteley (Bristol)

Introduction

I On eigen-functions and related quantities of particular integraloperators (e.g. un-normalised Markov transition kernels)

I Applications:I rare event estimationI stochastic controlI statistical mechanics/physics (neutron transport, nuclear

fission, particle motion etc.)

I Outline:I Introduction on particle methods and simple Feynman-Kac

models in discrete timeI Eigen-quantities in linear algebra: finite state spaces and

matrices.I generalisations for measurable spaces and integral operatorsI particle approximations of eigen-quantitiesI rare events estimation example

Preliminaries

I Let M(x , dy) be a homogeneous Markov probability kernel on(X,B (X)) and let X0 = x .

IPotential function:

G = eU

where U : X ! R.I The non-negative kernel

Q (x , dy) := eU(x)M(x , dy)

defines a linear operator on functions

Q (') (x) :=

ˆQ (x , dy)' (y)

Some Notation

Measures: say a measure µ on (X,B(X))

µ(') :=

ˆ'(x)µ(dx)

Integral kernels:I For K a (possibly un-normalised) Markov kernel on X ⇥ B (X),

K (') (x) :=

ˆK (x , dy)'(y)

andµK (·) :=

ˆµ(dx)K (x , ·)

I the n-fold iterate of kernel K : K n = K . . .K| {z }

n times

, K0 = Id

Think of kernel K being a matrix, µ a row vector and ' a vector!

Feynman-Kac models

I Following notation/style of [Del Moral 04]I Usually in Monte Carlo we are interested to compute

something like:

�n,x(') = Qn (') (x) = E

x

"

exp

n�1X

k=0

U (Xk

)

!

' (Xn

)

#

I Simple example: particle motion in absorbing mediumI 1 � eU(x) probability of absorption at location xI M(x , dx 0) describe Markov dynamics of neutronI Qn

�

1�

(x) is probability of survival after n steps starting atX0 = x .

Feynman-Kac models

I Feynman-Kac Models [Del Moral 04]:I more convenient to use a sequence of probability measuresI use the following sequence

⌘n

(') =Ex

h

exp⇣

P

n�1k=0 U (X

k

)⌘

' (Xn

)i

Ex

h

exp⇣

P

n�1k=0 U (X

k

)⌘i

I Update - prediction recursion:

⌘n+1(dy) =

⌘n

Q(x , dy)

⌘n

Q(1)

Feynman-Kac models

Measure valued recursion :

⌘n+1 := �(⌘

n

) =⌘n

Q

⌘n

Q(1),

or in update - prediction steps

⌘n+1 = ⌘

n

M, ⌘n

=eU⌘

n

⌘n

(eU)

Product formula:

�n,x(') = ⌘

n

(')n�1Y

k=0

⌘k

(eU)

A particle algorithm and some approximations

A simple particle algorithmI Initialization: sample i.i.d.

�

⇣ i0�

N

i=1 ⇠ µ,I For p = 1, ..., n: sample i.i.d.

�

⇣ ip

�

N

i=1

�

�

�

�

⇣ ip�1

�

N

i=1 ⇠N

X

j=1

exph

U⇣

⇣ jp�1

⌘i

P

N

j

0=1 exph

U⇣

⇣ j0

p�1

⌘iM⇣

⇣ jp�1, ·

⌘

Particle approximations, n � 0:

⌘Nn

:=1N

N

X

i=1

�⇣ in

, ⌘Nn

(') :=1N

N

X

i=1

'�

⇣ in

�

,

�Nn,x (') :=

n�1Y

k=0

⌘Nk

⇣

eU⌘

⌘Nn

(')

The eigen problem

Under some regularity assumptions, Q has:I an isolated, real, maximal eigen-value �?,I a positive (right) eigen-function h?,I a positive (left) eigen-measure ⌘?I All-together

Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1

I Do these quantities exist? Are they unique?I How can they be computed?I Is it possible to get extensions on

I Perron-Frobenious theoryI the power method from when Q, h?, ⌘? are matrix and vectors

resp.?

The eigen problem - why do we care?Applications:

IStatistical physics

I h? could be viewed as the Schrödinger ground energy state formolecules,

I diffusion or quantum Monte CarloI [Rousset 06], [R. Assaraf, M. Caffarel, & A. Khelif 00],

[Makrini, B. Jourdain, and T. Lelièvre. 07].

I In optimal control:I discrete time problems with Kullback-Leibler divergence term

in the stage costI Q is as a multiplicative Bellman operator

I h? is a logarithmic transformation of the value function.

I[Albertini & Runggaldier 88], [Todorov 08]

I can be related to discretisations of certain continuous timemodels

I[Flemming 82], [Sheu 84], [Dai Pra, Meneghini,

Runggaldier,1996], [Kappen 05], [Theodorou 10]

The eigen problem - why do we care?h? appears in large deviations theory of Markov chains;

I [Vere-Jones 67], [Ney & Nummelin 87], [Kontoyiannis & Meyn03]

I Let (Xn

; n � 0) is a Markov chain with transition kernel M,X0 = x , and G (x) := e↵U(x)

I Eigen-problem acts as multiplication Poisson eqn.

⇤(↵) = limn!1

1n

logEx

"

exp

↵n�1X

p=0

U(Xp

)

!#

,

h?(x) = limn!1

Ex

"

exp

↵n�1X

p=0

U(Xp

)� n⇤(↵)

!#

I Let ⇤0(↵) = c , �2↵ = ⇤00(↵) and some conditions on U

Px

"

n�1X

p=0

U(Xp

) > nc

#

⇠ h?(x)

↵p

2⇡�2↵

exp (�n⇤⇤(c))

The eigen problem: the “twisted” Markov kernel

I A related object of interest:

P?(x , dx0) :=

Q(x , dx 0)h?(x 0)h?(x)�?

. (1)

I Many applications of interest :I estimation of tail probabilities of Markov chains:

Icertain P? is optimal changes of measure

[Bucklew, Ney, Sadowski 90], [Dupuis & Wang 05]

I optimal control:I P? is optimally controlled Markov transition kernel.

I particle motion in absorbing media:I P? is the Markov transition kernel of a particle conditional on

long-term survival

[Del Moral, Miclo 03], [Del Moral & Doucet 04], [Rousset 06]

I branching processes: [Harris 51], [Athreya 00]

Eigen -values, -measures, -functions of Q�

= eUM�

We look to compute quantities that satisfy:

Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1

I In the finite state space caseI When do these quantities exist? Are they unique?I How can they be computed?

Basic Perron-Frobenious theory

Let Q be a square n ⇥ n matrix with real positive entries. Then:

1. there is one real and isolated eigenvalue �? on the spectralradius and for any other eigenvalue �, |�| < �?.

2. Perron vectors: for �? there exist unique positive right and lefteigenvectors h?, ⌘? of size n ⇥ 1 and 1 ⇥ n resp. that satisfy

Qh? = �?h?, ⌘?Q = �?⌘?, ⌘?h? = 1

3. We have:��n

? Qn ! h?⌘?

How to compute �?, h?, ⌘??

I The power method: hk

! h?,

hk+1 =

Qhk

kQhk

k

assuming h00h? 6= 0. Similar for ⌘?.I Approximation for the eigenvalue �

k

! �?, e.g. using Rayleighquotient:

�k

= arg min�

kQhk

� �hk

k2 =h⇤k

Qhk

h⇤k

hk

I Gelfand’s formula9Qk91/k ! �?

General State Space: regularity conditions for Q

Recall Q = eUM, M a Markov probability kernel, U a real functionon X

Regularity condition (A1): there exists a probability measure ⌫ suchthat

dQ(x , ·)d⌫

(y) = q(x , y)

and0 < ✏� q(x , y) ✏+ < 1

I Under (A1) Q is aperiodic and irreducible.I A bit restrictive but hopefully this can be relaxed, e.g.

[Kontoyiannis and Meyn 03] etc.

Perron-Frobenius Theory for Q

I [...,Nummelin 84] Under (A1) there exist: a maximal andisolated eigenvalue �?, a positive eigen-function, probabilityeigen-measure:

Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1

and these are unique.

I Multiplicative Ergodic Theorem: For any n � 1,

supx2X

sup|'|<1

�

��n

? Qn (') (x)� h?(x)⌘?(')�

� 2✓

✏+

✏�

◆2

⇢n

or

9��n

? Qn

? � h? ⌦ ⌘?9 2⇢n✓

✏+

✏�

◆2

where ⇢ = 1 � (✏�/✏+).

Perron-Frobenius Theory for Q

I [...,Nummelin 84] Under (A1) there exist: a maximal andisolated eigenvalue �?, a positive eigen-function, probabilityeigen-measure:

Q (h?) = �?h?, and ⌘?Q = �?⌘?, ⌘? (h?) = 1

and these are unique.I Multiplicative Ergodic Theorem: For any n � 1,

supx2X

sup|'|<1

�

��n

? Qn (') (x)� h?(x)⌘?(')�

� 2✓

✏+

✏�

◆2

⇢n

or

9��n

? Qn

? � h? ⌦ ⌘?9 2⇢n✓

✏+

✏�

◆2

where ⇢ = 1 � (✏�/✏+).

The twisted probability kernel (Doob’s h kernel)

I The MET is derived using the properties of the twisted Markovkernel

P?(x , dy) :=Q(x , dy)h?(y)

h?(x)�?. (2)

I P? is ergodic with a unique invariant probability distribution,denoted by ⇡? and for all n � 1,

supx2X

supA2B(X)

|Pn

? (x ,A)� ⇡?(A)| 2⇢n (3)

d⇡?/d⌘? = h? (4)

or9Pn

? � 1 ⌦ ⇡?9 2⇢n

where ⇢ = 1 � (✏�/✏+).

How to approximate ⌘?,�??I [Del Moral, Miclo 02], [Del Moral, Doucet 04], [Rousset 06]I Consider the forward recursion for n � 0:

⌘n+1 = �(⌘

n

) =⌘n

Q

⌘n

Q(1),

�n

= ⌘n

Q(1) = ⌘n

(eU)

I Recall

⌘?Q = �?⌘? ) ⌘?Q(1) = ⌘?(eU) = �?.

) � (⌘?) = ⌘?

I In fact, for any initial condition ⌘0

k⌘n+1 � ⌘?k C⇢n

|�n

� �?| C 0⇢n

How to approximate h?, P??

I Often this is enoughI when M is µ- reversible then h?

µ(h?)= d⌘?

dµI [Rousset 06], [Makrini, B. Jourdain, and T. Lelièvre. 07].

I When this does not apply but still within (A1)I Keep previous forward sequence (⌘

p

,�p

)p�0

I Find a sequence of functions (hp

)p�0 such that

Q (hp+1) = �

p

hp

, ⌘p

Q = �p+1⌘p+1, ⌘

p+1(hp+1) = 1

How to approximate h?, P??I Backward recursionI Set h

n,n(x) := 1, for 0 p < n.

hp,n(x) :=

Q (hp+1,n)

�p

=Q(n�p)(1)(x)Q

n�1`=p

�`,

andPp,n(x , dx

0) :=Q(x , dx 0)h

p,n

�p�1hp�1

I We can invoke the MET

9��n

? Q(n)? � h? ⌦ ⌘?9 2⇢n

✓

✏+

✏�

◆2

to deduce

khp,n � h?k C⇢p^(n�p)

9Pp,n � P?9 C 0⇢p^(n�p)

An ideal algorithm like the power method

ForwardInitialization: Set ⌘0 = µ,For p = 1, ..., 2n, :

Set ⌘p

= ⌘n

Q

�n

, �n

= ⌘n

Q(1)Backward

Initialization: Set h2n,2n(x) = 1, x 2 XFor p = 2n � 1, ..., n, :

Set hp,2n(x) = (�

p

)�1 Q (hp+1,2n) (x), x 2 X

Set Pp+1,n(x , dx 0) :=

Q(x ,dx 0)hp+1,n(x 0)

�p

h

p

(x) , x , x 0 2 X

khn,2n � h?k C⇢n

9Pp,n � P?9 C 0⇢n

A particle algorithm

I Forward

I Initialization: sample i.i.d.�

⇣ i0�

N

i=1 ⇠ µ,I For p = 1, ..., 2n: sample i.i.d.

�

⇣ ip

�

N

i=1

�

�

�

�

⇣ ip�1

�

N

i=1 ⇠N

X

j=1

exph

U⇣

⇣ jp�1

⌘i

P

N

j

0=1 exph

U⇣

⇣ j0

p�1

⌘iM⇣

⇣ jp�1, ·

⌘

,

I BackwardI Initialization: set h2n,2n(x) = 1, x 2 XI For p = 2n � 1, ..., n, set

hNp,2n(x) =

N

X

j=1

q(x , ⇣ jp+1)

P

N

i=1 q(⇣i

p

, ⇣ jp+1)

hNp+1,2n(⇣

j

p+1), x 2 X

Particle Approximations

I (Forward pass) Target eigen-measure and eigen-value

⌘Nn

:=1N

N

X

i=1

�⇣ in

, ⌘Nn

(') :=1N

N

X

i=1

'�

⇣ in

�

, �Nn

:= ⌘Nk

⇣

eU⌘

I (Backward pass) eigen-function

hNn,2n(x) =

N

X

j=1

q(x , ⇣ jn+1)

P

N

i=1 q(⇣i

n

, ⇣ jn+1)

hNn+1,2n(⇣

j

n+1)

I (Backward pass) twisted kernel:

PN

n,2n(x , dx0) :=

1hNn�1,2n(x)

N

X

j=1

q(x , ⇣ jn

)P

N

i=1 q(⇣i

n�1, ⇣j

n

)hNn,2n(⇣

j

n

)�⇣ jn

�

dx 0�

.

Lr error estimates

Under (A1) we have 8n � 1,N � 1,r � 1 there exist constantsB(r),C such that:

supx2X

EN

h

�

�

�

hNn,2n(x)� h?(x)

�

�

�

r

i1/r B

h

(r)pN

+ Ch

⇢n (5)

supx2X

supA2B(X)

EN

h

�

�

�

PN

n,2n (x ,A)� P? (x ,A)�

�

�

r

i1/r B

P

(r)pN

+ CP

⇢n,

(6)where ⇢ = (1 � (✏�/✏+)).

In part appeal to earlier results for particle smoothers [Del Moral,Doucet, Singh 10] and [Douc, Garivier, Moulines & Olsson 11]

An application on rare events estimation

I Let (Xn

; n � 0) be a Markov chain with transition Minitialized from some X0 = x .

I Also let U : X ! [�1, 1]I Our objective is for some � 2 (0, 1) and m � 1, to estimate

the deviation probability

⇡m

(�) := Px

0

@

m

X

p=1

U (Xp

) > m�

1

A . (7)

where Px

is the law of the chain.

Sequential Importance Sampling (IS) for rare events

I Let C be the collection of Markov transitions M: there exist0 < ✏�, ✏+ < 1 and a probability measure ⌫ such that ⌫ ⌧ ⌫

(C) ⌫ (·) ✏� M(x , ·) ✏+⌫ (·) , 8x ,ˆ ✓

d⌫

d ⌫(x)

◆2⌫ (dx) < 1,

I The sequential importance sampling (IS) estimator of ⇡m

(�) is

b⇡m

(�, L) :=1L

L

X

i=1

I

2

4

m

X

p=1

U�

X i

p

�

> m�

3

5

dPx

dPx

�

X i

0, ...,Xi

m

�

,

(8)where

��

X i

0,Xi

1, ...,�

; i = 1, ..., L

are L iid Markov chains,each with transition M and law denoted by P

x

Importance Sampling for rare events: twisting the kernelI AIM: the relative variance of IS

Ex

"

✓

b⇡m

(�, L)

⇡m

(�)� 1

◆2#

=1L

0

@

Ex

h

b⇡m

(�, 1)2i

⇡m

(�)2� 1

1

A (9)

not to grow exponentially fast in m.

I Consider

G↵0(x) := e↵0U(x), Q↵0

�

x , dx 0�

:= G↵0(x)M�

x , dx 0�

and

⇤?(↵) := limn!1

1n

logEx

2

4exp

0

@↵n�1X

p=0

U(Xp

)

1

A

3

5 , ↵ 2 R

I (t) := sup↵2R

[t↵� ⇤?(↵)] , t 2 R

Importance Sampling for rare events: twisting the kernelI AIM: the relative variance of IS

Ex

"

✓

b⇡m

(�, L)

⇡m

(�)� 1

◆2#

=1L

0

@

Ex

h

b⇡m

(�, 1)2i

⇡m

(�)2� 1

1

A (9)

not to grow exponentially fast in m.I Consider

G↵0(x) := e↵0U(x), Q↵0

�

x , dx 0�

:= G↵0(x)M�

x , dx 0�

and

⇤?(↵) := limn!1

1n

logEx

2

4exp

0

@↵n�1X

p=0

U(Xp

)

1

A

3

5 , ↵ 2 R

I (t) := sup↵2R

[t↵� ⇤?(↵)] , t 2 R

Importance Sampling with large deviations Bucklew et al. 901. I (t) is a non-negative, strictly convex function with I (t) = 0 if

and only if t = ⇤0?(0).2. For any � 2 (0, 1), the following large deviation principle holds

limm!1

1m

log ⇡m

(�) = � inft2[�,1)

I (t).

3. For any � 2 (0, 1) and M in C, the importance samplingestimator satisfies

limm!1

1m

logEx

h

b⇡m

(�, 1)2i

� �2 inft2[�,1)

I (t). (10)

4. For any � 2 (0, 1) and ↵ the unique solution of

⇤0? (↵) = �

the twisted kernel P↵? is the unique member of C for which

equality holds (asymptotically efficient.)

Importance Sampling for rare events using lack of bias

I Let�

Xp

; p = 0, ...,m�

be a non-homogeneous Markov chainwith transitions

X0 = x , Xp

⇠ PN

↵,n+p,2n�

Xp�1, ·

�

, p � 1.

I Then we have the following lack of bias property:

EN

2

4I

2

4

m

X

p=1

U�

Xp

�

> m�

3

5

m�1Y

p=0

�N↵,n+p

G↵�

Xp

�

hN↵,n,2n�

X0�

hN↵,n+m,2n�

Xm

�

3

5 = ⇡m

(�),

EN,↵ for expectation w.r.t. the joint law of

�

Xp

�

and the particlesystem.

Numerical example

Take X = [�c , c] and consider an ergodic Gaussian transitionkernel with support restricted to [�c , c],

M(x , dy) =exp

⇣

�12�

y � x

2�2⌘

⇣

1 � erf⇣

c+x/2p2

⌘⌘p2⇡

I[�c,c](y)dy ,

Consider U defined by

U (x) =

8

>

<

>

:

�1 x �1x x 2 (�1, 1)1 x � 1.

Numerical example

!6 !4 !2 0 2 4 6!8

!7

!6

!5

!4

!3

!2

!1

0

1

2

!10 !8 !6 !4 !2 0 2 4 6 8 10!1

!0.8

!0.6

!0.4

!0.2

0

0.2

0.4

0.6

0.8

1

Figure : Left: each of the solid curves shows an approximation of

[↵t � ⇤?(↵)] against ↵, with each curve corresponding to a different

value of t in the range [�0.8, 0.8]. The cross on each curve indicates

its maximum and thus approximates sup↵ [↵t � ⇤?(↵)] = I (t). Right:

⇤0?(↵) against ↵ approximated using finite differences.

Numerical example

0 5 10 15 20 25 30 35 40 45 5010

!16

10!14

10!12

10!10

10!8

10!6

10!4

10!2

100

0 5 10 15 20 25 30 35 40 45 500

10

20

30

40

50

60

70

80

90

100

Figure : Left: estimated value of ⇡m

(�) against m, for: �,� = 0.8;⇤,� = 0.9, and +, � = 0.99. Right: solid lines show sample relativevariance of the estimated value of ⇡

m

(0.9) against m for: �, ↵ = 1;+,↵ = 2; ⇤, ↵ = 4; ⇤, ↵ = 8; and ⇥,↵ = 16. Dashed line shows samplerelative variance of b⇡

m

(0.9, 1) in the case M = M.

Conclusions/Extensions

I Some interesting examples/extensions on stochastic control:I h

p

, h? are value functions for particular finite/ infinite horizonproblems resp.

I In the rare events example, can use also other ideas from SMCor adaptive IS [Dupuis and Wang 05] to improve things.

I When hp

(x), h?(x) are the inference objectiveI there is some interest in variance reduction

I This is a batch scheme with cost O(N2n)

I can this be reduced in some way?

ReferencesJ.A. Bucklew, P. Ney, J.S. Sadowsky (1990) Monte Carlo simulations and largedeviations theory for uniformly recurrent Markov chains J. Appl. Prob.

P. Del Moral and L. Miclo. (2003) Particle approximations of Lyapunovexponents connected to Schrodinger operators and Feynman-Kac semigroups.ESAIM Probab. Stat.

P. Del Moral, A. Doucet, and S.S. Singh. (2011) A backward particleinterpretation of Feynman-Kac formulae. ESAIM Math. Model. Numer. Anal.

R. Douc, Garivier, A. and Moulines, E. and Olsson, J. (2011) Sequential MonteCarlo smoothing for general state space hidden Markov models. Ann. Appl.Probab..

E. Nummelin. (1984) General irreducible Markov chains and non-negativeoperators. CUP.

I. Kontoyiannis and S. P. Meyn (2003) Spectral theory and limit theorems forgeometrically ergodic Markov processes,Ann. Appl. Probab. 13 (1), 304-362.

N. Whiteley and N. Kantas (2012). A particle method for approximatingprincipal eigen-functions and related quantities. Submitted.

N. Whiteley, N. Kantas and A. Jasra, (2012) Linear Variance Bounds for ParticleApproximations of Time-Homogeneous Feynman-Kac Formulae, SPA.

N. Whiteley (2013) Stability properties of some particle filters. Ann. Appl.Probab...

Appendix: relaxing (A1)

I Similar to [Whiteley, N.K., Jasra 12], [Whiteley 13]I Use assumptions from [Kontoyiannis & Meyn 03]:

I using multiplicative drift conditions , i.e. there exists bd

< 1and Lyapunov function V such that:

Q�

eV�

eV (1��)+b

d

IC

d .

I a stronger MET is derived for QI can be verified in general for following cases:

1. bounded functions U and non-ergodic kernels M2. unbounded above functions U and multiplicative ergodic

kernels M

I Can be verified in practice for realistic examples

Optimal Control

Let (Xn

; n � 0) be a controlled Markov chain initialized fromX0 = x and X

n

⇠ M f

n�1(Xn�1, ·)

V0(x) = inff 2Hn

Ef

x ,0

2

4

n�1X

p=0

⇣

U(Xp

) +KL⇣

M f

p

�

�

�

M⌘

(Xp

)⌘

+⌦(Xn

)

3

5 ,

V?(x) = inff 2HN

lim supn!1

1nEf

x ,0

2

4

n

X

p=0

⇣

U(Xp

) +KL⇣

M f

p

�

�

�

M⌘

(Xp

)⌘

3

5

Finite horizon setupI Let V

n

= ⌦, Q = e�UM, �p

= e�⇤p and V

p+1 = � log hp

thebackward recursion

Q (hn+1) = �

n

hn

corresponds to Bellman eqn.

Vp

(x) = U(x)�⇤p

+ inff

p

2H

n

KL⇣

M f

p

�

�

�

M⌘

(x) +M f

p (Vp+1) (x)

o

I For p = 1, . . . , nI Run forward particle system to compute particle

approximations for (⌘p

),

I For p = n, n � 1 . . . , 1I Run backwards in time to compute particle approximations for

hp

and Pp

Infinite horizon problem: particle value iteration

I Similarly, V?(x) = � log h?(x), &? = � log �?

Q (h?) = �?h?

is equivalent to a solution (V?,⇤?) of the average-costoptimality equation:

V?(x) + ⇤? = infh2H

h

U(x) + KL⇣

Mh

�

�

�

M⌘

(x) + Mh (V?) (x)i

.

I Use previous algorithm for n very largeI forward particle system up to time 2n to compute particle

approximations (⌘Np

),I then backwards to time n to compute particle approximations

hNn,2n and PN

n,2n

Example: Gaussian model

Consider the controlled dynamics:

Xn

=

X 1n

V 1n

�

=

1 ⌧0 1

�

Xn�1 +

⌧ ⌧2

0 ⌧

�

(Wn

+ An

)

consider the state-dependent-only part of the stage cost:

U(x) / (1 � I(��,�)(x1))

which penalises states outside (��, �).

Example: Gaussian model

!1 !0.5 0 0.5 1!1

0

1

2

3

4

5

x

C(x

)

!1 !0.5 0 0.5 1!1

0

1

2

3

4

5

x

V19(x

)

!1 !0.5 0 0.5 1!1

0

1

2

3

4

5

x

V15(x

)

!1 !0.5 0 0.5 1!1

0

1

2

3

4

5

x

V10(x

)

Figure : Estimated horizon average cost optimal value function Vn

(x)against x for various n. Here T = 20,

T

= C , ⌧ = 1,N = 500

Example: Cox-Ingersoll-Ross process

I Control free model for M is Euler discretisation of CIR

dXt

= ✓ (µ� Xt

) dt + �p

Xt

dWt

,

where {Wt

} is standard 1-D Brownian motion, ✓ > 0 is thereversion rate, µ > 0 is the level of mean reversion and � > 0specifies the volatility.

I Stage cost specified by

U(x) = 2I[0,10��](x) + I[10+�,1)(x), (11)

which penalises states outside (10 � �, 10 + �).

Example: Cox-Ingersoll-Ross process

4 6 8 10 12 14 16 18 20

0

2

4

6

8

10

12

14

16

18

20

Figure : Estimated infinite-horizon average cost optimal value functionV?(x) against x for: �, � = 5;⇥, � = 4;⇤, � = 3; +, � = 2.

Appendix : connection to some pde’s

Numerical solutions for some class of parabolic PDEs for t 2 [0,T ]

(v)t

+ a (v)x

+12�2 (v)

xx

+ 'v = 0, v(·,T ) = (·)

using Q(n�p)( )(x) =

hp,n(x)

n�1Y

l=p

�l

⇡ Etx

exp✓ˆ

T

t

U(Xs

)ds

◆

'(XT

)

�

= v(x , t)

where the expectation is taken conditional to Xt

= x w.r.t.

dXt

= a(Xt

)dt + �(Xt

)dWt

.

Appendix: a stochastic control exampleI For the following controlled Markov chain

dXt

= (a(Xt

) + B(Xt

)At

) dt + �(Xt

)dWt

,

let the total cost or value function for t � 0:

V (x , t) = infU2L2(·),s2(t,T )

Etx

ˆT

t

L(Xs

,As

)ds + (XT

)

�

,

with the stage cost being:

L(x ,A) = C (x) +12A

0R(x)A,

I IF we have BRB 0 = ��0 then the Hamilton-Jacobi-Bellmanequation implies

V (x , t) = � log v(x , t), C = �U, = � log' (12)

with v(x , t) as in the previous slide and for the optimal control:

A⇤(x , t) = R(x)�1B(x)0rx

v(x , t).

calculating principal eigenfunctions of non-negative...

Documents