adjoint methods in computational finance software tool support … · 2019-02-06 · adjoint...

STCE

Adjoint Methods in Computational FinanceSoftware Tool Support for Algorithmic Differentiation

Uwe Naumann

LuFG Informatik 12Software and Tools for Computational EngineeringRWTH Aachen University, [email protected]

www.stce.rwth-aachen.de

and

The Numerical Algorithms Group Ltd.Oxford, United [email protected]

1 / 44

STCE

Motivation: Cheap GreeksMathematical/Numerical Model

T = T (t, x , c(x)) : IR× IR× IR→ IR is given as the solution of the 1Ddiffusion equation

∂T

∂t= c(x) · ∂

2T

∂x2

over the domain Ω = [0, 1] and with initial condition T (0, x) = i(x) forx ∈ Ω and Dirichlet boundary T (t, 0) = b0(t) and T (t, 1) = b1(t) fort ∈ [0, 1].

The numerical solution is based on a central finite difference / implicit(backward) Euler integration scheme that exploits linearity of the residualr in T (single evaluation of constant Jacobian ∂r

∂T and factorization).

2 / 44

STCE

Motivation: Cheap GreeksJob

We aim to analyze sensitivities of the predicted T (x , c(x)) with respectto c(x) or even calibrate the uncertain c(x) to given observations O(x)for T (x , c(x)) at time t = 1 by solving the (possibly constrained) leastsquares problem

minc(x)

f (c(x), x) where f ≡∫

Ω(T (1, x , c(x))− O(x))2 dx .

Assuming a spatial discretization of Ω with n grid points we require

∂f

∂c=

∂f

∂T (1)· ∂T (1)

∂c∈ IRn (→ scalar adjoint integration at O(1))

and possibly

∂2f

∂c2∈ IRn×n (→ 2nd-order vector adjoint integrations at O(n))

3 / 44

STCE

Motivation: Cheap GreeksPerformance (Live ...)

Application of Algorithmic Differentiation (AD) tool dco/c++ inblack-box mode (→ “Getting Started” section in user guide):

∂f∂c (n=300, m=50)

central FD adjoint AD

run time (s) 15.2 0.7

∂2f∂c2 (n=100, m=50)

central FD adjoint AD

run time (s) 63.6 3.9

y=f(x)

x+hx

???

... while ignoring accuracy for the time being ...

4 / 44

STCE Nice to have?

MITgcm, (EAPS, MIT)

in collaboration with ANL,MIT, Rice, UColorado

J. Utke, U.N. et al: OpenAD/F:

A modular, open-source tool for

automatic differentiation of Fortran

codes . ACM TOMS 34(4), 2008.

Plot: A tangent-linear computation / finite difference approximation for64,800 grid points at 1 min each would keep us waiting for a month anda half ... :-((( We can do it in less than 10 minutes thanks to adjointscomputed by a differentiated version of the MITgcm :-)

5 / 44

STCE Flow ( ... through this presentation)

I AD Models

I AD Code

I Case Study: LIBOR

I AD of Numerical Methods

I Case Study: Local Volatilities

I Conclusion ( c©M.Towara with OpenFOAM)

6 / 44

STCE Foundations of what is coming next ...

U. Naumann:The Art of Differentiating Computer Programs.An Introduction to Algorithmic Differentiation.Number 24 in Software , Environments, and Tools,SIAM, 2012.

U. Naumann and O. Schenk, eds.:Combinatorial Scientific Computing.Chapman & Hall/CRC Computational Science SeriesTaylor and Francis Group, 2012.

7 / 44

STCE

Tangents and Adjoints by ADy = f (x) : IRn → IR

I Tangent-Linear Code f (1) (forward AD)

y (1) = ∇f (x)∈IRn

· x(1)

∈IRn

⇒ ∇f at O(n) · Cost(f )

Note: Approximation by bumping.

I Adjoint Code f(1) (reverse AD)

x(1) = y(1)∈IR· ∇f (x)

⇒ ∇f at O(1) · Cost(f )( c©J.Lotz)

8 / 44

STCE

“Hello World” Exampley = f (x) = sin(ex)

void f ( double &z ) z=exp ( z ) ;

z=s i n ( z ) ;

-1

-0.5

0

0.5

1

-3 -2 -1 0 1 2 3

sin(exp(x))

(generated with Gnuplot)

9 / 44

STCE Tangent Code:(

y (1)

y

)=

(f ′(x) · x (1)

f (x)

)

void t 1 f ( double &z , double &t 1 z ) z=exp ( z ) ;

z=s i n ( z ) ;

10 / 44

STCE Tangent Code:(

y (1)

y

)=

(f ′(x) · x (1)

f (x)

)

void t 1 f ( double &z , double &t 1 z ) z=exp ( z ) ;z=s i n ( z ) ;

t 1 z=exp ( z )∗ t 1 z ; // WRONG z !t 1 z=cos ( z )∗ t 1 z ;

11 / 44

STCE Tangent Code:(

y (1)

y

)=

(f ′(x) · x (1)

f (x)

)

void t 1 f ( double &z , double &t 1 z ) t 1 z=exp ( z )∗ t 1 z ;

z=s i n ( z ) ;z=exp ( z ) ;

t 1 z=cos ( z )∗ t 1 z ; // WRONG z !

12 / 44

STCE Tangent Code:(

y (1)

y

)=

(f ′(x) · x (1)

f (x)

)


z=exp ( z ) ;t 1 z=cos ( z )∗ t 1 z ;

z=s i n ( z ) ; // CORRECT z !

13 / 44

STCE Tangent Code:(

y (1)

y

)=

(f ′(x) · x (1)

f (x)

)


z=exp ( z ) ;

t 1 z=cos ( z )∗ t 1 z ;

z=s i n ( z ) ;

Driver:

. . .double z = . . . , t 1 z =1;t 1 f ( z , t 1 z ) ;cout << t 1 z << e n d l ;. . . -20

-15

-10

-5

0

5

10

15

20

25

0 0.5 1 1.5 2 2.5 3

cos(exp(x))*exp(x)


14 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)

void a 1 f ( double &z , double &a 1 z ) z=exp ( z ) ;

z=s i n ( z ) ;

( c©J.Lotz)

15 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)


z=s i n ( z ) ;

a 1 z=cos ( z )∗ a 1 z ; // WRONG z !

a 1 z=exp ( z )∗ a 1 z ;

16 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)


push ( z ) ;

z=s i n ( z ) ;

pop ( z ) ;

a 1 z=cos ( z )∗ a 1 z ;

a 1 z=exp ( z )∗ a 1 z ; // WRONG z !

17 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)

void a 1 f ( double &z , double &a 1 z ) push ( z ) ;

z=exp ( z ) ;

push ( z ) ;

z=s i n ( z ) ;

pop ( z ) ;

a 1 z=cos ( z )∗ a 1 z ;

pop ( z ) ;

a 1 z=exp ( z )∗ a 1 z ;

// WRONG FUNCTION VALUE!

18 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)

void a 1 f ( double &z , double &a 1 z ) push ( z ) ;

z=exp ( z ) ;

push ( z ) ;

z=s i n ( z ) ;

s t o r e ( z ) ;

pop ( z ) ;

a 1 z=cos ( z )∗ a 1 z ;

pop ( z ) ;

a 1 z=exp ( z )∗ a 1 z ;

r e s t o r e ( z ) ;

“You’re entering a world of pain ...”1 if MEM<24 Bytes ...

1John Goodman (“Walter”) in The Big Lebowski19 / 44

STCE Adjoint Code:(

yx(1)

)=

(f (x)

y(1) · f ′(x)

)

void a 1 f ( double &z , double &a 1 z ) . . .

Driver:

. . .double z = . . . , a 1 z =1;a 1 f ( z , a 1 z ) ;cout << a 1 z << e n d l ;. . . -20

-15

-10

-5

0

5

10

15

20

25

0 0.5 1 1.5 2 2.5 3

cos(exp(x))*exp(x)


20 / 44

STCE

Higher-Order ADy = f (x) : IRn → IR

I Second-Order Tangent-Linear Code (forward-over-forward AD)

y (1,2) = x(1)

∈IRn

T · ∇2f (x)∈IRn×n

· x(2)

∈IRn

⇒ ∇2f at O(n2) · Cost(f )

I Second-Order Adjoint Code (forward-over-reverse AD)

x(2)(1) = y(1) · ∇2f (x) · x(2)

⇒ ∇2f · x(2) at O(1) · Cost(f ) resp. ∇2f at O(n) · Cost(f )

I Higher-order tangent-linear (FoFo...FoF) and adjoint (FoFo...FoR)models are derived recursively

21 / 44

STCE

Case Study: LIBORPathwise Monte Carlo Greeks

Consider the LIBOR market model of Brace, Gatarek & Musiela (1997)as implemented by Glasserman & Zhao (1999) and used by Giles andGlasserman (2006) in their “Smoking Adjoints: Fast Monte CarloGreeks” article in Risk for the computation of

∆ ≡ ∂g

∂X (0)

and

Γ ≡ ∂2g

∂X (0)2

with discounted payout g ∈ IR and initialvalue of the underlying X (0) ∈ IRn (forwardLibor rates) in the context of the Euler in-tegration scheme.

. . .

X

X

X

Monte−Carlo

g

X

22 / 44

STCE

Case Study: LIBORPerformance of Computation of Greeks

... 15 swaptions expiring in N = 80 periods; dco exploiting structure

∆ (105 Monte Carlo path simulations)

function bumping adjoint AD

run time (s) 1.2 70.6 3.0relative run time 1 58.8 2.5

Γ (104 Monte Carlo path simulations)

function bumping adjoint AD

run time (s) 0.15 162 45relative run time / 80(Projection of Γ) 1/80 13.45 3.742

2e.g., cheap sum of columns23 / 44

STCE

AD of Numerical MethodsProblem Description

Suppose that some quant decides to replace certain numerical kernels(linear algebra, interpolation, quadrature, nonlinear programming, ...) inthe given simulation code by library routines (e.g. NAG Library).

Furthermore, suppose that this person is the lucky owner of an adjointversion of this simulation generated by some AD tool (e.g. dco).

Switching to the library will break the adjoint code unless the libraryroutines are available as source or are supported by the AD tool (NAGLibrary support by dco).

24 / 44

STCE

Example: Embedded Newton-Raphsonf (z(λ), λ) = sin(λ · ez ) = 0 (tangent-linear AD)

λ = P(x) // source

z = S(f , f ′, z0, λ) // library

y = p(z , x) // source -1

-0.5

0

0.5

1

-3 -2 -1 0 1 2 3

sin(exp(x))

⇓(λ

λ(1)

)= P(1)(x, x(1)) // AD applicable to source(

z

z(1)

)=??? // missing library routine :-((

y

y (1)

)= p(1)(z , z(1), x, x(1)) // AD applicable to source

25 / 44

STCE Developer’s Perspective on Library

zi+1 = zi −f (zi )

f ′(zi )for i = 0, . . .

1 0 2

(z

(1)i+1

zi+1

)=

(2 + f (zi )·f ′′(zi )(f ′(zi ))2

)· z(1)

i

zi − f (zi )f ′(zi )

for i = 0, . . .

Requires user to provide f ′′(zi ) · z(1)i .

→ dco

26 / 44

STCE And they all lived happily ever after ...

(λ

λ(1)

)= P(1)(x, x(1)) // dco applicable to source(

z

z(1)

)= S (1)(f , f ′, f ′′, z0, λ, λ(1)) // AD-intrinsic library routine :-)(

y

y (1)

)= p(1)(z , z(1), x, x(1)) // dco applicable to source

AD tool dco:

I delivers f ′ and f ′′ as well as P(1) and p(1)

I knows “intrinsic” S (1)

27 / 44

STCE

Example: NAG Least-Squares Solvere04gb

minx∈IRn

f (x,p) =m∑

i=1

(Fi (x,p))2, y = F (x,p) : IRn+np → IRm

1 E04GB A1S2 ( . . . , F , F A1S , ! F3 . . . , x , x A1S , ! i n i t i a l g u e s s / s o l u t i o n4 . . . )

1 F A1S ! u s e s g l o b a l p , p A1S2 ( . . . , x , x A1S , ! c u r r e n t p o i n t3 y , y A1S , ! c u r r e n t r e s i d u a l4 JAC , ! c u r r e n t J a c o b i a n5 JAC A1S , ! p r o j e c t e d H e s s i a n6 . . . )

28 / 44

STCE

Case Study: Local Volatility ModelPDE

Consider pricing a European call option by solving

∂V

∂t+

1

2· σ2 · S2 · ∂

2V

∂S2+ (r − γ) · S · ∂V

∂S− r · V = 0, t ∈ [0,T ],

for P = V (S ′, t ′) ∈ IR at some future time t ′ and for some current priceS ′ of the underlying asset.

The volatility σ = σ(S , t) is given as function of strike K , interestr = r(t), dividend γ = γ(t), maturity T , and implied volatilityΘ = Θ(S , t,K ,T ) ∈ IRmΘ×nΘ .

See Andersen, Brotherton-Radcliffe: The equity option volatility smile:an implicit finite difference approach. Journal of Computational Finance,2000, for details.

29 / 44

STCE

Case Study: Local Volatility ModelImplementation

P(1) = F (1)(r , r (1), γ, γ(1),K ,K (1),Θ,Θ(1)) (r(1), γ(1),K(1),Θ(1)) = F(1)(r , γ,K ,Θ,P(1))

r (1)∈IRmr

γ(1)

∈IRmγ θ(1) ∈ IRmθ×nθK (1)

∈IR

P (1) ∈ IR

Interpolate1D(1) Interpolate2D(1)

CrankNicolson(1)(· · ·

Query(1)

Factorize(1)

Substitute(1)

· · · )

r(1)∈IRmr

γ(1)∈IRmγ θ(1) ∈ IRmθ×nθ

K(1)∈IR

P(1) ∈ IR

Interpolate1D(1) Interpolate2D(1)

CrankNicolson(1)(

· · ·Query(1)

Factorize(1)

Substitute(1)

· · · )

30 / 44

STCE

Case Study: Local Volatility ModelPerformance (250× 500 PDE Mesh)

FD3 (s) AD (s) AD Mode Memory (gb)

∂P∂r ∈ IR11 4 12 R(F?) 2.75

+∂P∂γ ∈ IR11 4 ⇑4 R(F?) ⇑

+∂P∂Θ ∈ IR10×40 90 ⇑ R(F?) ⇑

Diag( ∂2P∂Θ2 ) ∈ IR10×40 100 100 FoF 0

∂2P∂S′∂Θ ∈ IR10×40 200 24 FoR 5.5

∂2P∂Θ2 ∈ IR(10×40)2

20000 1338 FoR 5.5

3(poor) approximation4t1 ⇒ t2 : no additional computation time / persistent memory requirement

31 / 44

STCE

Case Study: Local Volatility ModelAcknowledgments

I Jacques du Toit (The Numerical Algorithms Group)

I Claudia Yastremiz (Barclays Capital)

I Klaus Leppkes (STCE @ RWTH Aachen)

I Viktor Moseniks (The Numerical Algorithms Group and STCE @RWTH Aachen)

32 / 44

STCE (Non-Finance) Gallery

33 / 44

STCE Summary

I First-order Greeks at constant relative computational cost

I Second-order Greeks at linear relative computational cost

I AD tool dco

I supportsI first- and higher-order GreeksI growing set of NAG library routinesI “user-defined intrinsics”I loop- and subroutine call-level checkpointingI adjoint MPII dynamic data flow analysis

I written in high-end C++I relative adjoint cost amounts to 2-15 primal runs

... as illustrated by large number of successful applications

34 / 44

STCE Conclusion: Work Flow

f (small scale) :-( _ (blame user ...)

| |

(semi-automatic) (semi-automatic)

SOURCE TRANSFORMATION ___:-(___ OVERLOADING

(dcc) (dco/c++)

| |

:-) :-)

|_______________________________|

|

_________ TESTING / DEBUGGING

| (finite differences,

:-( tangent-linear code,

|_________ continuous adjoint)

|

:-)

|

...

35 / 44


...

|

F (large scale) __ :-))

| |

:-( |

| |

_________ PROFILING, CHECKPOINTING |

| (call tree reversal |

:-( bisection, revolve) |

|_________ TESTING / DEBUGGING |

| |

:-)) |

___________________|__________________|

|

...

36 / 44

STCEConcluding Song/Poem ...5

Optimality

I’m sittin’ in front of the computer screen.Newton’s second iteration is what I’ve just seen.It’s not quite the progress that I would expectfrom a code such as mine– no doubt it must be perfect!Just the facts are not supportive, and I wonder ...

minx∈IRn

f (x)

∇f (x∗) = 0

∇2f (xi )·δi = −∇f (xi )

xi−1 = xi + α · δi

5Lyrics: The Art of Differentiating Computer Programs. ... Page xviiMusic: Think of Fool’s Garden’s “Lemon Tree”

38 / 44

STCE Optimality

My linear solver is state-of-the-art.It does not get better wherever I start.For differentiation is there anything else?Perturbing the inputs– can’t imagine this fails.I pick a small Epsilon, and I wonder ...

A · x = b

A ≡ ∇2f (xi )

x ≡ δi

b = −∇f (xi )

bj ≈f (xi − ε · ej )− f (xi + ε · ej )

2 · ε

39 / 44

STCE Optimality

I wonder how, but I still give it a try.The next change in step size is bound to fly.’cause all I’d like to see is simply optimality.

Epsilon, in fact, appears to be rather small.A factor of ten should improve it all.’cause all I’d like to see is nearly optimality.

A DAD ADADA DAD ADADA DADAD.

ε :=ε

10

bj ≈f (xi − ε · ej )− f (xi + ε · ej )

2 · ε

???

ε := 100 · ε

bj ≈f (xi − ε · ej )− f (xi + ε · ej )

2 · ε

???

40 / 44

STCE Optimality

A few hours later my talk’s getting rude.The sole thing descending seems to be my mood.How can guessing the Hessian only take this muchtime?N squared function runs appear to be the crime.The facts support this thesis, and I wonder ...

Isolation due to KKTIsolation – why not simply drop feasibility?

minx∈IRn

f (x)

IRm 3 c(x) = 0

L(x, λ) = f (x)−λT ·c(x)

∇L(x, λ) = 0

∇2L(x, λ) s.p.d.

???

41 / 44

STCE Optimality

The guy next door’s been sayin’ again and again:An adjoint Lagrangian might relieve my pain.Though I don’t quite believe him, I surrender.

I wonder how but I still give it a try:Gradients and Hessians in the blink of an eye.Still all I’d like to see is simply optimality.Epsilon itself has finally disappeared.Reverse mode AD works, no matter how weird,and I’m about to see local optimality.

L(1)(x, λ, l(1)) = l(1)·∇L

⇓

L(2)(1)

(x, x(2), λ, l(1), l(2)(1)

)

42 / 44

STCE Optimality

Yes, I wonder, I wonder ...

I wonder how but I still give it a try:Gradient and Hessians in the blink of an eye.Still all I’d like to see ...I really need to see ...now I can finally see my cherished optimality:-)

?

43 / 44

adjoint methods in computational finance software tool support … · 2019-02-06 · adjoint...

Documents