tulane march 2017 talk

Simulating the Mean Efficientlyand to a Given Tolerance

Fred J. HickernellDepartment of Applied Mathematics, Illinois Institute of Technology

hickernell@iit.edu mypages.iit.edu/~hickernell

Thanks to Lan Jiang, Tony Jiménez Rugama, Jagadees Rathinavel,and the rest of the the Guaranteed Automatic Integration Library (GAIL) team

Supported by NSF-DMS-1522687

Thanks for your kind invitation

Introduction IID Monte Carlo Low Discrepancy Sampling Bayesian Cubature Examples References

Estimating/Simulating/Computing an Integral

Gaussian probability =

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx

option price =

Rdpayoff(x)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2

looooooomooooooon

PDF of Brownian motion at d times

Bayesian β̂j =

Rdβj prob(β|data)dβ =

Rd βj prob(data|β) probprior(β)dβş

Rd prob(data|β) probprior(β)dβ

Sobol’ indexj =

[0,1]2d

[output(x)´ output(xj, x

1´j)]output(x 1)dxdx 1

[0,1]d output(x)2 dx´[ş

[0,1]d output(x)dx]2

Rdg(x)dx = E[f (X)] =

Rdf (x)ν(dx) =?, µ̂n =

i=1wif (xi)

How to choose ν, txiuni=1, and twiu

ni=1 to make |µ´ µ̂n| small? (trio identity)

Given εa, how big must n be to guarantee |µ´ µ̂n| ď εa? (adaptive cubature)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx

option price =

Rdpayoff(x)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2

looooooomooooooon

Bayesian β̂j =

Sobol’ indexj =

[0,1]2d

i=1wif (xi)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx

option price =

Rdpayoff(x)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2

looooooomooooooon

Bayesian β̂j =

Sobol’ indexj =

[0,1]2d

i=1wif (xi)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx

option price =

Rdpayoff(x)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2

looooooomooooooon

Bayesian β̂j =

Sobol’ indexj =

[0,1]2d

i=1wif (xi)

Estimating/Simulating/Computing the MeanGaussian probability =

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx

option price =

Rdpayoff(x)

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2

looooooomooooooon

Bayesian β̂j =

Sobol’ indexj =

[0,1]2d

i=1wif (xi)

Given εa, how big must n be to guarantee |µ´ µ̂n| ď εa? (adaptive cubature) 2/16

Product Rules Using Rectangular Gridsµ =

Rdf (x)ν(dx) « µ̂n =

i=1wif (xi)

∣∣∣∣∣ż 1

0f (x)dx´

i=1wif (ti)

∣∣∣∣∣ = O(m´r), then∣∣∣∣ż[0,1]d

f (x)dx

i1=1¨ ¨ ¨

id=1wi1 ¨ ¨ ¨wid f (ti1 , . . . , tid)

∣∣∣∣∣= O(m´r) = O(n´r/d)

assuming rth derivatives in each direction exist.

But the computational costbecomes prohibitive for large dimensions, d:

d 1 2 5 10 100n = 8d 8 64 3.3E4 1.0E9 2.0E90

Product rules are typically a bad idea unless d is small.

i=1wif (xi)

∣∣∣∣∣ż 1

0f (x)dx´

i=1wif (ti)

∣∣∣∣∣ = O(m´r), then∣∣∣∣ż[0,1]d

f (x)dx

i1=1¨ ¨ ¨

id=1wi1 ¨ ¨ ¨wid f (ti1 , . . . , tid)

∣∣∣∣∣= O(m´r) = O(n´r/d)

assuming rth derivatives in each direction exist.

But the computational costbecomes prohibitive for large dimensions, d:

d 1 2 5 10 100n = 8d 8 64 3.3E4 1.0E9 2.0E90

Product rules are typically a bad idea unless d is small.

i=1wif (xi)

∣∣∣∣∣ż 1

0f (x)dx´

i=1wif (ti)

∣∣∣∣∣ = O(m´r), then∣∣∣∣ż[0,1]d

f (x)dx

i1=1¨ ¨ ¨

id=1wi1 ¨ ¨ ¨wid f (ti1 , . . . , tid)

∣∣∣∣∣= O(m´r) = O(n´r/d)

assuming rth derivatives in each direction exist. But the computational costbecomes prohibitive for large dimensions, d:

d 1 2 5 10 100n = 8d 8 64 3.3E4 1.0E9 2.0E90

Product rules are typically a bad idea unless d is small. 3/16

Monte Carlo Simulation in the News

Sampling with a computer can be fastHow big is our error?

IID Monte Carloµ =

i=1f (xi), xi

IID„ ν

µ´ µ̂n =µ´ µ̂n

std(f (X))/?

nlooooooomooooooon

CNF„(0,1)

nloomoon

DSC(txiu)

ˆ std(f (X))loooomoooon

VAR(f)

trio identity (Meng, 2017+)

µ´ µ̂n =

Rdf (x) (ν´ ν̂n)(dx)

ν̂n =1n

i=1δxi

P[|µ´ µ̂n| ď errn]99%for

by the ,

where is the sample variance

Central Limit Theorem Stopping Rule for IID Monte Carlo

i=1f (xi), xi

IID„ ν

std(f (X))/?

nlooooooomooooooon

CNF„(0,1)

nloomoon

DSC(txiu)

VAR(f)

P[|µ´ µ̂n| ď errn] « 99%

for errn =2.58ˆ 1.2σ̂

by the Central Limit Theorem (CLT),

where σ̂2 is the sample variance.

Central Limit Theorem Stopping Rule for IID Monte Carlo

i=1f (xi), xi

IID„ ν

std(f (X))/?

nlooooooomooooooon

CNF„(0,1)

nloomoon

DSC(txiu)

VAR(f)

P[|µ´ µ̂n| ď errn] « 99%

for errn =2.58ˆ 1.2σ̂

by the Central Limit Theorem (CLT),

where σ̂2 is the sample variance. But the CLT is only an asymptotic result, and1.2σ̂ may be an overly optimistic upper bound on σ.

Berry-Esseen Stopping Rule for IID Monte Carlo

i=1f (xi), xi

IID„ ν

std(f (X))/?

nlooooooomooooooon

CNF„(0,1)

nloomoon

DSC(txiu)

VAR(f)

P[|µ´ µ̂n| ď errn] ě 99%for Φ

n errn /(1.2σ̂nσ))

+ ∆n(´?

n errn /(1.2σ̂nσ), κmax) = 0.0025

by the Berry-Esseen Inequality,

where σ̂2nσ

is the sample variance using an independent sample from that used tosimulate the mean, and provided that kurt(f (X)) ď κmax(nσ) (H. et al., 2013;Jiang, 2016)

Adaptive Low Discrepancy Sampling Cubature

[0,1]df (x)dx

µ̂n =1n

i=1f (xi), xi Sobol’ or lattice

Normally n should be a power of 2

Let tpf (k)uk denote the coefficients of theFourier Walsh or complex exponentialexpansion of f .

Let tω(k)uk be someweights. Then

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

[0,1]df (x)dx

µ̂n =1n

Let tpf (k)uk denote the coefficients of theFourier Walsh or complex exponentialexpansion of f . Let tω(k)uk be someweights. Then

µ´ µ̂n =

0‰kPdual

pf (k)∥∥∥! pf(k)ω(k)

∥∥∥2‖tω(k)u0‰kPdual‖2

loooooooooooooooooooomoooooooooooooooooooon

CNFP[´1,1]

ˆ ‖tω(k)u0‰kPdual‖2loooooooooomoooooooooon

DSC(txiuni=1)=O(n´1+ε)

∥∥∥∥∥#

pf (k)ω(k)

∥∥∥∥∥2

looooooomooooooon

VAR(f)

[0,1]df (x)dx

µ̂n =1n

Assuming that the pf (k) do not decay erratically as kÑ∞, the discretetransform,

rfn(k)(

k, may be used to bound the error reliably (H. and Jiménez

Rugama, 2016; Jiménez Rugama and H., 2016; H. et al., 2017+):

|µ´ µ̂n| ď errn := C(n)ÿ

certaink

∣∣∣rfn(k)∣∣∣6/16

Bayesian Cubature—f Is Randomµ =

i=1wi f (xi)

Assume f „ GP(0, s2Cθ) (Diaconis, 1988;O’Hagan, 1991; Ritter, 2000; Rasmussen andGhahramani, 2003)

RdˆRdCθ(x, t)ν(dx)ν(dt)

RdCθ(xi, t)ν(dt)

i=1, C =

(Cθ(xi, xj)

)ni,j=1

Choosingw =(wi)n

i=1 = C´1c is optimal

b(c0 ´ cTC´1c

)yTC´1y

nlooooooooooooooomooooooooooooooon

CNF„N(0,1)

c0 ´ cTC´1cloooooooomoooooooon

yTC´1y

nlooooomooooon

VAR(f)

where y =(f (xi)

)ni=1.

But, θ needs to be inferred (by MLE), and C´1 typicallyrequires O(n3) operations

Bayesian Cubature—f Is Random

i=1wi f (xi)

Assume f „ GP(0, s2Cθ) (Diaconis, 1988;O’Hagan, 1991; Ritter, 2000; Rasmussen andGhahramani, 2003)

RdˆRdCθ(x, t)ν(dx)ν(dt)

RdCθ(xi, t)ν(dt)

i=1, C =

(Cθ(xi, xj)

)ni,j=1

Choosingw =(wi)n

i=1 = C´1c is optimal

P[|µ´ µ̂n| ď errn] = 99% for errn = 2.58c(

c0 ´ cTC´1c) yTC´1y

where y =(f (xi)

)ni=1. But, θ needs to be inferred (by MLE), and C´1 typically

requires O(n3) operations7/16

Gaussian Probability

exp(´ 1

2tTΣ´1t

(2π)d det(Σ)dt Genz (1993)

[0,1]d´1f (x)dx

For some typical choice of a, b, Σ, d = 3, εa = 0; µ « 0.6763Worst 10% Worst 10%

εr Method % Accuracy n Time (s)IID Monte Carlo 100% 8.1E4 1.8E´2

1E´2 Sobol’ Sampling 100% 1.0E3 5.1E´3Bayesian Lattice 100% 1.0E3 2.8E´3

IID Monte Carlo 100% 2.0E6 3.8E´11E´3 Sobol’ Sampling 100% 2.0E3 7.7E´3

Bayesian Lattice 100% 1.0E3 2.8E´3

1E´4 Sobol’ Sampling 100% 1.6E4 1.8E´2Bayesian Lattice 100% 8.2E3 1.4E´2

Bayesian lattice cubature uses covariance kernel C for which C is circulant,and operations on C require only O(n log(n)) operations 8/16

Asian Option Pricing

fair price =

Rde´rT max

j=1Sj ´ K, 0

e´xTΣ´1x/2

(2π)d/2 |Σ|1/2 dx « $13.12

Sj = S0e(r´σ2/2)jT/d+σxj = stock price at time jT/d,

min(i, j)T/d)d

Worst 10% Worst 10%εa = 1E´4 Method % Accuracy n Time (s)

Sobol’ Sampling 100% 2.1E6 4.3Sobol’ Sampling w/ control variates 97% 1.0E6 2.1

The coefficient of the control variate for low discrepancy sampling is different thanfor IID Monte Carlo (H. et al., 2005; H. et al., 2017+)

Sobol’ IndicesY = output(X), where X „ U[0, 1]d; Sobol’ Indexj(µ) describes how muchcoordinate j of input X influences output Y (Sobol’, 1990; 2001):

Sobol’ Indexj(µ) :=µ1

µ2 ´ µ23, j = 1, . . . , d

µ1 :=

[0,1)2d[output(x)´ output(xj, x

µ2 :=

[0,1)doutput(x)2 dx, µ3 :=

[0,1)doutput(x)dx

output(x) = ´x1 + x1x2 ´ x1x2x3 + ¨ ¨ ¨+ x1x2x3x4x5x6 (Bratley et al., 1992)

εa = 1E´3, εr = 0 j 1 2 3 4 5 6n 65 536 32 768 16 384 16 384 2 048 2 048

Sobol’ Indexj 0.6529 0.1791 0.0370 0.0133 0.0015 0.0015{Sobol’ Indexj 0.6528 0.1792 0.0363 0.0126 0.0010 0.0012

Sobol’ Indexj(pµn) 0.6492 0.1758 0.0308 0.0083 0.0018 0.0039

Summary

The error in simulating the mean can be decomposed as a trio identity(Meng, 2017+; H., 2017+)Knowing when to stop a simulation of the mean is not trivial (H. et al., 2017+)The Berry-Esseen inequality can tell us when to stop an IID simulationFourier analysis can tell us when to stop a low discrepancy simulationBayesian cubature can tell us when to stop a simulation if you can afford thecomputational costAll methods can be fooled by nasty functions, fRelative error tolerances and problems involving functions of integrals canbe handled (H. et al., 2017+)Our algorithms are implemented in the Guaranteed Automatic IntegrationLibrary (GAIL) (Choi et al., 2013–2015), which is under continuousdevelopment

Upcoming SAMSI Quasi-Monte Carlo Program

Thank you

Slides available at www.slideshare.net/fjhickernell/tulane-march-2017-talk

References I

Bratley, P., B. L. Fox, and H. Niederreiter. 1992. Implementation and tests of low-discrepancysequences, ACM Trans. Model. Comput. Simul. 2, 195–213.

Choi, S.-C. T., Y. Ding, F. J. H., L. Jiang, Ll. A. Jiménez Rugama, X. Tong, Y. Zhang, and X. Zhou.2013–2015. GAIL: Guaranteed Automatic Integration Library (versions 1.0–2.1).

Cools, R. and D. Nuyens (eds.) 2016. Monte Carlo and quasi-Monte Carlo methods: MCQMC,Leuven, Belgium, April 2014, Springer Proceedings in Mathematics and Statistics, vol. 163,Springer-Verlag, Berlin.

Diaconis, P. 1988. Bayesian numerical analysis, Statistical decision theory and related topics IV,Papers from the 4th Purdue symp., West Lafayette, Indiana 1986, pp. 163–175.

Genz, A. 1993. Comparison of methods for the computation of multivariate normal probabilities,Computing Science and Statistics 25, 400–405.

H., F. J. 2017+. Error analysis of quasi-Monte Carlo methods. submitted for publication,arXiv:1702.01487.

H., F. J., L. Jiang, Y. Liu, and A. B. Owen. 2013. Guaranteed conservative fixed width confidenceintervals via Monte Carlo sampling, Monte Carlo and quasi-Monte Carlo methods 2012, pp. 105–128.

H., F. J. and Ll. A. Jiménez Rugama. 2016. Reliable adaptive cubature using digital sequences,Monte Carlo and quasi-Monte Carlo methods: MCQMC, Leuven, Belgium, April 2014, pp. 367–383.arXiv:1410.8615 [math.NA].

References IIH., F. J., Ll. A. Jiménez Rugama, and D. Li. 2017+. Adaptive quasi-Monte Carlo methods. submittedfor publication, arXiv:1702.01491 [math.NA].

H., F. J., C. Lemieux, and A. B. Owen. 2005. Control variates for quasi-Monte Carlo, Statist. Sci. 20,1–31.

Jiang, L. 2016. Guaranteed adaptive Monte Carlo methods for estimating means of randomvariables, Ph.D. Thesis.

Jiménez Rugama, Ll. A. and F. J. H. 2016. Adaptive multidimensional integration based on rank-1lattices, Monte Carlo and quasi-Monte Carlo methods: MCQMC, Leuven, Belgium, April 2014,pp. 407–422. arXiv:1411.1966.

Meng, X. 2017+. Statistical paradises and paradoxes in big data. in preparation.

O’Hagan, A. 1991. Bayes-Hermite quadrature, J. Statist. Plann. Inference 29, 245–260.

Rasmussen, C. E. and Z. Ghahramani. 2003. Bayesian Monte Carlo, Advances in Neural InformationProcessing Systems, pp. 489–496.

Ritter, K. 2000. Average-case analysis of numerical problems, Lecture Notes in Mathematics,vol. 1733, Springer-Verlag, Berlin.

Sobol’, I. M. 1990. On sensitivity estimation for nonlinear mathematical models, Matem. Mod. 2,no. 1, 112–118.

. 2001. Global sensitivity indices for nonlinear mathematical models and their monte carloestimates, Math. Comput. Simul. 55, no. 1-3, 271–280.

Maximum Likelihood Estimation of the Covariance Kernelf „ GP(0, s2Cθ), Cθ =

(Cθ(xi, xj)

)ni,j=1

y =(f (xi)

)ni=1, µ̂n = cT

θ̂C´1θ̂y

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

P[|µ´ µ̂n| ď errn] = 99% for errn =2.58?

b(c0,θ̂ ´ c

Tθ̂C´1θ̂cθ̂)yTC´1

There is a de-randomized interpretation of Bayesian cubature (H., 2017+)

f P Hilbert space w/ reproducing kernel Cθ and with best interpolant rfy

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

= argminθ

z P Rn :∥∥rfz‖θ ď ‖rfy‖θ

)|µ´ µ̂n| ď

c0,θ̂ ´ cTθ̂C´1θ̂cθ̂

loooooooooomoooooooooon

‖error representer‖θ̂

yTC´1θ̂y

looooomooooon

‖rfy‖θ̂

if∥∥f ´rfy

∥∥θ̂ď

2.58∥∥rf∥∥

θ̂?n

(Cθ(xi, xj)

)ni,j=1

y =(f (xi)

)ni=1, µ̂n = cT

θ̂C´1θ̂y

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

b(c0,θ̂ ´ c

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

= argminθ

)|µ´ µ̂n| ď

yTC´1θ̂y

looooomooooon

‖rfy‖θ̂

if∥∥f ´rfy

∥∥θ̂ď

2.58∥∥rf∥∥

θ̂?n

(Cθ(xi, xj)

)ni,j=1

y =(f (xi)

)ni=1, µ̂n = cT

θ̂C´1θ̂y

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

b(c0,θ̂ ´ c

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

= argminθ

|µ´ µ̂n| ď2.58?

yTC´1θ̂y

looooomooooon

‖rfy‖θ̂

if∥∥f ´rfy

∥∥θ̂ď

2.58∥∥rf∥∥

θ̂?n

(Cθ(xi, xj)

)ni,j=1

y =(f (xi)

)ni=1, µ̂n = cT

θ̂C´1θ̂y

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

b(c0,θ̂ ´ c

θ̂ = argminθ

yTC´1θ y

[det(C´1θ )]1/n

= argminθ

)|µ´ µ̂n| ď

yTC´1θ̂y

looooomooooon

‖rfy‖θ̂

if∥∥f ´rfy

∥∥θ̂ď

2.58∥∥rf∥∥

θ̂?n

tulane march 2017 talk

Science

real talk with tulane university higher education -...

carleton biology talk : march 2014

street talk | march 2012 issue

3m talk march 2010

let's talk business march 2015

drum talk march 2016

parent talk magazine | march 2015

ryerson talk (march 20, 2012)

march 2013 stanford river talk

street talk march 2014

thani talk 2016 march

tarporley talk march 2012

trout talk march 2016

march ports talk

tapteal talk march 2017

peace talk - march 2011

tarporley talk march 09

quality enhancement plan suggestions march 19, 2009 the...

truck talk march 2015

titan talk march