arxiv:1607.07203v1 [stat.me] 25 jul 2016 · the parameter cascading method was proposed in ram-say...

arX

iv:1

607.

0720

3v1

[sta

t.ME

] 25

Jul

201

6

Stat Comput manuscript No.(will be inserted by the editor)

Laplace Based Approximate Posterior Inference for DifferentialEquation Models

Sarat C. Dass· Jaeyong Lee· Kyoungjae Lee· Jonghun Park

Received: 10 July 2015 / Accepted: date

Abstract Ordinary differential equations are arguably themost popular and useful mathematical tool for describingphysical and biological processes in the real world. Often,these physical and biological processes are observed with er-rors, in which case the most natural way to model such datais via regression where the mean function is defined by anordinary differential equation believed to provide an under-standing of the underlying process. These regression baseddynamical models are called differential equation models.Parameter inference from differential equation models posescomputational challenges mainly due to the fact that ana-lytic solutions to most differential equations are not avail-able. In this paper, we propose an approximation method forobtaining the posterior distribution of parameters in differ-ential equation models. The approximation is done in twosteps. In the first step, the solution of a differential equationis approximated by the general one-step method which is aclass of numerical numerical methods for ordinary differen-tial equations including the Euler and the Runge-Kutta pro-cedures; in the second step, nuisance parameters are marginal-ized using Laplace approximation. The proposed Laplaceapproximated posterior gives a computationally fast alter-native to the full Bayesian computational scheme (such asMakov Chain Monte Carlo) and produces more accurate and

Sarat C. DassDepartment of Fundamental and Applied Science, UniversitiTeknologi PETRONAS

Jaeyong LeeDepartment of Statistics, Seoul National University

Kyoungjae LeeDepartment of Statistics, Seoul National University1 Gwanak-ro, Gwanak-gu, Seoul, 151-747, KoreaTel.: +82-2-880-8138E-mail: [email protected]

Jonghun ParkDepartment of Industrial Engineering, Seoul National University

stable estimators than the popular smoothing methods (calledcollocation methods) based on frequentist procedures. Foratheoretical support of the proposed method, we prove thatthe Laplace approximated posterior converges to the actualposterior under certain conditions and analyze the relationbetween the order of numerical error and its Laplace ap-proximation. The proposed method is tested on simulateddata sets and compared with the other existing methods.

Keywords Ordinary differential equation· posteriorcomputation· Laplace approximation

1 Introduction

Ordinary differential equations (ODEs) are arguably the mostcommonly used mathematical tool for describing physicaland biological processes in the real world. Popular examplesinclude Lotka-Volterra equation (Alligood et al. 1997), SIR(Susceptible, Infected, Recovered) model (Kermack and McK-endrick 1927) and the continuously stirred tank reactor (CSTR)model (Schmidt 2005). The Lotka-Volterra equation is thedifferential equation describing the dynamics of predator-prey systems. The SIR model is an ODE model for diseaseepidemic describing the relation among the numbers of sus-ceptible, infected and recovered individuals in a closed pop-ulation. The CSTR model describes the surface temperaturechanges of an object at a rate proportional to its relative tem-perature to the surroundings. These are just a few examplesof ODEs.

The ODE model is the nonlinear regression model whoseregression function is expressed as the solution of an ODE.The ODE model depicts the statistical situation of most ap-plications where the parameters of an ODE need to be es-timated based on the noisy data. The statistical inferencefor ODE model, however, poses computational challengesmainly due to the lack of analytical solutions for most ODEs.

http://arxiv.org/abs/1607.07203v1

2 Sarat C. Dass, Jaeyong Lee, Kyoungjae Lee, Jonghun Park

Bard (1974) suggested to minimize an objective func-tion, a suitable measure of lack of fit, in which the solutionof ODE is approximated by numerical integration. The min-imization is carried out by a gradient-based method. But thesolutions are often divergent, stay at a local minimizer andare sensitive to initial values (Cao et al. 2011).

Varah (1982) proposed an estimation method with thefollowing two steps: in the first step, the regression functionis expressed by cubic splines with fixed knots and estimatedby least squares method using the data; in the second step,the parameters of the ODE are estimated by minimizing adistance measure between the ODE and the estimated re-gression function in the first step. Ramsay and Silverman(2005) introduced a two step iteration method where thefirst step of Varah is modified to a penalized least squaresmethod in which a roughness penalty term is introduced tomeasure the difference between the ODE and the estimatedmean function.

The parameter cascading method was proposed in Ram-say et al. (2007). In the parameter cascading approach, pa-rameters are grouped into (1) regularization parameters, (2)parameters in ODE and (3) regression coefficients in the ba-sis expansion of the regression function. Parameters in eachof the three groups are estimated in sequence. First, the re-gression coefficients are estimated given the structural pa-rameters and regularization parameters, then the structuralparameters are estimated given the regularization parame-ters, and finally the regularization parameters are estimatedbased on minimizing penalized least squares.

Gelman et al. (1996) proposed a Bayesian computationalmethod for the inference of pharmacokinetic models. Huanget al. (2006) suggested a hierarchical Bayesian procedure forthe estimation of parameters in a longitudinal HIV dynamicsystem. As it turns out, Bayesian computational schemes forODE models using Markov Chain Monte Carlo type proce-dures as in these two papers result in even bigger challenges.Each time the parameters are sampled from a candidate dis-tribution, numerical integration of ODE needs to be invokedto evaluate the full likelihood. Campbell (2007) adopted thecollocation method to obtain an approximation to the regres-sion function expressed by a differential equation as in Ram-sey et al. (2007). The collocation method was subsequentlycombined with parallel tempering (Geyer 1992) to overcomeinstability of the posterior surface. Incorporating temperingovercomes instabilities but slows down computational speedsignificantly. Recently, Gaussian processes (GP) have beenused to avoid the heavy computation of the numerical in-tegration. Dondelinger et al. (2013) introduced the adap-tive gradient matching (AGM) approach which has a linkto numerical integration but without the corresponding highcomputational cost. Wang and Barber (2014) introduced theGaussian process-ODE (GP-ODE) approach which providesa generative model and simpler graph model than AGM ap-

proach. Actually, GP-ODE approach makes an approxima-tion to allow a generative and proper graph model as Mac-donald et al. (2015) pointed out.

In this paper, to speed up the Bayesian computations, wepropose a Laplace approximated procedure (LAP) for poste-rior inference in differential equation models. The marginalposterior density of the ODE parameter is computed by theLaplace approximation (LA), in which the regression func-tion is approximated by a one-step numerical solver of ordi-nary differential equations. We use the Euler and the fourthorder Runge-Kutta procedures for illustrations. Finally,pos-terior inference is carried out by grid sampling or griddyGibbs sampling from the marginal posterior of the ODE pa-rameters depending on its dimension.

The proposed method has the following advantages. First,for an ODE model with the parameter dimension less than orequal to four, the posterior computations utilizes the MonteCarlo method (not the Markov Chain Monte Carlo method)based on independent sampling; thus, its posterior samplingis significantly faster than methods utilizing full Bayesiancomputations. Even for moderate parameter dimensions, theLAP runs and produces results within an acceptable compu-tational time frame.

The second advantage is that the LAP produces moreaccurate parameter estimates compared to the other existingmethods. In a simulation study, we compared the LAP withthe parameter cascading method (Ramsay et al., 2007), thedelayed rejection adaptive Metropolis algorithm (Haario etal., 2006), GP-ODE approach (Wang and Barber, 2014) andAGM approach (Dondelinger et al., 2013). In the FitzHugh-Nagumo model where the regression function changes morerapidly, the LAP estimator has better performance than thedelayed rejection adaptive Metropolis (DRAM), GP-ODEand AGM approach in the sense of the root mean squarederror (rmse) and the log-likelihood at the parameter esti-mates. The performance of LAP is comparable to the pa-rameter cascading (PC) method in the same sense. The lat-ter criteria judges whether the chosen procedure achieves aparameter estimate that is close to the maximum likelihoodby ascertaining the corresponding log-likelihood value.

Third, inference based on the LAP is numerically stable.Frequentist methods need to maximize the log-likelihoodsurface which has many ripples. So, depending on the start-ing points, optimization algorithms can be trapped in localmaximums. However, in many examples, the ripples of thelog-likelihood surface occur at the periphery of the param-eter space and disappear from the likelihood surface whenthe sample sizen becomes large.

The rest of the paper is organized as follows. In Sect. 2,we lay out inference framework of the differential equationmodels and the priors considered in this paper. The proposedposterior computations are described in Sect. 3. In Sect. 4,we prove that the approximated posterior converges to the

Laplace Based Approximate Posterior Inference for Differential Equation Models 3

true posterior under certain regularity conditions. In Sect.5, using the simulated data sets from three models, we ex-amine the quality of the LA based posterior. In the exampleswe considered, inference based on LAP generates stable andaccurate approximations of the true posterior. We apply theLAP to a real data set, U.S. Census data in Sect. 6. Discus-sions are presented in Sect. 7 whereas details of computa-tions and technical results are relegated to the Appendix.

2 Regression model defined by ODE

We consider the regression model

y(t) = x(t)+ ε(t),

wherey(t) is a p-dimensional vector of observation at timet ∈ [T0,T1],0 ≤ T0 < T1 < ∞ and ε(t) represents an errorterm assumed to arise fromNp(0,σ2Ip) with σ2 > 0 whereNp(µ ,Σ) denotes thep-dimensional normal distribution withmeanµ and covariance matrixΣ . The regression function,x(t), of the regression model is defined as the solution of adifferential equation

x(t) = f (x,u, t;θ ), t ∈ [T0,T1], (1)

where f is a p-dimensional smooth function ofx(t), knowninput functionu(t), time t, and the unknown parameterθ ∈Θ ⊆ Rq with q≥ 1; x(t) denotes the first derivative ofx(t)with respect to timet. The functionx is determined by theinitial value of x, x(T0), θ and the functionu(·). The un-known parameterθ needs to be estimated from observeddata ony(t)s andu(t)s which are given at certain pre-specifiedtime points.

We assume that observed data is collected at the timepointsT0 ≤ t1 < t2 < .. . < tn ≤ T1. Letting yi = y(ti), xi =

x(ti) andεi = ε(ti), we have the following regression model

yi = xi + εi , i = 1,2, . . . ,n, (2)

whereεi are drawn independently fromNp(0,σ2Ip).The value of eachxi , i = 1,2, · · · ,n, is determined by the

initial valuex1, θ andu(·) based on the differential equationmodel (1). When we need to emphasize this dependence,we will denotexi by xi ≡ xi(θ ,x1,u) or xi(θ ,x1) if x is notdependent onu. For simplicity of exposition, the input func-tion u(t) is not considered further in the rest of the paper,but analysis based on a known input function can be easilyaccommodated into our inference framework.

The differential equation (1) involves only the first orderderivatives, but can be used to describe those with higher or-der derivatives. For example, consider a second order equa-tion x(t) = f (x,x, t;θ ). By introducingz(t) = x(t), the dif-ferential equation model can be expressed as

X(t)≡

(x(t)z(t)

)=

(f (x, t;θ )

f (z,x, t;θ )

)≡ F(X, t;θ )

whereX(t)≡ (x(t)T ,z(t)T)T is now a vector with an addedcomponent for the dynamics ofz(t). Since any higher or-der differential equation models can be converted into a firstorder differential equation model based on adding extra dy-namical systems and variables, without loss of generality,we consider only the first order differential equation modelsfor developing our inference procedures in the remainder ofthis paper.

In the model (1) and (2), there are three unknowns,x1,θ and σ2, whose priors are denoted byπ(x1 | σ2), π(θ )andπ(σ2) (or π(τ2) with τ2 = 1/σ2), respectively. In thefollowing, we will take the following specific priors forτ2

andx1:

τ2 ∼ Gamma(a,b) (3)

x1 | τ2 ∼ Np(µx1,cτ−2Ip), (4)

wherec > 0 andGamma(a,b) is the gamma distributionwith parametersa,b> 0 and meana/b. The prior selectionfor (τ2,x1) is guided by conjugacy considerations which en-able components of the posterior to be integrated in closedform. One may select other types of priors for(τ2,x1). How-ever, for large sample sizes, like the ones considered in thispaper, the impact of these priors will be minimal since mostof the inference will be driven and guided by the likelihoodcomponent of the posterior.

3 Posterior Computation

3.1 Posterior ofθ , τ2 andx1

The full joint posterior ofθ , x1 andτ2 given the observationsyn = (y1,y2, . . . ,yn)

T has the expressionπ(θ ,τ2,x1 | yn)

∝ p(yn | θ ,τ2,x1)π(x1 | τ2)π(τ2)π(θ )

=

[n

∏i=1

det(τ−22π Ip)−1/2e−

τ22 ‖yi−xi(θ ,x1)‖

2

]

×det(2πcτ−2Ip)−1/2e−

τ22c ‖x1−µx1‖

2

×ba

Γ (a)(τ2)a−1e−bτ2

×π(θ )

∝ (τ2)12(n+1)p+a−1×e−

τ22 (ngn(x1,θ)+

‖x1−µx1‖2

c +2b)π(θ ),

wheregn(x1) = gn(x1,θ ) =∑ni=1‖yi −xi(θ ,x1)‖

2/n and‖x‖denotes the Euclidean norm of the vectorx, andπ(θ ) is anyprior on θ . The choice ofπ(θ ) can be arbitrary as it doesnot affect the inference onθ for large sample sizesn as iswell known.

In most cases,θ andτ2 are the parameters of primaryinterest whereasx1 is the nuisance parameter. The detailsof obtaining the posterior distributions ofθ andτ2 are out-lined as follows: In the first step, the posterior ofθ andτ2,π(θ ,τ2 | yn), is obtained by marginalizing (i.e., integrating


out)x1. In this marginalization step, two approximations areimplemented: (i) a one-step numerical method for calculat-ing eachxi , i = 1,2, · · · ,n, and (ii) the Laplace method forintegrating outx1. In the next step, as a consequence of con-jugacy, it can be shown that the posterior ofτ2 givenθ andyn follows a gamma distribution, which has the advantagethat it can be easily and directly sampled from. In the thirdstep, after marginalizingτ2, θ is sampled from its poste-rior distribution, π(θ | yn), using either grid sampling orgriddy Gibbs sampling depending on its dimensionq. Byeliminatingx1 andτ2 from the full posterior in the first twostages above, we reduce the dimension of the posterior fromp+q+1 to q, making it easier for thorough exploration ofits surface using grid based sampling as in the third stage.

3.2 Marginalization ofx1: Joint posterior ofθ andτ2

In the marginalization ofx1, we use two approximations.In the first approximation,xi(θ ,x1) is successively approxi-mated by a numerical procedure:

xi ≈ xi−1+(ti − ti−1)φ(xi−1, ti−1;θ ), i = 2, . . . ,n,

where different forms ofφ represent different numerical solversof differential equation. For example, the Euler method isrepresented by

φ(xi−1, ti−1;θ ) = f (xi−1, ti−1;θ );

while the 4-th order Runge-Kutta is represented by

φ(xi−1, ti−1;θ ) =16(ki−1,1+2ki−1,2+2ki−1,3+ ki−1,4), (5)

where

ki−1,1 = f (xi−1, ti−1;θ ),

ki−1,2 = f (xi−1+12

ki−1,1, ti−1+12(ti − ti−1);θ ),

ki−1,3 = f (xi−1+12

ki−1,2, ti−1+12(ti − ti−1);θ ),

ki−1,4 = f (xi−1+ ki−1,3, ti ;θ ).

Let h= max2≤i≤n(ti − ti−1) andxh be the approximation ofx. The global error of the numerical method is defined by

supt∈[T0,T1]

‖x(t)− xh(t)‖.

If the global error isO(hK) for some integerK, we callK tobe the order of the numerical method. Under some smooth-ness conditions, the order of the 4th order Runge-Kutta nu-merical procedure (given in (5)) isK = 4 (Mathews and Fink2004; Suli 2014).

In the second approximation, we integrate outx1 basedon its priorπ(x1 |τ2) defined in (4) and full likelihood using

Laplace approximation for the corresponding integral. Us-ing results from Tierney and Kadane (1986) and Azevedo-Filho and Shachter (1994), the marginal likelihood ofθ andτ2 can be approximated by

L(θ ,τ2) =

∫π(x1 | τ2)L(θ ,τ2,x1)dx1

∝∫(τ2)(n+1)p/2e

− τ22

(ngn(x1)+

‖x1−µx1‖2

c

)

dx1

∝ (τ2)(n+1)p/2e−τ22 u(θ) det

(ngn(x1)+

2c

Ip

)−1/2

×(τ2)−p/2(

1+O(n−3/2))

= (τ2)np/2e−τ22 u(θ)− 1

2 v(θ)(1+O(n−3/2)),

where

gn(x1) =∂ 2 gn(x1,θ )

∂x21

, x1 ≡ x1(θ ) is given by

x1(θ ) = argminx1

(ngn(x1,θ )+

‖x1− µx1‖2

c

),

u(θ ) = ngn(x1)+‖x1− µx1‖

2

c, and

v(θ ) = log det

(ngn(x1)+

2c

Ip

).

It follows from the last expression forL(θ ,τ2) that theapproximate posterior ofθ andτ2 givenyn, based on inde-pendent priorsπ(θ ) andGamma(a,b) on θ andτ2, respec-tively, is given by

π(θ ,τ2 | yn) ∝ π(θ )× (τ2)np2 +a−1e−τ2( 1

2 u(θ)+b) (6)

× det

(ngn(x1)+

2c

Ip

)−1/2

.

Details for the computation of ¨gn(x1) is given in the Ap-pendix. We used the gradient descent and Newton-Raphsonprocedures for obtaining the maximizer ˆx1(θ ) in our exam-ples.

3.3 Marginalization ofτ2: Posterior ofθ

We note from equation (6) that the posterior ofτ2 givenθandyn is proportional to

π(τ2 | θ ,yn) ∝ (τ2)np2 +a−1e−τ2( 1

2 u(θ)+b);

thus, the conditional posterior distribution ofτ2 givenθ andyn is given by

τ2 | θ ,yn ∼ Gamma(a∗,b∗),

wherea∗ = np/2+ a andb∗ = u(θ )/2+ b. Now, by inte-grating outτ2 from the product ofL(θ ,τ2) and the prior of


τ2, we get the marginal likelihood ofθ given by

L(θ ) ∝∫(τ2)

np2 +a−1e−τ2( 1

2u(θ)+b)dτ2

× det

(ngn(x1)+

2c

Ip

)−1/2

=Γ (np

2 +a)(

12u(θ )+b

)np2 +a

det

(ngn(x1)+

2c

Ip

)−1/2

.

Consequently, the posterior ofθ is

π(θ | yn) ∝π(θ )

(12 u(θ )+b

)np2 +a

det(

ngn(x1)+2c Ip)1/2

. (7)

To numerically approximateπ(θ | yn), we propose gridsampling or griddy Gibbs sampling depending on the di-mension ofθ . When the dimensionq of θ is not large (say,q≤ 4), the grid sampling is conceptually simple and numer-ically fast. Whenq is relatively large, we recommend thegriddy Gibbs sampling.

3.4 Posterior sampling ofθ

Whenq≤ 4, we recommend the grid sampling to sampleθfrom the marginal posteriorπ(θ | yn) of θ in (7). LetGΘ ⊂Θbe a grid set that coversΘ and letπd(θ | yn) be the discretedistribution with supportGΘ whose value atθ ∈ GΘ is pro-portional toπ(θ | yn). We will sampleθ from πd(θ | yn).

In practice, the choice of the grid matrixGΘ can be anontrivial task (Joshi and Wilson 2011). To choose a gridset, we adopt the reparametrization technique used by Rueet al. (2009). Letθ 0 be the initial guess for the center of thegrid set, and letΣ = H−1 whereH is the negative Hessianmatrix ofπ(θ |yn) atθ 0. If H is not a positive definite matrix,we replace the negative eigenvalues ofH with the minimumpositive eigenvalue of it. We expressθ with a standardizedvariablezby

θ (z) = θ 0+UD1/2z

whereΣ is diagonalized withΣ =UDUT , U = (u1, . . . ,uq)

andD = diag(λ j). λ j is the eigenvalue ofΣ , andu j is thecorresponding eigenvector,j = 1,2, . . . ,q. The grid pointsare selected for the parametrization ofz. We recommend thetwo step approach in choosing the range of the grid points.In the first step, the grid points for theith coordinatezi ischosen by dividing[−4,4] into 2M1 equal length intervalsresulting 2M1+1 points. Note[−4,4] comes from the roughnormal approximation. For each(2M1+1)q grid points, weevaluateπ(θ (zi)|yn), i = 1,2, . . . ,(2M1+1)q. With these val-ues, we determine the range[Ai ,Bi ] of each coordinatezi ,i = 1,2, . . . ,q. Ai and Bi are defined by the minimum andmaximum ofzi with π(θ (z1, . . . ,zi , . . . ,zq)|yn) > η whereη is a small number close to 0. In our examples, we used

η = 10−5. If the interval[−4,4] is not big enough to con-tain the mass of the posterior andAi andBi can not be se-lected, we perform the first step one more time with largerinterval than[−4,4]. The larger interval can be obtained byapproximating the marginal posterior with normal densitywith larger standard deviation.

After [Ai ,Bi ] are chosen, we move to the second step anddetermine the grid points for accurate computation. The pur-pose of the first step is to determine the grid set, andM1 ischosen as a small positive integer such that(2M1+1)q is notoverwhelmingly large computationally. In our examples, weusedM1 = 5.

In the second step,[Ai ,Bi ] is divided into 2M2 intervalsof equal length. The discrete approximation of the posterioris constructed by evaluating the posterior at(2M2+1)q gridpoints. Grid sampling is done first by samplingθ (i) from thediscrete approximation and the conditionally onθ (i), τ2 issampled fromGamma(np/2+ a,u(θ (i))/2+ b). In our ex-amples, we usedM2 = 15 or 25. Note that the samples fromthis algorithm are independent samples. Whenq is not verylarge, the algorithm is very fast.

We summarize the algorithm below.

1. (Step 1: Reparameterization step)Compute the initial guesses of the centerθ 0 and of theposterior covarianceΣ .Reparametrizeθ using the standardized variablezby

θ (z) = θ 0+UD1/2z

whereΣ =UDUT .2. (Step 2:Finding ranges ofzi )

For eachzi , divide the interval[−4,4] into 2M1 intervalsof equal length. Let

Ai = min{zi : π(zi | yn)≥ η}Bi = max{zi : π(zi | yn)≥ η}.

3. (Step 3: Grid sampling)Divide the intervals[Ai ,Bi ] into 2M2 intervals of equallength and construct grid points.1. For eachθ ∈ GΘ , calculateπ(θ |yn) using (7) and

constructπd(θ | yn).

2. Sampleθ (1),θ (2), . . . ,θ (N) iid∼ πd(θ | yn).

3. For eachi = 1,2, . . . ,N, sample

τ2(i) ∼ Gamma(np/2+a,u(θ (i))/2+b).

Whenq is large (q≥ 5), the construction of the discreteapproximationπd by evaluating the posterior at all the gridpoints can be computationally prohibitive. In this case, werecommend to replace the grid sampling by the griddy Gibbssampling in the above algorithm. In the griddy Gibbs sam-pling, the coordinates ofz is sampled from the conditionalposterior and it does not require the evaluation of the poste-rior at all grid points.


To improve accuracy of the numerical solution of dif-ferential equation, we divided the interval[ti−1, ti ] to m in-tervals and added intermittent time points in computingx.If the differential equation is smooth enough,m= 4 and 1usually suffice for Euler and 4th order Runge-Kutta methodnot to add error rate to that of the Laplace approximation, re-spectively. See Theorem 2. But in practice sometimes largervalues ofmare required. We apply larger values in turn, andif the change in the mean of the posterior is less than 0.1%,we stopped. In our examples, we used the sequence ofm as1,2,4,8,14,20,30, . . ..

4 Convergence of the approximated posterior

4.1 Convergence of the approximated posterior asmincreases

In this section, we show that the posterior with Laplace ap-proximation and numerical method,πLP

m , converges point-wise to the true posterior with an relative error ofO(n−3/2)

asm→ ∞, under some regular conditions.For convenience, letπm ≡ πLP

m . We assumeh ≡ ti+1 −

ti for all i = 2,3, . . . ,n and each[ti−1, ti ] is divided intomsegments; thus, the length of one segment ish/m. Let xm bethe approximation ofxby numerical method withmsegmentandxm(t1) = x(t1) for all m.

The theorem requires the following assumptions.

A1. {x(t) : t ∈ [T0,T1]} is a compact subset ofRp;A2. {y(t) : t ∈ [T0,T1]} is a bounded subset ofRp;A3. the Kth order derivative off (x, t;θ ) with respect tot

exists and is continuous inx andt, whereK is the orderof the numerical methodφ ; and

A4. the functionngn(x1)+‖x1−µx1‖2/chas the unique min-

imum x1.

Theorem 1 Suppose that f(x, t;θ ) is Lipschitz continuousin x, and A1 – A4 hold. Then, for sufficiently large n,

limm→∞

πm(θ ,σ2 | yn) = π(θ ,σ2 | yn)× (1+O(n−3/2)),

for all θ andσ2.

The proof of theorem is given in Appendix.

4.2 Suitable rate of step size with respect to sample size

In this section, we analyze the relation between the step sizeh/mand the approximation error rate of the posterior, whichis motivated by Xue et al. (2010). We assume that the num-ber of the observation goes to infinity andh/m= O(n−α).The large sample size and small step size give accurate in-ference, but they may cause heavy computation. We are in-terested in a reasonable choice of the step sizeh/m when

the sample sizen is growing. Here, reasonable choice meansthat it does not raise the relative error rateO(n−3/2) causedby the Laplace approximation.

Let K be the order of the numerical methodφ . If wedivide intervals[ti−1, ti ] into m segments,max1≤i≤n‖xi − xm

i ‖= O((h/m)K) = O(n−Kα).

Theorem 2 Suppose that f(x, t;θ ) is Lipschitz continuousin x, and A1−A3 hold. Let K be the order of the numeri-cal methodφ and h/m= O(n−α). If α ≥ 5/(2K), then, forsufficiently large n,

πm(θ ,τ2 | yn) = π(θ ,τ2 | yn)× (1+O(n−3/2)),

for all θ andτ2.

Theorem 2 says that if we seth/m= O(n−5/(2K)), thenumerical approximation does not raise the order of the rel-ative error caused by the Laplace approximation. Moreover,even if we takeh/m≪ n−5/(2K), it does not reduce the errorrateO(n−3/2) and only raise the computational cost.

5 Simulated Data Examples

In this section, we test our LAP inference with data sets sim-ulated from three ODE models. The data are generated withpredetermined parameter valueθ , the initial valuex1 anderror varianceσ2.

For the examples in 5.1, we use both the Euler and the4th order Runge-Kutta method to approximate ODE solu-tions. For the examples in 5.2 and 5.3, we use the 4th orderRunge-Kutta method to approximate ODE solutions. TheLAP inference can be extended to other numerical methodsby changing the functionφ(x, t;θ ).

5.1 Newton’s law of Cooling

5.1.1 Model description and data generation

English physicist Isaac Newton believed that temperaturechange of an object is proportional to the temperature differ-ence between the object and its surroundings. This intuitionis captured by Newton’s law of cooling, which is an ODEgiven by

x(t) = θ1(x(t)−θ2), (8)

wherex(t) is the temperature of the object in Celcius at timet, θ1 is a negative proportionality constant andθ2 is the tem-perature of the environment. See Incropera (2006) for thedetails. The solution of the ODE (8) is known and is

x(t) = θ2− (θ2− x1)eθ1t


wherex1 ≡ x(0). Since the analytic form of the solution isknown, it is not necessary to resort to the proposed approx-imate posterior computation method to fit the ODE modelwith (8). We have chosen this example as a testbed for theproposed method. We compare the true posterior without ap-proximation with the approximate posterior obtained by theproposed method.

The model parameters were fixed atx1 = 20,θ =(−0.5,80)T

andσ2 = 25, andy(ti) were generated atti = h(i−1) for i =1,2, . . . ,n. We generated 4 data sets with sample sizesn =

20,50,100,150,which have step sizesh= 0.75,0.3,0.15,0.1,respectively. The effect of sample size on the approximationis investigated below. The data set with sample sizen= 20and the true mean function is given in Figure 1.

The priors were set by

x1 | τ2 ∼ N(µx1 = y1,100/τ2)

τ2 ∼ Gamma(a,b)

θ = (θ1,θ2)∼Uni f orm(−200,0)×Uni f orm(−200,500).(9)

wherea= 0.1,b= 0.01 andy1 = 15.515.The true posterior ofθ andτ2 can be obtained as fol-

lows:

τ2 | θ ,yn ∼ Gamma(np2

+a,12

u(θ )+b)

θ | yn ∼1

(12u(θ )+b)

np2 +a

I(−200< θ1 < 0)

×I(−200< θ2 < 500),

where

u(θ ) = µ2x1/100+

n

∑i=1

z2i − (1/100+

n

∑i=1

e2θ1(i−1)h)−1

×(µx1/100+n

∑i=1

zieθ1(i−1)h)2,

zi = zi(θ ) = yi −θ2+θ2eθ1(i−1)h.

Since the dimension ofθ is only 2, the grid sampling isdeemed to be adequate for samplingθ . For this example, weended up settingM = 25 andh0 = (1,1)T whereh0 is thevector of step sizes for grid matrix. The center of the gridmatrix was chosen asθ 0 = (−0.547,80.933)T by parametercascading method. In total, we have 2,601 grid points. In therest of the paper, we got 10,000 posterior sample from eachexample.

5.1.2 Assessment of the performance of the approximateposteriors

The LAP inference has two approximations: Laplace ap-proximation for the marginal posterior ofθ andτ2 and nu-merical approximation method for the regression functionx.With this example, we investigate the quality of these two

approximations. In particular, we examine (1) the effect ofsample size on the Laplace approximation and (2) that of thenumerical approximation. For the numerical approximationpart, we compare the performance of the Euler method andthe 4th order Runge-Kutta method.

To see the effect of sample size on the Laplace approx-imation, the true posteriorπ(θ ,τ2 | yn) was compared withthe posterior with only Laplace approximationπLP(θ ,τ2 |

yn). Figure 2 shows the true posterior densities and Laplaceapproximated posterior densities of each parameter whenthe sample sizen = 20. Even when the sample size is assmall asn = 20, the Laplace approximated posterior den-sities are almost indistinguishable from the true posterior.Although we have not shown here, we tried the same com-parison plots for the samples with sample sizes as small as5 and 10 and concluded that the approximation is still good.Table 1 shows the similar story; that is, the summary statis-tics of the Laplace approximated posterior are quite close tothose of the true posterior. Table 1 shows only the summarystatistics ofθ1, but the same conclusion has been reached forθ2 andτ2.

To see the effect of the approximation due to the nu-merical methods, the true posteriorπ(θ ,τ2 | yn) was com-pared with the approximate posteriors obtained by apply-ing the Laplace and the numerical methods. In this exam-ple, we used the Euler method and 4th order Runge-Kuttamethod for the numerical method. The intervals between ob-servations[ti−1, ti ] were divided intom segments withm=1,2,4,8,14,20,30, . . .. The approximate posteriors are de-noted byπLP,E

m (θ ,τ2 | yn) and πLP,RKm (θ ,τ2 | yn) whereE

andRK stand for the Euler and Runge-Kutta, andm is thenumber of segments. Figure 3 shows the posterior densitieswith differentmand the true posterior density when the sam-ple sizen=20. The approximate posteriorsπLP,E

m andπLP,RKm

are shown in the first row and the second row, respectively.The approximate posteriorsπLP,RK

m are generally close to thetrue posterior even form= 1, butπLP,E

m show different be-havior. Forθ2 andτ2, πLP,E

m are close to the true posterioreven form= 1, but the marginal posterior ofθ1 of πLP,E

m

deviates from the true posterior. The deviation disappearsasm gets larger. These results can be also confirmed in Ta-ble 1 which includes the posterior summary statistics ofθ1

with different values ofmandn. We represent the results forthe Euler withm= 1,20,50,60 and the Runge-Kutta withm= 1,2. Based on these observations, we recommend theEuler withm= 50 and the Runge-Kutta withm= 1. In Sect.4, we present a theorem, a theoretical basis for this observa-tion. The Runge-Kutta method withm= 1 does not reducethe error rate obtained by Laplace method, while Euler withm= 1 does reduce the error rate and the larger value ofmis needed for the Euler method. The computation times forthe numerical methods with various values ofmandn in thisexample are shown in Table 2.


Fig. 1: The solid line is the true temperature as a function oftime from the Newton’s law of cooling model withx1 = 20, θ = (−0.5, 80)T andn= 20. The scatter plot of the generated data of temperatures and times is also drawn.

Fig. 2: The true posterior densities and the approximate posterior densities with Laplace approximation are shown. Thedata are generated fromNewton’s law cooling model with sample sizen= 20. The red lines represent true values of the parameters,θ = (−0.5,80)T andσ 2 = 25.


Fig. 3: The true posterior density and the approximate posteriors with Laplace approximation and numerical methods forthe Newton’s law ofcooling model are drawn whenn = 20. The true parameter values areθ = (−0.5,80)T andσ 2 = 25. Asm grows, the approximate posterior isgetting closer to the true posterior.

Table 1: Posterior summary statistics ofθ1 from the true posterior,Laplace approximated posterior, posterior with Laplace approximationand numerical approximation method with varying values of the num-ber of stepsmand sample sizesn in Newton’s law of cooling model.

θ1

n Case Mean Median 90% credible interval

n = 20

π -0.563 -0.555 (-0.734, -0.421)πLP -0.563 -0.555 (-0.734, -0.421)

πLP,E

m=1 -0.457 -0.453 (-0.565, -0.360)m=20 -0.563 -0.553 (-0.736, -0.423)m=50 -0.567 -0.557 (-0.740, -0.425)m=60 -0.567 -0.559 (-0.742, -0.425)

πLP,RK m=1 -0.569 -0.561 (-0.744, -0.427)m=2 -0.569 -0.561 (-0.744, -0.427)

n = 50

π -0.589 -0.585 (-0.711, -0.482)πLP -0.589 -0.585 (-0.711, -0.482)

πLP,E

m=1 -0.581 -0.581 (-0.585, -0.576)m=20 -0.589 -0.585 (-0.711, -0.482)m=50 -0.591 -0.586 (-0.711, -0.482)m=60 -0.591 -0.587 (-0.711, -0.482)

πLP,RK m=1 -0.592 -0.588 (-0.711, -0.482)m=2 -0.591 -0.588 (-0.711, -0.482)

Table 2: The computation times (s) for numerical methods with vary-ing values of step sizemand sample sizes in numerical approximationmethod for Newton’s law of cooling model.

n m Euler m 4th order Runge-Kutta

20

1 0.3701 1.107

20 1.80950 4.095

2 1.75060 4.663

50

1 4.9551 2.535

20 4.17850 9.243

2 3.82160 10.981

100

1 1.4541 4.618

20 8.01550 18.138

2 7.19260 21.793

150

1 1.9921 6.452

20 11.31550 25.936

2 9.94460 30.459

5.2 FitzHugh-Nagumo model

5.2.1 Model description and data generation

The action of spike potential in the giant axon of squid neu-rons is modeled by Hodgkin and Huxley (1952). FitzHugh


Table 3: Posterior summary statistics with varying values of step sizem in 4th order Runge-Kutta method for FitzHugh-Nagumo model.Inthe table, C.I. denotes the credible interval.

θ1 θ2

m Mean Median 90% C.I. Mean Median 90% C.I.1 0.199 0.190 (0.150, 0.248) 0.130 0.132 (-0.074, 0.350)2 0.198 0.198 (0.150, 0.247) 0.135 0.134 (-0.071, 0.352)4 0.198 0.198 (0.149, 0.246) 0.135 0.134 (-0.070, 0.352)

θ3 σ2

m Mean Median 90% C.I. Mean Median 90% C.I.1 3.057 3.057 (2.968, 3.143) 0.284 0.282 (0.241, 0.335)2 3.059 3.061 (2.972, 3.143) 0.285 0.283 (0.241, 0.335)4 3.060 3.061 (2.972, 3.143) 0.285 0.282 (0.241, 0.335)

(1961) and Nagumo et al. (1962) simplified this model withtwo variables. The reduced model with no external stimulusis given below:

x1(t) = θ3(x1(t)−13

x31(t)+ x2(t)),

x2(t) = −1θ3

(x1(t)−θ1+θ2x2(t)),

where−0.8< θ1,θ2 < 0.8,0< θ3 < 8, andx1(t) andx2(t)are the voltage across an membrane and outward currentsat timet and called the voltage and recovery variables, re-spectively. We use this parameter space for stable cyclicalbehavior of the system (Campbell 2007). With this example,we show that the Laplace approximated posterior inferenceworks well with appropriate choice ofm.

We generated a simulated data set with model parame-tersθ = (0.2,0.2,3)T,x1 = x(t1) = (−1,1)T andσ2 = 0.25.The time interval was fixed atti −ti−1 =0.2 for i = 2,3, . . . ,nwith n= 100. We divided[ti−1, ti ] into 100 segments and ap-plied the 4th order Runge-Kutta method to got the true meanfunction for simulated data.

For the prior, we hadx1 | τ2 ∼ N2(µx1 = y1,100/τ2I2),τ2 ∼ Gamma(a,b) and θ ∼ Uni f (A) wherea = 0.1,b =0.01,y1 = (−1.449,1.092)T andA= {(θ1,θ2,θ3) : −0.8<

θ1,θ2 < 0.8,0< θ3 < 8}.

5.2.2 Assessment of the performance of the approximateposteriors

We applied the procedure in Sect. 3, to choose the range andthe center of the grid matrix. For the final analysis, we setM = 15 andh0 = (7,7,4)T , so we have 29,791 grid points.The center of the grid matrixθ 0 was(0.199,0.131,3.056)T.Figure 4 shows the posterior densities with differentm. Inthis example, different values ofm shows slight changes inthe posterior approximations. Table 3 shows the posteriorsummary statistics with varying values of step size. We ap-plied the procedure to choosem described in Sect. 3 and inthe final analysism= 2 was used, and Table 4 contains thecomputation times form= 1,2 andn= 100,200.

Figure 5 contains the scatter plots of the observations,the true mean functions, 90% credible lines for the mean

Table 4: The computation times (s) with varying values of step sizemand sample size in numerical approximation method for the FitzHugh-Nagumo model.

n m 4th order Runge-Kutta

1001 111.1372 172.215

2001 131.462 200.973

functions, and the posterior mean functions as well as pre-diction values at 10 future time points whenm= 2.

5.2.3 Comparison with existing methods

We compare the performance of the LAP inference and theother existing methods: the parameter cascading method (Ram-say et al. 2007), the delayed rejection adaptive Metropolis(DRAM) algorithm (Haario et al. 2006) with numerical in-tegration, the Gaussian Process-ODE (GP-ODE) approach(Wang and Barber 2014) and the adaptive gradient match-ing (AGM) approach (Dondelinger et al. 2013). We gener-ate 100 simulated data set as above and compute the abso-lute bias, the standard deviation, the root mean squared error(rmse) and the log-likelihood to use as the measure of per-formance. However, for the GP-ODE approach (Wang andBarber 2014) and AGM approach (Dondelinger et al. 2013),only 20 data set were used because of their long computa-tion times. This long computation times is mainly due to thefact that the implementations of the two approaches werebased on the pure MATLAB codes. For the data genera-tion, θ = (0.2,0.2,3)T ,x1 = (−1,1)T , σ2 = 0.25,h = 0.2andn= 30 were used.

We are using the method of Ramsay et al. (2007) basedon parameter cascading (PC). PC is also called generalizedprofiling. We give the details below.

To represent the state of the ODE,x(t), the PC method-ology uses the collocation method: The collocation methoduses a series of basis expansion to represent thep dimen-sional vectorx(t), that is,

x(t) = (x1(t),x2(t), · · · ,xp(t)) = Φ(t)C (10)

whereΦ(t) = (Φ1(t), · · · ,ΦK(t)) is a set ofK bases evalu-ated at timet, and theK × p matrix C contains the coeffi-cients of the basis functions of each variable in its columns.In other words, expanding (10), thei-th component ofx(t)at timet has the basis function expansion

xi(t) = Φ(t)ci =K

∑k=1

cikΦk(t), (11)

whereci is a column vector of coefficientscik of lengthK,for i = 1,2, · · · , p. The ODE model whose parameters needto be estimated is given by

xi(t) = fi(x(t),θ ) (12)


Fig. 4: Approximate marginal posterior densities for each parameter with varying values ofm for FitzHugh-Nagumo model. The red lines representtrue values of the parameters,θ = (0.2,0.2,3)T andσ 2 = 0.25. Asm grows, the approximate posterior seems to be stabilized.

for i = 1,2, · · · , p.

PC involves a penalized likelihood criteriaJ≡ J(C,θ ,λ )which is based on the coefficients of basis expansionsC, theunknown parameterθ to be estimated andλ ≡ (λ1, · · · ,λp),the penalty (or smoothing) parameters. The criteriaJ is re-flects two competing goals based on two competing terms.The first term inJ(C,θ , |λ ) measures how well the statefunction values fit the data whereas the second term mea-sures how closely each of the state functions satisfy the cor-responding differential equation (12). The smoothing pa-rameters measures the weight of each competing term; whenλis, i = 1,2, · · · , p, are large, more and more emphasis is puton havingxi(t)s in (11) satisfy the differential equation in

(12), as opposed to fitting the data and vice versa whenλistend to zero.

PC optimization is based on two levels: An inner op-timization step nested within an outer optimization. In theinner optimization,θ andλ components are held fixed, andan inner optimization criterion is optimized with respect tothe coefficients in matrixC only. In effect, this makesJ =

J(C(θ ,λ ),θ ,λ ) a function ofθ and λ only. In the outeroptimization step,J is optimized with respect toθ keepingλ fixed. This essentially makesJ ≡ J(C(θ (λ ),λ ),θ (λ ),λ )now a function ofλ only. The smoothing parametersλ andnumber of basis functionsK are finally chosen based onnumerical stability of the parameter estimates. This is the


Fig. 5: Scatter plot of the observations generated from the FitzHugh-Nagumo model, and plots of 90% credible set lines and true states are drawnwhenm= 2. Predictions of 10 time points ahead are also drawn. The upper, lower and middle dotted lines are the 95% and 5% quantilesand meanof the posterior, respectively. The solid line in the middleis the true value of the statex(t), and the star-shaped points are the observations.

key idea underlying the generalized profiling or parametercascade algorithms in Ramsay (2007) and Cao and Ramsay(2009).

The implementation of the PC method was carried outusing theCollocInfer package (Hooker et al. 2014) in R.This package uses B-spline basis functions forΦk(t),k =

1, ...,K. B-spline basis functions are constructed by joiningpolynomial segments end-to-end at junctions specified byknots. Since our method usedm= 2, we set 2n−1 equallyspaced knots on[t1, tn] to get a twice number of knots thanthe data points. The finer knots gave negligible improve-ment in parameter estimate while slowing down the com-putational speed. We chose the three-order of B-spline basiswhich was used in Ramsay et al. (2007) for the same model.For the choice of the tuning parameterλ , we used both themanual procedure and the automatic procedure. We adoptedthe procedure of Ramsay et al. (2007) which tries larger val-ues ofλ and choosesλ manually which gives a stable result.The quartiles of the parameter estimates for 100 simulationdata sets were obtained asλ is varied from 10−2 to 106. Af-ter that, thisλ set at 105. For the automatic procedure, theforward prediction error (FPE) in Hooker et al. (2010) wasused. We divided the each data set into ten part, fromt1 tot10, andt11 to t20, and so on. For one data set, the averagedFPE was obtained asλ is varied from 10−2 to 106, and theoptimalλ which minimizes FPE was chosen.

The DRAM algorithm (Haario et al. 2006) is a variantof the standard Metoropolis-Hastings algorithm (Metropo-lis et al. 1953; Hastings 1970). We chose the DRAM algo-rithm with numerical integration to compare the computa-

tion time with our LAP inference. To implement it, we usedthemodMCMC function from theFME package (Soetaert andPetzoldt 2010) in R. The maximal number of tries for the de-layed rejection was fixed to 1, so actually we used the adap-tive Metropolis algorithm (Haario et al. 2001). The initialvalues were set by themodFit function which finds the bestfit parameters using optimization approaches. The varianceof the proposal distribution was set by sample covariance ofparameters(x1,θ ) scaled with 2.44/(p+q) in every 100 it-eration. We got 10,000 posterior sample from the DRAM al-gorithm. The DRAM was used here as a benchmark methodfor obtaining exact results based on Markov Chain MonteCarlo procedures but at the expense of computational time.

Gaussian processes (GP) have been used to avoid theheavy computation of the numerical integration. AGM (Don-delinger et al. 2013) and GP-ODE (Wang and Barber 2014)are two state-of-art paradigms for modelling the differen-tial equation models using GP. The gradient matching (GM)approach (Calderhead et al. 2009) was developed to inferthe differential equation models based on GP. However, GMapproach has disadvantages that the posterior of hyperpa-rameter of GP does not depend on the differential equa-tion system and it is not a generative model. Dondelinger etal. (2013) tried to remedy the former problem by substitut-ing the directed edges between the hyperparameter and theGP with the undirected edges. This modification improvedthe performance of the inference, but it is still not a gen-erative model. GP-ODE approach was developed by Wangand Barber (2014) to construct a simple generative model.They developed a different paradigm from gradient match-


ing approaches and argued the GP-ODE approach performsat least as well as the AGM. However, GP-ODE has beenshown to be conceptually problematic. Recently, Macdon-ald et al. (2015) pointed out that GP-ODE approach makesan undesirable approximation: GP-ODE eliminates the edgebetween the true state variablex(t) and the latent variablex(t) which should be same to the true state variable. Mac-donald et al. (2015) showed that AGM achieves better resultthan GP-ODE for the simple ODE model having missingvalues and comparable result for the FitzHugh-Nagumo sys-tem.

To compare our LAP inference with the GP based ap-proaches, we illustrated the results from both GP-ODE andAGM approaches. The MATLAB code for GP-ODE is avail-able from github, and Macdonald et al. (2015) provided theMATLAB code for the AGM approach. All parameters weresampled from griddy Gibbs sampling. The range for eachparameter componentθi was chosen by[θ R

i ±4sd(θ Ri )]where

θ Ri is the estimate from the parameter cascading method

(Ramsay et al. 2007). We devided it into 31 intervals ofequal length to set the same number of grid for each param-eter. For the variance function of GP, we chose squared ex-ponential functioncφ j (t, t

′) = σxj exp(−l j(t − t ′)2) and dis-

cretized the parametersσxj , l j over the ranges[0.1,1], [5,50]

with intervals 0.1,5, respectively. We got 10,000 posteriorsample from the GP-ODE and AGM approaches.

As we have concluded in the above simulation, we usedthe 4th order Runge-Kutta method withm= 2 for the LAPinference and got 10,000 posterior sample from each sim-ulation data set. The same grid set as GP-ODE was chosenfor fair comparison.

Table 5 shows the table of mean of the absolute biases,the standard deviations, the root mean squared errors (rmse)for θ , log-likelihoods with estimated parameters and com-putations times. The absolute bias term is calculated by

|Biass(θi)|= |θi − θ si |, i = 1,2,3

where Biass is the bias ins-th simulation andθ s is the esti-mate ofθ in s-th simulation andθ = (0.2,0.2,3)T. For theBayesian procedures, we use the posterior mean as the esti-mate of the parameter.

Table 5 shows that the LAP inference has better perfor-mance than the other methods in terms of rmse. The LAPhas lower rmse and higher log-likelihood than those of theDRAM method, while taking 25% less computational timecompared to DRAM. The PC method withλ = 105 hasthe fastest computation time and has a slightly higher log-likelihood value than that of LAP, while the automatic choiceof λ (PC FPE) has a comparable rmse and a relatively lowerlog-likelihood value as shown in Table 5. The GP-ODE andAGM do not perform well in terms of the computationalspeed and accuracy (as determined by rmse).

Table 5: The table of mean of the absolute biases, the standard devia-tions (sd), the root mean squared errors (rmse) forθ , log-likelihoodswith estimated parameters and computations times (s) in theFitzHugh-Nagumo model. The results for the Laplace approximated posterior(LAP) inference, parameter cascading (PC) method, delayedrejectionadaptive Metropolis (DRAM) algorithm, GP-ODE approach andadap-tive gradient matching (AGM) approach are shown. PC method withforward prediction error (FPE) criterion for the choice ofλ is denotedby PC FPE.

LAP PC PC FPE

Absolute biasθ1 0.179 0.256 0.264θ2 0.222 0.246 0.257θ3 0.598 0.815 0.762

sdθ1 0.220 0.290 0.369θ2 0.308 0.370 0.470θ3 0.679 0.825 0.913

rmseθ1 0.298 0.493 0.488θ2 0.400 0.578 0.576θ3 0.954 1.299 1.290

Log-likelihood -7.128 -7.059 -7.665Computation time 64.033 3.476 34.452

Software R and Fortran90 R and C/C++ R and C/C++

DRAM GP-ODE AGM

Absolute biasθ1 0.239 0.159 0.457θ2 0.512 0.193 0.168θ3 0.654 1.439 1.842

sdθ1 0.295 0.336 0.089θ2 0.621 0.411 0.267θ3 0.756 0.511 0.074

rmseθ1 0.397 0.405 0.472θ2 0.833 0.472 0.333θ3 1.071 1.563 1.844

Log-likelihood -8.551 -28.247 -25.567Computation time 85.327 6222.268 5235.615

Software R and C/C++ MATLAB MATLAB

We have also checked the values of log-likelihood at theparameter estimates for each method which are shown inTable 5. Note that LAP consistently achieves the higher log-likelihood value corresponding to its parameter estimates(which is comparable to PC and DRAM) than GP-ODE andAGM. Note that if there was a parameter estimate from an-other method, different from the LAP estimate but compa-rable in explaining the data, the log-likelihood at that pa-rameter value (for the other method) should be close to thelog-likelihood value corresponding to the LAP estimate. ButTable 5 shows that in terms of the log-likelihood values, thisis not the case; GP-ODE and AGM yield significantly lowerlog-likelihood values suggesting suboptimal parameter esti-mates from them.

To understand suboptimal parameter estimates, we notethat the Fitz-Hugh-Nagumo ODE model has large regionsof the likelihood corresponding to unidentifiable parame-ter values. However, this (large regions of the likelihoodwhere parameter values are unidentifiable) does not arisefor parameter values close to the maximum likelihood point(MLE) and for large sample sizen. Our method based onLaplace approximation finds this maximum likelihood esti-mate and hence the bias is of orderO(n−1/2). As seen fromthe log-likelihood values in Table 5, GP-ODE and AGMgive parameter estimates away from the MLE, and hencemay belong to these regions of unidentifiability. In other


words, parameter estimates from GP-ODE and AGM aregenuinely deviating away from the true parameter value.Over repeated simulation experiments, these genuine devia-tions get translated into the large overall biases and standarddeviations given in Table 5 (especially componentθ3).

The coverage probabilities of 95% credible interval ofthe LAP inference are comparable to those of the confi-dence intervals obtained by the other methods. The cov-erage probabilities forθ1,θ2,θ3 of the LAP inference are0.94,0.96,0.94, while those of the PC method and DRAMalgorithm are 0.84,0.91,0.81 and 0.96,0.93,0.97, respec-tively.

5.3 Predator-prey system

Fussmann et al. (2000) suggested a mathematical model forpredator-prey food chain between two microbials. The fol-lowing system of equations describes the predator-prey os-cillation between Brachionus calyciflorus and Chlorella vul-garis:

x1(t) = δ (N∗− x1(t))−θ1x1(t)x2(t)θ2+ x1(t)

x2(t) =θ1x1(t)x2(t)θ2+ x1(t)

−θ3x2(t)x4(t)θ4+ x2(t)

·1θ5

− δx2(t)

x3(t) =θ3x2(t)x3(t)θ4+ x2(t)

− (δ +θ6+θ7)x3(t)

x4(t) =θ3x2(t)x3(t)θ4+ x2(t)

− (δ +θ6)x4(t).

In the above model,x1,x2,x3,x4 represent the concentra-tions of nitrogen, Chlorella, reproducing Brachionus and to-tal Brachionus, respectively. The unit of Chlorella and Bra-chionus isµmolL−1. N∗ is the inflow concentration of nitro-gen, andδ is dilution rate. We have seven positive param-eters,θ = (θ1, . . . ,θ7)

T . θ1 andθ2 are the maximum birthrate and the half-saturation constant of Chlorella.θ3 andθ4 represent the maximum birth rate and the half-saturationconstant of Brachionus.θ5,θ6, andθ7 are the assimilationefficiency, the mortality and the decay of fecundity of Bra-chionus.

The dimension of the parameter is 7 which is too bigfor the grid sampling. Instead, we applied the griddy Gibbssampling method. We generated a simulated data set withmodel parametersθ =(3.3,0.43,2.25,1.5,2.5,0.055,0.4)T,x1 = (1,3,5,5)T , σ2 = 1 andN∗ = 8,δ = 0.68. We used theabsolute value of the data because the concentrations shouldbe positive. The parameter settings come from Cao et al.(2008). We just modified the scale ofx1,x2,θ2,θ4,θ5 andN∗ to control the scale ofx1 andx2. The time interval wasfixed atti − ti−1 = 0.1 for i = 2,3, . . . ,n wheren= 100. Weapplied the 4th order Runge-Kutta method to get the truemean function for simulated data withm= 1.

For the prior, we hadx1 | τ2 ∼ N4(µx1 = y1,100/τ2I4),τ2 ∼ Gamma(a,b) and θ ∼ Uni f (A) wherea = 0.1,b =

0.01,y1=(0.103,3.185,6.298,5.137)T andA= {(θ1, . . . ,θ7) :0< θ1,θ3,θ4,θ5 < 70,0< θ2,θ6,θ7 < 10}.

For this example, we ended up settingM = 15 andh0 =

(0.35,0.40,0.15,0.17,0.40,0.07,0.06)T, so we have 31 gridpoints for eachθ j , j = 1, . . . ,7. The center of the grid matrixθ 0 was chosen as(3.295,1.444,2.225,1.393,3.883,0.248,0.397)T.

Total 50,000 posterior sample was drawn by the griddyGibbs sampling and every 5-th draw was used as samplefor the posterior inference; finally we got 10,000 posteriorsample. It took 19.454 hours for this simulation. Figure 6shows the approximate marginal posterior densities for someparameters. The summary statistics for the posterior is givenat Table 6 with true value of the parametersθ andσ2.

Figure 7 contains the scatter plots of the observations forx1,x2,x3,x4, the true mean functions, 90% credible lines forthe mean functions, and the posterior mean functions as wellas prediction values at 10 future time points.

As commented by one of the referees, we have adjustedthe amount of error in our simulations to make the SNR(signal-to-noise-ratio) close to 10 to resemble real life sit-uations. For the predator-prey model, the scale of some pa-rameters were chosen to control the variance of signal. As aresult, on the Newton’s law of cooling model, the SNR forthe different dataset sizes were obtained as follows: whenn= 20, SNR= 10.493, whenn= 50, SRN= 8.313, whenn=100,SNR=7.660, whenn=150,SNR= 7.450. For theFitzHugh-Nagumo system, the SNR on species 1 is 8.598and on species 2 is 1.928. For the predator-prey system, theSNRs onx1,x2,x3,x4 are 5.712,6.112,5.696,8.369, respec-tively.

Table 6: Posterior summary statistics with step sizem= 1 in 4th orderRunge-Kutta method for Predator-prey model.

θ1 θ2 θ3 θ4 θ5

True value 3.3 0.43 2.25 1.5 2.5Mean 3.298 1.410 2.351 1.214 3.634Median 3.202 1.338 2.345 1.212 3.5635% quantile 2.828 0.911 2.145 0.985 3.03095% quantile 3.948 2.084 2.585 1.484 4.523

θ6 θ7 σ 2

True value 0.055 0.4 1Mean 0.229 0.450 0.940Median 0.211 0.445 0.9375% quantile 0.117 0.381 0.83495% quantile 0.416 0.525 1.056


Fig. 6: Approximate marginal posterior densities forθ1 (top left), θ2 (top right),θ3 (bottom left) andσ 2 (bottom right) for Predator-prey model.The step sizem= 1 in 4th order Runge-Kutta method is used. We omit the densities for the rest of the parameters, and the red lines representtruevalues of the parameters.

6 U.S. Census Data: logistic equation

A simple logistic equation describing the evolution of an an-imal population over time is

x(t) =θ1

θ2x(t)(θ2− x(t)), (13)

wherex(t) is the population size at timet, θ1 is the rate ofmaximum population growth,θ2 is the maximum sustain-able population sometimes called carrying capacity (Baca¨er2011). The analytic form of the solution to (13) can be found.See Law et al. (2003) for the details. In this example, how-ever, we will use only the differential equation (13) to fit themodel.

U.S. takes a census of its population every 10 years whichis mandated by the U. S. Constitution. It has important ram-ifications for many aspects. For instance, the census resultsare used in the decision of government program funding,congressional seat, and electoral votes. This data set repre-sents U.S. population from 1790 to 2010. The population isrepresented by one million units.

Since the census has been conducted every 10 years from1790 to 2010, we have totaln= 23 observations,(y1, . . . ,y23),with h= ti − ti−1 = 10, i = 1,2, . . . ,n.

We set the prior asx1 | τ2 ∼ N(µx1 = y1,100/τ2), τ2 ∼

Gamma(a,b)andθ ∼Uni f orm(0,1)×Uni f orm(300,1000),wherea= 0.1,b= 0.01 andy1 = 3.929. The lower limit ofθ2 was set to 300 which is slightly lower than the populationin year 2010,y23 = 308.746.

To apply the grid sampling method, we used the param-eter cascading estimate as a center of an initial grid set.For the final analysis, we setM = 35,h0 = (0.12,0.4)T andθ 0 = (0.020,532.125)T.

We tried several step sizesm and concluded that withm= 1 the posterior had been stabilized sufficiently. For thenumerical approximation, we used the 4th order Runge-Kuttamethod. The marginal posterior densities ofθ1, θ2 andσ2

whenm= 1 are given in Figure 8. Figure 9 includes the scat-ter plot of the observations, the 90% credible interval linesand posterior mean as well as prediction values of popula-tions at 10 future time points. Table 7 shows the summarystatistics for the posterior.


Fig. 7: Scatter plot of the observations generated from the Predator-prey model, and plots of 90% credible set lines and truex(t) values are drawnwhenm= 1. Predictions of 10 time points ahead are also drawn. The upper, lower and middle dotted lines are the 95% and 5% quantilesand meanof the posterior, respectively. The solid line in the middleis the true value of the statex(t), and the star-shaped points are the observations.

Fig. 8: The marginal posterior densities ofθ1, θ2 andσ 2 in the logistic model with U.S. census data when the step parameterm= 1.


Fig. 9: Scatter plot of the U.S. census, 90% credible interval lines and posterior mean when the step parameterm= 1. Prediction values ofpopulations at 10 future time points are also drawn.

Table 7: Posterior summaries withm= 1 for U.S. census data. C.I.denotes the credible interval.

θ1 θ2 σ2

Mean 0.020 534.528 28.276Median 0.020 532.125 26.31490% C.I. (0.019, 0.021) (482.817, 597.867) (16.430, 46.367)

7 Discussion

In this paper, we proposed a posterior computation method,the LAP, based on the Laplace method and numerical ap-proximation for ODE. There are three advantages of the pro-posed method. First, when the dimension of the parameteris small, the computation is fast. The main issue of the pro-posed method is computation time when the dimension ofθis high.

Second, the proposed method produces accurate estima-tor which has comparable or better performance than theother methods: the PC method, the DRAM, the GP-ODE

and the AGM. Although it is not entirely clear, we suspectthat the spline approximation of the PC method and the GPapproximation of the GP based approaches tox(t)may causeloss of efficiency. This issue also needs further investigation.

Third, the proposed method is numerically stable. Thefrequentist methods need to maximize the log-likelihood sur-face which has many ripples. However, in many examplesthe ripples of the log-likelihood surface occurs at peripheryof the parameter space and disappear in the likelihood sur-face as the sample sizen increases.

Referees pointed that there is a potential to use latticerule or sparse grid construction which can control the com-putational costs of the proposed method. It is an attractiveway to reduce the computation time of LAP whenq is large.However, there were several challenges that need to be over-come. For the lattice rule, the best way of transforming in-tegration domain to optimize its performance in the case ofODE models is not clear. It should be chosen carefully be-


cause poor transformation will cause the evaluations of ra-tios of densities under the lattice rule to be quite unstable.For the sparse grid, the existence of negative weights pre-vents computing the posterior probability on each grid point:we can compute the posterior moments only. To get the pos-terior probability on each grid point, the weights should bepositive everywhere. Furthermore, in our experiment, the es-timate from the sparse grid heavily depended on the rangeof the grid set and the accuracy of the sparse grid. We ap-plied the sparse grid construction to the proposed method forpredator-prey system. The Gauss-Legendre quadrature ruleon[0,1]was used with accuracy level 10. The integration do-main was transformed to the same domain in Sect. 5.3 usinglinear transformation. In this settings, we obtained the meanvalue(2.868,1.281,1.969,1.225,2.696,0.116,0.362)T for θ .The estimated mean or other moments were quite unstableto the choice of the domain and accuracy level.

Although we concluded that these problems are not easyto get around, the lattice rule and sparse grid are interest-ing idea to enhance the practical use of our LAP inference.Thus, we decided that applying the lattice rule or sparse gridto LAP inference deserves a separate investigation and pub-lication.

A Appendix

A.1 Computation of ¨gn(x1).

Recall that

gn(x1) =1n

n

∑i=1

‖yi −xi‖2,

wherexi = x(ti ) for i = 1,2, . . .,n andx(t) = (x1(t),x2(t), . . . ,xp(t))T .For the following discussion, we use the following convention for vec-tors and matrices. Suppose we have an array of real numbersai jk withindices i = 1,2, . . . , I , j = 1,2, . . .,J and k = 1,2, . . . ,K. Let (ai jk)(i)denote the column vector with dimensionI

(ai jk)(i) = (a1 jk,a2 jk, . . .,aI jk)T

and(ai jk )( j ,k) denote the matrix with dimensionsJ×K

(ai jk)( j ,k) =

ai,1,1 ai,1,2 . . . ai,1,Kai,2,1 ai,2,2 . . . ai,2,K. . . . . . . . . . . .

ai,J,1 ai,J,2 . . . ai,J,K

.

The indices in the the subscript with parenthesis are the indices run-ning in the vector or the matrix. The object with one running indexis a column vector, while the object with two running indicesa ma-trix where the first and the second running index are for the row andcolumn, respectively.

Note that

gn(x1) =1n

n

∑i=1

gni(x1),

wheregni(x1)= yTi yi −2xT

i yi +xTi xi . Thus, the(l ,k)thelement of ¨gn(x1)

is∂ 2gn

∂x1l ∂x1k=

1n

n

∑i=1

∂ 2gni

∂x1l ∂x1k.

Note∂gni

∂x1k=−2

p

∑j=1

yi j∂xi j

∂x1k+2

p

∑j=1

xi j∂xi j

∂x1k

and

∂ 2gni

∂x1l ∂x1k=−2

p

∑j=1

yi j∂ 2xi j

∂x1l ∂x1k+2

p

∑j=1

(∂xi j

∂x1l

∂xi j

∂x1k+xi j

∂ 2xi j

∂x1l ∂x1k

).

The above equation can be written in a matrix form

(∂ 2gni

∂x1l ∂x1k)(l ,k) = −2

p

∑j=1

(∂ 2xi j

∂x1l ∂x1k)(l ,k)yi j

+2(∂xi j

∂x1l)(l , j)(

∂xi j

∂x1k)( j ,k)

+2p

∑j=1

(∂ 2xi j

∂x1l ∂x1k)(l ,k)xi j

= 2(∂xi j

∂x1l)(l , j)(

∂xi j

∂x1k)( j ,k)

+2p

∑j=1

(∂ 2xi j

∂x1l ∂x1k)(l ,k)(xi j −yi j ).

Thus,

gn(x1) =2n

n

∑i=1

((

∂xi j

∂x1l)(l , j)(

∂xi j

∂x1k)( j ,k)+

p

∑j=1

(∂ 2xi j

∂x1l ∂x1k)(l ,k)(xi j −yi j )

).

The derivatives ofxi with respect tox1 can be computed by usingthe sensitivity equation for ODE. See Hooker (2009). Let

zjl (t) =∂x j(t)

∂x1lor Z(t) =

(∂x j (t)

∂x1l

)

( j ,l ), j , l = 1, . . ., p

be the sensitivity of the statex j with respect to the initial valuex1l . Thesensitivity equation is given by

zjl (t) =∂∂ t

∂x j (t)

∂x1l=

∂∂x1l

x j (t)

=p

∑u=1

∂ f j(x, t;θ )∂xu(t)

∂xu(t)∂x1l

=p

∑u=1

∂ f j(x, t;θ )∂xu(t)

zul(t),

or in matrix notation,

Z(t) =

(∂ f j(x, t;θ )

∂xu(t)

)

( j ,u)·Z(t) (14)

with an initial conditionZ(t1) = Ip. For givenθ and t, the coeffi-cient∂ f j(x, t;θ )/∂xu(t) is calculated easily. It is a linear ODE problemwhose initial condition is known. We can solve (14) using some nu-merical methods such as Runge-Kutta method.∂ 2xi j/(∂x1l ∂x1k) canbe computed similarly.

A.2 Proof of Theorem 1

Proof The results of Tierney and Kadane (1986) and Azevedo-Filhoand Shachter (1994) assume several regularity conditions such as theexistence of a unique global maximum as well as the existenceofhigher order derivatives (up to sixth order) of the likelihood function. Inparticular, our methods for approximating the ODE model work onlyunder the assumption of a unique maximum of the likelihood function.Thus, we assume that the likelihood surface does not includeany ridges(that is, continuum regions with equal values of the global maximum).


Using the result in Tierney and Kadane (1986) and Azevedo-Filhoand Shachter (1994), we have

πm(θ ,τ2 | yn) = c−1m

∫Lm(θ ,τ2,x1)π(θ ,τ2,x1)dx1

×(1+O(n−3/2)),

wherecm =∫

Lm(θ ,τ2,x1)π(θ ,τ2,x1)dx1dθdτ2. Note the full likeli-hoodL(θ ,τ2,x1) is

L(θ ,τ2,x1) ∝ e−τ22 ngn(x1)× (τ2)np/2,

andLm(θ ,τ2,x1) is the corresponding term withgn replaced bygmn . If

Lm(θ ,τ2,x1) converges toL(θ ,τ2,x1) asm→ ∞ for all θ ∈ Θ ,τ2 >0,x1 ∈R

p andyn, by the dominated convergence theorem,cm −→ c asm→ ∞. Thus,

limm→∞

πm(θ ,τ2 | yn) = c−1∫

L(θ ,τ2,x1)π(θ ,τ2,x1)dx1

×(1+O(n−3/2))

= π(θ ,τ2 | yn)× (1+O(n−3/2))

which is the desired result.To complete the proof, we need to proveLm(θ ,τ2,x1)−→ L(θ ,τ2,x1)

as m→ ∞, and it suffices to provengmn (x1) −→ ngn(x1) as m→ ∞.

Since we assume the Lipschitz continuity off , the ODE has a uniquesolution with initial conditionx(t1) = x1. Assumptions A1 and A3 im-plies

supx,t

‖dK

dtKf (x, t;θ )‖=: B< ∞

for some constantsB> 0. The local errors of theKth order numericalmethod are given by

‖x(ti )−x(ti−1)−hφ (xi−1, ti−1;θ )‖ ≤ B′hK+1, i = 2, . . . ,n

for someB′ > 0, which depends only on supt ‖dK f (x, t;θ )/(dtK)‖ ≤ B(Palais and Palais, 2009). Thus, the local errors are uniformly bounded.It implies the global errors uniformly bounded

‖xi −xhi ‖ ≤ChK

for some constantC > 0.Thus,

|ngn(x1)−ngmn (x1)| =

∣∣n

∑i=1

‖yi −xi‖2−

n

∑i=1

‖yi −xmi ‖

2∣∣

=n

∑i=1

(‖yi −xi‖+‖yi −xm

i ‖)

×∣∣‖yi −xi‖−‖yi −xm

i ‖∣∣

≤n

∑i=1

(2‖yi −xi‖+‖xi −xm

i ‖)‖xi −xm

i ‖

≤n

∑i=1

(2Cy +2Cx+‖xi −xm

i ‖)‖xi −xm

i ‖

≤n

∑i=1

(2Cy+2Cx+C

( hm

)K)

C( h

m

)K

≍ n( h

m

)K, asm−→ ∞, (15)

where supt∈[T0,T1]‖y(t)‖ < Cy < ∞, supt∈[T0,T1]

‖x(t)‖ < Cx < ∞. Thiscompletes the proof.

A.3 Proof of Theorem 2

Proof If α > 5/(2K), asn goes to infinity,n(h/m)K = O(n1−αK) =O(n−3/2) and it converges to to zero. UnderA1−A3, we have shownin the proof of Theorem 1 that|ngn(x1)−ngm

n (x1)|=O(n(h/m)K ). Forfixed τ2 > 0,

e−τ22 ngm

n (x1) = e−τ22 [ngn(x1)+ngm

n (x1)−ngn(x1)]

= e−τ22 ngn(x1)×e−

τ22 [ngm

n (x1)−ngn(x1)]

= e−τ22 ngn(x1)×e−

τ22 O(n( h

m)K )

= e−τ22 ngn(x1)×

(1+O

(n( h

m

)K))

becauseex = 1+O(x) for sufficiently smallx. It implies

πm(θ ,τ2 | yn) ∝∫

Lm(θ ,τ2,x1)π(θ ,τ2,x1)dx1× (1+O(n−3/2))

=∫

L(θ ,τ2,x1)π(θ ,τ2,x1)dx1× (1+O(n−3/2))

×

(1+O

(n( h

m

)K))

∝ π(θ ,τ2 | yn)× (1+O(n−3/2))×

(1+O

(n( h

m

)K))

for sufficiently largen. If α > 5/(2K), i.e., n(h/m)K ≤ n−3/2, (1+O(n−3/2))× (1+O(n(h/m)K )) is (1+O(n−3/2)).

References

1. Alligood K, Sauer T, Yorke J (1997) Chaos: An IntroductiontoDynamical Systems. Chaos: An Introduction to Dynamical Sys-tems, Springer

2. Azevedo-Filho A, Shachter RD (1994) Laplace’s method approx-imations for probabilistic inferencein belief networks with contin-uous variables. In: Proceedings of the Tenth InternationalConfer-ence on Uncertainty in Artificial Intelligence, Morgan KaufmannPublishers Inc., San Francisco, CA, USA, UAI’94, pp 28–36

3. Bacaer N (2011) Verhulst and the logistic equation (1838). In: AShort History of Mathematical Population Dynamics, Springer, pp35–39

4. Barber D, Wang Y (2014) Gaussian processes for bayesian esti-mation in ordinary differential equations. In: Proceedings of the31st International Conference on Machine Learning (ICML-14),pp 1485–1493

5. Bard Y (1974) Nonlinear parameter estimation. Academic Press6. Calderhead B, Girolami M, Lawrence ND (2009) Accelerating

bayesian inference over nonlinear differential equationswith gaus-sian processes. In: Advances in neural information processing sys-tems, pp 217–224

7. Campbell D (2007) Bayesian Collocation Tempering and Gen-eralized Profiling for Estimation of Parameters from DifferentialEquation Models. Canadian theses, McGill University (Canada)

8. Cao J, Ramsay JO (2009) Generalized profiling estimation forglobal and adaptive penalized spline smoothing. ComputationalStatistics & Data Analysis 53(7):2550–2562

9. Cao J, Fussmann GF, Ramsay JO (2008) Estimating a predator-prey dynamical model with the parameter cascades method. Bio-metrics 64(3):959–967

10. Cao J, Wang L, Xu J (2011) Robust estimation for ordinary differ-ential equation models. Biometrics 67(4):1305–1313

11. Dondelinger F, Filippone M, Rogers S, Husmeier D (2013) Odeparameter inference using adaptive gradient matching withgaus-sian processes. Journal of Machine Learning Research - Workshopand Conference Proceedings: The 16th International Conferenceon Artificial Intelligence and Statistics 31:216–228


12. FitzHugh R (1961) Impulses and physiological states in theoreti-cal models of nerve membrane. Biophysical journal 1:445–466

13. Fussmann GF, Ellner SP, Shertzer KW, Hairston Jr NG (2000)Crossing the hopf bifurcation in a live predator-prey system. Sci-ence 290(5495):1358–1360

14. Gelman A, Bois F, Jiang J (1996) Physiological pharmacokineticanalysis using population modeling and informative prior distri-butions. Journal of the American Statistical Association 91:1400–1412

15. Geyer CJ, Statistics MUMSO (1992) Markov Chain Monte CarloMaximum Likelihood. Defense Technical Information Center

16. Haario H, Saksman E, Tamminen J (2001) An adaptive metropolisalgorithm. Bernoulli pp 223–242

17. Haario H, Laine M, Mira A, Saksman E (2006) Dram: efficientadaptive mcmc. Statistics and Computing 16(4):339–354

18. Hastings WK (1970) Monte carlo sampling methods using markovchains and their applications. Biometrika 57(1):97–109

19. Hodgkin A, Huxley A (1952) A quantitative description ofmem-brane current and its application to conduction and excitation innerve. Journal of Physiology 117:500–544

20. Hooker G (2009) Forcing function diagnostics for nonlinear dy-namics. Biometrics 65(3):928–936

21. Hooker G, Ellner SP, Roditi LDV, Earn DJ (2010) Parameterizingstate–space models for infectious disease dynamics by general-ized profiling: measles in ontario. Journal of The Royal SocietyInterface p rsif20100412

22. Hooker G, Xiao L, Ramsay J (2014) CollocInfer:Collocation Inference for Dynamic Systems. URLhttp://CRAN.R-project.org/package=CollocInfer ,r package version 1.0.1

23. Huang Y, Liu D, Wu H (2006) Hierarchical Bayesian methodsfor estimation of parameters in a longitudinal HIV dynamic sys-tem. Biometrics 62(2):413–423, DOI 10.1111/j.1541-0420.2005.00447.x

24. Incropera FP (2006) Fundamentals of Heat and Mass Transfer.John Wiley & Sons

25. Joshi C, Wilson S (2011) Grid based bayesian inference forstochastic differential equation models. Technical Paper, TrinityCollege Dublin

26. Kermack WO, McKendrick A (1927) A Contribution to the Math-ematical Theory of Epidemics. Proceedings of the Royal Societyof London Series A, Containing Papers of a Mathematical andPhysical Character 115(772):700–721

27. Law R, Murrell DJ, Dieckmann U (2003) Population Growth inSpace and Time: Spatial Logistic Equations. Ecology 84(1):252–262

28. Macdonald B, Higham C, Husmeier D (2015) Controversy inmechanistic modelling with gaussian processes. In: Journal of Ma-chine Learning Research: Workshop and Conference Proceedings,Microtome Publishing, vol 37, pp 1539–1547

29. Mathews J, Fink K (2004) Numerical Methods Using MATLAB.Featured Titles for Numerical Analysis Series, Pearson PrenticeHall

30. Metropolis N, Rosenbluth AW, Rosenbluth MN, Teller AH, TellerE (1953) Equation of state calculations by fast computing ma-chines. The journal of chemical physics 21(6):1087–1092

31. Nagumo J, Arimoto S, Yoshizawa S (1962) An active pulsetransmission line simulating nerve axon. Proceedings of the IRE50:2061–2070

32. Palais RS, Palais RA (2009) Differential equations, mechanics,and computation, vol 51. American Mathematical Soc.

33. Ramsay JO, Silverman BW (2005) Functional Data Analysis, 2ndedn. Springer Series in Statistics, Springer

34. Ramsay JO, Hooker G, Campbell D, Cao J (2007) Parame-ter estimation for differential equations: a generalized smooth-ing approach. Journal of the Royal Statistical Society, Series B

69(5):741–796, DOI 10.1111/j.1467-9868.2007.00610.x35. Rue H, Martino S, Chopin N (2009) Approximate bayesian infer-

ence for latent gaussian models by using integrated nested laplaceapproximations. Journal of the Royal Statistical Society,Series B71(2):319–392

36. Schmidt LD (2005) The Engineering of Chemical Reactions, 2/E.Topics in Chemical Engineering - Oxford University Press, Ox-ford University Press, Incorporated

37. Soetaert K, Petzoldt T (2010) Inverse modelling, sen-sitivity and monte carlo analysis in R using packageFME. Journal of Statistical Software 33(3):1–28, URLhttp://www.jstatsoft.org/v33/i03/

38. Suli E (2014) Numerical solution of ordinary differential equa-tions. Lecture Notes at University of Oxford

39. Tierney L, Kadane JB (1986) Accurate Approximations forPos-terior Moments and Marginal Densities. Journal of the AmericanStatistical Association 81(393):82–86

40. Varah JM (1982) A spline least squares method for numericalparameter estimation in differential equations. SIAM Journal onScientific and Statistical Computing 3(1):28–46, DOI 10.1137/0903003

41. Xue H, Miao H, Wu H (2010) Sieve estimation of constant andtime-varying coefficients in nonlinear ordinary differential equa-tion models by considering both numerical error and measurementerror. Annals of statistics 38(4):2351

http://CRAN.R-project.org/package=CollocInfer

http://www.jstatsoft.org/v33/i03/

arxiv:1607.07203v1 [stat.me] 25 jul 2016 · the parameter cascading method was proposed in ram-say...

Documents