automatic differentiation in euroad workshop - finlay scott... · • current differentiation...
TRANSCRIPT
![Page 1: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/1.jpg)
Automatic Differentiation in
Finlay Scott & Iago Mosqueira
![Page 2: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/2.jpg)
Structure
• What is R? – history, strengths, limitations• Current differentiation options in R• How we have used AD with R• Next steps – AD in R
![Page 3: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/3.jpg)
What is R?
• R website - http://www.r-project.org/• “R is a free software environment for statistical
computing and graphics.”• Similar to S (Bell Laboratories)• Development started in 1993• Multi-platform (*nix, Windows, MacOS)• Scripting language• Object oriented and functional programming• Vectorised• Dynamically link to C / C++ / Fortran code
![Page 4: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/4.jpg)
R - what is it good for?• Wide variety of statistical and graphical
techniques:– linear and nonlinear modelling– classical statistical tests– time-series analysis– classification– clustering– ...
• Highly extensible• Excellent graphical capabilities
– Presentation– Data exploration
![Page 5: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/5.jpg)
Publication quality plots
![Page 6: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/6.jpg)
EconomicsBiology
Epidemiology
Data mining
High performance computing Optimisation
Genomics
Pharmacokinetics
Business analysisEnvironmental modelling
Biostatistics
Forecasting
Bayesian networksSpatial modelling
At recent useR conference
![Page 7: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/7.jpg)
NY Times article
![Page 8: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/8.jpg)
Why the interest?• Open Source and therefore ‘Free’
– Not just as in ‘Free beer’ but as in ‘Free software’
• Released under GNU General Public License• This means:
– Transparent (source code available)– Distributed peer review– ‘given enough eyeballs, all bugs are shallow’ – Eric
Raymond, The Cathedral & The Bazaar– Blurs boundary between developer and user
• Director of technology product marketing at SAS: “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”
![Page 9: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/9.jpg)
Why the interest (2)
• Highly extensible – ‘packages’• 1875 packages officially hosted on CRAN• Can ‘simply’ add particular functions or can
make more significant changes e.g. parrallelisation
• Link to C / C++ / Fortran• Link to databases (e.g. MySQL, SQLite)• People have added editors, GUIs, Sweave• Huge community• Costs nothing!
![Page 10: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/10.jpg)
Source: Hornik, 2008
![Page 11: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/11.jpg)
Differentiation and R• Default optimisation methods (including quasi-
Newton methods e.g. BFGS)• 20+ packages devoted to differentiation• Whenever derivatives are required, can supply
gradient function.• Otherwise, generally, numerical approximations
using finite differences are used.• All the associated problems – speed and
accuracy.• Some symbolic differentiation of expressions
(deriv) but simple (not Maple or Mathematica)• R would greatly benefit from generic AD
interface
![Page 12: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/12.jpg)
Demo of Hessian calculation in R
• Speelpenning’s example from ADOL-C
∏−
=
==1
0
)(n
iixxfy
• Can we calculate the Hessian in R?
• Use the numDeriv package
![Page 13: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/13.jpg)
![Page 14: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/14.jpg)
How we have used AD and R
• Fisheries modelling requires projection under various constraints (e.g. catch must not exceed quota)
• Fishing Effort as independent variable• Used ADOL-C libraries (Walther, Kowarz & Griewank)• Implemented simple solvers (Newton-Raphson method)• R dynamically links to the C++ code• But… not generic. It’s application specific• Can pass parameter values from R to C• But not functions• ‘Hard wired and hard work’ - for ‘average’ R user, writing
C / C++ is hard
![Page 15: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/15.jpg)
Example of using ADOL-C with R
User interface
C++
Function
ADOL-C Hessian calculation
Parameter values
Hessian
![Page 16: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/16.jpg)
What we would like…
• Generic interface to AD methods, in R• E.g. write a function in R, use AD to get
first and higher derivatives of that function.• Need is certainly there – useR 2009• ‘Finding a solution was fast, working out
that we had found a solution was the problem’
![Page 17: Automatic Differentiation in EuroAd Workshop - Finlay Scott... · • Current differentiation options in R • How we have used AD with R • Next steps – AD in R. ... • Can ‘simply’](https://reader033.vdocuments.us/reader033/viewer/2022051722/5aa2187f7f8b9a84398c9673/html5/thumbnails/17.jpg)
What next?
• Work is very long term• Many skilled people in R community who want to
help drive this forward• But assistance from the AD community would be
very helpful• If you are interested:
– John Nash: [email protected]– John Nash’s wiki:
http://macnash.telfer.uottawa.ca/optimx/doku.php– Martin Maechler: [email protected]– Developers mailing list: http://www.r-project.org/– [email protected]