introduction to ad-3.4, an automatic differentiation library in haskell
DESCRIPTION
Haskellの自動微分ライブラリ Ad-3.4 の紹介(の試み) If you don't see 21 slides in this presentation, try this one (re-uploaded): http://www.slideshare.net/nebuta/130329-ad-by-ekmettTRANSCRIPT
(An attempt of)Introduction to ad-3.4,
an automatic differentiation library in Haskell
by Nebuta
3/31/2013Ekmett study meeting
in Shibuya, Tokyo
Any comments or correction to the material are welcome
About myself
Nebuta (@nebutalab) https://github.com/nebuta
My research areas:A graduate student, studying biophysical chemistry and quantitative biology (2010−)Imaging live cells, analyzing microscopy images by Scala on ImageJ
My interest in softwares:Programming languages (Haskell, Scala, Ruby, etc)Image processing, data visualization, web designBrainstorming and lifehack methods that take advantage of IT, etc.
Where my interest in Haskell came from:MATLAB、ImageJで細胞の顕微鏡画像の解析 (2010年) → MATLAB, Javaはいまいち使いづらい → Scalaっていうイケてる言語がある(2011年) → 関数型? → Haskell 面白い!(2011年)
ad-3.4, an automatic differentiation library
Installation
array (≥0.2 & <0.5), base (4.*), comonad (≥3), containers (≥0.2 & <0.6), data-reify (0.6.*), erf (2.0.*),free (≥3), mtl (≥2), reflection (≥1.1.6),tagged (≥0.4.2.1), template-haskell (≥2.5 & <2.9)
Dependencies
Differentiation of arbitrary mathematical functions
$ sudo cabal install ad simple-reflect
For symbolic differentiation
What you can do
Taylor expansionCalculation of gradient, Jacobian, and Hessian, etc.
記号で微分するのに使う
How to use ad-3.4
>> :m + Numeric.AD>> diff sin 01.0>> :m + Debug.SimpleReflect>> diff sin x -- x :: Expr is defined in Debug.SimpleReflectcos x * 1>> diff (\x -> if x >= 0 then 1 else 0) 00.0
https://github.com/ekmett/ad/blob/master/README.markdown#examples
Not delta function nor undefined.
Derivative with a symbol!
※Derivative of atrigonometric function
Differentiation of a single-variable scalar function
Gradient>> grad (\[x,y] -> exp (x * y)) [x,y][0 + (0 + y * (0 + exp (x * y) * 1)),0 + (0 + x * (0 + exp (x * y) * 1))]>> grad (\[x,y] -> exp (x * y)) [1,1][2.718281828459045,2.718281828459045]
How to use (continued)
Taylor expansionPrelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp 0 d[exp 0 * 1,1 * exp 0 * (1 * d / 1),(0 * exp 0 + 1 * exp 0 * 1) * (1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x d[exp x * 1,1 * exp x * (1 * d / 1),(0 * exp x + 1 * exp x * 1) * (1 * d / 1 * d / (1 + 1))]Prelude Numeric.AD Debug.SimpleReflect> take 3 $ taylor exp x 0[exp x * 1,1 * exp x * (1 * 0 / 1),(0 * exp x + 1 * exp x * 1) * (1 * 0 / 1 * 0 / (1 + 1))]
•Taylor expansion is an infinite list!•No simplification, and slow in higher order terms
Taylor expansion (general)
Exponential function
Equality of functions>> sin x == sin xTrue>> diff sin xcos x * 1>> diff sin x == cos x * 1True>> diff sin x == cos x * 0.5 * 2False
And so on.
Cool! (no simplification, though...)
How to use (continued)
Cf. Mechanism of automatic differentiationhttp://en.wikipedia.org/wiki/Automatic_differentiation
??
(f + g)’ = f’ + g’
要は、合成関数の微分を機械的に順次適用していく、という認識で良いかと思われる
I don’t understand it yet.(What’s the difference from symbolic differentiation?)
Read a Wikipedia article
It seems to be mechanical, successive application of rules of differentiation for
composite functions.
I’ll try to appreciate the type and class organization
注意:Githubに上がっている最新バージョン(https://github.com/ekmett/ad)は4.0で、Hackage(http://hackage.haskell.org/package/ad-3.4)とは違います。ライブラリの構造が若干違うようです。
cabal unpack ad-3.4 でver. 3.4のソースをダウンロードするか、Hackageを見て下さい。
I’ll use ad-3.4 on Hackage, not ad-4.0 on Github
Package structure
いろいろな自動微分の”モード”の実装
ここでは複数のモードの実装のうち、デフォルトのモードがインポート&再エクスポートされている
型の定義
クラスの定義
http://new-hackage.haskell.org/package/ad-3.4
The starting point for exploration: diff functionNumeric.AD.Mode.Forward{-# LANGUAGE Rank2Types #-}diff :: Num a => (forall s. Mode s => AD s a -> AD s a) -> a -> adiff f a = tangent $ apply f a
Numeric.AD.Internal.Forward
{-# LANGUAGE Rank2Types, TypeFamilies, DeriveDataTypeable, TemplateHaskell, UndecidableInstances, BangPatterns #-}tangent :: Num a => AD Forward a -> atangent (AD (Forward _ da)) = datangent _ = 0
bundle :: a -> a -> AD Forward abundle a da = AD (Forward a da)
apply :: Num a => (AD Forward a -> b) -> a -> bapply f a = f (bundle a 1)
この部分の型:AD Forward a
微分対象の関数fは(AD s a->AD s a)型として表される
1変数スカラー関数の微分
どうやらAD型が鍵になるようだ。
AD typeNumeric.AD.Types
An instance of Mode class determines the behavior.
{-# LANGUAGE CPP #-}{-# LANGUAGE Rank2Types, GeneralizedNewtypeDeriving, TemplateHaskell, FlexibleContexts, FlexibleInstances, MultiParamTypeClasses, UndecidableInstances #-}
{-# ANN module "HLint: ignore Eta reduce" #-}
newtype AD f a = AD { runAD :: f a } deriving (Iso (f a), Lifted, Mode, Primal)
e.g.) AD (Forward 10 1) :: Num a => AD Forward a
Classes that have AD type as an instanceNumeric.AD.Types http://hackage.haskell.org/packages/archive/ad/3.4/doc/
html/Numeric-AD-Types.html#t:AD
(fがLiftedのインスタンス &&aがFloatingのインスタンス)のとき、 AD f a はFloatingのインスタンス。
AD s a -> AD s adiffの第一引数の型
は、以下の型に適合する*
Floating a => a -> a
*こういう言い方が正確か分からないが。
例:1変数実数スカラー関数 y = f(x)
Let’s look at some examples of values with AD type
import Numeric.ADimport Numeric.AD.Typesimport Numeric.AD.Internal.Classesimport Numeric.AD.Internal.Forward
f x = x + 3g x | x > 0 = x | otherwise = 0
d = diff (g . f)
*Main> :t (g . f)(g . f) :: (Num c, Ord c) => c -> c*Main> :t (g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a(g . f) :: (Num a, Ord a, Mode s) => AD s a -> AD s a :: (Num a, Ord a, Mode s) => AD s a -> AD s a*Main> :t ff :: Num a => a -> a*Main> :t f :: (Num a, Lifted s) => AD s a -> AD s af :: (Num a, Lifted s) => AD s a -> AD s a :: (Num a, Lifted s) => AD s a -> AD s a*Main> :t gg :: (Num a, Ord a) => a -> a*Main> :t dd :: Integer -> Integer
GHCi
adtest.hs
> :l adtest.hs[1 of 1] Compiling Main ( adtest.hs, interpreted )Ok, modules loaded: Main.[*Main]> :t applyapply :: Num a => (AD Forward a -> b) -> a -> b[*Main]> :t apply fapply f :: Num a => a -> AD Forward a[*Main]> :t apply f 0apply f 0 :: Num a => AD Forward a[*Main]> apply f 03
Let’s play around with apply and tangentGHCi
[*Main]> :t tangenttangent :: Num a => AD Forward a -> a[*Main]> :t tangent $ apply f 0tangent $ apply f 0 :: Num a => a[*Main]> tangent $ apply f 01
Since f :: Num t0 => t0 -> t0,b in the type of apply is restricted to AD Forward a
http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/Numeric-AD-Internal-Classes.html#t:Mode
http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/Numeric-AD-Internal-Classes.html#t:Lifted
Lifted class and Mode classChain rule of differentiation
Definition of different modes.Mode is a subclass of Lifted
Wrap a (Num a => a) value into t
e.g.) Forward is an instance of Lifted and Mode
Lifted class defines chain rules of differentiation
deriveLifted (in Numeric.AD.Internal.Classes)というTemplate Haskellの関数がLiftedのインスタンスを作ってくれる。
http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/src/Numeric-AD-Internal-Classes.html#deriveLifted
exp1 = lift1_ exp const log1 = lift1 log recip1
sin1 = lift1 sin cos1 cos1 = lift1 cos $ negate1 . sin1 tan1 = lift1 tan $ recip1 . square1 . cos1
A part of deriveLifted
deriveNumeric generates instances of Num, etc.
-- | @'deriveNumeric' f g@ provides the following instances:---- > instance ('Lifted' $f, 'Num' a, 'Enum' a) => 'Enum' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Eq' a) => 'Eq' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Ord' a) => 'Ord' ($g a)-- > instance ('Lifted' $f, 'Num' a, 'Bounded' a) => 'Bounded' ($g a)---- > instance ('Lifted' $f, 'Show' a) => 'Show' ($g a)-- > instance ('Lifted' $f, 'Num' a) => 'Num' ($g a)-- > instance ('Lifted' $f, 'Fractional' a) => 'Fractional' ($g a)-- > instance ('Lifted' $f, 'Floating' a) => 'Floating' ($g a)-- > instance ('Lifted' $f, 'RealFloat' a) => 'RealFloat' ($g a)-- > instance ('Lifted' $f, 'RealFrac' a) => 'RealFrac' ($g a)-- > instance ('Lifted' $f, 'Real' a) => 'Real' ($g a)
deriveNumeric (in Numeric.AD.Internal.Classes)
http://hackage.haskell.org/packages/archive/ad/3.4/doc/html/src/Numeric-AD-Internal-Classes.html#deriveNumeric
Definition of deriveNumericderiveNumeric :: ([Q Pred] -> [Q Pred]) -> Q Type -> Q [Dec]deriveNumeric f t = do members <- liftedMembers let keep n = nameBase n `elem` members xs <- lowerInstance keep ((classP ''Num [varA]:) . f) t `mapM` [''Enum, ''Eq, ''Ord, ''Bounded, ''Show] ys <- lowerInstance keep f t `mapM` [''Num, ''Fractional, ''Floating, ''RealFloat,''RealFrac, ''Real, ''Erf, ''InvErf] return (xs ++ ys)
lowerInstance :: (Name -> Bool) -> ([Q Pred] -> [Q Pred]) -> Q Type -> Name -> Q DeclowerInstance p f t n = do#ifdef OldClassI ClassI (ClassD _ _ _ _ ds) <- reify n#else ClassI (ClassD _ _ _ _ ds) _ <- reify n#endif instanceD (cxt (f [classP n [varA]])) (conT n `appT` (t `appT` varA)) (concatMap lower1 ds) where lower1 :: Dec -> [Q Dec] lower1 (SigD n' _) | p n'' = [valD (varP n') (normalB (varE n'')) []] where n'' = primed n' lower1 _ = []
primed n' = mkName $ base ++ [prime] where base = nameBase n' h = head base prime | isSymbol h || h `elem` "/*-<>" = '!' | otherwise = '1'
This looks an important part, but a bit difficult....
A pitfall?>> :t diff ((**2) :: Floating a => a -> a)diff ((**2) :: Floating a => a -> a) :: Floating a => a -> a>> :t diff ((**2) :: Double -> Double)
<interactive>:1:7: Couldn't match expected type `ad-3.4:Numeric.AD.Internal.Types.AD s a0' with actual type `Double' Expected type: ad-3.4:Numeric.AD.Internal.Types.AD s a0 -> ad-3.4:Numeric.AD.Internal.Types.AD s a0 Actual type: Double -> Double In the first argument of `diff', namely `((** 2) :: Double -> Double)' In the expression: diff ((** 2) :: Double -> Double)
You need to keep functions polymorphic for differentiation.
import Numeric.AD
func x = x ** 2func2 x = x ** 3
fs = [func, func2]test = map (\f -> diff f 1.0) fs
So, this does not compile. (there is a GHC extension to accept this, isn’t there..?)
Summary
Things that might be improved for application
• Simplification of terms
These might be a good practice for seasoned Haskellers (I’m not...)
• Improving the performance of higher order taylor expansion.
• Is automatic integration possible, maybe?
Sorry, but I’m still unclear about how Lifted actually executes differentiation
It seems to be done by Num instance of AD f a, viainstance (Lifted f, Num a) => Num (AD f a)...
An interesting library that shows how to use types and typeclasses effectively and beautifully.
感想
•他のEkmett氏のライブラリよりは具体的な対象がはっきりしている分、少しは分かりやすい。
•Haskellならではのコードの簡潔さのおかげで、ソースのどこを追っていけば良いかはわりと明確。
•とはいっても、私には若干(相当?)手に余る感じで、深いところまでは解説出来ませんでした。知らないGHC
拡張も多し。ただ、人に説明しようとすることで少しは理解が進んで、嬉しい。