Metaprogramming Patterns for Physics Simulationand Verification

Roy H. Stogner

The University of Texas at Austin

April 16, 2012

Roy H. Stogner Metaprogramming April 16, 2012 1 / 52

Computational Modeling and Verification

PECOS ResearchVerification, Validation, Uncertainty Quantification• Atmospheric entry vehicle, two key quantities of interest:

I Rate of recession of TPS (throughout peak heating regime)I Local peak heat flux to after-body

• Validation and uncertainty quantification for these QoI

Roy H. Stogner Metaprogramming April 16, 2012 2 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes

• New subphysicsmodels

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes• New subphysics

models

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes• New subphysics

models

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes• New subphysics

models

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes• New subphysics

models

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

PECOS Development

Experimental Data

QUESO

FIN-SlibMeshBoost

Turbulence

SHOCKINGMUTATION

GSL

Chemistry/Shocktube

Plug FlowABLATION1D

GSL

Ablation

Calibration/Validation

FIN-S

Inve

rsio

n

Forw

ard

QUESO/Dakota

Multiphysics Analysis

MASA

QoI

GRVYExternal Libraries

LDVFlow

Measurements

NASA EAST Shocktube

Measurements

Molecular Beam (O2)Heated Flow Titration (N2)

New Codes• New subphysics

models

• New developers

• New QoIpostprocessing

• New numerics

• New bugs

Roy H. Stogner Metaprogramming April 16, 2012 3 / 52

Computational Modeling and Verification

Method of Manufactured Solutions

Verification Process• “Manufacture” a closed form exact solution u, e.g. parameterized trig

functions:(Roy, 2002)

u(x, y, z, t) = u0 + ux fs

(auxπxL

)+ uy fs

(auyπyL

)+

+ uz fs

(auzπzL

)+ ut fs

(autπt

L

)• Insert u into model equation system F , evaluate source terms

• Use simulation code to solve modified physics F (u) ≡ F (u)− F (u)• Verify that solution uh → u at expected rate

Roy H. Stogner Metaprogramming April 16, 2012 4 / 52

Computational Modeling and Verification

Maple MMS: 3D Navier-Stokes Energy Term

Qe =−apxπpx

L

γ

γ − 1sin

�apxπx

L

� �u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��+

+apyπpy

L

γ

γ − 1cos

�apyπy

L

� �v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

��+

−apzπpz

L

γ

γ − 1sin

�apzπz

L

� �w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

��+

+aρxπρx2L

cos�aρxπx

L

� �u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

����u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��2+

+�w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

��2+

�v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

��2�

+

−aρyπρy2L

sin�aρyπy

L

� �v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

����u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��2+

+�w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

��2+

�v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

��2�

+

+aρzπρz2L

cos�aρzπz

L

� �w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

����u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��2+

+�w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

��2+�v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

��2�

+

+auxπux

2Lcos

�auxπx

L

����w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

��2+

�v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

��2+

+3�u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��2��

ρ0 + ρx sin�aρxπx

L

�+ ρy cos

�aρyπy

L

�+ ρz sin

�aρzπz

L

��+

+�p0 + px cos

�apxπx

L

�+ py sin

�apyπy

L

�+ pz cos

�apzπz

L

�� 2γ

(γ − 1)

�

+

−auyπuy

Lsin

�auyπy

L

� �v0 + vx cos

�avxπx

L

�+ vy sin

�avyπy

L

�+ vz sin

�avzπz

L

�� �ρ0 + ρx sin

�aρxπx

L

�+ ρy cos

�aρyπy

L

�+ ρz sin

�aρzπz

L

��·

·�u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��+

−auzπuz

Lsin

�auzπz

L

� �w0 + wx sin

�awxπx

L

�+ wy sin

�awyπy

L

�+ wz cos

�awzπz

L

�� �ρ0 + ρx sin

�aρxπx

L

�+ ρy cos

�aρyπy

L

�+ ρz sin

�aρzπz

L

��·

·�u0 + ux sin

�auxπx

L

�+ uy cos

�auyπy

L

�+ uz cos

�auzπz

L

��+

Roy H. Stogner Metaprogramming April 16, 2012 5 / 52

Computational Modeling and Verification

And more 3D Navier-Stokes Energy Term

Roy H. Stogner Metaprogramming April 16, 2012 6 / 52

Computational Modeling and Verification

Combinatorial Explosion

Explosion in source term size• Exponential complexity increase with more sophisticated models

I Basic Sutherland viscosity starts at a million characters

• Large CAS RAM requirements (128 GB insufficient for Sutherland)

• segfaults, incorrect output

Hierarchic MMS?• Manual simplification of sub-terms, one-by-one

• Recombination into final forcing function

• Still labor-intensive

Automatic Differentiation?• Adding complex dependencies to MASA?

Roy H. Stogner Metaprogramming April 16, 2012 7 / 52

Computational Modeling and Verification

Combinatorial Explosion

Explosion in source term size• Exponential complexity increase with more sophisticated models

I Basic Sutherland viscosity starts at a million characters

• Large CAS RAM requirements (128 GB insufficient for Sutherland)

• segfaults, incorrect output

Hierarchic MMS?• Manual simplification of sub-terms, one-by-one

• Recombination into final forcing function

• Still labor-intensive

Automatic Differentiation?• Adding complex dependencies to MASA?

Roy H. Stogner Metaprogramming April 16, 2012 7 / 52

Computational Modeling and Verification

Combinatorial Explosion

Explosion in source term size• Exponential complexity increase with more sophisticated models

I Basic Sutherland viscosity starts at a million characters

• Large CAS RAM requirements (128 GB insufficient for Sutherland)

• segfaults, incorrect output

Hierarchic MMS?• Manual simplification of sub-terms, one-by-one

• Recombination into final forcing function

• Still labor-intensive

Automatic Differentiation?• Adding complex dependencies to MASA?

Roy H. Stogner Metaprogramming April 16, 2012 7 / 52

Computational Modeling and Verification Generic Programming

Generic Programming

• Or more informally, “Duck Typing”: templating around unspecifieddata types that expected to be overloaded to act like mathematicalobjects, creating new code and new data types at compile time.

• We do this all the time for float/double/long double.

• So we can duct tape our templated typename to any class thatbehaves like we expect math to behave

Roy H. Stogner Metaprogramming April 16, 2012 8 / 52

Computational Modeling and Verification Generic Programming

Duck Typing

If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.

Unless it’s a dragon doing a duckimpression...

We ran into a couple of those.

Roy H. Stogner Metaprogramming April 16, 2012 9 / 52

Computational Modeling and Verification Generic Programming

Duck Typing

If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.

Unless it’s a dragon doing a duckimpression...

We ran into a couple of those.

Roy H. Stogner Metaprogramming April 16, 2012 9 / 52

Computational Modeling and Verification Generic Programming

Duck Typing

If it walks like a duck, quacks likea duck, looks like a duck, it mustbe a duck.

Unless it’s a dragon doing a duckimpression...

We ran into a couple of those.

Roy H. Stogner Metaprogramming April 16, 2012 9 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Multi-precision

Example

template <typename T, typename S>

struct ShadowNumber {.........T _val;

S _shadow;

};

• Simultaneous calculation with multiple floating point representations,to estimate rounding error:

ExampleShadowNumber<float, double> shadowed_float;

ShadowNumber<double, long double> shadowed_double;

Roy H. Stogner Metaprogramming April 16, 2012 10 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

template <unsigned int size, typename T>

struct NumberArray {.........

template <typename T2>

NumberArray<size,T>& operator+= (const NumberArray<size,T2>& a) {

for (unsigned int i = 0; i != size; ++i)

_data[i] += a[i]; return *this;

}.........T _data[size];

};

• Compile-time fixed array sizes - no heap memory

• Fixed length loops can unroll with no branching

• Mixed data subtypes

Roy H. Stogner Metaprogramming April 16, 2012 11 / 52

Computational Modeling and Verification Example Data Types

Overloaded Functions, Operators

Example

#define NumberArray_std_unary(funcname) \

template <unsigned int size, typename T> \

inline \

NumberArray<size, T> \

funcname (const NumberArray<size, T>& a) \

{ \

NumberArray<size, T> returnval; \

\

for (unsigned int i = 0; i != size; ++i) \

returnval[i] = funcname(a[i]); \

\

return returnval; \

}

NumberArray_std_unary(exp)

NumberArray_std_unary(sin)

NumberArray_std_unary(cos)

• Templates shouldreplace writing NTNf

repetitive codes usingunsafe macros

• Templated function andstruct names can’t betemplate arguments.

• We still need macros toavoid Nf repetitivetemplate functions.

• This gives usmultidimensional nth

derivatives in 300 linesof C++

Roy H. Stogner Metaprogramming April 16, 2012 12 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f);

// Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v);

// Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f);

// Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Example Data Types

Operator-Overloaded Vectors

Example

NumericArray<3,double> v;

v[1] = 2;

NumericArray<3,float> f;

f[2] = 1;

v += sin(f); // Great!

f += sin(v); // Tolerable!

v = v + sin(f); // Undefined?

What is the return type supposed to be for

template <unsigned int size, typename T1, typename T2>

operator+(const NumberArray<size,T1>& a,

const NumberArray<size,T2>& b)

std:: problem too - complex<float> + double won’t compile

Roy H. Stogner Metaprogramming April 16, 2012 13 / 52

Computational Modeling and Verification Computing new Data Types

Return Type Selection, Template Specialization

Let’s give it a template:

template <typename S, typename T>

struct CompareTypes {

// typedef something supertype;

};

And then we can specialize

Exampletemplate <>

struct CompareTypes<float,double> {

typedef double

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 14 / 52

Computational Modeling and Verification Computing new Data Types

Return Type Selection, Template Specialization

Let’s give it a template:

template <typename S, typename T>

struct CompareTypes {

// typedef something supertype;

};

Or, for templated classes, partially specialize

Exampletemplate <unsigned int size, typename T, typename T2>

struct CompareTypes<NumberArray<size,T>,T2> {

typedef NumberArray<size,typename CompareTypes<T,T2>::supertype>

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 14 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Here Be Dragons

A NumberArray of something, combined with anything else we haven’tseen before

Example

template <unsigned int size, typename T, typename T2>

struct CompareTypes<NumberArray<size,T>,T2> {

typedef NumberArray<size,

typename CompareTypes<T,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

NumberArray<3,float>,double

>::supertype) ==

typeid(NumberArray<3,double>))

how_much_we_rock = 10;

Roy H. Stogner Metaprogramming April 16, 2012 15 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Here Be Dragons

A NumberArray of something, combined with anything else we haven’tseen before

Example

template <unsigned int size, typename T, typename T2>

struct CompareTypes<NumberArray<size,T>,T2> {

typedef NumberArray<size,

typename CompareTypes<T,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

NumberArray<3,NumberArray<float> >,double

>::supertype) ==

typeid(NumberArray<3,NumberArray<double> >))

how_much_we_rock = 11; // That extra push over the cliff

Roy H. Stogner Metaprogramming April 16, 2012 15 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Here Be Dragons

A ShadowNumber of something, combined with anything else we haven’tseen before

Example

template <typename T, typename S, typename T2>

struct CompareTypes<ShadowNumber<T,S>,T2> {

typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,

typename CompareTypes<S,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

ShadowNumber<float,double>,double

>::supertype) ==

typeid(ShadowNumber<double,double>))

how_much_we_rock = 7; // C is still a passing grade

Roy H. Stogner Metaprogramming April 16, 2012 15 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Here Be Dragons

A ShadowNumber of something, combined with a NumberArray

Example

template <typename T, typename S, typename T2>

struct CompareTypes<ShadowNumber<T,S>,T2> {

typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,

typename CompareTypes<S,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

NumberArray<3,float>,ShadowNumber<float,double>

>::supertype) ==

typeid(NumberArray<3,ShadowNumber<float,double> >))

how_much_we_rock = -1; // This didn’t even compile

Roy H. Stogner Metaprogramming April 16, 2012 15 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Here Be Dragons

A ShadowNumber of something, combined with a NumberArray

Example

template <typename T, typename S, typename T2>

struct CompareTypes<ShadowNumber<T,S>,T2> {

typedef ShadowNumber<typename CompareTypes<T,T2>::supertype,

typename CompareTypes<S,T2>::supertype>

supertype;

};

if(typeid(CompareTypes<

NumberArray<3,float>,ShadowNumber<float,double>

>::supertype) ==

typeid(ShadowNumber<NumberArray<3,float>,NumberArray<3,double> >))

how_much_we_rock = -1; // This didn’t even compile

Roy H. Stogner Metaprogramming April 16, 2012 15 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Killing Dragons• Mathematically, these are isomorphic:

I (V ⊗W )n

I V n ⊗Wn

• Your compiler doesn’t understandisomorphisms:

I {float, double}[size]I {float[size]}, {double[size]}

• Need partial template specialization,SFINAE to disambiguate

Example

template <unsigned int size, typename T, typename T2, typename S2>

struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {

typedef NumberArray<size,

typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 16 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Killing Dragons• Mathematically, these are isomorphic:

I (V ⊗W )n

I V n ⊗Wn

• Your compiler doesn’t understandisomorphisms:

I {float, double}[size]I {float[size]}, {double[size]}

• Need partial template specialization,SFINAE to disambiguate

Example

template <unsigned int size, typename T, typename T2, typename S2>

struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {

typedef NumberArray<size,

typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 16 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Killing Dragons• Mathematically, these are isomorphic:

I (V ⊗W )n

I V n ⊗Wn

• Your compiler doesn’t understandisomorphisms:

I {float, double}[size]I {float[size]}, {double[size]}

• Need partial template specialization,SFINAE to disambiguate

Example

template <unsigned int size, typename T, typename T2, typename S2>

struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {

typedef NumberArray<size,

typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 16 / 52

Computational Modeling and Verification Computing new Data Types

Return Types: Killing Dragons• Mathematically, these are isomorphic:

I (V ⊗W )n

I V n ⊗Wn

• Your compiler doesn’t understandisomorphisms:

I {float, double}[size]I {float[size]}, {double[size]}

• Need partial template specialization,SFINAE to disambiguate

Example

template <unsigned int size, typename T, typename T2, typename S2>

struct CompareTypes<NumberArray<size,T>,ShadowNumber<T2,S2> > {

typedef NumberArray<size,

typename CompareTypes<T,ShadowNumber<T2,S2> >::supertype>

supertype;

};

Roy H. Stogner Metaprogramming April 16, 2012 16 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ − 1

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ − 1 ...no, wait, that’s C

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

“Dual Numbers” - [Clifford 1873],[Study 1903]

• Add a new element ε to R, close the field:{m∑i=0

aiεi : ai ∈ R,m <∞

}

• Take the quotient with ε2 ≡ 0D[R] ≡ {a+ bε : a, b ∈ R}

• (a+ bε)× (c+ dε) = (ac) + adε+ bcε+ bdε2 = ((ac) + (ad+ bc)ε)

• Extending to division, algebraic functions, transcendental functions,etc gives: for any function f(x),

f(a+ bε) = f(a) + bf ′(a)ε

• This time the isomorphism practically codes itself!

Roy H. Stogner Metaprogramming April 16, 2012 17 / 52

Computational Modeling and Verification Automatic Differentiation

Operator-Overloaded Forward Differentiation

Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code

Roy H. Stogner Metaprogramming April 16, 2012 18 / 52

Computational Modeling and Verification Automatic Differentiation

Operator-Overloaded Forward Differentiation

Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code

Roy H. Stogner Metaprogramming April 16, 2012 18 / 52

Computational Modeling and Verification Automatic Differentiation

Operator-Overloaded Forward Differentiation

Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code

Roy H. Stogner Metaprogramming April 16, 2012 18 / 52

Computational Modeling and Verification Automatic Differentiation

Operator-Overloaded Forward Differentiation

Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code

Roy H. Stogner Metaprogramming April 16, 2012 18 / 52

Computational Modeling and Verification Automatic Differentiation

Operator-Overloaded Forward Differentiation

Example

typedef double RawType

// typedef ShadowNumber<double, long double> RawType;

typedef DualNumber<RawType,RawType> FirstDerivType;

const FirstDerivType x(pi/6,1); // Initializing independent var

const FirstDerivType sinx = sin(x); // Caching just like normal

const FirstDerivType y = sinx*sinx; // Smart pow would work too

const double raw_y = raw_value(y); // No implicit down-conversions

double deriv = raw_value(y.derivatives());

assert(deriv == 2*sin(pi/6)*cos(pi/6)); // FP ops match hand-code

Roy H. Stogner Metaprogramming April 16, 2012 18 / 52

Computational Modeling and Verification Automatic Differentiation

“Hyper-dual Numbers” - [Fike 2009]

• Add two new elements ε1, ε2 to R• Take the quotient with ε21 ≡ ε22 ≡ 0

H[R] ≡ {a+ bε1 + cε2 + dε1ε2 : a, b, c, d ∈ R}

• (a+ bε1 + cε2 + dε1ε2)× (e+ fε1 + gε2 + hε1ε2) =(ae) + (af + be)ε1 + (ag + ce)ε2 + (ah+ de+ bg + cf)ε1ε2

• Extending gives: for any function f(x),f(a+bε1+bε2+cε1ε2) = f(a)+bf ′(a)(ε1+ε2)+(b2f ′′(a)+cf ′(a))ε1ε2

• adding εiεj = 0 to quotient would give vector gradients instead...

Roy H. Stogner Metaprogramming April 16, 2012 19 / 52

Computational Modeling and Verification Automatic Differentiation

Hyper-dual Numbers via Recursive Mathematics

• RecallI D[X] ≡ {a+ bε : a, b ∈ X}I f(a+ bε) = f(a) + bf ′(a)ε

• Applied directly to R we get first derivatives• Applied recursively:

I D[D[R]] ≡ {a+ cε2 : a, c ∈ D[R]}I D[D[R]] ≡ {(a+ bε1) + (c+ dε1)ε2 : a, b, c, d ∈ R}I f((a+ bε1) + (c+ dε1)ε2) = f(a+ bε1) + (c+ dε1)f

′(a+ bε1)ε2I = f(a) + bf ′(a)ε1 + (c+ dε1)(f

′(a) + bf ′′(a)ε1)ε2I = f(a) + bf ′(a)ε1 + cf ′(a)ε2 + (cbf ′′(a) + df ′(a))ε1ε2

• In other words, D[D[R]] ≡ H[R]• This isomorphism works in software, too!

Roy H. Stogner Metaprogramming April 16, 2012 20 / 52

Computational Modeling and Verification Automatic Differentiation

Hessian Differentiation via Recursive Templates

Example

typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;

typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;

NumberArray<3, FAD2> xyz;

xyz[0] =

FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());

xyz[1] =

FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());

xyz[2] =

FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());

const FAD2 u = some_function(xyz);

RawType laplacian_u = 0;

for (unsigned int i=0; i != 3; ++i)

laplacian_u += u.derivatives()[i].derivatives()[i];

Roy H. Stogner Metaprogramming April 16, 2012 21 / 52

Computational Modeling and Verification Automatic Differentiation

Hessian Differentiation via Recursive Templates

Example

typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;

typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;

NumberArray<3, FAD2> xyz;

xyz[0] =

FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());

xyz[1] =

FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());

xyz[2] =

FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());

const FAD2 u = some_function(xyz);

RawType laplacian_u = 0;

for (unsigned int i=0; i != 3; ++i)

laplacian_u += u.derivatives()[i].derivatives()[i];

Roy H. Stogner Metaprogramming April 16, 2012 21 / 52

Computational Modeling and Verification Automatic Differentiation

Hessian Differentiation via Recursive Templates

Example

typedef DualNumber<RawType,NumberArray<3,RawType> > FAD1;

typedef DualNumber<FAD1,NumberArray<3,FAD1> > FAD2;

NumberArray<3, FAD2> xyz;

xyz[0] =

FAD2(x,NumberArrayUnitVector<3,0,RawType>::value());

xyz[1] =

FAD2(y,NumberArrayUnitVector<3,1,RawType>::value());

xyz[2] =

FAD2(z,NumberArrayUnitVector<3,2,RawType>::value());

const FAD2 u = some_function(xyz);

RawType laplacian_u = 0;

for (unsigned int i=0; i != 3; ++i)

laplacian_u += u.derivatives()[i].derivatives()[i];

Roy H. Stogner Metaprogramming April 16, 2012 21 / 52

Computational Modeling and Verification Automatic Differentiation

MASA PDE Examples

Manufactured Solution

// Arbitrary manufactured solution

U.template get<0>() = u_0 + u_x * sin(a_ux * PI * x / L) +

u_y * cos(a_uy * PI * y / L);

// Why not U[0] and U[1]? Explanation to come...

U.template get<1>() = v_0 + v_x * cos(a_vx * PI * x / L) +

v_y * sin(a_vy * PI * y / L);

ADScalar RHO = rho_0 + rho_x * sin(a_rhox * PI * x / L) +

rho_y * cos(a_rhoy * PI * y / L);

ADScalar P = p_0 + p_x * cos(a_px * PI * x / L) +

p_y * sin(a_py * PI * y / L);

// Constitutive laws

Tensor GradU = gradient(U);

Tensor Tau = mu * (GradU + transpose(GradU) -

2./3. * divergence(U) * RawArray::identity());

FullArray q = -k * T.derivatives();

Roy H. Stogner Metaprogramming April 16, 2012 22 / 52

Computational Modeling and Verification Automatic Differentiation

MASA PDE Examples

Euler

// Gas state

ADScalar T = P / RHO / R;

ADScalar E = 1. / (Gamma-1.) * P / RHO;

ADScalar ET = E + .5 * U.dot(U);

// Constitutive laws

Tensor GradU = gradient(U);

Tensor Tau = mu * (GradU + transpose(GradU) -

2./3. * divergence(U) * RawArray::identity());

FullArray q = -k * T.derivatives();

// Conservation equation residuals

Scalar Q_rho = raw_value(divergence(RHO*U));

RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U)) +

P.derivatives());

Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U));

Roy H. Stogner Metaprogramming April 16, 2012 22 / 52

Computational Modeling and Verification Automatic Differentiation

MASA PDE Examples

Navier-Stokes

// Gas state

ADScalar T = P / RHO / R;

ADScalar E = 1. / (Gamma-1.) * P / RHO;

ADScalar ET = E + .5 * U.dot(U);

// Constitutive laws

Tensor GradU = gradient(U);

Tensor Tau = mu * (GradU + transpose(GradU) -

2./3. * divergence(U) * RawArray::identity());

FullArray q = -k * T.derivatives();

// Conservation equation residuals

Scalar Q_rho = raw_value(divergence(RHO*U));

RawArray Q_rho_u = raw_value(divergence(RHO*U.outerproduct(U) - Tau) +

P.derivatives());

Scalar Q_rho_e = raw_value(divergence((RHO*ET+P)*U + q - Tau.dot(U)));

Roy H. Stogner Metaprogramming April 16, 2012 22 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

Roy H. Stogner Metaprogramming April 16, 2012 23 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

Roy H. Stogner Metaprogramming April 16, 2012 23 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

Roy H. Stogner Metaprogramming April 16, 2012 23 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

Roy H. Stogner Metaprogramming April 16, 2012 23 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Automatic upconversion is dangerous!

Example

double problem(FirstDerivType x, SecondDerivType y) {

// This is desirable:

x *= 2.0;

// But now the compiler doesn’t see a problem with this!

return (x+y).derivatives().derivatives();

}

• Code with 2nd and higher derivatives still requires care

• For safe code: automatic upconversion only for built-in types,automatic downconversion for DualNumber, even to “HypodualNumber”

Roy H. Stogner Metaprogramming April 16, 2012 23 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!

I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:

I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)

I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:

I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:

I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:

I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:

I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:I Sparse vector data types

I Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizations

I Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Computational Modeling and Verification Automatic Differentiation

Lessons Learned

• Hyperduals are inefficient!I (1 +m)n data for n derivatives in m dimensions (e.g. (1 + 3)2 = 16)I Should require less than (e.g. 10)

n∑i=0

(m+ i− 1)!

(m− 1)!i!

• For efficient higher derivatives: partial specialization of recursiveDualNumber

• For efficient higher dimensions:I Sparse vector data typesI Functional metaprogramming for efficient compile-time optimizationsI Functional-to-imperative APIs for run-time operations

Roy H. Stogner Metaprogramming April 16, 2012 24 / 52

Numeric Functional Metaprogramming Functional Data Structures

Data Structures with Templates: Type ContainerExample

template <typename HeadType,

typename TailContainer=NullContainer,

typename Comparison=ValueLessThan>

struct Container

{

typedef HeadType head_type; // car and cdr would scare

typedef TailContainer tail_set; // off the Fortran folks

typedef Comparison comparison;

// These may be empty types or may have data

HeadType head;

TailContainer tail;.........};

struct NullContainer

{.........};

Roy H. Stogner Metaprogramming April 16, 2012 25 / 52

Numeric Functional Metaprogramming Functional Data Structures

Data Structures with Templates: Type Container

Example

Container<IntType<1> > no_data_one_static_int;

Container<IntType<1>,

Container<IntType<4> > no_data_two_static_ints;

Container<IntType<1,float>,

Container<IntType<4,float> > float_data_two_static_ints;

Container<IntType<1,double>,

Container<IntType<4>,NumberArray<3,double> > heterogenous_data;

Roy H. Stogner Metaprogramming April 16, 2012 26 / 52

Numeric Functional Metaprogramming Functional Data Structures

Data Structures with Templates: Type Container

Example

Container<IntType<1> > no_data_one_static_int;

Container<IntType<1>,

Container<IntType<4> > no_data_two_static_ints;

Container<IntType<1,float>,

Container<IntType<4,float> > float_data_two_static_ints;

Container<IntType<1,double>,

Container<IntType<4>,NumberArray<3,double> > heterogenous_data;

Roy H. Stogner Metaprogramming April 16, 2012 26 / 52

Numeric Functional Metaprogramming Functional Data Structures

Data Structures with Templates: Type Container

Example

Container<IntType<1> > no_data_one_static_int;

Container<IntType<1>,

Container<IntType<4> > no_data_two_static_ints;

Container<IntType<1,float>,

Container<IntType<4,float> > float_data_two_static_ints;

Container<IntType<1,double>,

Container<IntType<4>,NumberArray<3,double> > heterogenous_data;

Roy H. Stogner Metaprogramming April 16, 2012 26 / 52

Numeric Functional Metaprogramming Functional Data Structures

Data Structures with Templates: Type Container

Example

Container<IntType<1> > no_data_one_static_int;

Container<IntType<1>,

Container<IntType<4> > no_data_two_static_ints;

Container<IntType<1,float>,

Container<IntType<4,float> > float_data_two_static_ints;

Container<IntType<1,double>,

Container<IntType<4>,NumberArray<3,double> > heterogenous_data;

Roy H. Stogner Metaprogramming April 16, 2012 26 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting into a NullContainer is easy:

• If we’re inserting null, we’re still left with a NullContainer

• If not, we have a one-element container with the inserted element.

Examplestruct NullContainer

{.........template <typename ValueType, typename NewComparison>

struct Insert

{

typedef

typename IfElse<

(TypesEqual<ValueType,NullContainer>::value),

NullContainer,

Container<ValueType, NullContainer, NewComparison>

>::type type;

};.........};

C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?

Roy H. Stogner Metaprogramming April 16, 2012 27 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting into a NullContainer is easy:

• If we’re inserting null, we’re still left with a NullContainer

• If not, we have a one-element container with the inserted element.

Examplestruct NullContainer

{.........template <typename ValueType, typename NewComparison>

struct Insert

{

typedef

typename IfElse<

(TypesEqual<ValueType,NullContainer>::value),

NullContainer,

Container<ValueType, NullContainer, NewComparison>

>::type type;

};.........};

C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?

Roy H. Stogner Metaprogramming April 16, 2012 27 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting into a NullContainer is easy:

• If we’re inserting null, we’re still left with a NullContainer

• If not, we have a one-element container with the inserted element.

Examplestruct NullContainer

{.........template <typename ValueType, typename NewComparison>

struct Insert

{

typedef

typename IfElse<

(TypesEqual<ValueType,NullContainer>::value),

NullContainer,

Container<ValueType, NullContainer, NewComparison>

>::type type;

};.........};

C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?

Roy H. Stogner Metaprogramming April 16, 2012 27 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting into a NullContainer is easy:

• If we’re inserting null, we’re still left with a NullContainer

• If not, we have a one-element container with the inserted element.

Examplestruct NullContainer

{.........template <typename ValueType, typename NewComparison>

struct Insert

{

typedef

typename IfElse<

(TypesEqual<ValueType,NullContainer>::value),

NullContainer,

Container<ValueType, NullContainer, NewComparison>

>::type type;

};.........};

C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?

Roy H. Stogner Metaprogramming April 16, 2012 27 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting into a NullContainer is easy:

• If we’re inserting null, we’re still left with a NullContainer

• If not, we have a one-element container with the inserted element.

Examplestruct NullContainer

{.........template <typename ValueType, typename NewComparison>

struct Insert

{

typedef

typename IfElse<

(TypesEqual<ValueType,NullContainer>::value),

NullContainer,

Container<ValueType, NullContainer, NewComparison>

>::type type;

};.........};

C++ template syntax: karmic retribution for engineering majors whomocked CS majors’ Lisp syntax?

Roy H. Stogner Metaprogramming April 16, 2012 27 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations: Insert

Inserting in nonempty sorted sets is trickier:

• If we’re inserting null, we return our same set

• If we’re inserting a type smaller than our set head, then it becomesthe new head and our set becomes the new tail

• If we’re inserting a type larger than our set head, then the head staysunchanged and we insert into the tail

• If we’re inserting something equivalent to our set head, then the headmay get “upgraded”

• Templating around a comparison type to define “smaller”, “larger” and“equivalent” can give us set-like, multiset-like, or vector-like behavior

Roy H. Stogner Metaprogramming April 16, 2012 28 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations

• Union, Intersection, Difference with an empty container is trivial.

• With non-empty containers, Insert and Contains methods makerecursive definitions easy:

Example

template <typename Set2>

struct Union {

typedef typename

tail_set::Sorted::template Union<

typename Set2::Sorted::template Insert<

head_type

>::type

>::type type;

};

Roy H. Stogner Metaprogramming April 16, 2012 29 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations

• Union, Intersection, Difference with an empty container is trivial.

• With non-empty containers, Insert and Contains methods makerecursive definitions easy:

Example

template <typename Set2>

struct Union {

typedef typename

tail_set::Sorted::template Union<

typename Set2::Sorted::template Insert<

head_type

>::type

>::type type;

};

Roy H. Stogner Metaprogramming April 16, 2012 29 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations

• Union, Intersection, Difference with an empty container is trivial.

• With non-empty containers, Insert and Contains methods makerecursive definitions easy:

Example

template <typename Set2>

struct Union {

typedef typename

tail_set::Sorted::template Union<

typename Set2::Sorted::template Insert<

head_type

>::type

>::type type;

};

Roy H. Stogner Metaprogramming April 16, 2012 29 / 52

Numeric Functional Metaprogramming Functional Data Structures

Set Operations

• Union, Intersection, Difference with an empty container is trivial.

• With non-empty containers, Insert and Contains methods makerecursive definitions easy:

Example

template <typename Set2>

struct Union {

typedef typename

tail_set::Sorted::template Union<

typename Set2::Sorted::template Insert<

head_type

>::type

>::type type;

};

Roy H. Stogner Metaprogramming April 16, 2012 29 / 52

Numeric Functional Metaprogramming Functional Data Structures

Runtime Indexed ForEach

• Applying a standard functor to every set element

• Recursive compile-time metaprogram unrolls into a simple linearrun-time program

• f receives a non-compile-time-const v

Example

struct RuntimeForEach

{

template <typename Functor>

void operator()(const Functor &f) {

// f might want a reference, so we pass in a non-static copy

const typename head_type::value_type v = head_type::value;

f(v);

typename tail_set::RuntimeForEach()(f);

}

};

Roy H. Stogner Metaprogramming April 16, 2012 30 / 52

Numeric Functional Metaprogramming Functional Data Structures

Compile-time Indexed ForEach

• Applying a “template functor” to every set element

• Recursive compile-time metaprogram unrolls into a simple linearrun-time program

• f receives a compile-time-const HeadType

Example

struct ForEach

{

template <typename Functor>

void operator()(const Functor &f) {

f.operator()<HeadType>(v);

typename tail_set::ForEach()(f);

}

};

Roy H. Stogner Metaprogramming April 16, 2012 31 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• e.g. the vector [0 0 1 0 2 0] will have {3, 5} or a superset as index set

• Const-index O (1) access via template arguments

Example

template <typename T, typename IndexSet>

class SparseNumberArray {

public:

typedef IndexSet index_set;

static const unsigned int size = IndexSet::size;.........

T& raw_at(unsigned int i)

{ return _data[i]; }

.........private:

T _data[size];

};

Roy H. Stogner Metaprogramming April 16, 2012 32 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices

• Const-index O (1) access via template arguments

Example

template <typename T, typename IndexSet>

class SparseNumberArray {

public:

typedef IndexSet index_set;

static const unsigned int size = IndexSet::size;.........

T& operator[](index_value_type i)

{ return _data[IndexSet::runtime_index_of(i)]; }

.........private:

T _data[size];

};

Roy H. Stogner Metaprogramming April 16, 2012 32 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors• An IndexSet lists sparse indices with potentially non-zero data• Variable-index component access via runtime search through indices• Const-index O (1) access via template arguments

Example

template <typename T, typename IndexSet>

class SparseNumberArray {

public:

typedef IndexSet index_set;

static const unsigned int size = IndexSet::size;.........template <unsigned int i>

typename entry_type<i>::type& get() {

return _data[IndexSet::template IndexOf<i>::index];

}.........private:

T _data[size];

};

Roy H. Stogner Metaprogramming April 16, 2012 32 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vector OperationsSome operations are easily defined, no index set operations needed:

• Multiplication by a constant: for loop over all data, multiply datum

Other operations require simple set operations:

• Element-wise addition takes a Union of index sets

• Element-wise multiplication takes an Intersection of index sets

• Element-wise division takes the numerator’s index set, but asserts(compile-time!) that the divisor’s is a superset.

Others require repeated set operations:

• Consider a typical binary function, max(a,b) , applied element-wise totwo sparse vectors:

• For each element in the intersection of the index sets, we apply thefunction to the corresponding pair of vector data.

• For each element in each asymmetric difference of the index sets, weapply the function to a pair of the present vector datum and 0.

Roy H. Stogner Metaprogramming April 16, 2012 33 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·]

= [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Vectors

Consider recursion with dual numbers, for automatic differentiation

• x is only non-constant w.r.t. x; y is only non-constant w.r.t. y

• DualNumber x and y are given unit vector gradients

• Their sums, products, etc. then all have optimally sparse gradients.

We now have excellent sparse vectors-of-T!

[0 0 0 0 2 0] + [0 0 1 0 0 0] = [0 0 1 0 2 0]

[· · · · 2 ·] + [· · 1 · · ·] = [· · 1 · 2 ·]

• From 6 FLOPs down to 0

• But not good enough yet

Roy H. Stogner Metaprogramming April 16, 2012 34 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians

0 0 0 00 0 2 00 0 0 00 0 0 0

+

0 0 0 00 0 0 00 0 0 01 0 0 0

=

0 0 0 00 0 2 00 0 0 01 0 0 0

· · · ·· · 2 ·· · · ·· · · ·

+

· · · ·· · · ·· · · ·1 · · ·

=

· · · ·0 · 2 ·· · · ·1 · 0 ·

• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.

I Triangular matrices become full?I Diagonal matrices become full??

Roy H. Stogner Metaprogramming April 16, 2012 35 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians

0 0 0 00 0 2 00 0 0 00 0 0 0

+

0 0 0 00 0 0 00 0 0 01 0 0 0

=

0 0 0 00 0 2 00 0 0 01 0 0 0

· · · ·· · 2 ·· · · ·· · · ·

+

· · · ·· · · ·· · · ·1 · · ·

=

· · · ·0 · 2 ·· · · ·1 · 0 ·

• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.

I Triangular matrices become full?I Diagonal matrices become full??

Roy H. Stogner Metaprogramming April 16, 2012 35 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians

0 0 0 00 0 2 00 0 0 00 0 0 0

+

0 0 0 00 0 0 00 0 0 01 0 0 0

=

0 0 0 00 0 2 00 0 0 01 0 0 0

· · · ·· · 2 ·· · · ·· · · ·

+

· · · ·· · · ·· · · ·1 · · ·

=

· · · ·0 · 2 ·· · · ·1 · 0 ·

• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.

I Triangular matrices become full?I Diagonal matrices become full??

Roy H. Stogner Metaprogramming April 16, 2012 35 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians

0 0 0 00 0 2 00 0 0 00 0 0 0

+

0 0 0 00 0 0 00 0 0 01 0 0 0

=

0 0 0 00 0 2 00 0 0 01 0 0 0

· · · ·· · 2 ·· · · ·· · · ·

+

· · · ·· · · ·· · · ·1 · · ·

=

· · · ·0 · 2 ·· · · ·1 · 0 ·

• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.

I Triangular matrices become full?I Diagonal matrices become full??

Roy H. Stogner Metaprogramming April 16, 2012 35 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse StructuresConsider recursion with arrays, for matrices, hessians

0 0 0 00 0 2 00 0 0 00 0 0 0

+

0 0 0 00 0 0 00 0 0 01 0 0 0

=

0 0 0 00 0 2 00 0 0 01 0 0 0

· · · ·· · 2 ·· · · ·· · · ·

+

· · · ·· · · ·· · · ·1 · · ·

=

· · · ·0 · 2 ·· · · ·1 · 0 ·

• A SparseNumberArray-of-SparseNumberArray is sparse...• But only with a tensor-product sparsity structure.

I Triangular matrices become full?I Diagonal matrices become full??

Roy H. Stogner Metaprogramming April 16, 2012 35 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices

• Index-dependent data type makes variable-index component accessimpossible!

Example

template <typename IndexSet>

class SparseNumberStruct {

public:

typedef IndexSet index_set;

static const unsigned int size = IndexSet::size;

template <unsigned int i>

typename entry_type<i>::type& get() {

return _data.template data<UnsignedIntType<i> >(); }.........private:

IndexSet _data;

};

Roy H. Stogner Metaprogramming April 16, 2012 36 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structures• An IndexSet now stores data, not just static indices• Index-dependent data type makes variable-index component access

impossible!

Example

template <typename IndexSet>

class SparseNumberStruct {

public:

typedef IndexSet index_set;

static const unsigned int size = IndexSet::size;

template <unsigned int i>

typename entry_type<i>::type& get() {

return _data.template data<UnsignedIntType<i> >(); }.........private:

IndexSet _data;

};

Roy H. Stogner Metaprogramming April 16, 2012 36 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Operator-Overloaded Sparse Structure Operations

No operations are easily defined.

• C for loops don’t work over heterogenous structures.

• Templated functors must be defined for every operation.

Multiplication by a constant, using a functor and subfunctor:

Exampletemplate <typename T2, typename IndexSet>

inline

typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype

operator * (const SparseNumberStruct<IndexSet>& a, const T2& b)

{

typedef typename MultipliesType<SparseNumberStruct<IndexSet>,T2>::supertype type;

typename CompareTypes<SparseNumberStruct<IndexSet>,T2>::supertype

returnval;

typename IndexSet::ForEach()

(BinaryFunctor<MultipliesSubfunctor, IndexSet, ConstantDataSet<T2>, typename type::index_set>

(MultipliesSubfunctor(), a.raw_data(), ConstantDataSet<T2>(b), returnval.raw_data()));

return returnval;

}

Roy H. Stogner Metaprogramming April 16, 2012 37 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Templated Functors• A subfunctor class represents a single operation

• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides

proper return types

Example

struct MultipliesSubfunctor {

template <typename T1, typename T2>

struct Return {

typedef typename

SymmetricMultipliesType<T1,T2>::supertype type;

};

template <typename T1, typename T2>

typename Return<T1,T2>::type

operator()(const T1& x, const T2& y) const { return x * y; }

};

Roy H. Stogner Metaprogramming April 16, 2012 38 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types

• CompareTypes or similar classes (or C++11 decltype) providesproper return types

Example

struct MultipliesSubfunctor {

template <typename T1, typename T2>

struct Return {

typedef typename

SymmetricMultipliesType<T1,T2>::supertype type;

};

template <typename T1, typename T2>

typename Return<T1,T2>::type

operator()(const T1& x, const T2& y) const { return x * y; }

};

Roy H. Stogner Metaprogramming April 16, 2012 38 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides

proper return types

Example

struct MultipliesSubfunctor {

template <typename T1, typename T2>

struct Return {

typedef typename

SymmetricMultipliesType<T1,T2>::supertype type;

};

template <typename T1, typename T2>

typename Return<T1,T2>::type

operator()(const T1& x, const T2& y) const { return x * y; }

};

Roy H. Stogner Metaprogramming April 16, 2012 38 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides

proper return types

Example

struct MultipliesSubfunctor {

template <typename T1, typename T2>

struct Return {

typedef typename

SymmetricMultipliesType<T1,T2>::supertype type;

};

template <typename T1, typename T2>

typename Return<T1,T2>::type

operator()(const T1& x, const T2& y) const { return x * y; }

};

Roy H. Stogner Metaprogramming April 16, 2012 38 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Templated Functors• A subfunctor class represents a single operation• Template overloading of operator() accepts arbitrary input types• CompareTypes or similar classes (or C++11 decltype) provides

proper return types

Example

struct MultipliesSubfunctor {

template <typename T1, typename T2>

struct Return {

typedef typename

SymmetricMultipliesType<T1,T2>::supertype type;

};

template <typename T1, typename T2>

typename Return<T1,T2>::type

operator()(const T1& x, const T2& y) const { return x * y; }

};

Roy H. Stogner Metaprogramming April 16, 2012 38 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]

{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]

Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Sparse Data Structures

Sparse Matrix Operations

Example

A ≡[a bT

c D

]{L,U} ≡ {LUL(A), LUU (A)}

L =

[a 0T

c LUL(D− ca−1bT)

]U =

[1 a−1bT

0 LUU (D− ca−1bT)

]Then apply to sparse arrays of arrays:

• One compile-time pass to allocate output structure

• One run-time pass to do calculations

• Asymptotic complexity drops beautifully

• But now we have asymptotic code generation to worry about!

Roy H. Stogner Metaprogramming April 16, 2012 39 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equations• A templated functor is almost a full physics equation!

I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible

I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?

Example

#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \

struct physicsname { \

static const char* name() { return #physicsname; } \

typedef UIntSetConstructor<input1enum>::type inputset; \

typedef UIntSetConstructor<outputenum>::type outputset; \

\

template <typename inputtype> \

auto operator() (const inputtype& input1enum##_NAME) \

-> decltype(code) { \

return code; \

} \

}

Roy H. Stogner Metaprogramming April 16, 2012 40 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equations• A templated functor is almost a full physics equation!

I Define which physical quantities are inputs and outputs• Enumerated types are metaprogramming compatible

I Equations can be split into minimal chunks for modularityI Equations must be split into minimal chunks for Jacobian optimization?

Example

#define DeclareUnaryPhysics(physicsname, input1enum, outputenum, code) \

struct physicsname { \

static const char* name() { return #physicsname; } \

typedef UIntSetConstructor<input1enum>::type inputset; \

typedef UIntSetConstructor<outputenum>::type outputset; \

\

template <typename inputtype> \

auto operator() (const inputtype& input1enum##_NAME) \

-> decltype(code) { \

return code; \

} \

}

Roy H. Stogner Metaprogramming April 16, 2012 40 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equations

Example

DeclareUnaryPhysics(DensityFromSpeciesDensities,

DENSITIES_VAR, DENSITY_VAR,

rhoi.sum());

DeclareBinaryPhysics(MomentumFromVelocity,

DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,

U * rho);

DeclareBinaryPhysics(VelocityFromMomentum,

DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,

rhoU / rho);

DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,

SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,

SPECIFIC_ENERGY_VAR,

e + UdotU / 2);

• Functors get reused in every Physics set, for all input argument types

• Minimizing how much code must be entrusted to physics people

Roy H. Stogner Metaprogramming April 16, 2012 41 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equations

Example

DeclareUnaryPhysics(DensityFromSpeciesDensities,

DENSITIES_VAR, DENSITY_VAR,

rhoi.sum());

DeclareBinaryPhysics(MomentumFromVelocity,

DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,

U * rho);

DeclareBinaryPhysics(VelocityFromMomentum,

DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,

rhoU / rho);

DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,

SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,

SPECIFIC_ENERGY_VAR,

e + UdotU / 2);

• Functors get reused in every Physics set, for all input argument types

• Minimizing how much code must be entrusted to physics people

Roy H. Stogner Metaprogramming April 16, 2012 41 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equations

Example

DeclareUnaryPhysics(DensityFromSpeciesDensities,

DENSITIES_VAR, DENSITY_VAR,

rhoi.sum());

DeclareBinaryPhysics(MomentumFromVelocity,

DENSITY_VAR, VELOCITY_VAR, MOMENTUM_VAR,

U * rho);

DeclareBinaryPhysics(VelocityFromMomentum,

DENSITY_VAR, MOMENTUM_VAR, VELOCITY_VAR,

rhoU / rho);

DeclareBinaryPhysics(SpecificEnergyFromSpecificInternalEnergy,

SPECIFIC_INTERNAL_ENERGY_VAR, SPEED_SQUARED_VAR,

SPECIFIC_ENERGY_VAR,

e + UdotU / 2);

• Functors get reused in every Physics set, for all input argument types

• Minimizing how much code must be entrusted to physics people

Roy H. Stogner Metaprogramming April 16, 2012 41 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets

• Equations are now represented by data types

• Data types can be stuck into metaprogramming containers!

Example

typedef VectorConstructor<

DensityFromSpeciesDensities,

VelocityFromMomentum,

MomentumFromVelocity,

SpeedSquaredFromVelocity,

SpecificEnergyFromConserved,

SpecificInternalEnergyFromSpecificEnergy,

SpecificEnergyFromSpecificInternalEnergy,

LinearTranslationalRotationalEnergyFromTemperature,

SpecificInternalEnergyFromOnlyTransationalRotationalEnergy

>::type AllPhysics;

Roy H. Stogner Metaprogramming April 16, 2012 42 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets

• Equations are now represented by data types

• Data types can be stuck into metaprogramming containers!

Example

typedef VectorConstructor<

DensityFromSpeciesDensities,

VelocityFromMomentum,

MomentumFromVelocity,

SpeedSquaredFromVelocity,

SpecificEnergyFromConserved,

SpecificInternalEnergyFromSpecificEnergy,

SpecificEnergyFromSpecificInternalEnergy,

LinearTranslationalRotationalEnergyFromTemperature,

SpecificInternalEnergyFromOnlyTransationalRotationalEnergy

>::type AllPhysics;

Roy H. Stogner Metaprogramming April 16, 2012 42 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets

• Equations are now represented by data types

• Data types can be stuck into metaprogramming containers!

Example

typedef VectorConstructor<

DensityFromSpeciesDensities,

VelocityFromMomentum,

MomentumFromVelocity,

SpeedSquaredFromVelocity,

SpecificEnergyFromConserved,

SpecificInternalEnergyFromSpecificEnergy,

SpecificEnergyFromSpecificInternalEnergy,

LinearTranslationalRotationalEnergyFromTemperature,

SpecificInternalEnergyFromOnlyTransationalRotationalEnergy

>::type AllPhysics;

Roy H. Stogner Metaprogramming April 16, 2012 42 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets

• Equations are now represented by data types

• Data types can be stuck into metaprogramming containers!

Example

typedef VectorConstructor<

DensityFromSpeciesDensities,

VelocityFromMomentum,

MomentumFromVelocity,

SpeedSquaredFromVelocity,

SpecificEnergyFromConserved,

SpecificInternalEnergyFromSpecificEnergy,

SpecificEnergyFromSpecificInternalEnergy,

LinearTranslationalRotationalEnergyFromTemperature,

SpecificInternalEnergyFromOnlyTransationalRotationalEnergy

>::type AllPhysics;

Roy H. Stogner Metaprogramming April 16, 2012 42 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Variable Sets

• Variables are now represented by enumerated values

• Enums can be stuck into metaprogramming containers!

Example

typedef UIntSetConstructor<

DENSITIES_VAR,

VELOCITY_VAR,

TEMPERATURE_VAR

>::type PrimitiveVars;

typedef UIntSetConstructor<

DENSITIES_VAR,

MOMENTUM_VAR,

ENERGY_VAR

>::type ConservedVars;

Roy H. Stogner Metaprogramming April 16, 2012 43 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Variable Sets

• Variables are now represented by enumerated values

• Enums can be stuck into metaprogramming containers!

Example

typedef UIntSetConstructor<

DENSITIES_VAR,

VELOCITY_VAR,

TEMPERATURE_VAR

>::type PrimitiveVars;

typedef UIntSetConstructor<

DENSITIES_VAR,

MOMENTUM_VAR,

ENERGY_VAR

>::type ConservedVars;

Roy H. Stogner Metaprogramming April 16, 2012 43 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Variable Sets

• Variables are now represented by enumerated values

• Enums can be stuck into metaprogramming containers!

Example

typedef UIntSetConstructor<

DENSITIES_VAR,

VELOCITY_VAR,

TEMPERATURE_VAR

>::type PrimitiveVars;

typedef UIntSetConstructor<

DENSITIES_VAR,

MOMENTUM_VAR,

ENERGY_VAR

>::type ConservedVars;

Roy H. Stogner Metaprogramming April 16, 2012 43 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Variable Sets

• Variables are now represented by enumerated values

• Enums can be stuck into metaprogramming containers!

Example

typedef UIntSetConstructor<

DENSITIES_VAR,

VELOCITY_VAR,

TEMPERATURE_VAR

>::type PrimitiveVars;

typedef UIntSetConstructor<

DENSITIES_VAR,

MOMENTUM_VAR,

ENERGY_VAR

>::type ConservedVars;

Roy H. Stogner Metaprogramming April 16, 2012 43 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

• Consider each variable as a rednode

• Consider each equation as ablack node

I Each output is a directededge from equation to variable

I Each input is a directed edgefrom variable to equation

Then our physics now looks like agraph!

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

Roy H. Stogner Metaprogramming April 16, 2012 44 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

We can use metaprogramming toanswer questions about the graph.

• Given a set of input variables:I Which non-input variables in

the graph are nowdetermined?

I What are the dependencies ofeach variable?

• And a set of output variables:I Which equations must be

solved to evaluate them?I In which order?

Roy H. Stogner Metaprogramming April 16, 2012 45 / 52

Numeric Functional Metaprogramming Physics as a Graph Problem

Equation Sets as Bicolored Graphs

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

We can use metaprogramming toanswer questions about the graph.

• Given a set of input variables:I Which non-input variables in

the graph are nowdetermined?

I What are the dependencies ofeach variable?

• And a set of output variables:I Which equations must be

solved to evaluate them?I In which order?

Roy H. Stogner Metaprogramming April 16, 2012 45 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

SolveList

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

45

6

12

3

7

Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.

1 Find all the dependencies ofyour output variables

I Unreachable output variablesbecome a compile-time error

I Unnecessary non-inputnon-output variables are leftout

2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list

Roy H. Stogner Metaprogramming April 16, 2012 46 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

SolveList

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

45

6

12

3

7

Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.

1 Find all the dependencies ofyour output variables

I Unreachable output variablesbecome a compile-time error

I Unnecessary non-inputnon-output variables are leftout

2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list

Roy H. Stogner Metaprogramming April 16, 2012 46 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

SolveList

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

45

6

12

3

7

Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.

1 Find all the dependencies ofyour output variables

I Unreachable output variablesbecome a compile-time error

I Unnecessary non-inputnon-output variables are leftout

2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list

Roy H. Stogner Metaprogramming April 16, 2012 46 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

SolveList

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

45

6

12

3

7

Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.

1 Find all the dependencies ofyour output variables

I Unreachable output variablesbecome a compile-time error

I Unnecessary non-inputnon-output variables are leftout

2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list

Roy H. Stogner Metaprogramming April 16, 2012 46 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

SolveList

T

ei

ρi

(ρe)i

ρe

ρ

~u

~u · ~u

ρ~u · ~u

ρE

45

6

12

3

7

Given input variables, outputvariables, and equation functor classtypes connecting them, we want aSolveList: an ordered set of whichequations to solve in what order.

1 Find all the dependencies ofyour output variables

I Unreachable output variablesbecome a compile-time error

I Unnecessary non-inputnon-output variables are leftout

2 Greedy search throughequations (in the same order asfor dependency finding!), addingsolveable equations to list

Roy H. Stogner Metaprogramming April 16, 2012 46 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables

3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types

2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equation

I Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Evaluations on the Graph

State Updating

With user-specified types on all variables, SolveList evaluation is easy:

1 Create a Container type holding the specified variable types

2 Instantiate this State state; set values on the user input variables3 Do a ForEach loop over the SolveList

I Pass input variables from state, assign output variables to state foreach equation

With user-specified types on input variables only, we first need ametaprogramming pass:

1 Create a Container type holding the specified input variable types2 Do a metaprogramming loop over the SolveList

I Pass input variable types to each equationI Insert output variable types into the State

3 Do a ForEach loop at runtime as before

Roy H. Stogner Metaprogramming April 16, 2012 47 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

Differentiation looks already solved!

1 Collect your Physics

2 Choose your input variables

3 Evaluate the SolveList

4 Use DualNumber input variable types

5 Take derivatives() of output variables

6 Cheer

Roy H. Stogner Metaprogramming April 16, 2012 48 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Equation Derivatives

But how well does that solve differentiation?• Forward differentiation issues:

1 Needlessly expensive for many-input-variable, few-output variablegraphs

• or subgraphs!

• Adjoint differentiation issues:1 Needlessly expensive for many-output-variable, few-input variable

graphs• or subgraphs!

2 Typically requires additional temporary data• particularly with operator overloading...

• Optimal Jacobian Accumulation issues:1 NP-complete!

• don’t want to stress-test compilers...

• Heuristic: “Eh, Good Enough” Jacobian Accumulation

Roy H. Stogner Metaprogramming April 16, 2012 49 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives

• Ni inputs, No outputs gives Ni ·No partial derivatives

• Small equations are efficiently evaluated with sparse forward AD

• Equation nodes are out of the picture

• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj

An intermediate variable M can be “condensed away”:

• For all outgoing edges to output variable O:

• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)

I (starting from D(I,O) ≡ 0 when creating a new edge)

Keep condensing until all remaining edges are directly from user inputs touser outputs

Roy H. Stogner Metaprogramming April 16, 2012 50 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives

• Ni inputs, No outputs gives Ni ·No partial derivatives

• Small equations are efficiently evaluated with sparse forward AD

• Equation nodes are out of the picture

• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj

An intermediate variable M can be “condensed away”:

• For all outgoing edges to output variable O:

• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)

I (starting from D(I,O) ≡ 0 when creating a new edge)

Keep condensing until all remaining edges are directly from user inputs touser outputs

Roy H. Stogner Metaprogramming April 16, 2012 50 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives

• Ni inputs, No outputs gives Ni ·No partial derivatives

• Small equations are efficiently evaluated with sparse forward AD

• Equation nodes are out of the picture

• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj

An intermediate variable M can be “condensed away”:

• For all outgoing edges to output variable O:

• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)

I (starting from D(I,O) ≡ 0 when creating a new edge)

Keep condensing until all remaining edges are directly from user inputs touser outputs

Roy H. Stogner Metaprogramming April 16, 2012 50 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives

• Ni inputs, No outputs gives Ni ·No partial derivatives

• Small equations are efficiently evaluated with sparse forward AD

• Equation nodes are out of the picture

• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj

An intermediate variable M can be “condensed away”:

• For all outgoing edges to output variable O:

• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)

I (starting from D(I,O) ≡ 0 when creating a new edge)

Keep condensing until all remaining edges are directly from user inputs touser outputs

Roy H. Stogner Metaprogramming April 16, 2012 50 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing one NodeEach equation node can be used to define local partial derivatives

• Ni inputs, No outputs gives Ni ·No partial derivatives

• Small equations are efficiently evaluated with sparse forward AD

• Equation nodes are out of the picture

• Partial derivatives are directed edges with weights D(Vi, Vj)connecting variables Vi and Vj

An intermediate variable M can be “condensed away”:

• For all outgoing edges to output variable O:

• For all incoming edges from input variable I:• D(I,O)+ = D(I,M) ·D(M,O)

I (starting from D(I,O) ≡ 0 when creating a new edge)

Keep condensing until all remaining edges are directly from user inputs touser outputs

Roy H. Stogner Metaprogramming April 16, 2012 50 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing the Graph

Ia

Ib

Ic

Ma

Mb

Mc Md

Me

Mf

Oa

Ob

Oc

A

B

C D

E

F

G

H

I

• In what order do we condense these equations?I N -factorial possibilitiesI Some possibilities are cheaper than othersI NP -complete optimization problem

Eh, good enough

Roy H. Stogner Metaprogramming April 16, 2012 51 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing the Graph

Ia

Ib

Ic

Ma

Mb

Mc Md

Me

Mf

Oa

Ob

Oc

A

B

C D

E

F

G

H

I

• Forward differentiation:I Evaluates equations inputs-to-outputsI Effectively does Ma,Mb,Mc,Md,Me,Mf

I 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations

Eh, good enough

Roy H. Stogner Metaprogramming April 16, 2012 51 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing the Graph

Ia

Ib

Ic

Ma

Mb

Mc Md

Me

Mf

Oa

Ob

Oc

A

B

C D

E

F

G

H

I

• Reverse differentiation:I Evaluates equations outputs-to-inputsI Effectively does Mf ,Me,Md,Mc,Mb,Ma

I Still 3 + 3 + 3 + 6 + 9 + 9 = 33 multiply+add operations

Eh, good enough

Roy H. Stogner Metaprogramming April 16, 2012 51 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing the Graph

Ia

Ib

Ic

Ma

Mb

Mc Md

Me

Mf

Oa

Ob

Oc

A

B

C D

E

F

G

H

I

• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md

I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!

Eh, good enough

Roy H. Stogner Metaprogramming April 16, 2012 51 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Chain Rule Application: Condensing the Graph

Ia

Ib

Ic

Ma

Mb

Mc Md

Me

Mf

Oa

Ob

Oc

A

B

C D

E

F

G

H

I

• Greedy (lazy?) differentiation:I Pick cheapest condensations firstI Does Mc,Ma,Mb,Me,Mf ,Md

I Still 2 + 3 + 3 + 3 + 3 + 9 = 23 multiply+add operations: 30% cheaper!

Eh, good enough

Roy H. Stogner Metaprogramming April 16, 2012 51 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Any Questions?

Lots of My Questions:• How much code for non-MASA

apps can be replaced bystandard libraries?

I Boost MTLI SacadoI Phalanx

• Can physics functors bedecomposed automatically?

I Precompilation script?

• Can we “roll up” sparsestructure algorithms?

Roy H. Stogner Metaprogramming April 16, 2012 52 / 52

Numeric Functional Metaprogramming Differentiation on the Graph

Any Questions?

Lots of My Questions:• How much code for non-MASA

apps can be replaced bystandard libraries?

I Boost MTLI SacadoI Phalanx

• Can physics functors bedecomposed automatically?

I Precompilation script?

• Can we “roll up” sparsestructure algorithms?

Roy H. Stogner Metaprogramming April 16, 2012 52 / 52