1 master

Probability and Stochastic processesThe Liouville equation

Brownian motionThe Master equation

Overview of the course

Complex systems: many particle systems with complexpatterns of interactionsObjective: physical theory that gives quantitative predictionsto be confronted with observations and experimentsNeed mathematical model that indicates how some variablesevolve in time and their connection to measureable quantitiesRandomness can appear in several ways

Finite precision on initial conditions (important with sensitivedependence on initial conditions) e.g. coin tossingLack of information about all relevant variables or inability toprocess them e.g. Brownian motionStochastic character of evolution laws e.g. animal behaviour(arguably depending on physical and chemical processes thatconstitute its brain and body, but not directly derivable fromthem)

A Annibale Dynamical Analysis of Complex Systems CCMCS04



Assume that individual degrees of freedom behave randomlyaccording to certain probabilistic rules

Consider many identical copies of the same system withdifferent realizations of the randomness: ensemble

Expect averages over ensembles exist and can be calculated

Get statistical properties of the motion, that can beexperimentally investigated by repeating experiments manytimes (or making observation time very long)

Stochastic models fully described by probability distribution tofind system at time t in a certain configuration s

In thermal equilibrium probability distribution is given by theGibbs-Boltzmann p(s) eH(s)/T : equilibrium models aredefined by an energy function s H(s) (no notion of time)




Non-equilibrium models are defined by a set of transition ratesand the probability distribution is obtained by solving theMaster equation

Analytical solutions of ME hardly available, two strategies:numerical integration or Perturbative theory (e.g. VanKampen system size expansion, Kramers-Moyal expansion etc)to cast ME into Fokker-Planck eqn

For many-particle systems often possible to deduce equationsfor a small set of (macroscopic) variables that followapproximately a deterministic law

eliminated variables are felt as a superimposed effective noise,often referred as fluctuations (basis of Langevin approach)

Stochastic approach needed to study fluctuations (importanton nanoscales) and to determine range of validity ofmacroscopic laws




Stochastic dynamics: Objectives

At the end of this section youll be able to:

1 Derive the Liouville equation for systems evolvingdeterministically

2 Write the Chapman-Kolmogorov equation for Markovprocesses

3 Derive the Master equation for Markov processes4 Use ME to derive equations for the average and fluctuations5 Use detailed balance to prove convergence to equilibrium of

ergodic systems6 Find the solution of a master equation via spectral

decomposition for systems which satisfy detailed balance7 understand the difference between equilibrium and steady

states




Outline

1 Probability and Stochastic processes

2 The Liouville equation

3 Brownian motionDimansional analysis and Scaling

4 The Master equationDerivationEquations for the momentsDirac notationEquilibrium dynamicsSpectral decompositionNon-equilibrium dynamics




Probability density

Stochastic variable X = variable whose value is unknown

Stochastic process X(t) = time evolution of stochastic var

Consider system which can be described in terms of X

P1(x, t) = prob. density that X has value x at time t

P2(x1, t1;x2, t2) = prob. density that X has value x1 at timet1 and x2 at time t2

Pn(x1, t1; . . . ;xn, tn) = prob. density that X has value x1 attime t1, . . . , and xn at time tn

Pn 0 n (Non-negative)dxn Pn(x1, t1; ...;xn1, tn1;xn, tn) =

Pn1(x1, t1; ...;xn1, tn1) (Marginal)dx1 P1(x1, t1) = 1 (Normalization)




Moments

Time-dependent moments

x(t1)x(t2)...x(tn) =dx1dx2...dxn Pn(xn, tn; ...;x2, t2;x1, t1)x1x2...xn

Stationary processes:

Pn(x1, t1;x2, t2; ...;xn, tn) = Pn(x1, t1+T ;x2, t2+T ; ...;xn, tn+T ) n, T

P1(x1, t1) = P1(x1) (x1(t1) = M)x1(t1)x2(t2) = C(|t1 t2|)

Equilibrium stationarityA Annibale Dynamical Analysis of Complex Systems CCMCS04



Connected correlator

If value of x2 at t2 independent of x1 at t1

P2(x1, t1;x2, t2) = P1(x1, t1)P1(x2, t2)

x1(t1)x2(t2) =dx1dx2 P2(x2, t2;x1, t1)x1x2

=

dx1dx2 P1(x2, t2)P1(x1, t1)x1x2 = x1(t1)x2(t2)

Connected correlator

x1(t1)x2(t2) x1(t1)x2(t2)measures degree of correlation between two measures taken atdifferent times




Markov processes

Markov property:

P1|n1(xn, tn|xn1, tn1; ...;x1, t1) =transition probability

P1|1(xn, tn|xn1, tn1), t1 < ... < tnMarkov process fully determined by P1 and P1|1P3(x1, t1;x2, t2;x3, t3) = P2(x1, t1;x2, t2)P1|2(x3, t3|x1, t1;x2, t2)

= P1(x1, t1)P1|1(x2, t2|x1, t1)P1|1(x3, t3|x2, t2)

Int. over x2, divide by P1(x1, t1): Chapman-Kolmogorov eqn

P1|1(x3, t3|x1, t1) =dx2 P1|1(x3, t3|x2, t2)P1|1(x2, t2|x1, t1)

For t1 = t2 = t : P1|1(x3, t3|x1, t) =dx2 P1|1(x3, t3|x2, t)P1|1(x2, t|x1, t)

satisfied by P1|1(x2, t|x1, t) = (x2 x1)A Annibale Dynamical Analysis of Complex Systems CCMCS04



Deterministic evolution

Assume system can be described by set of variables Xevolving according to autonomous ODE (Newtons,Hamiltons, Schrodingers etc)

d

dtX(t) = f(X(t))

e.g. classical system, N particles, 3-dim; q = (q1, . . . , q3N ),p = (p1, . . . , p3N ), briefly X = (q,p)

Set of possible values of X determines phase space

Each possible state of the system determines a point in phasespace

Deterministic evolution: state X at time t univocally assignedfrom initial state X0 at time 0 as X(t,X0), solution of ODEwith initial condition X(0) =X0




Randomness in deterministic evolution

1-dim for simplicity

ddtx(t) = f(x(t))x(0) = x0

x(t, x0)

Deterministic evolution so

P (x, t|x0, 0) = (x x(t, x0))However, x0 determined through measurements, subjects toerrors and finite precisioninitial conditions should not be given as a point in phasespace, but as distribution P (x0, 0)

P (x, t) =x0

(x x(t, x0))P (x0, 0) = (x x(t, x0))




The Liouville equation

P (x, t)

t=

t(x x(t, x0)) =

x(x x(t, x0))dx(t, x0)

dt

= x(x x(t, x0))f(x(t, x0)) =

x[(x x(t, x0))f(x)]

Distribution in phase space evolves according continuityequation

P (x, t)

t=

x[P (x, t)f(x)]

In some situations if initial condition is sharply peaked aroundx0, P (x, t) remains peaked around x(t, x0)But this is not true for sensitive dependence on initialcondition and stochastic dynamicsprobabilistic approach required when P (x, t) spreads over timeevolution A Annibale Dynamical Analysis of Complex Systems CCMCS04



Dimansional analysis and Scaling

The Brownian motion

historically first phenomenological theory of how fluctuatingphenomena arise

observation by Brown in 1827: a pollen grain suspended inwater is found in very animated and irregular motion

Spectacular evidence on macroscopic scale for discrete oratomic nature of matter on the micro-scale

Paradigm theory for many-body systems in classical statisticalmechanics (noise, thermal bath, separation of time scalebetween degrees of freedom, fluctuation-dissipation etc.)first explanations:

Einstein (1905), Smoluchowski (1906): neglect inertiaLangevin (1908): account for inertia

1950s: clear that can apply theory of Brownian motion to anyobservable in a macroscopic system generalized BM





Einsteins explanation

motion caused by frequent impacts on the pollen grains bymolecules of the liquid

too complicated statistical descritpionAssumptions:

motion of each particle independent of othersmotion of the same particle in successive time intervals areindependent (timpact tobs)

Simple: isotropy; can in fact look at one dimension

x(t+ ) = x(t) + (t)

random, distributed according to () = ()d() = 1,

d () = 0,

d 2() = a2

induces distribution of x, P (x, t), withdxP (x, t) = 1





Diffusion equation

Markov assumption

P (x, t+ ) =

dx P (x, t)(x x) =

dP (x, t)()

decays very rapidly, P broad

P (x, t+ ) =

d()[P (x, t)

xP (x, t) +

1

22

2

x2P (x, t) + . . .]

= P (x, t) +1

2a2

2

x2P (x, t)

small P (x, t+ ) = P (x, t) + PtP (x, t)

t= D

2P (x, t)

x2, D = lim

01

2

d()2





Solution by Fourier Transform

Solve in Fourier space for P (x, 0) = (x)

G(q, t) =

dxP (x, t)eiqx, G(q, 0) = 1

G(q, t)

t= Dq2G(q, t) G(q, t) = eDq2tG(q, 0) = eDq2t

P (x, t) =1

2pi

dq G(q, t)eiqx =1

4piDtex

2/4Dt

Momentsx = 0, x2 = 2Dt

one of central results in statistical physics: x(t) t fordiffusion





Contains many major concepts central in dynamical analysis ofstochastic processes

independence of the pushes on the previous history:Markovian property

Kramers-Moyal expansion: approximation which effectivelyreplaces a process whose sample path need not be continuouswith one whose paths are continuous.

Fokker-Planck equation: equation for the probabilitydistribution to find the system in a certain state

Alternative: regard x(t) as a continuous function of time, buta random function: write a stochastic differential equation forthe path (initiated by Langevin)

Tools for non-equilibrium analysis, such as Dimensionalanalysis and Scaling





Dimensional analysis

Get moments from dimensional analysis without solving eqn

P (x, t)

t= D

2P (x, t)

x2

Clearly, x = 0, as there is no biasx2 non trivial, should depend on D and tLet L=unit of length, T=unit of time

[x] = L, [t] = T, [D] = L2/T

[x2] = L2 x2 = C DtC e.g. from equation for moments

d

dtx2 = 2D

Dimensional analysis works for much more complex problemsVery simple: should be used as first resort





Scaling

[P ] = L1 e.g. fromdxP (x, t) = 1

so,DtP (x, t) dimensionless

using x, t,D, can form dimensionless quantity = x/Dt

DtP (x, t) = ()

Scaling ansatz

P (x, t) =1Dt

()

Plug into eqn, get an ODE

2 + + = 0 () = (4pi)1/2e2/4

P (x, t) =1

4piDtex

2/4Dt




DerivationEquations for the momentsDirac notationEquilibrium dynamicsSpectral decompositionNon-equilibrium dynamics

The Master equation

Define transition probability per unit time

Wt(x|x) = P1|1(x

, t+ |x, t)

|=0Assume can Taylor expand P1|1(x, t+ |x, t) for small

P1|1(x, t+ |x, t) = (x x) + Wt(x|x) +O(2)Need correction to preserve normalization

P1|1(x, t+ |x, t) = c(x x) + Wt(x|x) +O(2)dx P1|1(x, t+ |x, t) = 1 c = 1

dxWt(x|x)

Define escape rate a(0)(x, t) =dxWt(x|x) so

P1|1(x, t+ |x, t) =[1 a(0)(x, t)

](xx)+ Wt(x|x)+O(2)





The Master equation - 2

Recall

P1(x, t+ ) =

dx

P2(x,t+ ;x,t)

P1(x, t)P1|1(x, t+ |x, t)

For small

P1(x, t+ ) = P1(x, t)[1 a(0)(x, t)] + dx P1(x, t)W (x|x)

For 0 get continuous time Master equation

tP1(x, t) =

dx [W (x|x)P1(x, t)W (x|x)P1(x, t)]

For discrete set of states, let pn(t) = P1(n, t)

dpn(t)

dt=n

[Wnnpn(t)Wnnpn(t)]





Comments

Master Equation is a gain-loss equation

Not invariant for t t irreversible dynamics towardssteady state where transitions cannot cause further changes toprobability distribution (P/t = 0)

Broad applicability (Markov process), only needed transitionprobability over short time

ME also applies to all transition probabilities P1|1(x, t|x0, t0)Given X(t), defined by P1(x1, t1) and P1|1(x2, t2|x1, t1), canalways extract sub-ensemble X?(t) withP ?1 (x1, t1) = P (x1, t1|x0, t0) andP ?1|1(x2, t2|x1, t1) = P1|1(x2, t2|x1, t1),





Equation for the average

for arbitrary function f(x):

d

dtf(x) =

f(x)

P1(x, t)

tdx =

f(x)[W (x|x)P1(x, t)

W (x|x)P1(x, t)]dxdx =

[f(x) f(x)]W (x|x)P1(x, t)dxdx

f(x) = x, def a()(x, t) =dx (xx)Wt(x|x) = 0, 1...

dxdt

=

dx a(1)(x)P1(x, t) = a(1)(x, t)

if a(1) is linear, a(1)(x, t) = a(1)(x, t), eqn closestx = a(1)(x, t) deterministic, ODE

if a(1) not linear, expand about xA Annibale Dynamical Analysis of Complex Systems CCMCS04




Equation for the variance

If a(1) not linear need higher order moments. For f(X) = X2

d

dtX2 =

dx dx (x2 x2)W (x|x)P1(x, t)

=

dx dx [(x x)2 + 2x(x x)]W (x|x)P1(x, t)

= a(2)(X)+ 2Xa(1)(X)

but generally depends on higher order moments..

often approximation required to close eqns e.g. neglectfluctuations X2 ' X2





Bra and KetReminder: for vector A in 3-dim Euclidean space have

A = A1e1 +A2e2 +A3e3 =

A1A2A3

em en = mnGen. to N -dim vector space over complex numbers, A CN

ket : |A = A1|1+A2|2+ . . .+AN |N =

A1A2...AN

Hermitian conjugate |A+ = A|bra : A| = A?11|+A?22|+. . .+A?N N | = (A?1, A?2, . . . , A?N )Inner (gen. scalar) product: A|B A|A = i |Ai|2Identity operator 1: |A = ii|A|i i |ii| = 1





Master equation in Dirac notation

pn(t) =m

[Wmnpm(t)Wnmpn(t)]

Define Lnm = Wmn mn

n Wnn

pn(t) =m

Lnmpm(t)

Associate with each possible configuration n = 1 . . .M a basisvector |n: n |nn| = 1; m|n = mnregard pn(t) as n-th component of a state vector |P (t)

|P (t) =

p1(t)p2(t)...pN (t)

so, |P (t) = n pn(t)|n, i.e. pn(t) = n|P (t)





L matrices

Recallpn(t) =

m

Lnmpm(t)

Rates collected in matrix

L =mn

Lnm|nm|, n|L|m = Lnm

t |P (t) = L|P (t)projection state vector I| = (1, . . . , 1) = nn|1 =

n pn(t) = I|P (t) tI|P (t) = 0 I|L = 0

= 0 is always an eigenvalue with left eigenstate I| andright eigenstate |Peq, as t|Peq = 0





How do we choose the rates?

if dynamics converges to equilibrium steady stateequilibrium probabilities pn pn() generally known, e.g.canonical ensemble|Peq =

n pn|n indep on time, so L|Peq = 0

also, equilibrium dynamics is reversible, i.e. a trajectory overtime t = mt, n0 . . . nm is as likely as nm . . . n0this requires Detailed Balance (DB), i.e. no net probabilitycurrent:

Wmnpm = Wnmpn n,mDB guarantes that pn is a steady state:

m

Wnmpn =m

Wmnpm pn = 0

but is a stronger condition





Metropolis and Glauber rates

Focus on canonical ensemble, pn = Z1eH(n)

DB: exp[H(n)]Wnm = exp[H(m)]Wmnchoice not unique

Metropolis rates

Wnm =

{1 H(m) H(n)e[H(m)H(n)] H(m) > H(n)

Glauber

Wnm =1

1 + e[H(m)H(n)]





Convergence to equilibrium

If rates are in detailed balance with pn can show for ergodicsystems that pn(t) pn from any pn(0)Focus on canonical ensemble,

pn() = 1ZeH(n)

Kullback-Leibler distance D(p||q) = n pn ln pnqnF (t) =

n

pn(t) lnpn(t)

pn() =n

pn(t)[ln pn(t) + H(n) + lnZ]

F (t) is a Liapunov functionF (t) 0 (= 0 iff pn(t) = pn())F (t) 0 (= 0 iff pn(t) = pn())

dFdt =

n [ln pn(t) + H(n) + 1]

dpndt





Use Master equation and DB: WnneH(n) = WnneH(n)

dF

dt= 1

2

nn

WnneH(n)[(ln pn(t)+H(n))(ln pn(t)+H(n))]

[eH(n)+ln pn (t) eH(n)+ln pn(t)] 0

Used identity: (ex ey)(x y) 0 (x, y), equality iff x = ySystem must reach a point where

eH(n)pn = eH(n)pn or Wnn = 0

pn = (n)eH(n), with (n) = (n) n, n : Wnn 6= 0

Sn set of states dynamically accessible from n:(n) = Z1n n Snergodic: (n) = Z1 n Boltzmann unique equilibrium





Steady state

Solution of ME hardly available

Formal solution to ME: |P (t) = eLt|P (0)Need to diagonalize L: usually non-trivial, L non-hermitianEasy when DB holds

stationary distribution of ME can be obtained by iterating

p2 =W21W12

p1; p3 =W32W23

W21W12

p1; . . .

withn

pn = 1

non-stationary distribution pn(t) can be otained from spectraldecomposition of ME





Spectral decomposition of ME

U symmetric complete orthonormal set of eigenvectorsi|U = ii|, U |i = i|i, i = 0, . . . ,M 1

|P (t) = eUt|P (0) =i

eit|ii|P (0)

=i

m

eit|ii|mm|P (0)

pn(t) =i

m

eitn|ii|mpm(0)

pn(t) =

M1i=0

Mm=1

pnpm

pm(0)i|meitn|i, i 0 i





Solution to the ME

Stationary solution

let (0) = 0 : pn = limt pn(t) =

Mm=1

pnpm

pm(0)0|mn|0m

pm(t) = 1 t m|0 = 0|m = pm

Time-dependent solution

pn(t) = pn +M1i=1

Mm=1

pnpm

pm(0)i|meitn|i

Equilibrium given by eigenvector with zero eigenvalueRelaxation time given by second smallest eigenvalue (inabsolute value) = 1/mini>0|(i)|





Non-equilibrium dynamicsSome systems reach steady state (SS), but this does NOTsatisfy DB with L (e.g. biased random walker on cycle)

m

Wmnpm =m

Wnmpn but Wmnpm 6= Wnmpn

These are non-equilibrium systems: for these, SS can NOT ingeneral be described by Gibbs distribution

P eq(C) = Z1eE(C)/T , Z =C

eE(C)/T

non-eq. systems do not normally have an energy functionE(C), but are described by their transition rateslack of energy function common in irreversible processes,where basic dynamical quantities, e.g. number of particles arenot conserved (aggregation, fragmentation, adsorption)however possible sometimes to construct energy functionwhich describes SS (conservative process, e.g. biased RW)





Choice of rates makes part of the modelling

If DB does not hold, time-dependent solution available only inspecial cases

when not, a numerical integration might be possible

or Perturbation theory: Kramers-Moyal (small jump)expansion, Van Kampens (large size) expansion etc





Textbooks

N.G. Van Kampen (2007)Stochastic Processes in Physics and Chemistry, Elsevier, 3rdEdition

Linda E ReichlA Modern Course in Statistical Physics, Wiley-VCH 2009





Recent research papers

P Moretti, A Baronchelli, A Barrat, R Pastor-Satorras (2011)Complex Networks and Glassy Dynamics: walks in the energylandscape J. Stat. Mech. P03032

B Waclaw, RJ Allen, M Evans (2010)A dynamical phase transition in a model for evolution withmigration, Phys. Rev. Lett. 105, 268101

J Currie, M Castro, G Lythe, E Palmer, C Molina-Paris (2012)A stochastic T cell response criterion JR Soc Interface9:2856-2870


Probability and Stochastic processesThe Liouville equationBrownian motionDimansional analysis and Scaling

The Master equationDerivationEquations for the momentsDirac notationEquilibrium dynamicsSpectral decompositionNon-equilibrium dynamics

1 master

Documents

particle systems

coursecomplex systems

time t

physical theory

equilibrium ofergodic

longstochastic models

relevant variables

observation time