maximum entropy, maximum entropy production and their application to physics and biology

Maximum Entropy,

Maximum Entropy Production

and their

Application to Physics and Biology

Roderick C. Dewar

Research School of Biological Sciences

The Australian National University

Part 1: Maximum Entropy (MaxEnt) – an overview

Part 2: Applying MaxEnt to ecology

Part 3: Maximum Entropy Production (MEP)

Part 4: Applying MEP to physics & biology

• The problem: to predict “complex system” behaviour

• The solution: statistical mechanics

- Boltzmann microstate counting (maximum probability)

- Gibbs algorithm (MaxEnt)

• Applications of MaxEnt to equilibrium systems

- micro-canonical, canonical, grand-canonical distributions

• Physical interpretation of MaxEnt

- frequency interpretation

- information theory interpretation (Jaynes)

• Extension to non-equilibrium systems (Jaynes)

• General properties of MaxEnt distributions

Part 1: MaxEnt – an overview

system

energy in

What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …

environment

matter in

many interacting degrees of freedom

energy out

matter out

opennon-equilibrium

Poleward heat transport

SW

LW

Latitudinal heat

transport H = ?

170 W m-2

300 W m-2

T

Cold plate, Tc

Hot plate, Th

Ra < 1760

conduction

T

Cold plate, Tc

Hot plate, Th

Ra > 1760

convection

H = ?

Turbulent heat flow (Raleigh-Bénard convection)

Fsw

Flw + H + E C, H20, O2, N

T,

Ecosystem energy & mass fluxes

system

energy in

What is the problem? - to predict the macroscopic behaviour of systems having many interacting degrees of freedom cells, plants, ecosystems, economies, climates …

environment

matter in

many interacting degrees of freedom

energy out

matter out

opennon-equilibrium

many degrees of freedom statistical mechanicsstatistical mechanics

Global Circulation Models, Dynamic Ecosystem Models ….

W(A) = number of microstates that give macrostate A

Microstate i1 Macrostate A = less detailed description

Ludwig Boltzmann (1844 - 1906)

The most probable macrostate A is the one with the largest W(A) (assume microstates are a priori equiprobable)

SB(A) = kBlog W(A) = Boltzmann entropy of macrostate A

The most probable macrostate is the one of maximum entropy

Boltzmann microstate counting

Microstate i2

Example: N independent distinguishable particles with fixed total energy E

Macrostate A = {nj particles are in state j}

Microstate i = {the mth particle is in state jm}ε3

ε2

ε1

j

jjj

jj

jj nnnnNN εβαloglogΦ

j

jn

NAW

!

!Nn

jj

Enj

jj ε: maximise S = kBlog W subject to

j

jj

jj

ZN

nβεexp

1

βεexp

βεexp

0

Φ

jn

(large N)

Boltzmann entropy Clausius entropy

β 1/kBT

Given E, Smax = kBlogWmax = kB(βE + NlogZ)

under δE = δQ, Smax changes by δSmax = kBβ(δQ)

cf. Clausius thermodynamic entropy δSTD = δQ/T

Smax STD

BUT: microstate counting only works for non-interacting particles

pi = probability that system is in microstate i Macroscopic predictions via

J Willard Gibbs (1839 - 1903)

Gibbs algorithm

i

iiQpQ

The Gibbs algorithm

(MaxEnt)

Maximise H = -i pi log pi with respect to {pi} subject to the

constraints (C) on the system

But how do we construct pi ?

‘minimise the index of probability of

phase’

• Closed, isolated

• Closed

• Open

Three applications of MaxEnt (equilibrium systems)

• Microcanonical

• Canonical

• Grand-canonical

System constraints (C) Distribution (pi)

Example 1: closed, isolated system in equilibrium

C: N and E fixed

Microstate i = any N-particle state with total energy Ei restricted to E

Precise description of i and Ei depends on microscopic physics (CAN include particle interactions)

1

1

NE

iip

,ΩMaximise

i

NE

ii ppH log

,Ω

1

subject to

NE

iii

NE

ii ppp

,Ω,ΩαlogΦ

11

NEpi ,Ω

10

ip

Φbasis for Boltzmann’s microstate counting

Example 2: closed system in equilibrium

Microstate i = any N-particle state (no restriction on Ei)

E

Maximise subject to

C: N and fixedE

iC

i EZ

p βexp 1

0

ip

Φ

β 1/kBT

N

iii ppH

Ωlog

1

1

1

N

iip

Ω

EEp

N

iii

Ω

1

N

iii

N

iii

N

ii Epppp

ΩΩΩβαlogΦ

111

Hmax STD

Example 3: open system in equilibrium

Microstate i = any physically allowed microscopic state (no restriction on Ei or Ni)

E N

Maximise

1i

ip

Ω

log1i

ii ppH subject to EEpi

ii

NNpi

ii

C: and fixedEN

iiGC

i NEZ

p γβexp 1

0

ip

Φ β 1/kBT

γ -μ/kBT

ΩΩΩΩ

γβαlogΦ1111 i

iii

iii

iii

i NpEpppp

Hmax STD

Frequency interpretation (Venn, Pearson, Fisher …)

System has Ω a priori equiprobable microstates

N independent identical systems, ni = no. of systems in state i

pi describes a physical property of the real world (frequency)

! ... !!

!

Ωnnn

N

21

W = no. of microstates giving {n1,n2 … nΩ}

pi = ni /N = frequency of microstate i

NHppWN i

ii as loglogΩ

1

1

MaxEnt coincides with large-N limit of Maximum Probability for multinomial W

• pi represents our state of knowledge of the real world

• basic axioms for uncertainty H associated with pi the unique uncertainty function is

• Applies to any discrete set of outcomes i

pi = 1/6 (i = 1…6) H = log 6maximum uncertainty :

pi = 0 (i = 1,2...5), p6 = 1 H = 0

minimum uncertainty :

Ω

1log

iii ppH

Information theory interpretation (Shannon 1948, Jaynes 1957 …)

Claude Shannon(1916-2001)

Jaynes (1957b, 1978)

Q (= ΣipiQi) reproducible under C

it is sufficient to encode only the information C into pi …

all information other than C is thrown away

… but this is precisely what MaxEnt does!

MaxEnt = max H subject to C

H = -i pi log pi = missing information about i

Behaviour that is experimentally reproducible under conditions C must be

theoretically predictable from C aloneEdwin Jaynes (1922-1998)

Assumed constraints C

experimental conditionsconservation laws

microstates (e.g. QM)

Reproducible behaviour Q

Max H subject to C pi

The prediction game

i

iiQpQ

Observed behaviour Qobs

Qobs Q missing constraint

MaxEnt

testC C'

ESSENTIAL PHYSICS PREDICTION

OBSERVATION

Edwin Jaynes (aged 14 months)

Information theory interpretation of MaxEnt

general algorithm for predicting reproducible behaviour

under given constraints

can be extended to non-equilibrium systems

(same principle, different constraints)

‘Maximum caliber principle’ (Jaynes 1980, 1996)

Γ micropaths

ΓΓ logmax ppH

cf. Feynman path integral formalism of QM!

A B

The second law in a nutshell

AB reproducible WB WA' = WA SB SA

WB

WA WA'

. .microscopic

path in phase-space

after Jaynes 1963, 1988

S = kBlog W

Liouville Theorem(Hamiltonian dynamics)

reproducible macroscopic change

Some general properties of MaxEnt distributions

λexpλ

1

kkiki f

Zp

1i

ip

m k,Ffpf ki

kiik 1

k

kk FλZH λlogmax

k

kZ

Fλ

λlog

kjkjkj

kjk

jjk Cffff

ZFC

λλ

λlog

λ

2

subject to m + 1 constraints C

i

ii ppH logmax

Fλ

k

k F

FHF

maxλ FHmax

i kkik fZ λexpλ

;

λ max2

kjkjj

kjk B

FF

FH

FB

1CB

Response-fluctuation & reciprocity relations:

Stability-convexity relation:

Constitutive relation: Orthogonality:

Partition function:

Summary of Lecture 1 …

The problem

to predict the behaviour of non-equilibrium systems with many degrees of freedom

The proposed solution

MaxEnt: a general information-theoretical algorithm for predicting reproducible behaviour under given constraints

Boltzmann

Gibbs

Shannon

Jaynes

maximum entropy, maximum entropy production and their application to physics and biology

Documents

maximum entropy maxent

maximum entropy production

microstate i1 macrostate

probable macrostate

degrees of freedom cells

microstate i2 example

overviewthe problem

biologythe problem