entropy-information and irreversibility

EntropyInformation and Irreversibility

Yun [email protected]

Teodor [email protected]

Sachith [email protected]

December 16, 2015AMSTERDAM UNIVERSITY COLLEGE

mailto:[email protected]



Contents

1 Introduction 5

2 Irreversibility 72.1 The Second Law . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Arrows of Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.2.1 Cosmological Arrow of Time . . . . . . . . . . . . . . . . . . . . . . 82.2.2 Other Arrows of Time . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Heat Engines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3.1 The Carnot Cycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.4 Brownian ratchet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.4.1 Efficiency of the Brownian ratchet . . . . . . . . . . . . . . . . . . . . 132.4.2 Mechanical and statistical (ir)reversibility . . . . . . . . . . . . . . . . 14

3 Entropy and Information 173.1 Statistical interpretation of Entropy . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.1 Derivation of a statistical expression for the entropy in a canonical en-semble . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.1.2 Consequences of the statistical interpretation of entropy . . . . . . . . 183.2 Entropy as Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.2.1 Basics of Information theory . . . . . . . . . . . . . . . . . . . . . . . 193.2.2 Entropy in information theory . . . . . . . . . . . . . . . . . . . . . . 213.2.3 Gibbs Inequality and Maximum information Entropy . . . . . . . . . . 223.2.4 Application of the Information Entropy to a simple physical system . . 23

3.3 Entropy as Ignorance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Entropy and Irreversibility in Quantum Mechanics 274.1 Von Neumann entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2 The measurement problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.3 Quantum decoherence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

5 Exercises 315.1 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315.2 Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3

4 CONTENTS

Chapter 1

Introduction

The phenomenon of the Arrow of Time has remained one of the most elusive and ambiguousconcepts in Physics, despite numerous theoretical advancements in the field. Thermodynamics,Relativity and Quantum Mechanics have all exposed fundamental aspects of reality, but primar-ily in the form of time-symmetric phenomena; in most of the realm of Physics, there exists nodistinction between the past and the future. Despite this, there remains a distinctive, explicitArrow of Time in our perception of reality - we remember the past but not the future. Thephenomenon of Entropy was developed in the 19th Century amidst rapid advances in the fieldof Thermodynamics. This, in combination with the Second Law of Thermodynamics, gives usa time-asymmetric physical phenomenon, and is where we can begin to articulate a physicalArrow of Time. We derived the Carnot efficiency, and showed that the Brownian ratchet is nota perpetuum mobile and has an efficiency not exceeding the Carnot efficiency.

After considering entropy as irreversibility and the arrow of time problem, we introducedthe two general interpretations of entropy, the statistical and information interpretation or Shan-non’s entropy, illustrating the second law of thermodynamics in terms of information entropy,by considering the evolution of a simple ideal gas model through time. The statistical and in-formation interpretation of entropy finally converged to the most general way of thinking aboutentropy, as ignorance about the initial conditions of the system.

We also introduced the concept of entropy in quantum mechanics, which is a natural ex-tension of the Gibbs entropy and is equivalent to the Shannon entropy in information theory.Irreversible process such as quantum decoherence is examined in, which also adds evidence tothe asymmetric property of the universe.

“Entropy isn’t whatit used to be.”

- ANONYMOUS

5

6 CHAPTER 1. INTRODUCTION

Chapter 2

Irreversibility

In this section we will introduce the Second Law of Thermodynamics, perhaps the most impor-tant application of entropy. Through it, we can begin to explore the concept of irreversibility,and its implications. The elusive issue of the Arrow of Time can also begin to be discussedwith these new tools in our repertoire, of which we briefly introduce some manifestations ofhere. Finally, we’ll take a look at the beginnings of entropy, how it came about and why it issuch a profoundly important subject, of which we still know very little about.

2.1 The Second LawThe Second Law of Thermodynamics dictates that the entropy of a closed system can neverdecrease, or

dS

dt≥ 0 (2.1)

where

S = kB ln Ω (2.2)

is the entropy of the system, kB is the Boltzmann constant and Ω is the multiplicity of thesystem. What this tells us about the physical state of the system is that once a closed systemhas evolved to a state with higher multiplicity, it will never, of its own accord, evolve back toone with lower multiplicity thereafter; an increase in entropy is an irreversible process.

Boltzmann entropy, as articulated in Eq.(2.2) proves to be a statistical explanation as towhy systems evolve the way they do, and why irreversibility is inherent in nature. By makingentropy a function of the multiplicity of the system, Eq.(2.2) illustrates how increasing entropyis merely the evolution of a system to its most probable state i.e. the state with the highestmultiplicity; the Second Law (2.1) is a statistical law. This proves to be in accordance withthe fact that most other physical phenomena are CPT-symmetric1, at least on a microscopic

1CPT-symmetry refers to the physical theorem that every physical phenomenon displays symmetry with re-gards to charge (C), parity (P) and time (T). It is the idea that if these three quantities were to be reversed - in aparallel Universe let’s say - then the evolution of that Universe would be identical to our’s.

7

8 CHAPTER 2. IRREVERSIBILITY

scale, as it allows for microscopic, fluctuations of decreasing entropy while emphasising thestatistical unlikelihood of such fluctuations on a macroscopic scale.

Given the irreversibility of an increase in entropy, how can we relate this to energy con-servation, and (engine) efficiency? With regards to the First Law of Thermodynamics (energyconservation), the Second Law adds a further limitation to the energy relations within a system.Not only must energy be conserved, but there is also some fundamental feature of nature thatirreversibly limits the extractable work of a system. You can not merely extract work from asystem with a temperature (and hence internal energy from U = 3

2NkBT ), which completely

adheres to the First Law. Instead, you need a system within which there exists a temperaturedifference, which results in an internal flow of heat from higher to lower temperature, as wellas an internal increase in entropy. This flow of energy is the foundation of the heat enginesdevised in the 19th Century, the efficiencies of which are strictly limited by this phenomenon.

2.2 Arrows of Time

Other than the Second Law, there exist various other observable time-asymmetric phenomena,such as the expansion of the Universe, Quantum Mechanical wave-function collapse and nu-clear decay. The interesting thing is that all of these can be broken down to be consequentof the Second Law of Thermodynamics. Exploration of these other arrows of time, however,are useful as they can help us garner what may be causing this unique direction of increasingentropy in time we experience in reality.

2.2.1 Cosmological Arrow of Time

Astrophysicists have observed the relentless expansion of the Universe at an ever increasing rate[1], to which one can attribute an Arrow of Time to, as this expansion proves to be asymmetricin time. In this model of the Universe, there exist two boundary conditions, namely a uniquepoint of lowest entropy - the Big Bang - and a point of highest entropy - the hypothetical heatdeath of Universe [2]. While the Big Bang had a huge mass-energy, and hence a relatively highentropy associated with that, all of this was densely configured into a small volume, leading toa small configuration entropy. Given this, it is very easy to analogize the cosmological Arrowof Time with the thermodynamical one - as the Universe expands, the number of configurations(and hence multiplicity Ω) of the Universe’s mass-energy increases, leading to a higher entropythan at the boundary condition that is the Big Bang. The hypothetical heat death of the Universecan also be thought of as a unique point of very high entropy. As the Universe expands, andeverything becomes more spread out, the density - and thereby the average temperature andinternal energy - decreases until it is infinitely spread out .

Boltzmann had an interesting hypothesis as to why the boundary condition of lowest en-tropy at the Big Bang occurred. He proposed (extending his Boltzmann Brains hypothesis) thatthe Universe came to being as a statistical fluctuation of low entropy, which we observe often ona microscopic scale. This fluctuation from a uniformly spread exterior Universe and resulted inthe Big Bang (remember, boundary condition of low entropy). In this model, the Universe natu-

2.3. HEAT ENGINES 9

rally tends towards equilibrium, namely in the direction of increasing entropy. This is coherentwith his perspective of entropy itself (Eq.2.2), namely one that is a statistical consequence of asystem tending towards equilibrium. Just as a system of particles with different temperaturestend towards the configuration in which energy (and thereby temperature) is equally dispersedamongst all the particles to bring the state as a whole to equilibrium. Similarly, from this Boltz-mannian perspective, the scenario of low entropy of the Big Bang (equivalent to the situation oflow entropy of the particles with varying temperatures) will increase its entropy in an attemptto re-achieve equilibrium.

A consequence of this model you can think of is the fact that there would exist two directionsin time in which entropy increases, both emanating outwards from the unique point of lowestentropy. If this in fact were the model of the Universe, then perhaps there exist, as proposedby Barbour et al. [3] local arrows of time depending on ‘which side’ of the point of lowestentropy you reside in. Either way, our best bet at articulating a physical Arrow of Time withthe expansion of the Universe in mind is in relation to the direction of increasing entropy.

2.2.2 Other Arrows of TimeIn coherence with the trend that most physical phenomena, Quantum Mechanical wave-functionevolution proves to be time-symmetric. However, the collapse of the wave-function, a conse-quence of observation on a quantum mechanical system proves to be asymmetric in time. Al-though the Boltzmann entropy (2.2) is no longer applicable in the quantum mechanical world,given the wave-particle duality and the consequent ambiguity regarding multiplicity, other in-terpretations of entropy2 beyond the scope of this book can be used to demonstrate the increasein entropy that results from the collapse of a wave-function. Extending this profound effect ofobservation on the quantum state of reality, some have proposed [4] that the act of observa-tion itself is what drives entropy to increase. In this model, anything that leaves a trace (andis thus observable and measurable to be studied by physicists) by definition must increase theentropy of a system. Entropy is just as likely to decrease, according to Maccone, and there isno fundamental asymmetry in time with regards to physical phenomena, but such a scenariois impossible to observe, as observation is dependent on interaction with a system, which willincrease the entropy of the system.

While the above theories are yet unproven attempts to explain the cause of this relentlessincrease of entropy we observe in our Universe, they are useful as they can all be analogized tothe notorious Second Law of Thermodynamics, and utilise areas beyond Thermodynamics inan attempt to explain entropy.

2.3 Heat EnginesMassive developments in the field of Thermodynamics occurred amidst the industrial revolutionof Europe when many scientists where concerned with maximising efficiency of engines. Anengine is merely a device used to convert one form of energy to utilisable work. In the 19th

2See Section 4.1 on von Neumann entropy


Figure 2.1: The P-V diagram for the Carnot Cycle

Century, engine technology centred around using heat transfer between reservoirs of differenttemperatures to extract mechanical work.

2.3.1 The Carnot Cycle

A pioneer in this field, Sadi Carnot (1796-1832), developed the renowned Carnot Cycle (Carnot,1824) which defines an upper limit for any energy conversion cycle, by theorising a hypothet-ical, fully reversible heat cycle. Reversible in this case implies there is no energy loss, andany work extracted from the system is put back into it to revert it to its initial state (that is, thetemperatures of the reservoirs return to their initial temperatures). In order to ensure the cycleas a whole remains wholly reversible, it must not lose energy as ‘waste’ (unutilisable) products,such as friction and sound.

To formulate this cycle (Fig.2.1), we start with reservoirs R1 and R2 with temperatures T1and T2, with T1 > T2 and a piston connected to a cylinder with a mono-atomic ideal gas totransfer heat between R1 and R2.

We begin by isothermally expanding the cylinder at a temperature T1, leading to decreasedpressure within the cylinder and heat Q1 transferred from R1 to the gas. Thermal contact withR1 is then blocked, resulting in the adiabatic expansion of the cylinder, until the temperatureof the gas within is T2.

The cylinder is then put in thermal contact with R2 and isothermally compressed at a tem-perature T2, resulting in heat Q2 transferred to R2. Thermal contact is then blocked betweenthe cylinder and R2, and further adiabatic compression is done on the cylinder, until it is at atemperature T1. At this point all state variables of the system are what they were at the start:internal energy U = 1

2NkT1 and entropy S = S1.

Now we’ll calculate the efficiency of the Carnot Engine. Recall that

2.3. HEAT ENGINES 11

η =W

Qin

(2.3)

where W is the useful work done by the system, and Qin is the heat put into the system.Recall that,

∆U = W +Qnet

and for a Carnot Engine

∆U = W −Q1 +Q2 (2.4)

where Q1 is the heat let into the system from R1 and Q2 is the heat let out of the systeminto R2. By combining Eq.(2.3) and Eq.(2.4),

we can get

ηCarnot =Q1 −Q2

Q1

= 1− Q2

Q1

(2.5)

Now to calculate Q1 and Q2, we can use the relationship

Q1 = −W1 =

∫ V2

V1

P · dV

and as we’re dealing with a mono-atomic ideal gas where PV = NkT

Q1 = NkT1

∫ V 2

V1

dV

V

= NkT1 ln(V2V1

)(2.6)

Following a similar derivation for Q2 = NkT2 ln V3V4

(note that Q2 > 0 as it is the amountof heat given out by the engine to R2). With this in mind and by using the fact that

V3V4

=V1V2

for a Carnot Engine, we have

ηCarnot = 1−NkT2 ln

(V3V4

)NkT1 ln

(V2V1

)= 1− T2

T1(2.7)


which is the maximum efficiency for a heat engine, because it has the lowest entropy in-crease, namely zero. In reality, a Carnot Engine is unattainable, as a heat cycle will never befully reversible, but the Carnot cycle assigns an upper limit for the efficiency of heat engines.

The key thing about this engine in the context of irreversibility is to note that the totalincrease in entropy of the Carnot Engine is 0. The fact that the Carnot Engine is reversibleand its total change in entropy is 0 is no coincidence - it is indicative of the fact that entropyis in fact a consequence of irreversibility, and a wholly reversible process will by definitionhave no net change in entropy. Once again we have a physical process that remains wholysymmetric in time - at any point in the Carnot cycle you can reverse the direction of time and itis indistinguishable to reversing the direction of the cycle.

2.4 Brownian ratchetA Brownian ratchet, as shown in Fig.2.2, is a device that hypothetically extracts useful workfrom random Brownian motion. In the compartment on the right, vanes are attached to an axlewhich will move as air molecules bombarding on the vanes at temperature T1. In the left thereare a ratchet and a pawl which allows it to turn in only one direction. It is also filled by thesame air molecules at temperature T2. The two compartments are connected by an axle, andthe wheel in between can be used to lift some weights.

Figure 2.2: Brownian ratchet

Since the air molecules in the box move in completely random directions, it is equally pos-sible for the vanes to turn in either direction. However, due to the ratchet and pawl mechanismon the left which prohibits motion in one direction, intuitively it seems that this device wouldbe propelled to move spontaneously in one direction forever. This is certainly unrealistic, andin fact if we examine it closely, we would realise that a simple ratchet and pawl combination isnot enough. Another element, namely a spring, must be introduced in order to press the pawlagainst the ratchet. Without this spring force, we have no guarantee that the ratchet will notmove backwards.

Further, assume we have an ideal ratchet so that there is no energy dissipation due to fric-tion, and all parts in this device are made of elastic materials. Every time the vanes gather

2.4. BROWNIAN RATCHET 13

enough energy to move the ratchet by one tooth, the pawl would also jiggle along. As a resultof the spring and the fact that everything is perfectly elastic, this jiggling motion will nevercome to an end. So for the ratchet to function in the way we desire, there must also be adamping mechanism which captures this energy and convert it to heat.

It is not hard to see that, given these modifications, this device will draw energy from thebox on the right, and this energy will then be used to do work on the load as well as dissipatein the form of heat in the left box. In other words, T1 will decrease while T2 will increase overtime.

2.4.1 Efficiency of the Brownian ratchetIt would be illuminating to study the behaviour of this device from the point of view of statisti-cal mechanics. Assume that the energy required to lift the spring on the pawl in order to movethe ratchet by one tooth is ε, and that each tooth corresponds to an angle θ. Assume also thatthe torque on the wheel due to the load is L, so that for every turn of angle θ the amount ofwork required for the wheel is Lθ.

In the left compartment, the ratchet and the pawl are actually subject to the same bombard-ment of air molecules as the vanes in the right. Therefore, it is possible that such Brownianmotion pushes the pawl up and set the ratchet in motion. Hence there are three possible statesfor the system:

1. Ratchet moves forward by θ2. Ratchet stays put3. Ratchet moves backward by θ

In order for the ratchet to move forward, the amount of energy needed is ε+Lθ, since workneed to be done to life the load as well as push pack the spring on the pawl. When the ratchetmoves backwards, however, the pawl only needs to gather energy ε to lift up, and the load willhelp the ratchet to slip back by θ while simultaneously give off energy Lθ as heat. Hence wecan derive the Boltzmann factors for the three states:

State Energy Boltzmann factor

Forward motion ε+ Lθ e− ε+Lθ

kT1

No motion 0 1

Backward motion ε e− εkT2

In the trivial case where no load is attached to the wheel, or L = 0, the partition function is

Z = 1 + e− εkT1 + e

− εkT2

and the possibilities of forward and backward motions are 1Ze− εkT1 and 1

Ze− εkT2 respectively.

Clearly, if T1 = T2, there will be equal probability for forward and backward motions to occur,


and as a result they will balance each other. While the device might jiggle around now and then,in general it is in a state of dynamic equilibrium. On the other hand, if one of the compartmentshave a higher temperature, the device will be moving in such a way so that the temperatures willbe equal to each other after some time. If we put the system in isolation from the environment,in other words there will be no heat input to maintain the temperature difference, the ratchetundergoes an irreversible change, and new entropy is generated as heat flows from the side ofhigher temperature to the side of lower temperature in the form of mechanical work.

When L 6= 0, the partition function is

Z = 1 + e− ε+Lθ

kT1 + e− εkT2

and the possibilities of forward and backward motions are 1Ze− ε+Lθ

kT1 and 1Ze− εkT2 respectively.

Given this condition a new “balance” will occur when

exp

(−ε+ Lθ

kT1

)= exp

(− ε

kT2

)ε+ Lθ

kT1=

ε

kT2

T1 =ε+ Lθ

εT2 (2.8)

As expected, this device can only do work when T1 > T2.

We can show that the Brownian ratchet will be able to lift the weight with an efficiency of

η =Lθ

Lθ + ε=T1 − T2T1

= 1− T2T1

as following from (2.8). This is the same as the Carnot efficiency (Feynman 1989) (see alsoEquation (2.7)), a theoretical maximum for converting heat into useful work. Apparently, theBrownian ratchet is not a a perpetuum mobile, and the laws of thermodynamics are not violated.Moreover, without continuous heat supply, the device will function in the direction towards anew equilibrium.

2.4.2 Mechanical and statistical (ir)reversibilityBeing a mechanical device and abiding by Newton’s laws, the Brownian ratchet, however,does not exhibit reversibility as other mechanical processes. The reason is that it is poweredthermodynamically, which follows the statistical law of entropy.

Now imagine putting the Brownian ratchet in a absolute isolation from the environment, asdescribed above we would expect this device to eventually reach a state of equilibrium where itis equally likely to move forward to backward. In this case we see no arrow of time, since thechanges are entirely time-independent. Alternatively, if the ratchet is in contact of the rest of the

2.4. BROWNIAN RATCHET 15

universe, then if the temperature is to be maintained, some heat exchange must be expected,so that the ratchet undergoes an irreversible change with new entropy generated. We mightconclude that this property is a result of the statistical nature of systems containing a very largenumber of particles (Feynman 1989).

Mechanical processes as described by Newton’s laws are perfectly time-independent, andare therefore reversible. The one-way behaviour of large systems (such as the universe) followsfrom an emergent statistical law, and is irreversible in the sense that events of higher statisti-cal likelihood is always the preferred one to occur. Moreover, everything that is part of thisevolving system also exhibits similar time-dependent behaviours.

“Let us draw an arrow arbitrarily. If aswe follow the arrow we find more and more

of the random element in the state of theworld, then the arrow is pointing towards

the future; if the random element decreasesthe arrow points towards the past I shall

use the phrase ‘time’s arrow’ to express thisone-way property of time which has no

analogue in space.”

- SIR ARTHUR STANLEY EDDINGTON (1928)

Chapter 3

Entropy and Information

3.1 Statistical interpretation of EntropyFormally entropy is defined as:

S = k ln Ω (3.1)

Where Ω is the multiplicity of the macrostate with the biggest number of microstates or possiblearrangements of the particles in the system. The use of (3.1), is however mainly limited toevaluating entropy changes in a system, instead of evaluating the entropy of the system in aparticular state. This difficulty arises when one tries to compute the multiplicity of a state inthe system Ω. The goal of this section is therefore to use the laws of statistical mechanics toderive a more useful expression for the entropy in a system in term’s of the canonical ensemble.

3.1.1 Derivation of a statistical expression for the entropy in a canonicalensemble

We consider a canonical ensemble with a total of NA systems, such that there are NS systemswith energy ES .The multiplicity of this ensemble can be written as :

Ω =NA!∏s

NS!(3.2)

,with the additional requirement that: ∑s

NS = NA (3.3)

Using the statistical definition of entropy eq.(3.1) we can write the entropy per system in thiscanonical ensemble as:

S =k

NA

ln Ω (3.4)

17

18 CHAPTER 3. ENTROPY AND INFORMATION

Using (3.2) and Sterling’s approximation in logarithmic form (lnN ! = N lnN −N ) we couldwrite the entropy per system as:

S =k

NA

ln Ω =k

NA

lnNA!∏s

NS!=

k

NA

(lnNA!− ln∏s

NS!)

=k

NA

(NA lnNA −NA −∑s

lnNs!)

=k

NA

(∑s

Ns lnNA −∑s

Ns −∑s

(Ns lnNs −Ns))

=k

NA

(∑s

Ns lnNA −∑s

Ns −∑s

Ns lnNs +∑s

Ns)

=k

NA

(∑s

Ns lnNA −∑s

Ns lnNs)

The last expression can we rewritten, if we note that the probability that a system is observedin state s is Ns

NA. So:

S = −k∑s

Ps lnPs (3.5)

On the other hand the Boltzmann factor of state s of the system is e−βEs , so the probabilityPs = e−βEs

Z.If we denote the energy of one particle in state s as εs equation (3.5) can be

rewritten as:

S = −k ln∑s

e−βεs +1

T

∑s

Ese−βεs

∑s

e−βεs(3.6)

Equation (3.6) is a useful way to calculate the absolute entropy of a system from first principles.If the system is a single particle than ES is changed with εs in (3.6).

3.1.2 Consequences of the statistical interpretation of entropyHere we will consider the implications of the statistical interpretation of entropy, particularly(3.1) and 3.6 to systems of Bose and Fermi gases. In the simplest case we consider a gas atT = 0K. At this temperature both the Fermi and Bose gases are in their ground state. Inthe case of Fermions at this temperature the Fermi-Dirac distribution becomes a step function,meaning that all energy levels up to εF = µ are going to be occupied while all higher energylevels will remain unoccupied. That means that as we add Fermions, each Fermions is goingto go to the lowest energy available state, until the last Fermion goes to the state with energyεf . This means that there is only one possible arrangement for each energy state, so Ω = 1 andfrom (3.1), we can see that S = 0. In the case a Bose-Einstein distribution all the Bosons willbe in their lowest energy state, and since they are indistinguishable Ω = 1, therefore requiring

3.2. ENTROPY AS INFORMATION 19

again that S = 0.

The same can predictions about the value of S can also be seen from equation (3.6). Ifwe take the lowest-energy single state E0 = 0 and assume that E0 id non-degenerate , in thelimit as limT→0 S = k log 1 = 0. As the temperature of the systems increases the particleswill tend to occupy higher energy levels, and there will be more ways in which the particlescan be arranged so they would form the same macrostate. From (3.6) we can also see that asT increases the entropy also increases, except when we encounter a phase transition. From thestatistical interpretation we can also see that entropy is additive. Suppose we have two systemsat equilibrium, such that system 1 has entropy S1, and multiplicity Ω1 and system 2 has entropyS2 and equilibrium multiplicity Ω2. When we combine the two systems at equilibrium we willhave:

Ω = Ω1 × Ω2 (3.7)

Therefore the combined system’s entropy using (3.7) and the statistical definition of entropy(3.1) will be:

S = k ln Ω = k ln(Ω1 × Ω2) = k ln Ω1 + k ln Ω2 = S1 + S2 (3.8)

This, can of coarse in a similar way be generalized if we consider the case of N combinedsystems at equilibrium. Now that we have analyzed the statistical definition of entropy let’slook at the definition of information entropy.

3.2 Entropy as Information

3.2.1 Basics of Information theory

Information is a usable measure we get from observing an event having probability p.Meaningwe are not interested about any particular features of the even, just whether the even has oc-curred. Thus information of an event will be defined in terms of the probability that the evenhas occurred p.Via an axiomatic approach, namely by naming a couple of properties that ourmeasure if information has to satisfy, we can develop a theory of information. The four axiomsthat a proper information measure ought to satisfy are therefore:

1. Information is a non-negative quantity: I(p) ≥ 0.

2. If an event has a probability of occurrence of 1, than we get no information from theoccurrence of the event: I(1) = 0

3. If two independent events occur, even A with probability p1, event B with probabilityp2, then the information we get from observing the two events is the sum of the information’sof each of the events separately , I(p1 × p2) = I(p1) + I(p2).


4. It will be required that the information measure be a continuous function of probability,namely small changes in probability measures result in small changes in information.

Now that we have the axioms that the information measure satisfies our next goal is to derivea useful formula that directly gives us the information gained from an event with probability ofoccurring p. First notice that from axiom (3), we have:

I(p× p) = I(p) + I(p) = 2× I(p) = I(p2) (3.9)

Thus repeating this process:

I(pn) = I(p) + I(p) · · · = n× I(p) (3.10)

So, using induction, one can show (see problem (1)) that in general I(pmn ) = m

nI(p), and from

axiom (4) because I(P ) is continuous, for 0 < p ≤ 1 and a real number a > 0 we get:

I(pa) = a× I(p) (3.11)

Thus, from (3.11) we can derive the nice property:

I(p) = − logb p = logb(1

p) (3.12)

This in general will be the way that we measure information, the base b of the logarithm de-termines the units we are using, using different bases for the logarithm results in informationmeasures that are just constant multiples of each other (logb2 p = logb2 b1× logb1 p).Thus somecommon units are bits corresponding to log2, trits or log3 units and Hartleys or log10 units. Wedo not need to bother about the units that much, so we will use the notation log p, and we willtypically think in terms of log2 p.To illustrate this consider flipping a fair coin, where the prob-ability of getting heads or tails is the same 1

2, the information obtained from flipping a fair coin

will thus be I = − log2(12) = 1, thus flipping a fair coin gives us 1 bit of information. Similarly

flipping a fair coin n times or equivalently flipping n fair coins, will give us log2(12)n = n bits

of information. We could enumerate a sequence of 10 flips as, a string hhhtththtt , where hstands for heads and t for tails, or by using 1 for heads and 0 for tails 1110010100, thus 10 flipsgive us 10 bits of information and we need 10 bits to specify the information contained in theflips.This reassures that the axiomatic definition of information measure is correct.

Now that we have a rigorous definition of information given by equation 3.12 consider theproblem of finding average amount of information per symbol in a given system. To be moreconcrete consider the following problem:

Suppose we have n symbols x1, x2, x3, · · · , xn and some source providing us with thesesymbols. Further suppose that the source emits the symbols with probabilities p1, p2, · · · , pn,and furthermore assume that the symbols are emitted independently, so that the newly emittedsymbol does not in any way depend on the symbols that have been previously emitted.What isnow the average amount of information we get from each symbol in this stream? Precisely theanswer of this question will lead to a definition of information entropy, which was devised byClaude Shannon in 1948.


3.2.2 Entropy in information theoryTo develop the definition of entropy let’s solve the problem posed in the previous section andthen give a generalized interpretation which is going to be valid for any probability distribution,be it continuous or discrete. Note that if we observe the symbol xi we get log2(

1pi

) of informa-tion from this particular observation, in the long run after N observations we will observe thesymbol xi approximately N × pi. Thus after N observations we will get a total information of:

I =n∑i=1

(N × pi) log2(1

pi) (3.13)

Than the average information per symbol observed is:

I =n∑i=1

pi log2(1

pi) (3.14)

Since limx→0 x log21x

= 0, we can for our purposes define pi log 1pi

to be 0 whenever pi =0.Equation (3.14) brings us to the fundamental definition of entropy in information theory. If wehave a probability distribution P = p1, p2, · · · , pn we define the entropy of this distributionas:

S(P ) =n∑i=1

pi log2(1

pi) (3.15)

Of course than the generalization for a continuous probability distribution P (x) is:

S(P ) =

∫P (x) log(

1

P (x))dx (3.16)

Here we use the conventional notation for entropy, by denoting it with letter S, however inmany information theory books the preferred notation is H in honor of Ralph Hartley oneof the founders of information theory. Another useful way to think about entropy of infor-mation of a particular probability distribution is to view it as an expected value of the in-formation contained in this distribution.Thus given a discrete probability distribution P =p1, p2, · · · , pn with pi ≥ 0 and

∑ni=1 pi = 1, or a continuous distribution P (x), such that

P (x) ≥ 0 and∫P (x)dx = 1, we can define the expected value of a associated discrete set

G = g1, g2, · · · , gn or a function G(x) as:

E(G) =n∑i=1

gipi (3.17)

in the discrete case or:E(G) =

∫G(x)P (x)dx (3.18)

in the continuous case.With these definitions we can finally write:

S(P ) = E(I(p)) (3.19)


or the entropy of a probability distribution is just the expected value of the information of thedistribution. Now that we have a reasonable understanding of what information entropy means,we can explore some of the properties of the function S(P ) and explore the connections be-tween the definition of entropy in thermodynamics and the definition of entropy in informationtheory.In the next section we consider the problem of finding the probability distribution thatmaximizes total information entropy. This is in a way analogous to maximizing the value ofthe multiplicity Ω when a system of particles is in a thermodynamic equilibrium.

3.2.3 Gibbs Inequality and Maximum information EntropyIn this section we will prove the Gibbs inequality which will be later in order to find a probabil-ity distribution which maximizes the total information entropy, or equivalently the expectationvalue of the information of the distribution.

To prove the Gibbs inequality we will use the fact that lnx ≤ x − 1, where equality holdsif and only if x = 1 (for a proof see problem 2).Now if we are given two discrete probabilitydistributions P = p1, p2, · · · , pn and Q = q1, q2, · · · , qn where pi, qi ≥ 0 and

∑i pi =∑

i gi = 1 we have the following inequality:

=n∑i=1

pi lnqipi≤

n∑i=1

pi(qipi− 1)

=n∑i=1

(qi − pi) = (n∑i=1

qi)− (n∑i=1

pi) = 1− 1 = 0

Equality holds if and only if pi = qi for all i. This result is called the Gibbs inequality and itis easy to see that it holds for any base, not just base e since loga x = lnx

ln a, and we can convert

any base to a to a base e up to a constant.

Now that we have the Gibbs inequality in our toolkit we can use it to find the probabilitydistribution which would maximize the entropy function. Suppose that we have the followingprobability distribution P = p1, p2, · · · , pn, then:

S(P )− log n =n∑i=1

pi log1

pi− log n =

n∑i=1

pi log1

pi− log n

n∑i=1

pi

=n∑i=1

pi log1

pi−

n∑i=1

pi log n =n∑i=1

pi(log1

pi− log n)

=n∑i=1

pi(log1

pi+ log

1

n) =

n∑i=1

pi log1/n

pi≤ 0

In the last step we have used Gibbs inequality, so equality holds if and only if pi = 1n

for all i.The result obtained gives us the following bounds on the information entropy of a system:

0 ≤ S(P ) ≤ log n (3.20)


From equation (3.20) it is easy to see that S(P ) = 0 if we have a probability distributionP = p1, p2, · · · , pn, where exactly one of the pi’s is equal to 1 and the rest are equal to 0, andS(P ) = log n if we have a probability distribution where all events have the same probabilityof occurring 1

n. Thus, the maximum of the entropy function is the logarithm of the number of

possible events and it occurs when all events are equally likely.

3.2.4 Application of the Information Entropy to a simple physical system

Since we have determined the bounds of the information entropy function the goal, we are nowready to apply this concept to the simple model of an ideal gas. The conclusions drawn will bea form of the second law of thermodynamics, namely as time progresses it is very likely for thesystem to reach equilibrium in the macrostate with a highest information entropy.

First consider a static model of an ideal gas of N point particles, such that at time t0 all theparticles are in a cubical box of length L, and therefore volume V = L3.Assume that throughsome measurement mechanics we are always able to determine the position of the particlessufficiently well as to be able to locate them in a box with sides 1

100L, or a volume 106 times

smaller than the original volume V , therefore in the original box we have 106 such small boxes.For each of the 106 boxes we can assign a probability pi of finding any specific gas particle inthat box by counting the number of particles ni in the box and dividing by the total numberof particles N , so pi = ni

N. Note that this is a valid probability distribution since pi ≥ 0 and∑

i pi =∑s niN

= NN

= 1, because each particle is contained only in one box. From thisprobability distribution we can calculate the entropy of this system:

S(P ) =106∑i=1

pi log1

pi=

106∑i=1

niN

logniN

(3.21)

If the particle are evenly distributed in the boxes each box will contain ni = N106

particles.Therefore, using 3.21 the entropy of the system is going to be:

S(P ) =106∑i=1

N106

Nlog

N106

N= log 106 (3.22)

The first thing to note about the result obtained is that it depends on the relative strength ofour measuring device, meaning if we were able to measure each particle sufficiently well as tobe able to determine its position to an accuracy of 1

1000L than the information entropy would

have increased and instead of log 106 it would have been log 109. Naturally the measurementscale for a physical system in the quantum limits will be bounded by Heisenberg’s uncertaintyprinciple. In this simplified model we have assumed that the particles have only one property,the position, if we want to talk about states of particles all we can do is specify the box wherethe particle is at t0, and if there are M such boxes than the information entropy of the system


would be logM .

Let’s now modify the model so that we allow the particles to move in the box as time pro-gresses. Than a configuration of the system at a certain time t would be a list of 106 numbersbi which is going to specify the number of particles in each of the 106 boxes. Also suppose thatthe motions of the particles are such that for each particle there is an equal probability that itwill move into any given new small box during one macroscopic time step. Let’s analyze thissystem carefully and deduce see whether this system will eventually reach the state of maxi-mum information entropy.

Since there are 106 boxes there are 106 possible configurations where all the particles arein one box. Hence exactly one of the probabilities pi is 1 and the rest are equal to zero, clearlyfrom 3.20 this is the configuration with lowest entropy, so Sallinonebox = 0.If we now considera pair of boxes, the number of configurations with all the particles evenly distributed betweenthe two boxes is : (

106

2

)=

106!

2!× (106 − 2)!≈ 5× 1011 (3.23)

Since only two boxes contain particles the probability, and the particles are evenly distributedthe probability that a particle is in each of the two boxes is 1

2, so the entropy of each of the

configurations is:

S =1

2log 2 +

1

2log 2 = log 2 (3.24)

Since there are at least 106 + 5× 1011 configurations, if we start the system in a configurationof entropy 0 the probability that the system is going to be in a configuration of S ≥ log 2 (sincethe probability is proportional to the number of configurations) is going to be:

≥ 5× 1011

106 + 5× 1011= (1− 106

5× 1011) ≈ 1− 10−5 (3.25)

Which means that if we start at a state with a very low entropy the probability that the system isspontaneously going to increase its entropy is very high, as we could have expected. Next, let’scalculate the number of configurations if the particles are distributed almost equally betweenall the boxes, but half of the boxes are short by one particle and half the boxes have an extraparticle. To calculate the number of such configurations we will use Sterling’s approximation(for an elaborate calculation see problem 3):(

106

106

2

)=

106!

(106

2!)2≈ 103×105 (3.26)

From equation (3.26), using essentially the same procedure as before one can calculate that ifwe start with entropy 0 the probability that the system will be in a configuration with a higherentropy is:

≥ 103×105

106 + 103×105 = (1− 106

103×105 ) ≈ 1− 10−3×105

(3.27)

3.3. ENTROPY AS IGNORANCE 25

Similar arguments can be made if we start with any number of configurations that produceentropy less than log 106 (see problem 4), as we can see this argument is analogous with thestatement about the second law of thermodynamics derived from the statistical interpretation ofentropy, namely that the system will always in equilibrium be in the macrostate with a highestnumber of arrangements or highest multiplicity (3.1). Here if we are given any macroscopicsystem which is free to change configurations, and given any configuration with less entropythan the maximum, there will be tendency as time progresses for the system to change into aconfiguration of a higher entropy until the maximum has been reached.

3.3 Entropy as IgnoranceAs we have seen in the previous two sections, the statistical interpretation of entropy and en-tropy as information offer generalized interpretations of entropy and the second law of thermo-dynamics.This provides us with the most general interpretation of entropy, namely as ignoranceabout a given system. The previous two sections have shown us that the entropy is maximizedwhen the system is in equilibrium, and throughout the process of getting to the equilibriumvalue, as entropy increases throughout time we lose all the information about the initial condi-tions of the system. This generalization of entropy as ignorance, allows us to prove that entropyis up to a constant a unique function that satisfies three key properties, meaning that the statis-tical interpretation and the interpretation of entropy as the expectation value of a informationof the system are equivalent. The three properties that the entropy function needs to satisfy arethe following:1. Entropy is maximum for equal probabilities.2. Entropy is unaffected by extra states for zero probability.3. Entropy changes for conditional probabilities

We won’t give the general proof that both the statistical interpretation and the informationinterpretation of entropy satisfy the three properties, since they can be found in [11], howeverthe proofs for properties 1 and 2 can be found in section 1.2.3 .

“It was not easy for a person brought upin the ways of classical thermodynamics

to come around to the idea that gainof entropy eventually is nothing more nor

less than loss of information”

- GILBERT NEWTON LEWIS (1981)

Chapter 4

Entropy and Irreversibility in QuantumMechanics

The arrow of time problem is also present in the field of quantum mechanics. In order to betterunderstand the behaviour of quantum systems, in particular the collapse of wave functions fromsuperposition into a single state, John von Neumann introduced the density matrix to describea quantum system of a mixed state. Assume the system can be found in orthogonal states |ψi〉with probability pi, the density operator is defined as

ρ =∑i

pi |ψi〉〈ψi| (4.1)

where the sum of all probabilities∑i

pi = 1.

The density matrix is given by

ρmn = 〈um|ρ|un〉 (4.2)

where uj’s are orthonormal bases. This provides us a powerful tool to examine quantum sys-tems of mixed states.

4.1 Von Neumann entropyJohn von Neumann also proposed an extension of the classical Gibbs entropy in quantum sta-tistical mechanics. For a quantum mechanical system described by a density matrix ρ, the vonNeumann entropy is

S = −tr(ρ ln ρ) (4.3)

where tr denotes the trace of the matrix.

It follows from (4.1) and (4.2) that the von Neumann entropy can be expressed as

S = −∑i

pi ln pi

27

28 CHAPTER 4. ENTROPY AND IRREVERSIBILITY IN QUANTUM MECHANICS

which is equivalent to the Shannon entropy in information theory.

4.2 The measurement problemFor a quantum system described by its wave function |ψ〉, the Schrodinger’s equation

ihd

dt|ψ〉 = H |ψ〉

describes how the system will evolve in time, where H denotes the Hamiltonian. As it unfolds,a vast multitude of possibilities become available and exist in a state of superposition. However,upon measurement, only one of the eigenstates can be perceived and recorded. The wave func-tion |ψ〉 is said to have “collapsed” in this process. The problem of how and why this collapsehappens is usually referred to as the “measurement problem”, and is open to interpretation tillthis day.

The most widely accepted interpretations include the Copenhagen interpretation proposedby Bohr and Heisenberg and the many-worlds interpretation proposed by Hugh Everett andpopularised by Bryce DeWitt. The main difference between the two lies at the differentiationbetween the quantum and the classical worlds (Zurek 2002). According to the Copenhagen in-terpretation, all measurements of a quantum system must by done by classical instruments, andthe collapse of the wave function happens when the quantum system is interfered by the clas-sical environment. On the other hand, the many-world interpretation claims to do without thisdichotomy between quantum and classical. The entire universe can be described by quantumtheory alone, and every time one quantum system interacts with another (such as an observercarrying out an measurement), the set of universe branches following different possibilities.

Figure 4.1: An illustration of the different interpretations

Whichever interpretation we take, we can show that the act of the observer, or the measure-ment, is an irreversible process. According to the Copenhagen interpretation, measurement

4.3. QUANTUM DECOHERENCE 29

results in the collapse of wave functions, which cannot restore themselves immediately after.The many-worlds interpretation predicts an increasing assemble of possible universes. Thusone might wonder, how does this irreversibility reflect from entropy?

4.3 Quantum decoherenceOne of the key facts we know so far is that macroscopic systems cannot be isolated fromtheir environments. However, since the Schrodinger’s equation only applies to closed systems,information will “leak” from the system into the environment, and quantum decoherence occursas a result (Zurek 2002). If we take the many-worlds interpretation, after measurement theobserver has the ability to know for sure what state the system is in, but has no access to theother possible branches of the universe at all. This loss of coherence must be accompanied bya change in entropy.

Diosi (2003) showed that there is indeed an increase in entropy upon measurement due toinformation loss. If the states of a system of n particles (n 1) changes from ρ1 to ρ2, thechange in von Neumann entropy is ∆S = −ntr(ρ1 ln ρ1) + ntr(ρ2 ln ρ2). With more advancedBayesian analysis one can show that there is an arrow of time associated with information loss(Diosi 2003).

A macroscopic system can be treated as a quantum system which is under continuous ob-servation (Joos 2007). In a sense the idea is quite similar to the one in Section 2.4.2: the factthat a system is part of the whole universe gives it the property to evolve in time as the universedoes.

“The opposite of a correct statement is afalse statement. The opposite of a profound

truth may well be another profound truth.

- NIELS BOHR (1885-1962)

30 CHAPTER 4. ENTROPY AND IRREVERSIBILITY IN QUANTUM MECHANICS

Chapter 5

Exercises

5.1 ProblemsQuestion 1Show using mathematical induction, and the axioms developed for information theory that:

I(pmn ) =

m

nI(p)

Question 2Prove that lnx ≤ x − 1, where equality holds if and only if x = 1. (Hint Use the Mean ValueTheorem)

Question 3Complete the steps in (3.26). (Hint use Sterling’s approximation to simplify the factorials).

Question 4Using the simple model of the gas developed in the text, what is the probability that if westart in the configuration of entropy log 2 that in the future we will end up in a state of higherentropy?

Solutions can be found on the next page!

31

32 CHAPTER 5. EXERCISES

5.2 Solutions

Question 1We will start by showing that:

I(pm) = m× I(p) (5.1)

For m = 2 the statement holds by axiom three:

I(p2) = I(p) + I(p) = 2× I(P )

Assume that form m = k the statement is true, and:

I(pk) = k × I(p) (5.2)

For m = k + 1 using (5.2) and axiom 3. we have:

I(pk+1) = I(pk × p) = I(pk) + I(p) = k × I(p) + I(p) = (k + 1)× I(p)

Hence, by induction (5.1) holds. Next note that:

I(p) = I((p1n )n) = n× I(p

1n ) (5.3)

So from (5.3):

I(p1n ) =

1

n× I(p) (5.4)

Hence in general using (5.1) and (5.4)

I(pmn ) = I((p

1n )m) = m× I(p

1n ) =

m

n× I(p) (5.5)

Question 2First note that for x > 1 : ∫ x

1

1

t2dt ≤

∫ x

1

1

xdt ≤

∫ x

1

dt (5.6)

Hence from (5.6) we get for x > 1:lnx ≤ x− 1

Also note that for x = 1 the equality holds. Now suppose that 0 < x ≤ 1. Define the functionf(x) = lnx − x + 1, and note that df(x)

dx= 1

x− 1 ≥ 0. So f(x) is a non-decreasing function

on the interval 0 < x ≤ 1,hence on that interval f(x) ≤ f(1) = 0. From the last inequality itfollows that lnx ≤ x− 1

5.2. SOLUTIONS 33

Question 3Using Sterling’s approximation:(

106

106

2

)=

106!

(106

2!)2≈√

2π(106)106e−10

6√

106

(√

2π(106

2)106

2 e−106

2

√106

2)2

=

√2π(106)10

6e−10

6√

106

2π(106

2)106e−106 106

2

=2106+1

√106

√2π√

106≈ 2106 ≈ 103×105

Question 4The number of configurations of entropy log 2 is 5 × 1011 and the number of configurationswith higher entropy is at least 103×105 (computed previously). Therefore the probability thatthe system will end up in a higher entropy state if it starts with entropy log 2 is bigger than:

≥ 103×105

5× 1011 + 103×105 = (1− 5× 1011

103×105 ) ≈ 1− 10−3×105+11 (5.7)

Which is as expected a bit smaller than the probability computed with (3.27) .

“I got 99 problems but entropyfrom the perspective of information

and irreversibility ain’t one”

- ANONYMOUS

34 CHAPTER 5. EXERCISES

Bibliography

[1] Hubble, Edwin Powell. The observational approach to cosmology. Oxford: ClarendonPress, 1937.

[2] Thomson, William. XLVII. On a universal tendency in nature to the dissipation of mechan-ical energy. The London, Edinburgh, and Dublin Philosophical Magazine and Journal ofScience 4.25 (1852): 304-306.

[3] Barbour, Julian, Tim Koslowski, and Flavio Mercati. Identification of a gravitational arrowof time. Physical review letters 113.18 (2014): 181101.

[4] Maccone, Lorenzo. Quantum solution to the arrow-of-time dilemma. Physical review let-ters 103.8 (2009): 080401.

[5] Clyde, Henry. Statistical interpretation of Temperature and Entropy. Retrieved from http ://www.physics.udel.edu/ glyde/PHY S813/Lectures/chapter9.pdf

[6] Diosi, Lajos. Probability of intrinsic time-arrow from information loss. Decoherence andEntropy in Complex Systems. Springer Berlin Heidelberg, 2004. 125-135.

[7] Feynman, Richard P., Robert B. Leighton, and Matthew Sands. The Feynman Lectures onPhysics, Desktop Edition Volume I. Vol. 1. Basic Books, 2013.

[8] J. D. Fast. Entropy: The Signifcance of the concept of entropy and its Applications inScience and Technology. New York: McGraw-Hill, 1962.

[9] Joos, Erich. Dynamical consequences of strong entanglement. Quantum Decoherence.Birkhuser Basel, 2007. 177-192.

[10] R. Landauer. Irreversibility and heat generation in the computing process. IBM J. Res.Develop., 5:183-191,1961.

[11] Sethna, James. Statistical mechanics: entropy, order parameters, and complexity. Vol. 14.Oxford University Press, 2006.

[12] T. Carter. An introduction to information theory and entropy. Santa Fe 2014.

[13] Zurek, Wojciech H. Decoherence and the transition from quantum to classical–REVISITED. arXiv preprint quant-ph/0306072 (2003).

35

entropy-information and irreversibility

Documents