a soft intro to mean field games - amazon s3

The optimizing agent The transported density Symmetric N-player games

A Soft Intro to Mean Field Games

Barnabe Monnot

December 11, 2015

The optimizing agent The transported density Symmetric N-player games

What we are looking at

Game theory is complicated enough with only two players, butwhen the number explodes...

I Use ’population’ arguments (congestion games...)

I Use ’algorithmic’ arguments (learning...)

I Play on networks (local effects...)

Many connections between one another.

Today we are looking at way to move to the continuous usingPDEs and mean field arguments.

The optimizing agent The transported density Symmetric N-player games

If you like pina coladas...

The optimizing agent The transported density Symmetric N-player games

The Beast

MFG “basic” system∂u∂t

+ ν∆u + H(x ,∇u) + V [m] = 0∂m∂t− ν∆m + div (m∇pH(x ,∇u)) = 0

m|t=0 = m0, u|t=T = g [mT ]

The optimizing agent The transported density Symmetric N-player games

Why it is worth looking at

Very general model, extends far beyond economics. The previoussystem admits as particular cases famous equations such as theEuler equations or the Vlasov ones.

Piggybacks on very active fields of research, such as optimaltransport or stochastic control.

Has found a good deal of numerical methods for solving thesetypes of games, taking it out of the theoretical forest (pedestriancrossing animation).

The optimizing agent The transported density Symmetric N-player games

Derivation of the MFG system

We will derive The Beast from a simple model. Two steps:

I Agents optimize their choices given the density of otherplayers (our m) ⇒ control in feedback, gives backwardequation.

I The densities are transported by these choices ⇒ forwardequation.

The optimizing agent The transported density Symmetric N-player games

Plan

I The optimizing agent

I The transported density

I Symmetric N-player games

The optimizing agent The transported density Symmetric N-player games

The agent’s problem

We have a nice space Y (take Rd , or call it the beach), with adistance d . Look at time interval [0,T ].

For all t ∈ [0,T ], suppose the density of the players on Y , mt isgiven.

The agent controls (using α ∈ A) the stochastic equation

dXt = αtdt +√

2νdBt

where Bt is a Brownian motion in Rd .

The optimizing agent The transported density Symmetric N-player games

The agent’s problem (cont.)

The agent solves the following stochastic control probleminfα E

[ ∫ T0

(L(Xt , αt) + V [mt ](Xt)

)dt + g [mT ](XT )

]dXt = αtdt +

√2νdBt

X0 = x

Define Hamiltonian H

H(x , p) = minα∈A{L(x , α) + α · p}

Hamilton-Jacobi-Bellman

The value function u satisfies{∂u∂t + ν∆u + H(x ,∇u) + V [m] = 0

u|t=T (x) = g [mT ](x)

The optimizing agent The transported density Symmetric N-player games

A closer look at the V

V is the cost incurred from the density of other players. Can give ageneral example:

V [m](x) = V0(x)︸ ︷︷ ︸I like the bar

+

I don’t like people︷ ︸︸ ︷f (x ,m(x)) +

∫YW (x , y)m(dy)︸ ︷︷ ︸

But not too alone either

BUT

I With congestion term f (x ,m(x)), no existence of equilibrium(need V [m] ∈ C (Y ))

I With distance term∫Y W (x , y)m(dy), no uniqueness (need

V [m] to be monotonous in m, e/gV [m](x ,m(x)) ≥ V [m](y ,m(y)) if m(x) ≥ m(y))

The optimizing agent The transported density Symmetric N-player games

Plan

I The optimizing agent

I The transported density

I Symmetric N-player games

The optimizing agent The transported density Symmetric N-player games

Movement of agents

So far we have considered the measure m as given, but if agentsmove, m does too.

Agents find an optimal control α(x) such that

∇pH(x , p) = α(x)

This drift is their motion.

The optimizing agent The transported density Symmetric N-player games

Transport of the measure

Now, for all t, mt = L(Xt), with m0 = L(X0) given. So mt is thesolution of

∫Y φ dmt =

∫Y E[φ(X x

t )] dm0(x)

dXt = αtdt +√

2νdBt

m0 = L(X0)

φ is a test function (∈ C∞c (Y ))

Fokker-Planck equation

mt is governed by{∂m∂t − ν∆m + div (mα) = 0

m|t=0 = m0

The optimizing agent The transported density Symmetric N-player games

The Beast, dissected

Clear where this system is coming from?∂u∂t

+ ν∆u + H(x ,∇u) + V [m] = 0∂m∂t− ν∆m + div (m∇pH(x ,∇u)) = 0

m|t=0 = m0, u|t=T = g [mT ]

The optimizing agent The transported density Symmetric N-player games

The linear-quadratic case

Let L(x , α) = |α|22 , then

H(x , p) = minα

|α|2

2+ p · α = −|p|

2

2

The MFG system now reads∂u∂t + ν∆u− |∇u|

2

2 + V [m] = 0∂m∂t − ν∆m − div (m∇u) = 0

m|t=0 = m0, u|t=T = g [mT ]

The optimizing agent The transported density Symmetric N-player games

Plan

I The optimizing agent

I The transported density

I Symmetric N-player games

The optimizing agent The transported density Symmetric N-player games

Symmetric N-player games

We can look at games with N players and see what happens whenN → +∞.

We assume all players are identical, choose an action xi ∈ Y . Costfunction for player i is

Ji (xi , x−i ) = F (xi , x−i )

F is symmetric in the sense that for all permutations σ on N − 1,F (xi , x−i ) = F (xi , (xσ(j))j 6=i ).

If N → +∞, then the number of variables in F also goes toinfinity...

The optimizing agent The transported density Symmetric N-player games

Ashes to ashes, points to measures

The trick: map positions x = (x1, ..., xN) ⊂ Y of N players to theempirical measure mN

x ∈ P(Y )

mNx =

1

N

∑i∈N

δxi

Then if x = y up to a permutation, mNx = mN

y .

(Some like to think of x and y as belonging to the quotient spaceY /GN)

The optimizing agent The transported density Symmetric N-player games

Distances on P(Y )

Who says limits says distances, and we have a couple options forour space of measures P(Y ).

Let m, m in P(Y ). Define the 1-Wasserstein distance by

W1(m, m) =

{inf∫Y×Y d(x , y) γ(dx , dy)

γ ∈ Π(m, m)

Π(m, m) is the set of couplings between m and m, i/e jointdistributions whose marginals are m and m.

Connections with optimal transport here...

The optimizing agent The transported density Symmetric N-player games

Deblais et remblais

Problem: X and Y have the same volume, find the cheapesttransportation plan between them.

Figure: Pulled directly from Topics on Optimal Transportation, C. Villani

The optimizing agent The transported density Symmetric N-player games

W1 for empirical measures

To nail our transition from Y to P(Y ), we show our distancestranslate naturally.

Take x and y in Y N , define d a distance such that

d(x , y) = minσ∈GN

1

N

∑i∈N

d(xi , yσ(i))

Then W1(mNx ,m

Ny ) = d(x , y) (uses Birkhoff’s theorem on

bistochastic matrices).

The optimizing agent The transported density Symmetric N-player games

Looking back

I We started from a space Y of players’ positions.

I We moved from Y to P(Y ) by using empirical measures.

I We defined a distance on the space P(Y ), the 1-Wassersteindistance.

I This distance metrizes the weak topology on P(Y ), i/e ifmn ⇀

∗ m⇒W1(mn,m)→ 0.

⇒ We can take limits in P(Y )!

The optimizing agent The transported density Symmetric N-player games

The limit of functions with N variables

Let FN : Y N −→ R symmetric, FN(x) = FN(mNx ). Under nice

assumptions, there is a function F ∈ C (P(Y )) such that:

supx∈Y N

|FN(mNx )− F (mN

x )| −→N→∞ 0

The optimizing agent The transported density Symmetric N-player games

Back to games

We now have a strong conceptual background to look at limits ofgames.

Assume each player has cost function of the form FNi (x1, ..., xN).

Our previous theorem says we can send N to infinity.

Asymptotically, players’ costs will be close to

F(xi ,

1

N − 1

∑j 6=i

δxj

)for some F : Y × P(Y )→ R.

The optimizing agent The transported density Symmetric N-player games

Nash equilibrium in mean field

We arrive at our mean field theorem.

Mean field equation

Assume that for all N, XN = (xN1 , ..., xNN ) is a Nash equilibrium for

the game (FN1 , ...F

NN ). Then, up to a subsequence, the sequence of

empirical measures (mN) converges to a measure m such that∫YF (y , m) dm(y) = inf

m∈P(Y )

∫YF (y , m) dm(y)

We did the reasoning for pure strategies, can extend to mixed⇒ guarantees existence of NE.

The optimizing agent The transported density Symmetric N-player games

Where to go

Once we have all this, we can tie up the two parts (derivation ofthe MFG system and symmetric N-player games).Assume N players, each one controls system in the form∫ T

0Li (x1(t), ..., xN(t), αi (t))dt + gi (x1(T ), ..., xN(T ))

Can assume

I Li (x1(t), ..., xN(t), αi (t)) = |α|22 + F

(1

N−1∑

j 6=i δxj

)I gi (x1(T ), ..., xN(T )) = g

(xi ,

1N−1

∑j 6=i δxj

)

The optimizing agent The transported density Symmetric N-player games

Nash equilibrium

We assume there is a map UNi such that

UNi (xi , t, (xj)j 6=i ) = UN(xi , t, (xj)j 6=i )

and it satisfies a bunch of HJB equations∂UNi

∂t + 12

∣∣∣∇xiUNi

∣∣∣2 − F(

1N−1

∑j 6=i δxj

)+∑

j 6=i ∇xjUNj · ∇xjU

Ni = 0

UNi = gi , t = T

In other words, we have a direction for the player to follow, andthe optimal control αi = −∇xiU

Ni is a Nash equilibrium (can prove

that).

The optimizing agent The transported density Symmetric N-player games

The optimizing agent The transported density Symmetric N-player games

Suspended disbelief

We know we can push N to infinity and the symmetric functionUN will have a limit U, but we need to do it in the space ofprobabilities P(Y ).

Using estimates on ∇xiUN , we can switch to that space (actually

using the 2-Wasserstein distance...)

We then give ourselves the initial measure m0 and push it forwardusing the obtained U (and its derivatives).

The optimizing agent The transported density Symmetric N-player games

Resulting system

With a bit more work we obtain:−∂u

∂t+ 1

2|Du|2 = F (m)

∂m∂t− div (mDu) = 0

m|t=0 = m0, u(x ,T ) = g(x ,m(T ))

The optimizing agent The transported density Symmetric N-player games

References

Good references out there, I used:

I P.L Lions videos at the College de France (some are inEnglish)

I P. Cardaliaguet’s notes (online)

I Lecture notes from class taught by G. Carlier in Dauphine

Cocorico!

Thank you!