Kernel Mean Embeddings

George Hron, Adam Scibior

Department of EngineeringUniversity of Cambridge

Reading Group, March 2017

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Kernels

A kernel k is a positive definite function X × X →, that is for alla1, . . . , an ∈, x1, . . . , xn ∈ X ,

n∑i ,j=1

aiajk(xi , xj) ≥ 0

Intuitively a kernel is a measure of similarity between two elementsof X .

Kernels

Commonly used kernels:

I polynomial(〈x1, x2〉+ 1)d

I Gaussian RBF

e−‖x−y‖2

2σ2

I Laplace

e−‖x−y‖σ

RKHS

A reproducing kernel Hilbert space (RKHS) for a kernel k is onespanned by functions k(x , ·) for all x ∈ X with the inner productdefined by

〈k(x1, ·), k(x2, ·)〉 = k(x1, x2)

which is well-defined on all of H by linearity of the inner product.

The equation above is known as the kernel trick and lets us treatH as an implicit feature space, where we never have to explicitlyevaluate the feature map.

RKHS

H has the reproducing property, that is for all f ∈ H and all x ∈ X ,

〈f , k(x , ·)〉 = f (x)

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Expectations in RKHS

Goal: evaluating expected values of functions from an RKHS.

As with the kernel trick, it turns out that this is possible in termsof inner products, making the computation analytically tractable,

Ep

[f (x)] =

∫X

p (dx)f (x) =

∫X

p (dx)〈k (x , ·), f 〉Hk

?= 〈∫X

p (dx)k (x , ·), f 〉Hk=: 〈µp , f 〉Hk

µp is so called kernel mean embedding (KME) of distribution p .

Note: X assumed measurable throughout the whole presentation.

Expectations in RKHS

Goal: evaluating expected values of functions from an RKHS.

As with the kernel trick, it turns out that this is possible in termsof inner products, making the computation analytically tractable,

Ep

[f (x)] =

∫X

p (dx)f (x) =

∫X

p (dx)〈k (x , ·), f 〉Hk

?= 〈∫X

p (dx)k (x , ·), f 〉Hk=: 〈µp , f 〉Hk

µp is so called kernel mean embedding (KME) of distribution p .

Note: X assumed measurable throughout the whole presentation.

Expectations in RKHS

Goal: evaluating expected values of functions from an RKHS.

As with the kernel trick, it turns out that this is possible in termsof inner products, making the computation analytically tractable,

Ep

[f (x)] =

∫X

p (dx)f (x) =

∫X

p (dx)〈k (x , ·), f 〉Hk

?= 〈∫X

p (dx)k (x , ·), f 〉Hk=: 〈µp , f 〉Hk

µp is so called kernel mean embedding (KME) of distribution p .

Note: X assumed measurable throughout the whole presentation.

Existence of KME

Riesz representation theorem: For every bounded linearfunctional T : H → R (resp. T : H → C), there exists a uniqueg ∈ H such that T (f ) = 〈g , f 〉H,∀f ∈ H.

Expectation is a linear functional, and ∀f ∈ H we have,

Epf (x) ≤

∣∣∣∣Ep f (x)

∣∣∣∣ ≤ Ep

∣∣f (x)∣∣ = E

p

∣∣〈f , k (x , ·)〉H∣∣ ≤‖f ‖H E

p

∥∥k (x , ·)∥∥H

Thus if Ep

√k (x , x) <∞, then µp ∈ H exists, and

Ep f (x) = 〈µp , f 〉H,∀f ∈ H.

Existence of KME

Riesz representation theorem: For every bounded linearfunctional T : H → R (resp. T : H → C), there exists a uniqueg ∈ H such that T (f ) = 〈g , f 〉H,∀f ∈ H.

Expectation is a linear functional, and ∀f ∈ H we have,

Epf (x) ≤

∣∣∣∣Ep f (x)

∣∣∣∣ ≤ Ep

∣∣f (x)∣∣ = E

p

∣∣〈f , k (x , ·)〉H∣∣ ≤‖f ‖H E

p

∥∥k (x , ·)∥∥H

Thus if Ep

√k (x , x) <∞, then µp ∈ H exists, and

Ep f (x) = 〈µp , f 〉H,∀f ∈ H.

Existence of KME

Riesz representation theorem: For every bounded linearfunctional T : H → R (resp. T : H → C), there exists a uniqueg ∈ H such that T (f ) = 〈g , f 〉H,∀f ∈ H.

Expectation is a linear functional, and ∀f ∈ H we have,

Epf (x) ≤

∣∣∣∣Ep f (x)

∣∣∣∣ ≤ Ep

∣∣f (x)∣∣ = E

p

∣∣〈f , k (x , ·)〉H∣∣ ≤‖f ‖H E

p

∥∥k (x , ·)∥∥H

Thus if Ep

√k (x , x) <∞, then µp ∈ H exists, and

Ep f (x) = 〈µp , f 〉H,∀f ∈ H.

Estimating KME

Because µp is generally unknown, it has to be estimated.

A natural (and minimax optimal) estimator is the sample mean,

µp :=1

N

N∑n=1

k (xn, ·)

and in particular,

µp (x) = 〈µp , k (x , ·)〉H =1

N

N∑n=1

k (xn, x)N→∞−−−−→ E

p (x ′)k (x ′, x)

Estimating KME

Because µp is generally unknown, it has to be estimated.

A natural (and minimax optimal) estimator is the sample mean,

µp :=1

N

N∑n=1

k (xn, ·)

and in particular,

µp (x) = 〈µp , k (x , ·)〉H =1

N

N∑n=1

k (xn, x)N→∞−−−−→ E

p (x ′)k (x ′, x)

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Representing Distributions via KME

A positive definite kernel is called characteristic if,

T : M+1 (X )→ H

p 7→ µp

is injective; M+1 (X ) is the set of probability measures on X . [5, 6]

Characteristic kernels

Proving a kernel is characteristic is non-trivial in general, butsufficient conditions exist. Three well known examples:

I Universality : If k is continuous, X compact, and Hk dense inC(X ) wrt L∞, then k is characteristic. [6, 8]

I Integral strict positive definiteness: A bounded measurablekernel k is called integrally strictly positive definite if∫X∫X k (x , y)µ(dx)µ(dy) > 0 for all non–zero finite signed

Borel measures µ on X . [22]

I Some stationary kernels: For X = Rd , a stationary kernel k ischaracteristic iff supp Λ (ω) = Rd , where Λ (ω) is the spectraldensity of k (cf. Bochner’s theorem). [22]

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

t-test

Have {x1, . . . , xN}iid∼ N (µ1, σ

21), and {y1, . . . , yM}

iid∼ N (µ2, σ22),

where all parameters are unknown.

Declare H0 : µ1 = µ2;H1 : µ1 6= µ2. Then for,

µ1 :=1

N

N∑n=1

xn µ2 :=1

M

M∑m=1

ym

µi ∼ N (µi ,σ2iN ), ∀i ∈ {1, 2}, and µ1 − µ2 ∼ N (µ1 − µ2,

σ21N +

σ22

M ).

Hence tH0∼ t ν , ν :=

(s21/N+s22/M)2

(s21/N)2

N−1+

(s22/M)2

M−1

, where,

t :=(µ1 − µ2)− 0√

s21N +

s22M

, s21 :=

∑Ni=1(xi − µ1)2

N− 1, s22 :=

∑Mi=1(yi − µ2)2

M− 1.

t-test

Have {x1, . . . , xN}iid∼ N (µ1, σ

21), and {y1, . . . , yM}

iid∼ N (µ2, σ22),

where all parameters are unknown.

Declare H0 : µ1 = µ2;H1 : µ1 6= µ2. Then for,

µ1 :=1

N

N∑n=1

xn µ2 :=1

M

M∑m=1

ym

µi ∼ N (µi ,σ2iN ), ∀i ∈ {1, 2}, and µ1 − µ2 ∼ N (µ1 − µ2,

σ21N +

σ22

M ).

Hence tH0∼ t ν , ν :=

(s21/N+s22/M)2

(s21/N)2

N−1+

(s22/M)2

M−1

, where,

t :=(µ1 − µ2)− 0√

s21N +

s22M

, s21 :=

∑Ni=1(xi − µ1)2

N− 1, s22 :=

∑Mi=1(yi − µ2)2

M− 1.

t-test

Have {x1, . . . , xN}iid∼ N (µ1, σ

21), and {y1, . . . , yM}

iid∼ N (µ2, σ22),

where all parameters are unknown.

Declare H0 : µ1 = µ2;H1 : µ1 6= µ2. Then for,

µ1 :=1

N

N∑n=1

xn µ2 :=1

M

M∑m=1

ym

µi ∼ N (µi ,σ2iN ), ∀i ∈ {1, 2}, and µ1 − µ2 ∼ N (µ1 − µ2,

σ21N +

σ22

M ).

Hence tH0∼ t ν , ν :=

(s21/N+s22/M)2

(s21/N)2

N−1+

(s22/M)2

M−1

, where,

t :=(µ1 − µ2)− 0√

s21N +

s22M

, s21 :=

∑Ni=1(xi − µ1)2

N− 1, s22 :=

∑Mi=1(yi − µ2)2

M− 1.

t-test

http://grasshopper.com/blog/the-errors-of-ab-testing-your-conclusions-can-make-things-worse/

More examples

Arthur Gretton, Gatsby Neuroscience Unit

More examples

Arthur Gretton, Gatsby Neuroscience Unit

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Recap

Kernel mean embedding,

Ep (x)

[f (x)] = 〈f , µp 〉H, ∀f ∈ H

For a characteristic kernel k and the corresponding RKHS H,

µp = µq iff p = q .

This means we might be able to distinguish distributions bycomparing the corresponding kernel mean embeddings.

Recap

Kernel mean embedding,

Ep (x)

[f (x)] = 〈f , µp 〉H, ∀f ∈ H

For a characteristic kernel k and the corresponding RKHS H,

µp = µq iff p = q .

This means we might be able to distinguish distributions bycomparing the corresponding kernel mean embeddings.

Recap

Kernel mean embedding,

Ep (x)

[f (x)] = 〈f , µp 〉H, ∀f ∈ H

For a characteristic kernel k and the corresponding RKHS H,

µp = µq iff p = q .

This means we might be able to distinguish distributions bycomparing the corresponding kernel mean embeddings.

Maximum Mean Discrepancy (MMD)

Measure distance between mean embeddings by the worst casedifference of expected values [9],

MMD(p , q ,H) := supf ∈H,‖f ‖H≤1

(E

p (x)[f (x)]− E

q (y)[f (y)]

)= sup

f ∈H,‖f ‖H≤1

(〈µp − µq , f 〉H

)

Notice that 〈µp − µq , f 〉H ≤∥∥µp − µq ∥∥H‖f ‖H, with equality iff

f ∝ µp − µq . Hence,

MMD(p , q ,H) =∥∥µp − µq ∥∥H

Maximum Mean Discrepancy (MMD)

Measure distance between mean embeddings by the worst casedifference of expected values [9],

MMD(p , q ,H) := supf ∈H,‖f ‖H≤1

(E

p (x)[f (x)]− E

q (y)[f (y)]

)= sup

f ∈H,‖f ‖H≤1

(〈µp − µq , f 〉H

)

Notice that 〈µp − µq , f 〉H ≤∥∥µp − µq ∥∥H‖f ‖H, with equality iff

f ∝ µp − µq . Hence,

MMD(p , q ,H) =∥∥µp − µq ∥∥H

Witness function

Arthur Gretton, Gatsby Neuroscience Unit

Witness function

Arthur Gretton, Gatsby Neuroscience Unit

Estimation

Recall: empirical kernel mean embedding estimator,

µp =1

N

N∑n=1

k (xn, ·)

We can estimate the square of MMD by substituting the empirical

estimator. For {xi}Ni=1iid∼ p and {yi}Ni=1

iid∼ q ,

MMD2

=1

N(N− 1)

N∑i=1

N∑j 6=i

(k (xi , xj) + k (yi , yj)

)− 1

N2

N∑i=1

N∑j=1

(k (xi , yj) + k (xj , yi )

)

Estimation

Recall: empirical kernel mean embedding estimator,

µp =1

N

N∑n=1

k (xn, ·)

We can estimate the square of MMD by substituting the empirical

estimator. For {xi}Ni=1iid∼ p and {yi}Ni=1

iid∼ q ,

MMD2

=1

N(N− 1)

N∑i=1

N∑j 6=i

(k (xi , xj) + k (yi , yj)

)− 1

N2

N∑i=1

N∑j=1

(k (xi , yj) + k (xj , yi )

)

Estimation

Proof sketch:

MMD2(p , q ,H) =∥∥µp − µq ∥∥2H =

∥∥µp ∥∥2H +∥∥µq ∥∥2H − 2〈µp , µq 〉H

where,

∥∥µp ∥∥2H = 〈µp , µp 〉H = Ex∼p

µp (x) = Ex∼p〈µp , k (x , ·)〉H

= Ex∼p

Ex∼p

k (x , x) ≈ 1

N(N− 1)

N∑i=1

N∑j 6=i

k (xi , xj)

Similarly for the other terms.

Estimation

Proof sketch:

MMD2(p , q ,H) =∥∥µp − µq ∥∥2H =

∥∥µp ∥∥2H +∥∥µq ∥∥2H − 2〈µp , µq 〉H

where,

∥∥µp ∥∥2H = 〈µp , µp 〉H = Ex∼p

µp (x) = Ex∼p〈µp , k (x , ·)〉H

= Ex∼p

Ex∼p

k (x , x) ≈ 1

N(N− 1)

N∑i=1

N∑j 6=i

k (xi , xj)

Similarly for the other terms.

Estimation

Proof sketch:

MMD2(p , q ,H) =∥∥µp − µq ∥∥2H =

∥∥µp ∥∥2H +∥∥µq ∥∥2H − 2〈µp , µq 〉H

where,

∥∥µp ∥∥2H = 〈µp , µp 〉H = Ex∼p

µp (x) = Ex∼p〈µp , k (x , ·)〉H

= Ex∼p

Ex∼p

k (x , x) ≈ 1

N(N− 1)

N∑i=1

N∑j 6=i

k (xi , xj)

Similarly for the other terms.

Considerations

I The distribution of MMD2

under H0 assymptoticallyapproaches an infinite sum of shifted chi-squared randomvariables multiplied by eigenvalues of the RKHS.

I Approximations to the sampling distribution [1, 10, 11, 12].

I Calculation of the naive MMD2

estimator is O(N2).I Linear time approximations [2, 26].

I Performance is dependent on choice of the kernel. There is nouniversally best performing kernel.

I Previously heuristical, recently replaced by hyperparameteroptimisation based on test power, or Bayesian evidence [4, 24].

I Empirical kernel mean estimator might be suboptimal.I Better estimators exist [4, 16, 17].

Considerations

I The distribution of MMD2

under H0 assymptoticallyapproaches an infinite sum of shifted chi-squared randomvariables multiplied by eigenvalues of the RKHS.

I Approximations to the sampling distribution [1, 10, 11, 12].

I Calculation of the naive MMD2

estimator is O(N2).I Linear time approximations [2, 26].

I Performance is dependent on choice of the kernel. There is nouniversally best performing kernel.

I Previously heuristical, recently replaced by hyperparameteroptimisation based on test power, or Bayesian evidence [4, 24].

I Empirical kernel mean estimator might be suboptimal.I Better estimators exist [4, 16, 17].

Considerations

I The distribution of MMD2

under H0 assymptoticallyapproaches an infinite sum of shifted chi-squared randomvariables multiplied by eigenvalues of the RKHS.

I Approximations to the sampling distribution [1, 10, 11, 12].

I Calculation of the naive MMD2

estimator is O(N2).I Linear time approximations [2, 26].

I Performance is dependent on choice of the kernel. There is nouniversally best performing kernel.

I Previously heuristical, recently replaced by hyperparameteroptimisation based on test power, or Bayesian evidence [4, 24].

I Empirical kernel mean estimator might be suboptimal.I Better estimators exist [4, 16, 17].

Considerations

I The distribution of MMD2

under H0 assymptoticallyapproaches an infinite sum of shifted chi-squared randomvariables multiplied by eigenvalues of the RKHS.

I Approximations to the sampling distribution [1, 10, 11, 12].

I Calculation of the naive MMD2

estimator is O(N2).I Linear time approximations [2, 26].

I Performance is dependent on choice of the kernel. There is nouniversally best performing kernel.

I Previously heuristical, recently replaced by hyperparameteroptimisation based on test power, or Bayesian evidence [4, 24].

I Empirical kernel mean estimator might be suboptimal.I Better estimators exist [4, 16, 17].

Optimised MMD and model criticism

Arthur Gretton’s twitter.

Optimised MMD and model criticism

MMD tends to lose test power with increasing dimensionality. [18]

Pick the kernel such that the test power is maximised, [24]

p H1

MMD2−MMD2

√Vm

>cα/m −MMD2

√Vm

m→∞−−−−→ 1− Φ

(cα

m√Vm− MMD2

√Vm

)

where cα is an estimator of the theoretical rejection threshold cα.

Optimised MMD and model criticism

MMD tends to lose test power with increasing dimensionality. [18]

Pick the kernel such that the test power is maximised, [24]

p H1

MMD2−MMD2

√Vm

>cα/m −MMD2

√Vm

m→∞−−−−→ 1− Φ

(cα

m√Vm− MMD2

√Vm

)

where cα is an estimator of the theoretical rejection threshold cα.

Optimised MMD and model criticism

Witness function evaluated on a test set, ARD weights highlighted [24].

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Integral Probability metrics

Have two probability measure p , and q . Integral ProbabilityMetric (IPM) defines a discrepancy measure,

dH(p , q ) := supf ∈H

∣∣∣∣∣ Ep (x)

(f (x)

)− E

q (y)

(f (y)

)∣∣∣∣∣where the space of functions H must be rich enough such thatdH(p , q ) = 0 iff p = q .

f ∗ := argmaxf ∈H∣∣Ep (f (x))− Eq (f (y))

∣∣ is the witness function.

MMD is an IPM where H is the unit ball in characteristic RKHS.

Integral Probability metrics

Have two probability measure p , and q . Integral ProbabilityMetric (IPM) defines a discrepancy measure,

dH(p , q ) := supf ∈H

∣∣∣∣∣ Ep (x)

(f (x)

)− E

q (y)

(f (y)

)∣∣∣∣∣where the space of functions H must be rich enough such thatdH(p , q ) = 0 iff p = q .

f ∗ := argmaxf ∈H∣∣Ep (f (x))− Eq (f (y))

∣∣ is the witness function.

MMD is an IPM where H is the unit ball in characteristic RKHS.

Integral Probability metrics

Have two probability measure p , and q . Integral ProbabilityMetric (IPM) defines a discrepancy measure,

dH(p , q ) := supf ∈H

∣∣∣∣∣ Ep (x)

(f (x)

)− E

q (y)

(f (y)

)∣∣∣∣∣where the space of functions H must be rich enough such thatdH(p , q ) = 0 iff p = q .

f ∗ := argmaxf ∈H∣∣Ep (f (x))− Eq (f (y))

∣∣ is the witness function.

MMD is an IPM where H is the unit ball in characteristic RKHS.

Stein Discrepancy

Construct H such that Eq (f (y)) = 0,∀f ∈ H.

dH(p , q ) = Sq (p , T ,H) := supf ∈H

∣∣∣∣∣ Ep (x)

[(T f )(x)

]∣∣∣∣∣where T is a real–valued operator and Eq [(T f )(y)] = 0, ∀f ∈ H.

A standard choice of T when x = x ∈ R is the Stein’s operator,

(T f )(x) = Aq f (x) := sq (x)f (x) +∇x f (x)

Stein Discrepancy

Construct H such that Eq (f (y)) = 0,∀f ∈ H.

dH(p , q ) = Sq (p , T ,H) := supf ∈H

∣∣∣∣∣ Ep (x)

[(T f )(x)

]∣∣∣∣∣where T is a real–valued operator and Eq [(T f )(y)] = 0, ∀f ∈ H.

A standard choice of T when x = x ∈ R is the Stein’s operator,

(T f )(x) = Aq f (x) := sq (x)f (x) +∇x f (x)

Kernelised Stein Discrepancy (KSD)

Assume that p (x), and q (y) are differentiable continuous densityfunctions; q might be unnormalised [3, 15].

Use the Stein’s operator in the unit ball of the product RKHS FD,

Sq (p ,Aq ,FD) := supf ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Ep (x)

[Tr(Aq f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Ep (x)

[Tr(f (x)sq (x)T +∇2f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣∣D∑

d=1

Ep (x)

[fd(x)sq ,d(x) +

∂fd(x)

∂xd

]∣∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣〈f ,βq 〉FD

∣∣∣ =∥∥∥βq

∥∥∥FD

(s.t. βq ∈ FD)

〈f , g〉FD :=∑D

d=1〈fd , gd〉F , βq := Ep [k (x , ·)uq (x) +∇xk (x , ·)].

Kernelised Stein Discrepancy (KSD)

Assume that p (x), and q (y) are differentiable continuous densityfunctions; q might be unnormalised [3, 15].

Use the Stein’s operator in the unit ball of the product RKHS FD,

Sq (p ,Aq ,FD) := supf ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Ep (x)

[Tr(Aq f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Ep (x)

[Tr(f (x)sq (x)T +∇2f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣∣D∑

d=1

Ep (x)

[fd(x)sq ,d(x) +

∂fd(x)

∂xd

]∣∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣〈f ,βq 〉FD

∣∣∣ =∥∥∥βq

∥∥∥FD

(s.t. βq ∈ FD)

〈f , g〉FD :=∑D

d=1〈fd , gd〉F , βq := Ep [k (x , ·)uq (x) +∇xk (x , ·)].

KSD vs. alternative GoF tests

Variational Inference using KSDNatural idea: minimise KSD between the true andan approximate posterior distribution → a particular case ofOperator Variational Inference. [19]

Quick detour: For the true posterior p (x) = p (x)/Zp andan approximation q (x) = q (x)/Zq ,

S(p , q ) = supf ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [Tr(Ap f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [Tr(Ap f (x)−Aq f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [f (x)T(sp (x)− sq (x))]∣∣∣∣∣

= Eq (x)

Eq (x)

k (x , x)(sp (x)− sq (x))T(sp (x)− sq (x))

→ q (x) =∝ q (x , ε), e.g. q (x , ε) = N (x | Tθ(ε), σ2I ) N (ε | 0, I )

Variational Inference using KSDNatural idea: minimise KSD between the true andan approximate posterior distribution → a particular case ofOperator Variational Inference. [19]

Quick detour: For the true posterior p (x) = p (x)/Zp andan approximation q (x) = q (x)/Zq ,

S(p , q ) = supf ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [Tr(Ap f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [Tr(Ap f (x)−Aq f (x)

)]∣∣∣∣∣= sup

f ∈FD,‖f ‖FD≤1

∣∣∣∣∣ Eq (x) [f (x)T(sp (x)− sq (x))]∣∣∣∣∣

= Eq (x)

Eq (x)

k (x , x)(sp (x)− sq (x))T(sp (x)− sq (x))

→ q (x) =∝ q (x , ε), e.g. q (x , ε) = N (x | Tθ(ε), σ2I ) N (ε | 0, I )

Learning to Sample using KSD

Two papers [15, 25] on optimising a set of particles, resp. a set ofsamples from a model (amortisation), repeatedly using KL-optimalperturbations x t = x t−1 + εt f (x).

Read Yingzhen’s blog post [13] and a recent paper [14]!

Random walk through the latent space of a GAN trained with KSD adversary. [25]

Outline

RKHS

Kernel Mean Embeddings

Characteristic kernels

Two Sample Testing

MMD

Kernelised Stein Discrepancy

Kernel Bayes’ Rule

Tensor product

If H and K are Hilbert spaces then so is H⊗K.

If {ai}∞i=1 spans H and {bj}∞j=1 spans K then {ai ⊗ bj}∞i ,j=0 spansH⊗K.

For any f , f ′ ∈ H and g , g ′ ∈ K we have

〈f ⊗ g , f ′ ⊗ g ′〉 = 〈f , f ′〉〈g , g ′〉

H ⊗ K is isomporphic to a space of bounded linear operatorsK → H

(f ⊗ g)g ′ = f 〈g , g ′〉

Covariance operatorsCovariance operators CXY : K → H are defined as

CXY = E[kX (X , ·)⊗ kY (Y , ·)]

CXX = E[kX (X , ·)⊗ kX (X , ·)]

For any f ∈ H and g ∈ K

〈f ,CXY g〉 = 〈CYX f , g〉 = E[f (X )g(Y )]

Empirical estimators

(Xi ,Yi ) ∼i .i .d . P(X ,Y )

CXY =1

N

N∑i=1

kX (Xi , ·)⊗ kY (Yi , ·)

CXX =1

N

N∑i=1

kX (Xi , ·)⊗ kX (Xi , ·)

Conditional embeddings

LetµX |Y=y = E[kX (X , ·)|Y = y ]

We seek an operator UX |Y such that for all y

µX |Y=y = UX |Y kY (y , ·)

which we call the conditional mean embedding.

Conditional embeddings

Lemma ([21])

For any f ∈ H let h(y) = E[f (X )|Y = y ] and assume that h ∈ K.Then

CYY h = CYX f

Proof.

Take any g ∈ K.

〈g ,CYY h〉 = EY∼P(Y )

[g(Y )h(Y )] =

EY∼P(Y )

[g(Y ) EX∼P(X |Y )

[f (X )|Y ]] = EY∼P(Y )

[ EX∼P(X |Y )

[g(Y )f (X )|Y ]]

EX ,Y∼P(X ,Y )

[g(Y )f (X )] = 〈g ,CYX f 〉

Conditional embeddings

Whenever CYY is invertible we have

UX |Y = CXYC−1YY

in practice we use a regularized version

U εX |Y = CXY (CYY + εI )−1

which can be estimated by

U εX |Y = CXY (CYY + εI )−1

Kernel Bayes’ rule

We could compute posterior embedding by

µX |Y=y = UX |Y kY (y , ·)

but we may want a different prior.

Setting:

I let P(X ,Y ) be the joint distribution of the model

I let Q(X ,Y ) be a different distribution such thatP(Y |X = x) = Q(Y |X = x) for all x

I we have a sample (Xi ,Yi ) ∼ Q and Xj ∼ P(X )

I we want to estimate the posterior embeddingEX∼P(X |Y=y)[kX (X , ·)] for some particular y

Kernel sum rule

Sum ruleP(X ) =

∑Y

P(X ,Y )

Kernel sum ruleµX = UX |YµY

CorollaryCXX = UXX |YµY

Kernel product rule

Product ruleP(X ,Y ) = P(X |Y )P(Y )

Kernel product ruleCXY = UXYCYY

Kernel Bayes’ rule

Observe that

CXY = UY |XCXX

CYY = UYY |XµX

and so

UX |Y = CXYC−1YY = (UY |XCXX )(UYY |XµX )−1

lets us express UX |Y in terms of UY |X .

Reminder

Setting:

I let P(X ,Y ) be the joint distribution of the model

I let Q(X ,Y ) be a different distribution such thatP(Y |X = x) = Q(Y |X = x) for all x

I we have a sample (Xi ,Yi ) ∼ Q and Xj ∼ P(X )

I we want to estimate the posterior embeddingEX∼P(X |Y=y)[kX (X , ·)] for some particular y

Kernel Bayes’ rule

We have

UPY |X = UQ

Y |X

UPX |Y 6= U

QX |Y

but

UPX |Y = (UP

Y |XCPXX )(UP

YY |XµPX )−1

= (UQY |XC

PXX )(UQ

YY |XµPX )−1

so

µPX |Y=y = (UQY |XC

PXX )(UQ

YY |XµPX )−1kY (y , ·)

Kernel Bayes’ rule

In practice we use the following estimator

µX |Y=y = (UQY |X C

PXX )((UQ

YY |X µPX )2 + εI )−1(UQ

YY |X µPX )kY (y , ·)

which can be written as

µX |Y=y = A(B2 + δI )−1BkY (y , ·)A = UQ

Y |X CPXX = CQ

YX (CQXX + εI )−1CP

XX

B = UQYY |X µ

PX = CQ

YYX (CQXX + εI )−1µPX

For ε = N−13 and δ = N−

427 it can be shown [7] that∥∥∥µX |Y=y − µX |Y=y

∥∥∥ = Op(N−427 )

Kernel Bayes’ rule

When is Kernel Bayes’ Rule useful?

I when densities aren’t tractable (ABC)

I when you don’t know how to write a model but you know howto pick a kernel

I perhaps it can perform better than alternatives even if theabove aren’t satisfied

Other topics

Other topics combining KMEs with Bayesian inference

I adaptive Metropolis-Hastings using KME [20]

I Hamiltonian Monte Carlo without gradients [23]

I Bayesian estimation of KMEs [4]

References I

M. A. Arcones and E. Gine.

On the Bootstrap of U and V Statistics.

Ann. Statist., 20(2):655–674, 06 1992.

K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton.

Fast Two-Sample Testing with Analytic Representations ofProbability Measures.

ArXiv e-prints, 2015.

K. Chwialkowski, H. Strathmann, and A. Gretton.

A Kernel Test of Goodness of Fit.

In Proceedings of the International Conference on Machine Learning(ICML), 2016.

S. Flaxman, D. Sejdinovic, J. P. Cunningham, and S. Filippi.

Bayesian learning of kernel embeddings.

In UAI, 2016.

References IIK. Fukumizu, F. R. Bach, and M. I. Jordan.

Dimensionality Reduction for Supervised Learning with ReproducingKernel Hilbert Spaces.

JMLR, 5:73–99, 2004.

K. Fukumizu, A. Gretton, X. Sun, and B. Scholkopf.

Kernel Measures of Conditional Dependence.

In Proceedings of the 20th International Conference on NeuralInformation Processing Systems, NIPS’07, pages 489–496, 2007.

K. Fukumizu, L. Song, and A. Gretton.

Kernel Bayes’ rule: Bayesian inference with positive definite kernels.

Journal of Machine Learning Research, 14:3753–3783, 2013.

References IIIA. Gretton, K. M. Borgwardt, M. Rasch, B. Scholkopf, and A. J.Smola.

A Kernel Method for the Two-sample-problem.

In Proceedings of the 19th International Conference on NeuralInformation Processing Systems, NIPS’06, pages 513–520, 2006.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scholkopf, andA. Smola.

A Kernel Two-sample Test.

JMLR, 13:723–773, Mar. 2012.

A. Gretton, K. Fukumizu, Z. Harchaoui, and B. K. Sriperumbudur.

A Fast, Consistent Kernel Two-sample Test.

In Proceedings of the 22Nd International Conference on NeuralInformation Processing Systems, NIPS’09, pages 673–681, 2009.

References IVW. Hoeffding.

Probability Inequalities for Sums of Bounded Random Variables.

Journal of the American Statistical Association, 58(301):13–30,1963.

N. L. Johnson, S. Kotz, and N. Balakrishnan.

Continuous Univariate Distributions.

Wiley, 1994.

Y. Li.

Stein Variational Gradient Descent: A General Purpose BayesianInference Algorithm, August 2016.

Y. Li, R. E. Turner, and Q. Liu.

Approximate Inference with Amortised MCMC.

arXiv preprint arXiv:1702.08343, 2017.

References VQ. Liu, J. D. Lee, and M. I. Jordan.

A Kernelized Stein Discrepancy for Goodness-of-fit Tests and ModelEvaluation.

arXiv preprint arXiv:1602.03253, 2016.

K. Muandet, B. Sriperumbudur, K. Fukumizu, A. Gretton, andB. Scholkopf.

Kernel mean shrinkage estimators.

Journal of Machine Learning Research, 17(48):1–41, 2016.

K. Muandet, B. Sriperumbudur, and B. Scholkopf.

Kernel mean estimation via spectral filtering.

In Advances in Neural Information Processing Systems, pages 1–9,2014.

References VIA. Ramdas, S. J. Reddi, B. Poczos, A. Singh, and L. Wasserman.

On the high-dimensional power of linear-time kernel two-sampletesting under mean-difference alternatives.

arXiv preprint arXiv:1411.6314, 2014.

R. Ranganath, D. Tran, J. Altosaar, and D. Blei.

Operator Variational Inference.

In Advances in Neural Information Processing Systems, pages496–504, 2016.

D. Sejdinovic, H. Strathmann, M. L. Garcia, C. Andrieu, andA. Gretton.

Kernel adaptive Metropolis-Hastings.

In ICML, 2014.

References VIIL. Song, J. Huang, A. Smola, and K. Fukumizu.

Hilbert space embeddings of conditional distributions withapplications to dynamical systems.

In ICML, 2009.

B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Scholkopf, andG. R. Lanckriet.

Hilbert Space Embeddings and Metrics on Probability Measures.

JMLR, 11:1517–1561, 2010.

H. Strathmann, D. Sejdinovic, S. Livingstone, Z. Szabo, andA. Gretton.

Gradient-free Hamiltonian Monte Carlo with efficient kernelexponential families.

In NIPS, 2015.

References VIIID. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas,A. Smola, and A. Gretton.

Generative models and model criticism via optimized maximummean discrepancy.

arXiv preprint arXiv:1611.04488, 2016.

D. Wang and Q. Liu.

Learning to Draw Samples: With Application to Amortized MLE forGenerative Adversarial Learning.

arXiv preprint arXiv:1611.01722, 2016.

Q. Zhang, S. Filippi, A. Gretton, and D. Sejdinovic.

Large-Scale Kernel Methods for Independence Testing.

ArXiv e-prints, 2016.