computer vision group prof. daniel cremers · l = f =1, n =0.1 l =0.3, f =1.08, n =0.0005 n =0.89 f...

Computer Vision Group Prof. Daniel Cremers

7. Gaussian Processes (contd.)

PD Dr. Rudolph TriebelComputer Vision Group

Machine Learning for Computer Vision

Prediction with a Gaussian Process

In the case of only one test point we have

Now we compute the conditional distribution

where

This defines the predictive distribution.!2

x⇤

K(X,x⇤) =

0

B@k(x1,x⇤)

...k(xN ,x⇤)

1

CA = k⇤

µ⇤ = kT⇤ K

�1t

⌃⇤ = k(x⇤,x⇤)� kT⇤ K

�1k⇤

p(y⇤ | x⇤, X,y) = N (y⇤ | µ⇤,⌃⇤)



Implementation

•Cholesky decomposition is numerically stable •Can be used to compute inverse efficiently

!3

Algorithm 1: GP regression

Data: training data (X,y), test data x⇤Input: Hyper parameters �2

f , l, �2n

Kij k(xi,xj)

L cholesky(K + �2yI)

↵ LT \(L\y)E[f⇤] kT

⇤ ↵v L\k⇤var[f⇤] k(x⇤,x⇤)� vTvlog p(y | X) � 1

2yT↵�

Pi logLii � N

2 log(2⇡)

Precomputed during Training

Test Phase

n

l = �f = 1, �n = 0.1

l = 0.3,

�f = 1.08,

�n = 0.0005

�n = 0.89

�f = 1.16

l = 3



Varying the Hyperparameters

•20 data samples •GP prediction with

different kernelhyper parameters

!4

−8 −6 −4 −2 0 2 4 6 8−3

−2

−1

0

1

2

3

−8 −6 −4 −2 0 2 4 6 8−3

−2

−1

0

1

2

3

−8 −6 −4 −2 0 2 4 6 8−3

−2

−1

0

1

2

3




The squared exponential covariance function can be generalized to

where M can be: • : this is equal to the above case

• : every feature dimension has its own length scale parameter

• : here Λ has less than D columns

!5

k(xp,xq) = �2f exp(�

1

2(xp � xq)

TM(xp � xq)) + �2n�pq

M = l�2I

M = diag(l1, . . . , lD)�2

M = ⇤⇤T + diag(l1, . . . , lD)�2

M = I M = diag(1, 3)�2

M =

✓1 �1�1 1

◆+ diag(6, 6)�2




!6

−20

2

−20

2−2

−1

0

1

2

input x1input x2

outp

ut y

−20

2

−20

2−2

−1

0

1

2

input x1input x2

outp

ut y

−20

2

−20

2−2

−1

0

1

2

input x1input x2

outp

ut y



Estimating the Hyperparameters

To find optimal hyper parameters we need the marginal likelihood:

This expression implicitly depends on the hyper parameters, but y and X are given from the training data. It can be computed in closed form, as all terms are Gaussians. We take the logarithm, compute the derivative and set it to 0. This is the training step.

!7

p(y | X) =

Zp(y | f , X)p(f | X)df





!8

p(y | X) =1p

(2⇡)n|K|exp

✓�1

2yTK�1y

◆

<latexit sha1_base64="ODEVN8PieCc8S1XRFQqByISj9lg=">AAAC53icZVJLaxsxENZuX6n7iNMeexE1BhuasGsK7aUQ2kshBFKIE0NkG608a4tI2q00W2w2+xt6K732L/XWf9Jjtc6SOs6AxMc8vtHMpyRX0mEU/QnCe/cfPHy087j15Omz57vtvRdnLiusgKHIVGZHCXegpIEhSlQwyi1wnSg4Ty4/1fHzb2CdzMwprnIYaz43MpWCo3dN2yrvMc1xkaTlqmJazuioTz9QllouyrgqmftqsewNWC77E0Ovjq6qisEyZwpS7O3f5A2q/zST06NJuR9veJiV8wX2p+1OdBCtjd4FcQM6pLGT6V4g2CwThQaDQnHnLuIox3HJLUqhoGqxwkHOxSWfw4WHhmtw43K9lop2vWdG08z6Y5CuvZsVJdfOrXTiM+uXuu1Y7byJdTeDuKwZ3VZ/TN+PS2nyAsGI6/ZpoShmtN48nUkLAtXKAy6s9BNQseB+f+j1ucVUizl7Ywvlb5HpenpnCp2AhZkfXs0zX77QA6go7VK/Fr8gR5sX04aFSuOQK8/RYg5QZIXxjcpjvjzmaOXS/w3ndYuqlpcl3hbhLjgbHMQef3nbOfzYCLRDXpHXpEdi8o4cks/khAyJIL/J3yAIwlCG38Mf4c/r1DBoal6SWxb++get/+0m</latexit><latexit sha1_base64="ODEVN8PieCc8S1XRFQqByISj9lg=">AAAC53icZVJLaxsxENZuX6n7iNMeexE1BhuasGsK7aUQ2kshBFKIE0NkG608a4tI2q00W2w2+xt6K732L/XWf9Jjtc6SOs6AxMc8vtHMpyRX0mEU/QnCe/cfPHy087j15Omz57vtvRdnLiusgKHIVGZHCXegpIEhSlQwyi1wnSg4Ty4/1fHzb2CdzMwprnIYaz43MpWCo3dN2yrvMc1xkaTlqmJazuioTz9QllouyrgqmftqsewNWC77E0Ovjq6qisEyZwpS7O3f5A2q/zST06NJuR9veJiV8wX2p+1OdBCtjd4FcQM6pLGT6V4g2CwThQaDQnHnLuIox3HJLUqhoGqxwkHOxSWfw4WHhmtw43K9lop2vWdG08z6Y5CuvZsVJdfOrXTiM+uXuu1Y7byJdTeDuKwZ3VZ/TN+PS2nyAsGI6/ZpoShmtN48nUkLAtXKAy6s9BNQseB+f+j1ucVUizl7Ywvlb5HpenpnCp2AhZkfXs0zX77QA6go7VK/Fr8gR5sX04aFSuOQK8/RYg5QZIXxjcpjvjzmaOXS/w3ndYuqlpcl3hbhLjgbHMQef3nbOfzYCLRDXpHXpEdi8o4cks/khAyJIL/J3yAIwlCG38Mf4c/r1DBoal6SWxb++get/+0m</latexit><latexit sha1_base64="ODEVN8PieCc8S1XRFQqByISj9lg=">AAAC53icZVJLaxsxENZuX6n7iNMeexE1BhuasGsK7aUQ2kshBFKIE0NkG608a4tI2q00W2w2+xt6K732L/XWf9Jjtc6SOs6AxMc8vtHMpyRX0mEU/QnCe/cfPHy087j15Omz57vtvRdnLiusgKHIVGZHCXegpIEhSlQwyi1wnSg4Ty4/1fHzb2CdzMwprnIYaz43MpWCo3dN2yrvMc1xkaTlqmJazuioTz9QllouyrgqmftqsewNWC77E0Ovjq6qisEyZwpS7O3f5A2q/zST06NJuR9veJiV8wX2p+1OdBCtjd4FcQM6pLGT6V4g2CwThQaDQnHnLuIox3HJLUqhoGqxwkHOxSWfw4WHhmtw43K9lop2vWdG08z6Y5CuvZsVJdfOrXTiM+uXuu1Y7byJdTeDuKwZ3VZ/TN+PS2nyAsGI6/ZpoShmtN48nUkLAtXKAy6s9BNQseB+f+j1ucVUizl7Ywvlb5HpenpnCp2AhZkfXs0zX77QA6go7VK/Fr8gR5sX04aFSuOQK8/RYg5QZIXxjcpjvjzmaOXS/w3ndYuqlpcl3hbhLjgbHMQef3nbOfzYCLRDXpHXpEdi8o4cks/khAyJIL/J3yAIwlCG38Mf4c/r1DBoal6SWxb++get/+0m</latexit><latexit sha1_base64="ODEVN8PieCc8S1XRFQqByISj9lg=">AAAC53icZVJLaxsxENZuX6n7iNMeexE1BhuasGsK7aUQ2kshBFKIE0NkG608a4tI2q00W2w2+xt6K732L/XWf9Jjtc6SOs6AxMc8vtHMpyRX0mEU/QnCe/cfPHy087j15Omz57vtvRdnLiusgKHIVGZHCXegpIEhSlQwyi1wnSg4Ty4/1fHzb2CdzMwprnIYaz43MpWCo3dN2yrvMc1xkaTlqmJazuioTz9QllouyrgqmftqsewNWC77E0Ovjq6qisEyZwpS7O3f5A2q/zST06NJuR9veJiV8wX2p+1OdBCtjd4FcQM6pLGT6V4g2CwThQaDQnHnLuIox3HJLUqhoGqxwkHOxSWfw4WHhmtw43K9lop2vWdG08z6Y5CuvZsVJdfOrXTiM+uXuu1Y7byJdTeDuKwZ3VZ/TN+PS2nyAsGI6/ZpoShmtN48nUkLAtXKAy6s9BNQseB+f+j1ucVUizl7Ywvlb5HpenpnCp2AhZkfXs0zX77QA6go7VK/Fr8gR5sX04aFSuOQK8/RYg5QZIXxjcpjvjzmaOXS/w3ndYuqlpcl3hbhLjgbHMQef3nbOfzYCLRDXpHXpEdi8o4cks/khAyJIL/J3yAIwlCG38Mf4c/r1DBoal6SWxb++get/+0m</latexit>





!9

p(y | X) =1p

(2⇡)n|K|exp

✓�1

2yTK�1y

◆


log p(y | X) = �1

2log((2⇡)n|K|)� 1

2yTK�1y

<latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit>





!10

p(y | X) =1p

(2⇡)n|K|exp

✓�1

2yTK�1y

◆


log p(y | X) = �1

2log((2⇡)n|K|)� 1

2yTK�1y

<latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit><latexit sha1_base64="t6BkxRxHdez7JfxyCuCeRwvXcEE=">AAAC3XicZVJbaxNBFJ6stxovTfXRl4MhkIAtu0HQF6Hoi1AKFZo20E3C7OxsMnQuy8ysJGz30Tfx1b/kX/BP+KqPnk0XTdMDM3x85zbnfJPkUjgfhj9bwZ279+4/2HnYfvT4ydPdzt6zM2cKy/iIGWnsOKGOS6H5yAsv+Ti3nKpE8vPk8kPtP//MrRNGn/pVzieKzrXIBKMeqVkniaWZQ96PFfWLJCtXVaxECuMBvIP9OLOUlVFVDiuo4/r9YZyLwVTD1dHVYNP9P316ejQt96MNZtbphgfh2uA2iBrQJY2dzPZaLE4NKxTXnknq3EUU5n5SUusFk7xqx4XjOWWXdM4vEGqquJuU62VU0EMmhcxYPNrDmt3MKKlybqUSjKyf6LZ9NfnP19t0+mVd0W3199nbSSl0Xniu2XX7rJDgDdT7hlRYzrxcIaDMCpwA2ILi4jyqcqNSLWH6yhYSb2ZUPb3ThUq45SkOL+cG0xdqyCuAHuBacEEOmhdDUwWEdp5KrNGOHffMFBoblcd0eUy9FUv8EQ4FC6s2yhJti3AbnA0PIsSfXncP3zcC7ZAX5CXpk4i8IYfkIzkhI8LID/KL/CZ/glnwJfgafLsODVpNznNyw4LvfwFTCelT</latexit>

@ log p(y | X)

@✓i=

1

2yTK�1 @K

@✓iy � 1

2tr

✓K�1 @K

@✓i

◆

<latexit sha1_base64="KbLzMChXMKXna/o0wuiG7N97qgg=">AAADWXiclVLbattAEF3ZvbjqzWke+zLUGBxogmQK7UshtH0ohEAKcWKIHLNaraxtdrVid1RshD60FPotXdmicZy+dEDiMDPnzM4lLqSwGAQ/vU73wcNHj3tP/KfPnr942d97dWF1aRifMC21mcbUcilyPkGBkk8Lw6mKJb+Mbz438csf3Fih83NcFXym6CIXqWAUnWve/x2lhrIqKqhBQSVEUi+gGEWKYhan1aqOlEhgelBvpWDGkc5FDR9hww7ralzfUq7PT66rw7CGHe2Tf4rc8uAQfNiWdLFYLys0dSR5iqP/kI2MWGR4MO8PgqNgbXAfhC0YkNbO5nseixLNSsVzZJJaexUGBc6qRp9JXvtRaXlB2Q1d8CsHc6q4nVXrRdQwdJ4EUm3clyOsvduMiiprVyp2mU3XdjfWOP/GhttBXDaKdqc+ph9mlciLEnnONuXTUgJqaHYNiTCcoVw5QJkRrgNgGXXDQ3cRd5Sa80nemlK6P9Oq6d7mpYq54YlrXi60o2dqzGuAIbixuAFZaF8MrQqI3CKVTsOPLEemy9wVqk7p8pSiEUt3jdatNah9t5Zwdwn3wcX4KHT427vB8ad2QT3ymrwhIxKS9+SYfCVnZEKY98X77lkPO7+6XrfX9TepHa/l7JM71t3/A/xuFlw=</latexit><latexit sha1_base64="KbLzMChXMKXna/o0wuiG7N97qgg=">AAADWXiclVLbattAEF3ZvbjqzWke+zLUGBxogmQK7UshtH0ohEAKcWKIHLNaraxtdrVid1RshD60FPotXdmicZy+dEDiMDPnzM4lLqSwGAQ/vU73wcNHj3tP/KfPnr942d97dWF1aRifMC21mcbUcilyPkGBkk8Lw6mKJb+Mbz438csf3Fih83NcFXym6CIXqWAUnWve/x2lhrIqKqhBQSVEUi+gGEWKYhan1aqOlEhgelBvpWDGkc5FDR9hww7ralzfUq7PT66rw7CGHe2Tf4rc8uAQfNiWdLFYLys0dSR5iqP/kI2MWGR4MO8PgqNgbXAfhC0YkNbO5nseixLNSsVzZJJaexUGBc6qRp9JXvtRaXlB2Q1d8CsHc6q4nVXrRdQwdJ4EUm3clyOsvduMiiprVyp2mU3XdjfWOP/GhttBXDaKdqc+ph9mlciLEnnONuXTUgJqaHYNiTCcoVw5QJkRrgNgGXXDQ3cRd5Sa80nemlK6P9Oq6d7mpYq54YlrXi60o2dqzGuAIbixuAFZaF8MrQqI3CKVTsOPLEemy9wVqk7p8pSiEUt3jdatNah9t5Zwdwn3wcX4KHT427vB8ad2QT3ymrwhIxKS9+SYfCVnZEKY98X77lkPO7+6XrfX9TepHa/l7JM71t3/A/xuFlw=</latexit><latexit sha1_base64="KbLzMChXMKXna/o0wuiG7N97qgg=">AAADWXiclVLbattAEF3ZvbjqzWke+zLUGBxogmQK7UshtH0ohEAKcWKIHLNaraxtdrVid1RshD60FPotXdmicZy+dEDiMDPnzM4lLqSwGAQ/vU73wcNHj3tP/KfPnr942d97dWF1aRifMC21mcbUcilyPkGBkk8Lw6mKJb+Mbz438csf3Fih83NcFXym6CIXqWAUnWve/x2lhrIqKqhBQSVEUi+gGEWKYhan1aqOlEhgelBvpWDGkc5FDR9hww7ralzfUq7PT66rw7CGHe2Tf4rc8uAQfNiWdLFYLys0dSR5iqP/kI2MWGR4MO8PgqNgbXAfhC0YkNbO5nseixLNSsVzZJJaexUGBc6qRp9JXvtRaXlB2Q1d8CsHc6q4nVXrRdQwdJ4EUm3clyOsvduMiiprVyp2mU3XdjfWOP/GhttBXDaKdqc+ph9mlciLEnnONuXTUgJqaHYNiTCcoVw5QJkRrgNgGXXDQ3cRd5Sa80nemlK6P9Oq6d7mpYq54YlrXi60o2dqzGuAIbixuAFZaF8MrQqI3CKVTsOPLEemy9wVqk7p8pSiEUt3jdatNah9t5Zwdwn3wcX4KHT427vB8ad2QT3ymrwhIxKS9+SYfCVnZEKY98X77lkPO7+6XrfX9TepHa/l7JM71t3/A/xuFlw=</latexit><latexit sha1_base64="KbLzMChXMKXna/o0wuiG7N97qgg=">AAADWXiclVLbattAEF3ZvbjqzWke+zLUGBxogmQK7UshtH0ohEAKcWKIHLNaraxtdrVid1RshD60FPotXdmicZy+dEDiMDPnzM4lLqSwGAQ/vU73wcNHj3tP/KfPnr942d97dWF1aRifMC21mcbUcilyPkGBkk8Lw6mKJb+Mbz438csf3Fih83NcFXym6CIXqWAUnWve/x2lhrIqKqhBQSVEUi+gGEWKYhan1aqOlEhgelBvpWDGkc5FDR9hww7ralzfUq7PT66rw7CGHe2Tf4rc8uAQfNiWdLFYLys0dSR5iqP/kI2MWGR4MO8PgqNgbXAfhC0YkNbO5nseixLNSsVzZJJaexUGBc6qRp9JXvtRaXlB2Q1d8CsHc6q4nVXrRdQwdJ4EUm3clyOsvduMiiprVyp2mU3XdjfWOP/GhttBXDaKdqc+ph9mlciLEnnONuXTUgJqaHYNiTCcoVw5QJkRrgNgGXXDQ3cRd5Sa80nemlK6P9Oq6d7mpYq54YlrXi60o2dqzGuAIbixuAFZaF8MrQqI3CKVTsOPLEemy9wVqk7p8pSiEUt3jdatNah9t5Zwdwn3wcX4KHT427vB8ad2QT3ymrwhIxKS9+SYfCVnZEKY98X77lkPO7+6XrfX9TepHa/l7JM71t3/A/xuFlw=</latexit>




The log marginal likelihood is not necessarily concave, i.e. it can have local maxima.

The local maxima can correspond to sub-optimal solutions.

!11

100 101

10−1

100

characteristic lengthscale

nois

e st

anda

rd d

evia

tion

−5 0 5−2

−1

0

1

2

input, x

outp

ut, y

−5 0 5−2

−1

0

1

2

input, x

outp

ut, y



Automatic Relevance Determination

•We have seen how the covariance function can be generalized using a matrix M

•If M is diagonal this results in the kernel function

•We can interpret the as weights for each feature dimension

•Thus, if the length scale of an input dimension is large, the input is less relevant

•During training this is done automatically

!12

k(x,x0) = �f exp

1

2

DX

i=1

⌘i(xi � x0i)

2

!

⌘i

li = 1/⌘i



Automatic Relevance Determination

During the optimization process to learn the hyper-parameters, the reciprocal length scale for one parameter decreases, i.e.:

This hyper parameter is not very relevant!!13

3-dimensional data, parameters as they evolve during training

⌘1 ⌘2 ⌘3

⌘1

⌘2

⌘3

Computer Vision Group Prof. Daniel Cremers

Gaussian Processes -Classification



Remember the Visualisation

!15

Linear Regression

Logistic Regression

Bayesian Logistic Regression

Bayesian Linear Regression

probabilistic reasoning

from regression toclassification



Kernelization as a New Dimension

!16

Linear Regression

Logistic Regression



Kernelization

Kernel Regression




!17

Linear Regression

Logistic Regression



Kernelization

Kernel Regression

GP Regression




!18

Linear Regression

Logistic Regression



Kernel Regression

GP Regression GP Classification

Kernel Classification



Gaussian Processes For Classification

In regression we have , in binary classification we have To use a GP for classification, we can apply a sigmoid function to the posterior obtained from the GP and compute the class probability as:

If the sigmoid function is symmetric:then we have . A typical type of sigmoid function is the logistic sigmoid:

!19

y 2 Ry 2 {�1; 1}

p(y = +1 | x) = �(f(x))

�(�z) = 1� �(z)

p(y | x) = �(yf(x))

�(z) =1

1 + exp(�z)



Application of the Sigmoid Function

Function sampled from a Gaussian Process

!20

Sigmoid function applied to the GP function

Another symmetric sigmoid function is the cumulative Gaussian:

�(z) =

Z z

�1N (x | 0, 1)dx



Visualization of Sigmoid Functions

The cumulative Gaussian is slightly steeper than the logistic sigmoid

!21



The Latent Variables

In regression, we directly estimated f asand values of f where observed in the training data. Now only labels +1 or -1 are observed and

f is treated as a set of latent variables.

A major advantage of the Gaussian process classifier over other methods is that it marginalizes over all latent functions rather than maximizing some model parameters.

!22

f(x) ⇠ GP(m(x), k(x,x0))



Class Prediction with a GP

The aim is to compute the predictive distribution

!23

p(y⇤ = +1 | X,y,x⇤) =

Zp(y⇤ | f⇤)p(f⇤ | X,y,x⇤)df⇤

�(f⇤)



A Simple Example

•Red: Two-class training data •Green: mean function of •Light blue: sigmoid of the mean function

!26

p(f | X,y)



But There Is A Problem...

•The likelihood term is not a Gaussian! •This means, we can not compute the

posterior in closed form. •There are several different solutions in the

literature, e.g.: •Laplace approximation •Expectation Propagation •Variational methods

!27

p(f | X,y) =p(y | f)p(f | X)

p(y | X)



Laplace Approximation

Consider a general distribution

where

!28

p(z) =1

Zf(z)

<latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit>

Z =

Zf(z)dz

<latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit>




Consider a general distribution

where

Aim: approximate this with a normal distribution

!29

p(z) =1

Zf(z)

<latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit><latexit sha1_base64="CxQntPZAFSCFZVcJEk+flDxQNb4=">AAACkXicZVHbahsxEJW3l6TuJU4DfemLqDGkUMKuKTR9CJj0pVACKdRJaGyMVjtri+iySLPFzmZ/pq/tD/VvOusuJXEGJA5nNGc0Z9JCq4Bx/KcTPXj46PHW9pPu02fPX+z0dl+eBVd6CWPptPMXqQiglYUxKtRwUXgQJtVwnl59avLnP8AH5ew3XBUwNWJuVa6kQKJmvVfF/vVbfsQnuReySurqe50TM+v144N4Hfw+SFrQZ22cznY7cpI5WRqwKLUI4TKJC5xWwqOSGurupAxQCHkl5nBJ0AoDYVqtB6j5gJiM587TscjX7O2KSpgQViall0bgImzmGvJ/bnA7ictGMWz0x/xwWilblAhW/mufl5qj441HPFMeJOoVASG9ogm4XAgyCMnJO0qN7dk7X2q6pTPN9MGWJgUPGQ2v547KF2YINecDTraQQYG3P+atClc2oNCk0Z0EQOlKS42qE7E8EejVkrYY6moY111aS7K5hPvgbHiQEP76vj86bhe0zV6zN2yfJewDG7HP7JSNmWQ37Cf7xX5He9HHaBS1b6NOW7PH7kT05S9fsMpZ</latexit>

Z =

Zf(z)dz

<latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit><latexit sha1_base64="KJ+LtQckQPa24U0jctbxbT8D8lA=">AAACi3icZVFtaxNBEN6cL63Ralo/+mUxBCpIuYuFFrFQFMEvhQqmLTYh7O3NJUv35didK0mP+yd+1f/kv3EuPaRNB3Z5mJdnZp5JC60CxvHfTvTo8ZOnG5vPus9fbL181dveOQuu9BJG0mnnL1IRQCsLI1So4aLwIEyq4Ty9+tLEz6/BB+XsD1wWMDFiZlWupEByTXu9n/yIj5VFnu/evOPZzbTXj/filfGHIGlBn7V2Ot3uyHHmZGnAotQihMskLnBSCY9Kaqi74zJAIeSVmMElQSsMhEm1Gr3mA/JkPHeeHg2x8t6tqIQJYWlSyjQC52E91jj/xwZ3g7hoGMNaf8wPJ5WyRYlg5W37vNQcHW/U4ZnyIFEvCQjpFW3A5Vx4IZE0vMfUCJ6996WmXzrTbB9saVLwkNHyeuaofG6GUHM+4CQLCRR4OzFvWbiyAYUmju44AEpXWmpUnYjFiUCvFnS/UFfDuO7SWZL1IzwEZ8O9hPD3/f7x5/ZAm+wNe8t2WcIO2DH7xk7ZiEl2zX6x3+xPtBV9iD5Gn25To05b85rds+jrP/9qx44=</latexit>

q(f) = N (f | f̂ , A�1)<latexit sha1_base64="LhwFSevBBtaPgSh2BN28G4QY9cc=">AAACv3icZVHbbtNAEN2YWwm3FB55WRFFaqVS2RESvCC1wAMvRUUibaU6ROP1Oll1L2Z3jBJZ/iG+htfyNYxTg9J2JFtnz9mZ2ZmTlVoFjOPLXnTn7r37D7Ye9h89fvL02WD7+UlwlRdyIpx2/iyDILWycoIKtTwrvQSTaXmaXXxs9dOf0gfl7DdclXJqYG5VoQQgUbPBpx87qQFcZEVdNLv8PV+fBOj6S7OhpEbl/N8xXQAS1+zxw+/166TZnQ2G8X68Dn4bJB0Ysi6OZ9s9keZOVEZaFBpCOE/iEqc1eFRCy6afVkGWIC5gLs8JWjAyTOv1uA0fEZPzwnn6LPI1u5lRgwlhZTK62b443NRa8r822hRx2VYMN/pj8W5aK1tWKK24al9UmqPj7UZ5rrwUqFcEQHhFE3CxAA8Cae/XKrUm5Xu+0vQXzrTTB1uZTHqZ0/B67ih9Ycay4XzEaS20oMC7F/OuClc2IGiq0U+DROEqS43qI1geAXq1JM9DU4/jpk+2JDdNuA1OxvsJ4a9vhgcfOoO22Ev2iu2whL1lB+wzO2YTJtgv9ptdsj/RYTSPbFReXY16Xc4Ldi2i1V82X93h</latexit><latexit sha1_base64="LhwFSevBBtaPgSh2BN28G4QY9cc=">AAACv3icZVHbbtNAEN2YWwm3FB55WRFFaqVS2RESvCC1wAMvRUUibaU6ROP1Oll1L2Z3jBJZ/iG+htfyNYxTg9J2JFtnz9mZ2ZmTlVoFjOPLXnTn7r37D7Ye9h89fvL02WD7+UlwlRdyIpx2/iyDILWycoIKtTwrvQSTaXmaXXxs9dOf0gfl7DdclXJqYG5VoQQgUbPBpx87qQFcZEVdNLv8PV+fBOj6S7OhpEbl/N8xXQAS1+zxw+/166TZnQ2G8X68Dn4bJB0Ysi6OZ9s9keZOVEZaFBpCOE/iEqc1eFRCy6afVkGWIC5gLs8JWjAyTOv1uA0fEZPzwnn6LPI1u5lRgwlhZTK62b443NRa8r822hRx2VYMN/pj8W5aK1tWKK24al9UmqPj7UZ5rrwUqFcEQHhFE3CxAA8Cae/XKrUm5Xu+0vQXzrTTB1uZTHqZ0/B67ih9Ycay4XzEaS20oMC7F/OuClc2IGiq0U+DROEqS43qI1geAXq1JM9DU4/jpk+2JDdNuA1OxvsJ4a9vhgcfOoO22Ev2iu2whL1lB+wzO2YTJtgv9ptdsj/RYTSPbFReXY16Xc4Ldi2i1V82X93h</latexit><latexit sha1_base64="LhwFSevBBtaPgSh2BN28G4QY9cc=">AAACv3icZVHbbtNAEN2YWwm3FB55WRFFaqVS2RESvCC1wAMvRUUibaU6ROP1Oll1L2Z3jBJZ/iG+htfyNYxTg9J2JFtnz9mZ2ZmTlVoFjOPLXnTn7r37D7Ye9h89fvL02WD7+UlwlRdyIpx2/iyDILWycoIKtTwrvQSTaXmaXXxs9dOf0gfl7DdclXJqYG5VoQQgUbPBpx87qQFcZEVdNLv8PV+fBOj6S7OhpEbl/N8xXQAS1+zxw+/166TZnQ2G8X68Dn4bJB0Ysi6OZ9s9keZOVEZaFBpCOE/iEqc1eFRCy6afVkGWIC5gLs8JWjAyTOv1uA0fEZPzwnn6LPI1u5lRgwlhZTK62b443NRa8r822hRx2VYMN/pj8W5aK1tWKK24al9UmqPj7UZ5rrwUqFcEQHhFE3CxAA8Cae/XKrUm5Xu+0vQXzrTTB1uZTHqZ0/B67ih9Ycay4XzEaS20oMC7F/OuClc2IGiq0U+DROEqS43qI1geAXq1JM9DU4/jpk+2JDdNuA1OxvsJ4a9vhgcfOoO22Ev2iu2whL1lB+wzO2YTJtgv9ptdsj/RYTSPbFReXY16Xc4Ldi2i1V82X93h</latexit><latexit sha1_base64="LhwFSevBBtaPgSh2BN28G4QY9cc=">AAACv3icZVHbbtNAEN2YWwm3FB55WRFFaqVS2RESvCC1wAMvRUUibaU6ROP1Oll1L2Z3jBJZ/iG+htfyNYxTg9J2JFtnz9mZ2ZmTlVoFjOPLXnTn7r37D7Ye9h89fvL02WD7+UlwlRdyIpx2/iyDILWycoIKtTwrvQSTaXmaXXxs9dOf0gfl7DdclXJqYG5VoQQgUbPBpx87qQFcZEVdNLv8PV+fBOj6S7OhpEbl/N8xXQAS1+zxw+/166TZnQ2G8X68Dn4bJB0Ysi6OZ9s9keZOVEZaFBpCOE/iEqc1eFRCy6afVkGWIC5gLs8JWjAyTOv1uA0fEZPzwnn6LPI1u5lRgwlhZTK62b443NRa8r822hRx2VYMN/pj8W5aK1tWKK24al9UmqPj7UZ5rrwUqFcEQHhFE3CxAA8Cae/XKrUm5Xu+0vQXzrTTB1uZTHqZ0/B67ih9Ycay4XzEaS20oMC7F/OuClc2IGiq0U+DROEqS43qI1geAXq1JM9DU4/jpk+2JDdNuA1OxvsJ4a9vhgcfOoO22Ev2iu2whL1lB+wzO2YTJtgv9ptdsj/RYTSPbFReXY16Xc4Ldi2i1V82X93h</latexit>

fnew = f � (rr )�1r <latexit sha1_base64="wPiO0NKQB1a58e+E1I1BEGzq9CA=">AAACvHicZVHbbtNAEN2YWwmXpvDIy4ooUpFoZUdI8AIq8MJLpSCRtlITovF6nKyyF2t3XBJZ/h6+hlcQf8M6tWibjuTV2TM7Mz5z0kJJT3H8txPduXvv/oOdh91Hj5883e3tPTvxtnQCx8Iq685S8KikwTFJUnhWOASdKjxNl5+b/OkFOi+t+UbrAqca5kbmUgAFatb7ONFAizSv8npWGfxR8/f8iuIHfH9iIFXQniMvX32vDpL66j7r9ePDeBP8Nkha0GdtjGZ7HTHJrCg1GhIKvD9P4oKmFTiSQmHdnZQeCxBLmON5gAY0+mm10VrzQWAynlsXPkN8w16vqEB7v9ZpeNnI8Nu5hvyfG1xP0qrp6LfmU/5uWklTlIRGXI7PS8XJ8madPJMOBal1ACCcDAq4WIADQWHpNzo1DmWvXanCKaxu1HtT6hQdZkG8mttQvtBDrDkf8LCWsCDP2z/mbRcujSdQoUd34pGELU0YVB3D6hjIyVUw3NfVMK67wZZk24Tb4GR4mAT89U3/6FNr0A57wV6yfZawt+yIfWEjNmaC/WS/2G/2J/oQZdEy0pdPo05b85zdiOjiH0XC3LE=</latexit><latexit sha1_base64="wPiO0NKQB1a58e+E1I1BEGzq9CA=">AAACvHicZVHbbtNAEN2YWwmXpvDIy4ooUpFoZUdI8AIq8MJLpSCRtlITovF6nKyyF2t3XBJZ/h6+hlcQf8M6tWibjuTV2TM7Mz5z0kJJT3H8txPduXvv/oOdh91Hj5883e3tPTvxtnQCx8Iq685S8KikwTFJUnhWOASdKjxNl5+b/OkFOi+t+UbrAqca5kbmUgAFatb7ONFAizSv8npWGfxR8/f8iuIHfH9iIFXQniMvX32vDpL66j7r9ePDeBP8Nkha0GdtjGZ7HTHJrCg1GhIKvD9P4oKmFTiSQmHdnZQeCxBLmON5gAY0+mm10VrzQWAynlsXPkN8w16vqEB7v9ZpeNnI8Nu5hvyfG1xP0qrp6LfmU/5uWklTlIRGXI7PS8XJ8madPJMOBal1ACCcDAq4WIADQWHpNzo1DmWvXanCKaxu1HtT6hQdZkG8mttQvtBDrDkf8LCWsCDP2z/mbRcujSdQoUd34pGELU0YVB3D6hjIyVUw3NfVMK67wZZk24Tb4GR4mAT89U3/6FNr0A57wV6yfZawt+yIfWEjNmaC/WS/2G/2J/oQZdEy0pdPo05b85zdiOjiH0XC3LE=</latexit><latexit sha1_base64="wPiO0NKQB1a58e+E1I1BEGzq9CA=">AAACvHicZVHbbtNAEN2YWwmXpvDIy4ooUpFoZUdI8AIq8MJLpSCRtlITovF6nKyyF2t3XBJZ/h6+hlcQf8M6tWibjuTV2TM7Mz5z0kJJT3H8txPduXvv/oOdh91Hj5883e3tPTvxtnQCx8Iq685S8KikwTFJUnhWOASdKjxNl5+b/OkFOi+t+UbrAqca5kbmUgAFatb7ONFAizSv8npWGfxR8/f8iuIHfH9iIFXQniMvX32vDpL66j7r9ePDeBP8Nkha0GdtjGZ7HTHJrCg1GhIKvD9P4oKmFTiSQmHdnZQeCxBLmON5gAY0+mm10VrzQWAynlsXPkN8w16vqEB7v9ZpeNnI8Nu5hvyfG1xP0qrp6LfmU/5uWklTlIRGXI7PS8XJ8madPJMOBal1ACCcDAq4WIADQWHpNzo1DmWvXanCKaxu1HtT6hQdZkG8mttQvtBDrDkf8LCWsCDP2z/mbRcujSdQoUd34pGELU0YVB3D6hjIyVUw3NfVMK67wZZk24Tb4GR4mAT89U3/6FNr0A57wV6yfZawt+yIfWEjNmaC/WS/2G/2J/oQZdEy0pdPo05b85zdiOjiH0XC3LE=</latexit><latexit sha1_base64="wPiO0NKQB1a58e+E1I1BEGzq9CA=">AAACvHicZVHbbtNAEN2YWwmXpvDIy4ooUpFoZUdI8AIq8MJLpSCRtlITovF6nKyyF2t3XBJZ/h6+hlcQf8M6tWibjuTV2TM7Mz5z0kJJT3H8txPduXvv/oOdh91Hj5883e3tPTvxtnQCx8Iq685S8KikwTFJUnhWOASdKjxNl5+b/OkFOi+t+UbrAqca5kbmUgAFatb7ONFAizSv8npWGfxR8/f8iuIHfH9iIFXQniMvX32vDpL66j7r9ePDeBP8Nkha0GdtjGZ7HTHJrCg1GhIKvD9P4oKmFTiSQmHdnZQeCxBLmON5gAY0+mm10VrzQWAynlsXPkN8w16vqEB7v9ZpeNnI8Nu5hvyfG1xP0qrp6LfmU/5uWklTlIRGXI7PS8XJ8madPJMOBal1ACCcDAq4WIADQWHpNzo1DmWvXanCKaxu1HtT6hQdZkG8mttQvtBDrDkf8LCWsCDP2z/mbRcujSdQoUd34pGELU0YVB3D6hjIyVUw3NfVMK67wZZk24Tb4GR4mAT89U3/6FNr0A57wV6yfZawt+yIfWEjNmaC/WS/2G/2J/oQZdEy0pdPo05b85zdiOjiH0XC3LE=</latexit>

A = K�1 +W<latexit sha1_base64="Sl1t+eeH6M9Gx5k/U8ZUBOM8Nas=">AAACh3icZVFbaxNRED5Zb228tfroy2AICGrdDaJ9EVp9EaRQwTSFJi1nTybJoeeynDMrCcv+D1/1X/lvnE0XadOBc/iYyzcz3+SF0ZHS9G8nuXP33v0HW9vdh48eP3m6s/vsJPoyKBwqb3w4zWVEox0OSZPB0yKgtLnBUX75pYmPfmKI2rsftCpwYuXc6ZlWkth1fgif4Nt59Tar4TWMLnZ66V66NrgNshb0RGvHF7sdNZ56VVp0pIyM8SxLC5pUMpBWBuvuuIxYSHUp53jG0EmLcVKtx66hz54pzHzg5wjW3usVlbQxrmzOmVbSIm7GGuf/WP96kJYNY9zoT7P9SaVdURI6ddV+VhogD40yMNUBFZkVA6mC5g1ALWSQili/G0yN2NM3oTT8K2+b7aMrbY4Bp7y8mXsuX9gB1gB9YFlYoAjtxNCygHaRpGGO7jgiKV86blQdyeWRpKCXfLtYV4O07vJZss0j3AYng72M8ff3vYPP7YG2xAvxUrwSmfgoDsRXcSyGQokgfonf4k+ynbxLPiT7V6lJp615Lm5YcvgPZhDF+w==</latexit><latexit sha1_base64="Sl1t+eeH6M9Gx5k/U8ZUBOM8Nas=">AAACh3icZVFbaxNRED5Zb228tfroy2AICGrdDaJ9EVp9EaRQwTSFJi1nTybJoeeynDMrCcv+D1/1X/lvnE0XadOBc/iYyzcz3+SF0ZHS9G8nuXP33v0HW9vdh48eP3m6s/vsJPoyKBwqb3w4zWVEox0OSZPB0yKgtLnBUX75pYmPfmKI2rsftCpwYuXc6ZlWkth1fgif4Nt59Tar4TWMLnZ66V66NrgNshb0RGvHF7sdNZ56VVp0pIyM8SxLC5pUMpBWBuvuuIxYSHUp53jG0EmLcVKtx66hz54pzHzg5wjW3usVlbQxrmzOmVbSIm7GGuf/WP96kJYNY9zoT7P9SaVdURI6ddV+VhogD40yMNUBFZkVA6mC5g1ALWSQili/G0yN2NM3oTT8K2+b7aMrbY4Bp7y8mXsuX9gB1gB9YFlYoAjtxNCygHaRpGGO7jgiKV86blQdyeWRpKCXfLtYV4O07vJZss0j3AYng72M8ff3vYPP7YG2xAvxUrwSmfgoDsRXcSyGQokgfonf4k+ynbxLPiT7V6lJp615Lm5YcvgPZhDF+w==</latexit><latexit sha1_base64="Sl1t+eeH6M9Gx5k/U8ZUBOM8Nas=">AAACh3icZVFbaxNRED5Zb228tfroy2AICGrdDaJ9EVp9EaRQwTSFJi1nTybJoeeynDMrCcv+D1/1X/lvnE0XadOBc/iYyzcz3+SF0ZHS9G8nuXP33v0HW9vdh48eP3m6s/vsJPoyKBwqb3w4zWVEox0OSZPB0yKgtLnBUX75pYmPfmKI2rsftCpwYuXc6ZlWkth1fgif4Nt59Tar4TWMLnZ66V66NrgNshb0RGvHF7sdNZ56VVp0pIyM8SxLC5pUMpBWBuvuuIxYSHUp53jG0EmLcVKtx66hz54pzHzg5wjW3usVlbQxrmzOmVbSIm7GGuf/WP96kJYNY9zoT7P9SaVdURI6ddV+VhogD40yMNUBFZkVA6mC5g1ALWSQili/G0yN2NM3oTT8K2+b7aMrbY4Bp7y8mXsuX9gB1gB9YFlYoAjtxNCygHaRpGGO7jgiKV86blQdyeWRpKCXfLtYV4O07vJZss0j3AYng72M8ff3vYPP7YG2xAvxUrwSmfgoDsRXcSyGQokgfonf4k+ynbxLPiT7V6lJp615Lm5YcvgPZhDF+w==</latexit><latexit sha1_base64="Sl1t+eeH6M9Gx5k/U8ZUBOM8Nas=">AAACh3icZVFbaxNRED5Zb228tfroy2AICGrdDaJ9EVp9EaRQwTSFJi1nTybJoeeynDMrCcv+D1/1X/lvnE0XadOBc/iYyzcz3+SF0ZHS9G8nuXP33v0HW9vdh48eP3m6s/vsJPoyKBwqb3w4zWVEox0OSZPB0yKgtLnBUX75pYmPfmKI2rsftCpwYuXc6ZlWkth1fgif4Nt59Tar4TWMLnZ66V66NrgNshb0RGvHF7sdNZ56VVp0pIyM8SxLC5pUMpBWBuvuuIxYSHUp53jG0EmLcVKtx66hz54pzHzg5wjW3usVlbQxrmzOmVbSIm7GGuf/WP96kJYNY9zoT7P9SaVdURI6ddV+VhogD40yMNUBFZkVA6mC5g1ALWSQili/G0yN2NM3oTT8K2+b7aMrbY4Bp7y8mXsuX9gB1gB9YFlYoAjtxNCygHaRpGGO7jgiKV86blQdyeWRpKCXfLtYV4O07vJZss0j3AYng72M8ff3vYPP7YG2xAvxUrwSmfgoDsRXcSyGQokgfonf4k+ynbxLPiT7V6lJp615Lm5YcvgPZhDF+w==</latexit>




where and To compute an iterative approach using Newton’s method has to be used.

The Hessian matrix A can be computed as

where is a diagonal matrix which depends on the sigmoid function.

!30

p(f | X,y) ⇡ q(f | X,y) = N (f | f̂ , A�1)

f̂ = argmaxf

p(f | X,y)

A = �rr log p(f | X,y)|f=f̂

second-order Taylor expansion

f̂

A = K�1 +W

W = �rr log p(y | f)




•Yellow: a non-Gaussian posterior •Red: a Gaussian approximation, the mean is

the mode of the posterior, the variance is the negative second derivative at the mode

!31

Now that we have we can compute:

From the regression case we have:

where

This reminds us of a property of Gaussians that we saw earlier!

p(f | X,y)



Predictions

!32

p(f⇤ | X,y,x⇤) =


⌃⇤ = k(x⇤,x⇤)� kT⇤ K

�1k⇤

p(f⇤ | X,x⇤, f) = N (f⇤ | µ⇤,⌃⇤)

µ⇤ = kT⇤ K

�1f

Linear in f



Gaussian Properties (Rep.)

If we are given this: I. II. Then it follows (properties of Gaussians):

III. IV.

where

!33

p(x) = N (x | µ,⌃1)

p(y | x) = N (y | Ax+ b,⌃2)

p(y) = N (y | Aµ+ b,⌃2 +A⌃1AT )

p(x | y) = N (x | ⌃(AT⌃�12 (y � b) + ⌃�1

1 y),⌃)

⌃ = (⌃�11 +AT⌃�1

s A)�1

V[f⇤ | X,y,x⇤] = k(x⇤,x⇤)� kT⇤ (K +W�1)�1k⇤



Applying this to Laplace

It remains to compute

Depending on the kind of sigmoid function we • can compute this in closed form (cumulative

Gaussian sigmoid)

• have to use sampling methods or analytical approximations (logistic sigmoid)

!34

E[f⇤ | X,y,x⇤] = k(x⇤)TK�1f̂

p(y⇤ = +1 | X,y,x⇤) =

Zp(y⇤ | f⇤)p(f⇤ | X,y,x⇤)df⇤



A Simple Example

•Two-class problem (training data in red and blue) •Green line: optimal decision boundary •Black line: GP classifier decision boundary •Right: posterior probability

!35



Summary • Kernel methods solve problems by implicitly mapping

the data into a (high-dimensional) feature space

• The feature function itself is not used, instead the algorithm is expressed in terms of the kernel

• Gaussian Processes are Normal distributions over functions

• To specify a GP we need a covariance function (kernel) and a mean function

• More on Gaussian Processes:http://videolectures.net/epsrcws08_rasmussen_lgp/

!36

computer vision group prof. daniel cremers · l = f =1, n =0.1 l =0.3, f =1.08, n =0.0005 n =0.89 f...

Documents