svd++ and pmf - pontificia universidad católica de...
TRANSCRIPT
![Page 1: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/1.jpg)
SVD++ and Probabilistic Matrix Factorization
Claudio Rojas Bórquez
![Page 2: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/2.jpg)
ReferencesThis presentation is a summary of the following papers:
• Yehuda, Koren. Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model. Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. Pages 426-434 [2008]
• Salakhutdinov, Ruslan. Probabilistic Matrix Factorization. [2007]
2
![Page 3: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/3.jpg)
In this presentation• Collaborative filtering models
• SVD++
• Probabilistic matrix approximation
3
![Page 4: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/4.jpg)
Collaborative Filtering• CF is a methodology where past transactions are
analysed in order to establish connections between users and products.
• The two more successful approaches to CF are latent factor models, represented by SVD, and neighbourhood models (Yehuda, 2008)
• Both models have strengths and weaknesses.
4
![Page 5: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/5.jpg)
Latent Factor models• What it does?
Transform items and users to the same latent factor space, thus making them directly comparable. The most common way to do so is SVD
The approximating matrix is the best approximation in the Frobenius norm (more on this later)
5
M = U⌃V T ! M̃ = U ⌃̃V T
![Page 6: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/6.jpg)
Latent Factor models• The drawback of latent factor models are basically two:
• They show poor performance at detecting strong associations among a small set of closely related items.
• Their results is little to no explanation on the reasons behind a recommendation which is bad for the user who wants to know why an item is being recommended to him/her.
• SVD also is sensible to sparse data which is very common in the field.
6
![Page 7: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/7.jpg)
Neighborhood models• Neighborhood method takes in account either
items or users that are similar between themselves, this is usually made using a similarity score
• This method is very good at detecting localised relationships which is exactly where Latent method behave poorly.
• They are blind to the big structure.
7
sij
![Page 8: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/8.jpg)
Implicit feedback• Neither of both previous models include implicit
feedback, this in one of the goals the author wants to achieve with the SVD++ model.
• There are many kinds of implicit feedback, some are not so obvious, for example in the case of the Netflix’s data that a user rates a movie, without regard of the actual rating, may be considered implicit feedback.
8
![Page 9: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/9.jpg)
SVD++ model• This model tries to mix strengths of the latent model
as well of the neighbourhood model.
• The prediction made by this model has 3 parts
9
r̂ui = µ+ bu + bi
+ qTi
0
@pu + |N(u)|� 12
X
j2N(u)
yj
1
A
+ |Rk(i;u)|� 12
X
j2Rk(i;u)
(ruj � buj)wij + |Nk(i;u)|� 12
X
j2Nk(i;u)
cij
![Page 10: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/10.jpg)
1st tier
r̂ui = µ+ bu + bi
+ qTi
0
@pu + |N(u)|� 12
X
j2N(u)
yj
1
A
+ |Rk(i;u)|� 12
X
j2Rk(i;u)
(ruj � buj)wij + |Nk(i;u)|� 12
X
j2Nk(i;u)
cij
The first term is the basis rate, it takes in account a global mean and the bias of both user and item.
![Page 11: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/11.jpg)
2nd tier
r̂ui = µ+ bu + bi
+ qTi
0
@pu + |N(u)|� 12
X
j2N(u)
yj
1
A
+ |Rk(i;u)|� 12
X
j2Rk(i;u)
(ruj � buj)wij + |Nk(i;u)|� 12
X
j2Nk(i;u)
cij
The second term is similar to the original SVD but takes in account the implicit feedback present in the set of rated items
N(u)
![Page 12: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/12.jpg)
r̂ui = µ+ bu + bi
+ qTi
0
@pu + |N(u)|� 12
X
j2N(u)
yj
1
A
+ |Rk(i;u)|� 12
X
j2Rk(i;u)
(ruj � buj)wij + |Nk(i;u)|� 12
X
j2Nk(i;u)
cij
The third and fourth terms are the neighborhood terms. The former is the weighted bias of the basis rate and the actual rate and the latter is the local effect of the implicit feedback
![Page 13: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/13.jpg)
Actual formulation• If then the problem that is solved is:
• This optimisation can be performed using gradient descent.
13
eui = rui � r̂ui
minX
u,i
e2ui + �kparamsk2
![Page 14: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/14.jpg)
SVD++ performance
![Page 15: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/15.jpg)
SVD++ performance
![Page 16: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/16.jpg)
PMF (Probabilistic matrix factorization)
• This method tries to make a matrix decomposition as SVD but it has two main differences:
• It only takes in account the non zero elements, i.e performs well with sparse data
• Scales linearly with the number of observations
16
![Page 17: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/17.jpg)
PMF• This model view the rating as a probabilistic
graphical model:
17
![Page 18: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/18.jpg)
p(U |�2u) =
Y
i
N (Ui|0,�2uI) p(V |�2
v) =Y
j
N (Vj |0,�2vI)
p(R|U, V,�2) =Y
i
Y
j
⇥N (Rij |UT
i Vj ,�2)⇤Iij
![Page 19: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/19.jpg)
PMF• Given prior for the latent factors for users and items
the equivalent problem is minimise the square error given by:
• It’s posible to put priors to the parameters which receives the name of adaptative PMF
19
E =1
2
X
i
X
j
Iij(Rij � UTi Vj) +
�u
2
X
i
kUik2 +�v
2kVjk2
![Page 20: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/20.jpg)
PMF performance
![Page 21: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/21.jpg)
Constrained PMF
21
![Page 22: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/22.jpg)
Constrained PMF
22
![Page 23: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/23.jpg)
Constrained PMF
![Page 24: SVD++ and PMF - Pontificia Universidad Católica de Chiledparra.sitios.ing.uc.cl/.../CRojas_SVDpp-PMF.pdf · 2016-06-20 · PMF (Probabilistic matrix factorization) • This method](https://reader034.vdocuments.us/reader034/viewer/2022050421/5f9015ccff58e832767f88fc/html5/thumbnails/24.jpg)
Constrained PMF
24