10-603/15-826a: multimedia databases and data mining svd - part ii (more case studies) c. faloutsos

10-603/15-826A:Multimedia Databases and Data

Mining

SVD - part II (more case studies)

C. Faloutsos

Multi DB and D.M. Copyright: C. Faloutsos (2001) 2

Outline

Goal: ‘Find similar / interesting things’

• Intro to DB

• Indexing - similarity search

• Data Mining


Indexing - Detailed outline• primary key indexing• secondary key / multi-key indexing• spatial access methods• fractals• text• Singular Value Decomposition (SVD)• multimedia• ...


SVD - Detailed outline• Motivation• Definition - properties• Interpretation• Complexity• Case studies• SVD properties• More case studies• Conclusions


SVD - detailed outline• ...• Case studies• SVD properties• more case studies

– google/Kleinberg algorithms– query feedbacks

• Conclusions


SVD - Other properties - summary

• can produce orthogonal basis (obvious) (who cares?)

• can solve over- and under-determined linear problems (see C(1) property)

• can compute ‘fixed points’ (= ‘steady state prob. in Markov chains’) (see C(4) property)


SVD -outline of properties

• (A): obvious• (B): less obvious• (C): least obvious (and most powerful!)


Properties - by defn.:

A(0): A[n x m] = U [ n x r ] [ r x r ] VT [ r x m]

A(1): UT [r x n] U [n x r ] = I [r x r ] (identity matrix)

A(2): VT [r x n] V [n x r ] = I [r x r ]

A(3): k = diag( 1k, 2

k, ... rk ) (k: ANY real

number)A(4): AT = V UT


Less obvious properties


B(1): A [n x m] (AT) [m x n] = ??




B(1): A [n x m] (AT) [m x n] = U 2 UT

symmetric; Intuition?




B(1): A [n x m] (AT) [m x n] = U 2 UT

symmetric; Intuition?‘document-to-document’ similarity matrix

B(2): symmetrically, for ‘V’

(AT) [m x n] A [n x m] = V 2 VT

Intuition?


Less obvious propertiesA: term-to-term similarity matrix

B(3): ( (AT) [m x n] A [n x m] ) k= V 2k VT

and

B(4): (AT A )

k ~ v1 12k v1

T for k>>1 where

v1: [m x 1] first column (eigenvector) of V

1: strongest eigenvalue



B(4): (AT A )

k ~ v1 12k v1

T for k>>1

B(5): (AT A )

k v’ ~ (constant) v1

ie., for (almost) any v’, it converges to a vector parallel to v1

Thus, useful to compute first eigenvector/value (as well as the next ones, too...)


Less obvious properties - repeated:


B(1): A [n x m] (AT) [m x n] = U 2 UT

B(2): (AT) [m x n] A [n x m] = V 2 VT

B(3): ( (AT) [m x n] A [n x m] ) k= V 2k VT

B(4): (AT A )

k ~ v1 12k v1

T

B(5): (AT A )



Least obvious properties


C(1): A [n x m] x [m x 1] = b [n x 1]

let x0 = V (-1) UT bif under-specified, x0 gives ‘shortest’ solutionif over-specified, it gives the ‘solution’ with the

smallest least squares error(see Num. Recipes, p. 62)


Least obvious propertiesIllustration: under-specified, eg [1 2] [w z] T = 4 (ie, 1 w + 2 z = 4)

1 2 3 4

1

2all possible solutionsx0

w

z shortest-length solution


Least obvious propertiesIllustration: over-specified, eg [3 2]T [w] = [1 2]T (ie, 3 w = 1; 2 w = 2 )

1 2 3 4

1

2reachable points (3w, 2w)

desirable point b


Least obvious properties - cont’d


C(2): A [n x m] v1 [m x 1] = 1 u1 [n x 1]

where v1 , u1 the first (column) vectors of V, U. (v1 == right-eigenvector)

C(3): symmetrically: u1T A = 1 v1

T

u1 == left-eigenvectorTherefore:


Least obvious properties - cont’d


C(4): AT A v1 = 12 v1

(fixed point - the usual dfn of eigenvector for a symmetric matrix)


Least obvious properties - altogether


C(1): A [n x m] x [m x 1] = b [n x 1]

then, x0 = V (-1) UT b: shortest, actual or least-squares solution

C(2): A [n x m] v1 [m x 1] = 1 u1 [n x 1]

C(3): u1T A = 1 v1

T

C(4): AT A v1 = 12 v1


Properties - conclusions


B(5): (AT A )


C(1): A [n x m] x [m x 1] = b [n x 1]

then, x0 = V (-1) UT b: shortest, actual or least-squares solution

C(4): AT A v1 = 12 v1



– Kleinberg/google algorithms– query feedbacks

• Conclusions


Kleinberg’s algorithm

• Problem dfn: given the web and a query• find the most ‘authoritative’ web pages for

this query

Step 0: find all pages containing the query terms

Step 1: expand by one move forward and backward



• Step 1: expand by one move forward and backward



• on the resulting graph, give high score (= ‘authorities’) to nodes that many important nodes point to

• give high importance score (‘hubs’) to nodes that point to good ‘authorities’)

hubs authorities



observations

• recursive definition!

• each node (say, ‘i’-th node) has both an authoritativeness score ai and a hubness score hi



Let E be the set of edges and A be the adjacency matrix: the (i,j) is 1 if the edge from i to j exists

Let h and a be [n x 1] vectors with the ‘hubness’ and ‘authoritativiness’ scores.

Then:



Then:

ai = hk + hl + hm

that is

ai = Sum (hj) over all j that (j,i) edge exists

or

a = AT h

k

l

m

i



symmetrically, for the ‘hubness’:

hi = an + ap + aq

that is

hi = Sum (qj) over all j that (i,j) edge exists

or

h = A a

p

n

q

i



In conclusion, we want vectors h and a such that:

h = A a

a = AT hRecall properties:

C(2): A [n x m] v1 [m x 1] = 1 u1 [n x 1]

C(3): u1T A = 1 v1

T



In short, the solutions toh = A aa = AT h

are the left- and right- eigenvectors of the adjacency matrix A.

Starting from random a’ and iterating, we’ll eventually converge

(Q: to which of all the eigenvectors? why?)



(Q: to which of all the eigenvectors? why?)

A: to the ones of the strongest eigenvalue, because of property B(5):

B(5): (AT A )



Kleinberg’s algorithm - results

Eg., for the query ‘java’:

0.328 www.gamelan.com

0.251 java.sun.com

0.190 www.digitalfocus.com (“the java developer”)


Kleinberg’s algorithm - discussion

• ‘authority’ score can be used to find ‘similar pages’ (how?)

• closely related to ‘citation analysis’, social networs / ‘small world’ phenomena


google/page-rank algorithm

• closely related: imagine a particle randomly moving along the edges (*)

• compute its steady-state probabilities

(*) with occasional random jumps



• ~identical problem: given a Markov Chain, compute the steady state probabilities p1 ... p5

1 2 3

45



• Let A be the transition matrix (= adjacency matrix, row-normalized : sum of each row = 1)

1 2 3

45

1

1/2 1/2

1

1

1/2 1/2

p1

p2

p3

p4

p5

p1

p2

p3

p4

p5

=



• A p = p

1 2 3

45

1

1/2 1/2

1

1

1/2 1/2

p1

p2

p3

p4

p5

p1

p2

p3

p4

p5

=

A p = p



• A p = p

• thus, p is the eigenvector that corresponds to the highest eigenvalue (=1, since the matrix is row-normalized)


Kleinberg/google - conclusions

SVD helps in graph analysis:

hub/authority scores: strongest left- and right- eigenvectors of the adjacency matrix

random walk on a graph: steady state probabilities are given by the strongest eigenvector of the transition matrix




• Conclusions


Query feedbacks[Chen & Roussopoulos, sigmod 94]sample problem:estimate selectivities (e.g., ‘how many movies

were made between 1940 and 1945?’for query optimization,LEARNING from the query results so far!!


Query feedbacks

Idea #1: consider a function for the CDF (cummulative distr. function), eg., 6-th degree polynomial (or splines, or whatever)

year

count, so far


Query feedbacks

GREAT Idea #2: adapt your model, as you see the actual counts of the actual queries

year

count, so faractual

original estimate


Query feedbacks

year

count, so faractual

original estimate


Query feedbacks

year

count, so faractual

original estimate

a query


Query feedbacks

year

count, so faractual

original estimate

new estimate


Query feedbacks

Eventually, the problem becomes:- estimate the parameters a1, ... a7 of the

model- to minimize the least squares errors from the

real answers so far.Formally:


Query feedbacks

Formally, with n queries and 6-th degree polynomials:

X11 X12 X17

Xn1 Xn2 Xn7

b1

b2

bn

a1

a2

a7

=


Query feedbacks

where xi,j such that Sum (xi,j * ai) = our estimate for the # of movies and bj: the actual

X11 X12 X17

Xn1 Xn2 Xn7

b1

b2

bn

a1

a2

a7

=


Query feedbacks

In matrix form:

X11 X12 X17

Xn1 Xn2 Xn7

b1

b2

bn

a1

a2

a7

=

X a = b

1st query

n-th query


Query feedbacks

In matrix form:

X a = band the least-squares estimate for a is

a = V UT baccording to property C(1)

(let X = U VT )


Query feedbacks - enhancements

the solutiona = V UT b

works, but needs expensive SVD each time a new query arrives

GREAT Idea #3: Use ‘Recursive Least Squares’, to adapt a incrementally.

Details: in paper - intuition:



Intuition:

x

b a1 x + a2

least squares fit



Intuition:

x

b a1 x + a2

least squares fit

new query



Intuition:

x

b a1 x + a2

least squares fit

new query

a’1 x + a’2



the new coefficients can be quickly computed from the old ones, plus statistics in a (7x7) matrix

(no need to know the details, although the RLS is a brilliant method)



GREAT idea #4: ‘forgetting’ factor - we can even down-play the weight of older queries, since the data distribution might have changed.

(comes for ‘free’ with RLS...)


Query feedbacks - conclusions

SVD helps find the Least Squares solution, to adapt to query feedbacks

(RLS = Recursive Least Squares is a great method to incrementally update least-squares fits)




• Conclusions


Conclusions

• SVD: a valuable tool

• given a document-term matrix, it finds ‘concepts’ (LSI)

• ... and can reduce dimensionality (KL)

• ... and can find rules (PCA; RatioRules)


Conclusions cont’d

• ... and can find fixed-points or steady-state probabilities (google/ Kleinberg/ Markov Chains)

• ... and can solve optimally over- and under-constraint linear systems (least squares / query feedbacks)


References

• Brin, S. and L. Page (1998). Anatomy of a Large-Scale Hypertextual Web Search Engine. 7th Intl World Wide Web Conf.

• Chen, C. M. and N. Roussopoulos (May 1994). Adaptive Selectivity Estimation Using Query Feedback. Proc. of the ACM-SIGMOD , Minneapolis, MN.


References cont’d

• Kleinberg, J. (1998). Authoritative sources in a hyperlinked environment. Proc. 9th ACM-SIAM Symposium on Discrete Algorithms.

• Press, W. H., S. A. Teukolsky, et al. (1992). Numerical Recipes in C, Cambridge University Press.

10-603/15-826a: multimedia databases and data mining svd - part ii (more case studies) c. faloutsos

Documents

v t r x n v n x r

u n x r r x r v t r

u t r x n u n x r

r x r a3

v u t slide

v t b4

r x r identity matrix

t intuition