presented by ashish chawla and vinit asher the pagerank citation ranking: bringing order to the web...
TRANSCRIPT
PRESENTED BY ASHISH CHAWLA AND VINIT ASHER
The PageRank Citation Ranking: Bringing Order to the Web
Lawrence Page and Sergey Brin, Stanford University. 1998.
Agenda
Introduction
Background
Link Structure
Propagation of Ranking
Simplified Page Rank Calculation
Problems in Ranking
Page Rank Definition
Computing Page Rank
Mathematical Basics
Implementation Details
Convergence
Searching with
PageRank
Introduction
Challenges in Information Retrieval on Web Large # of documents Heterogeneous and Unstructured WWW
Is hypertext provides auxiliary information (other than the text of web
pages)
Objective Take advantage of this link structure.
Background
Academic Citations link to other well known papers peer reviewed have quality control
Web : Homogeneous in their quality, usage, citation & length Quality measure (subjective to the user) Importance of a page is a quantity that isn’t intuitively
possible to capture
What does a user want?
Most applicable documents firstWhat is the job of a retrieval system?
Present more relevant documents upfrontNotion: Quality/Importance of Web Pages
Difficult to classify (depends on user)
We deal with the overall importance of a page, rather than individual sections of the page.
Link Structure
Forward LinksBack LinksWeb has 150 million
pages and 1.7 billion links (probably more now)
Use the concept of citation analysis Highly linked pages are
more “important" than pages with few links
Propagation of Ranking
Page Rank: a page has high rank if the sum of the ranks of its back-links is high
Some notationsu Web Page
Fu Set of pages u points to (Forward links)
Bu Set of pages that point to u (Backlinks)
Nu = |Fu| Number of links from u
c Normalization factor
Simple Ranking function
uBv vN
vRcuR
)()(
Simplified Page Rank Calculation
Problem in Ranking?
Rank Sink: Two web pages that point to each other but to no
other page. Third page which points to one of them. loop will accumulate rank but never distribute it
(since there are no outedges).
Page Rank Definition
Let E(u) be some vector over the Web pages that corresponds to a source of rank. Then, the PageRank of a set of Web pages is an assignment, R’, to the Web pages which satisfies
such that c is maximized and ||R’||1 = 1 (||R’||
1 denotes the L1 norm of R’).
)()(
)('
ucEN
vRcuR
uBv v
Computing Page Rank
initialize vector over web pages
Loop:
new ranks sum of normalized backlink ranks
compute normalizing factor
add escape term
control parameter
while stop when converged
SR 0
ii ARR 1
111 ii RRd
dERR ii 11
ii RR 1
Random Surfer Model
Random SurferClicks at random basis“Surfer” periodically gets bored.
Solution to Random Surfer Model
Escape term: E(u) can be thought of as the random surfer gets bored periodically and jumps to a different page – not staying in the loop forever.
We term this E to be a vector over all the web pages that accounts for each page’s escape probability (user defined parameter).
)()(
)('
ucEN
vRcuR
uBv v
Another Problem – Dangling Links
What are dangling links? Links that point to any page with no outgoing links. Pages not downloaded yet.
Why is this a problem? We don’t know how to distribute weight to these.
What do we do ? Remove them from the system
Mathematical Basics
What is eigen vector and eigen value? Given a vector v in the n-dimensional vector space, we
can linear transform it to another vector space using a transformation matrix A. The transformed vector is Av.
An eigen vector is a vector that is scaled by a linear transformation, but not moved. The scaling factor is the eigen value. Eigen values and eigen vectors are not unique. We can compute them by Ax = x where is the eigen value of A and x is the corresponding eigen vector.
An eigenvector is a vector that 'points' in the same direction (has invariant direction cosines) under some transform. The eigenvalue is a number that describes how the magnitude of the eigenvector is scaled by the transform.
Mathematical Basics
A is designated to be a matrix, u and v correspond to the columns of this matrix.
Given that A is a matrix, and R be a vector over all the Web pages, the dominant eigenvector is the one associated with the maximal eigenvalue.
Example
AT=
Example (contd..)
R = c A R = M R c: eigenvalue R : eigen vector of A
A =
R = Normalized =
A x = λ x| A - λI | x = 0
Implementation
Web crawler keeps a database of URLs so that it can discover all URLs on the web
To implement PageRank, the web crawler builds an index of the URLs as it crawls
Problems??? Infinitely large sites Incorrect HTML Sites are down Web is always changing
PageRank Implementation
Convert each URL into unique integer IDLink structure sorted by the IDsRemove dangling linksMake a initial assignment of ranks and
iterate until convergenceAdd the dangling links backIterate the process again to assign weights to
all dangling linksLink database A, is normally kept in RAM
Convergence Properties
PageRank will scale very well for large collections as the scaling factor is roughly linear in log n.
Convergence Properties
Here we interpret web as a expander like graph.
A graph is said to be expander if every subsets of nodes S has a neighborhood that is larger than some factor α times |S|
Mathematically we verify the same if the largest eigenvalue is sufficiently larger than the second-largest eigenvalue
Searching with PageRank
Two search engines implemented using PageRank. Title based search engine
Matches titles of web pages with the given query Ranks the results using PageRank Works well for general queries having a large result set
Full text search engine (Google) Scans the entire document for a match with the given
query Performs rank merging.
Types of Results
Information based result Finds a site which contains great deal of information Propagates textual matching score through the link
structureCommon Case result
Most commonly used site (often commercial) relevant to the search query
PageRank results in good representation of the common case
Personalized PageRank
E vector Corresponds to a distribution of web pages Provides flexibility in adjustment of PageRanks
Uniform E causes highly linked web pages to achieve a very high ranking
Single page E results in important pages not related to the homepage to achieve a low PageRank
E consisting of root level pages of all web servers is a good compromise between uniform E and single page E
Applications
Estimating Web Traffic Looking at differences between PageRank and actual
usage statistics, it is possible to find things that people often look at, but do not want to link to their web pages
Backlink Predictor Citation counts tends to get stuck in the local web
pages Using random surfer model, PageRank quickly finds
the site homepage, and gives preference to its children resulting in an efficient, broad search
Hence PageRank potentially acts as a better backlink predictor since it builds up the entire website information faster
Other Applications
Spam detection and prevention
Sort the backlinks based on their importance
Issues
Users are not random walkers.Starting point distribution (actual usage data
as starting vector).Bias towards main pages.Linkage spam.No query specific rank.
Conclusion
PageRank is a global ranking of all webpages, regardless of their content based solely on their location in the Web’s graph structure
PageRank can be used to separate a small set of commonly used documents
Full database is consulted only when small database is not adequate to answer the queries
Personalized PageRank can be used to create a view of Web from a particular user’s perspective
Google Architecture..