Authoritative Sources in a Hyperlinked EnvironmentAuthoritative Sources in a Hyperlinked Environment

Hui HanHui Han

CSE dept, PSUCSE dept, PSU

10/15/0110/15/01

Main IdeaMain Idea

Return more authoritative information, using the algorithm based on link structure analysis, esp. the relationship between a set of relevant authoritative pages and the set of “hub” pages that join them together in the link structure

The types of queriesThe types of queries

Specific queries – scarcity problem

“Does Netscape support the JDK 1.1 code-signing API” Broad-topic queries – abundance problem

“Find information about the Java programming language.”

Solution: find a small set of the most “authoritative” relevant pages.

Similar-page queries “Find pages ‘similar’ to java.sun.com

Linked Based AnalysisLinked Based Analysis

Limitations of text based analysis– Text-based ranking function

Eg. Could www.harvard.edu be recognized as one of the most authoritative pages, since many other webpages contain “harvard” more often.

– Pages are not sufficiently self – descriptive Usually the term “search engine” doesn’t appear on

search engine web pages

Limitations of Basic Link-based AnalysisLimitations of Basic Link-based Analysis

Basic model: Of all pages containing the query string, return those with the greatest number of in-links

Large number of links are created primarily for navigational purposes

Difficulty in finding a balance between relevance and popularity– Popular sites like www.yahoo.com would be considered as highly

authoritative on any query strings it contained.

P QConferred Authority

Hub-Authority methodHub-Authority method

A link-based model for the conferral of authority, using the method that consistently identifies relevant, authoritative WWW pages for broad search topics.

Difference from Clustering– Distinguishing pages related to different

meanings or senses of a query term

Step1: Constructing a Focused Subgraph Step1: Constructing a Focused Subgraph of the WWWof the WWW

Properties of a ideal collection S of pages

– S is relatively small

– S is rich in relevant pages

– S contains most(or many) of the strongest authorities

Construction of S– Collect the t highest-ranked pages for the query from a text-

based search engine (Altavista, hotbot) as root set R– Expand R to base set S

a strong authority is quite likely to be pointed to by at least one page in R

– Keep only transverse links

Step2: Compute Hubs and AuthoritiesStep2: Compute Hubs and Authorities

Authoritative pages should– have large in-degree– have a considerable overlap in the sets of pages that

point to them, since they share a common topic

“Hub” pages “pull together”(link to) authorities on a common topic

An Iterative AlgorithmAn Iterative Algorithm

G : the subgraph induced on the pages in S

x(p) : authority weight of Page Py(p): hub weight of Page P

Normalize: pS (x(p))2 = 1

pS (y(p))2 = 1 I operation:

x(p) = q : (q,p) Ey(p)

O operation:

y(p) = q : (q,p) Ex(p)

Hub and Authority exhibit “mutually reinforcing relationship”

An Iterative Algorithm(cont.)An Iterative Algorithm(cont.)

{x(p) } is a vector x with authority weight for each page in G

{y(p) } is a vector y with hub weight for each page in G

Filter out the top c authorities and top c hubs

Basic knowledge of MatrixBasic knowledge of Matrix

M: symmetric n*n matrix :vector : a number

If for some vector , M = , we say,The set of all such is a subspace of Rn Eigenspace

associated with ;

These 1(M), 2(M), … are eigenvalues, while 1(M), 2(M), … are eigenvectors

i(M) belongs to the subspace of i(M)

If we assume |1(M) > 2(M)|, we refer to 1(M) as the principal eigenvector, and all other i(M) as non-principal eigenvector.

Convergence Proof of Iterate ProcedureConvergence Proof of Iterate Procedure

Theorem1. The sequences x1, x2, x3, … and y1, y2, y3, … converge to x* and y* respectively.

Proof: G=(V,E); V={p1, p2, …, pn}; A is the adjacency matrix of graph G; Aij = 1 if (pi, pj) is an edge of G.

I & O operations can be written as:x ATy y Ax K loops,

So, x (1) AT Ax (0); x(0) = AT z x* … x (k) (AT A)k-1 AT zy* … y (k) (AAT)k z

“if is a vector not orthogonal to the principle eigenvector 1(M), the unit vector in the direction of Mk converges to 1(M) as k increases without bound”

Convergence Proof of Iterate Procedure(cont.)Convergence Proof of Iterate Procedure(cont.)

A is called an orthogonal matrix if AAT = AT A = E.

Theorem2: x* is the principal eigenvector of ATA, and y* is the principal eigenvector of AAT.

Experiment finds that k=20 is sufficient for the convergence of vectors.

Experimental resultsExperimental results Most of these authorities do not contain any occurrences of

the initial query string , except http://www.roadhead.com

Similar-Page QueriesSimilar-Page Queries

The strongest authorities in the local region of the link structure near p, can server as a broad-topic summary of the pages related to p.

A small difference in initial request to the search engine: “Find t pages pointing to p”, to get R

Multiple sets of Hubs and AuthoritiesMultiple sets of Hubs and Authorities

Why?– The query string may have several very different

meanings. Eg.“java”– The string may arise as a term in the context of multiple

technical communities. Eg. “randomized algorithms”– The string may refer to a highly polarized issue,

involving groups that are not likely to link to one another. Eg. “abortion”

Idea:– The NON-principle eigenvectors of ATA and AAT

provide us with a natural way to extract additional densely linked collections of hubs and authorities from the base set S.

Multiple sets of Hubs and AuthoritiesMultiple sets of Hubs and AuthoritiesExperimental result1Experimental result1

For the query “jaguar*”, the strongest collections of authoritative sources concerned the Atari Jaguar product, the NFL football team from Jacksonville, and the automobile.

Multiple sets of Hubs and AuthoritiesMultiple sets of Hubs and AuthoritiesExperimental result2Experimental result2

For the query “randomized algorithms”, none of the strongest collections of hubs and authorities are precisely on the query topic. They include home pages of theoretical computer scientists, compendia of mathesmatical software and pages on wavelets.

Multiple sets of Hubs and AuthoritiesMultiple sets of Hubs and AuthoritiesExperimental result3Experimental result3

For the query “abortion”, there is a natural separation.

ConclusionConclusion

A technique for locating high-quality information related to a broad search topic on the www, based on a structural analysis of the link topology surrounding “authoritative” pages on the topic.

Related work.– Standing, influence in social networks, scientific

citations, etc.– Hypertext and WWW rankings– …