knowledge extraction from the web monika henzinger steve lawrence
TRANSCRIPT
Knowledge Extraction from the Web
Monika Henzinger
Steve Lawrence
Outline
• Hyperlink analysis in web IR• Sampling the web:
– Web pages– Web hosts
• Web graph models• Focused crawling• Finding communities
Hyperlink analysis in web information retrieval
Graph structure of the web
• Web graph– Each web page is a node
– Each hyperlink is a directed edge
• Host graph– Each host is a node
– If there are k links from host A to host B, there is an edge with weight k from A to B.
Hyperlink analysis in Web IR
• Idea: Mine structure of the web graph to
improve search results
• Related work:
– Classic IR work (citations = links) a.k.a.
“Bibliometrics” [K’63, G’72, S’73,…]
– Socio-metrics [K’53, MMSM’86,…]
– Many Web related papers use this approach
[PPR’96, AMM’97, S’97, CK’97, K’98, BP’98,…]
Google’s approach
• Assumption: A link from page A to page B is a recommendation of page B by the author of A(we say B is successor of A)
Quality of a page is related to its in-degree
• Recursion: Quality of a page is related to– its in-degree, and to
– the quality of pages linking to it
PageRank [BP ‘98]
Definition of PageRank
• Consider the following infinite random walk (surf):– Initially the surfer is at a random page
– At each step, the surfer proceeds
• to a randomly chosen web page with probability d
• to a randomly chosen successor of the current page with
probability 1-d
• The PageRank of a page p is the fraction of steps
the surfer spends at p in the limit.
PageRank (cont.)
By previous theorem:• PageRank = stationary probability for this
Markov chain, i.e.
where n is the total number of nodes in the graph
Euv
voutdegreevPageRankdn
duPageRank
),(
)(/)()1()(
Query Results= Start Set Forward SetBack Set
Neighborhood graph
An edge for each hyperlink, but no edges within the same host
Result1
Result2
Resultn
f1
f2
fs
...
b1
b2
bm
… ...
• Subgraph associated to each query
HITS [Kleinberg’98]
• Goal: Given a query find:
– Good sources of content (authorities)
– Good sources of links (hubs)
Repeat until HUB and AUTH converge:
Normalize HUB and AUTH
HUB[v] := AUTH[ui] for all ui with Edge(v, ui)
AUTH[v] := HUB[wi] for all wi with Edge(wi, v)
HITS details
w1
wk
......Aw2
u1
uk
u2
......H
v
PageRank vs. HITS
• Computation: – Once for all documents
and queries (offline)
• Query-independent – requires combination with query-dependent criteria
• Hard to spam
• Computation:– Requires computation for
each query
• Query-dependent
• Relatively easy to spam• Quality depends on
quality of start set• Gives hubs as well as
authorities
PageRank vs. HITS
• [Lempel] Not rank-stable: O(1) changes in graph can change O(N2) order-relations
• [Ng, Zheng, Jordan01] “Value”-Stable: change in k nodes (with PR values p1,…pk) results in p* s.t.
• Not rank-stable
• “value”-stability depends on gap g between largest and second largest eigenvector in ATA: change of O(g) in ATA results in p* s.t.dppp
k
jj /2||||
1
*
)1(|||| * pp
Random sampling of web pages
Random sampling of web pages
• Useful for estimating:
– Web properties: Percentage of pages in a domain,
in a language, on a topic, indegree distribution …
– Search engine comparison: Percentage of pages in
a search engine index (index size)
Let’s do the random walk!
• Perform PageRank random walk
• Select uniform random sample from resulting pages
• Can’t jump to a random page; instead, jump to a
random page on a random host seen so far.• Problem:
– Starting state bias: finite walk only approximates PageRank.
“Quality-biased” sample of the web
Page Freq. Freq. RankWalk2 Walk1 Walk1
www.microsoft.com/ 3172 1600 1www.microsoft.com/windows/ie/default.htm 2064 1045 3www.netscape.com/ 1991 876 6www.microsoft.com/ie/ 1982 1017 4www.microsoft.com/windows/ie/download/ 1915 943 5www.microsoft.com/windows/ie/download/all.htm 1696 830 7www.adobe.com/prodindex/acrobat/readstep.html 1634 780 8home.netscape.com/ 1581 695 10www.linkexchange.com/ 1574 763 9www.yahoo.com/ 1527 1132 2
Most frequently visited pages
Site Frequency Frequency RankWalk 2 Walk 1 Walk 1
www.microsoft.com 32452 16917 1home.netscape.com 23329 11084 2www.adobe.com 10884 5539 3www.amazon.com 10146 5182 4www.netscape.com 4862 2307 10excite.netscape.com 4714 2372 9www.real.com 4494 2777 5www.lycos.com 4448 2645 6www.zdnet.com 4038 2562 8www.linkexchange.com 3738 1940 12www.yahoo.com 3461 2595 7
Most frequently visited hosts
Sampling pages nearly uniformly
• Perform PageRank random walk
• Sample pages from walk s.t.
• Don’t know PageRank(p):
• PR: PageRank computation of crawled graph
• VR: VisitRatio on crawled graph
“Nearly uniform” sample of the web
)(/1)crawled is |sampled is Pr( pPageRankpp
Sampling pages nearly uniformly
• “Nearly uniform” sample:
– Recall:
– A page is well-connected if it can be reached by
almost every other page by short paths (O(n1/2) steps)
– For short paths in a well-connected graph:
constant
crawled) is |sampled is Pr(crawled) is Pr()sampled is Pr(
pppp
)(
) of visitsofnumber ()crawled is Pr(
pPageRankL
pEp
)(/1)crawled is |sampled is Pr( pPageRankpp
Sampling pages nearly uniformly
• Problems:– Starting state bias: finite walk only approximates
PageRank.
– Dependence, especially in short cycles
Synthetic graphs: in-degree
Synthetic graphs: PageRank
Experiments on the real web
• Performed 3 random walks in Nov 1999 (starting from 10,258 seed URLs)
• Small overlap between walks – walks disperse well (82% visited by only 1 walk)
Walk#visited URLsunique URLs
12,702,939 990,2512 2,507,004 921,1143 5,006,745 1,655,799
Percentage of pages in domains
05
1015
202530
3540
4550
com edu org net jp gov de uk
CrawlWalk 1 UniformWalk 2 UniformWalk 3 UniformWalk 1 PRWalk 2 PRWalk 3 PRWalk 1 VRWalk 2 VRWalk 3 VR
Estimating search engine index size
• Choose a sample of pages p p1,p2,p3… pn according to near
uniform distribution
• Check if the pages are in search engine index S [BB’98]:– Exact match
– Host match
• Estimate for size of index S is the percentage of sampled
pages that are in S, i.e.
where I[pj in S] = 1 if pj is in S and 0 otherwise
][1
)( SpIn
Svj
j
Result set for index size (fall ’99)
Random sampling of sites
Publicly indexable web
• We analyzed the “publicly indexable web”• Excludes pages that are not indexed by the
major search engines due to– Authentication requirements– Pages hidden behind search forms– Robots exclusion standard
Random sampling of sites
• Randomly sample IP addresses (2564 or about 4.3 billion)• Test for a web server at the standard port• Many machines and network connections are temporarily
unavailable - recheck all addresses after one week• Many sites serve the same content on multiple IP
addresses for load balancing or redundancy– Use DNS - only count one address in publicly indexable web
• Many servers not part of the “publicly indexable web”– Authorization requirements, default page, sites “coming soon”,
web-hosting companies that present their homepage on many IP addresses, printers, routers, proxies, mail servers, etc.
– Use regular expressions to find a majority, manual inspection
Feb 99 results
• Manually classified 2,500 random web servers
• 83% of sites commercial• Percentage of sites in areas
like science, health,and government is relatively small– Would be feasible and very
valuable to create specialized services that are very comprehensive and up to date
• 65% of sites have a majority of pages in English
Metadata analysis
• Analyzed simple HTML meta tag usage on the homepage of the 2,500 random servers– 34% of sites had description or keywords tags
• Low usage of this simple standard suggests that acceptance and widespread use of more complex standards like XML and Dublin Core may be very slow– 0.3% of sites contained Dublin Core tags
Web graph models
Inverse power laws on the web
• Fraction of pages with k in-links =
1.2, k
Properties with inverse power law
1. indegree of web pages2. outdegree of web pages3. indegree of web pages, off-site links only4. outdegree of web pages, off-site links only5. size of weakly connected components6. size of strongly connected components7. indegree of hosts8. outdegree of hosts9. number of hyperlinks between host pairs10.PageRank11.…
Category specific web
• All US company homepages• Histogram with exponentially
increasing size buckets (constant size on log scale)
• Strong deviation from pure power law
• Unimodal body, power law tail
Web graph model [BA ’99]
• Preferential attachment model:• Start with nodes• At each timestep:
– add 1 node v and – m edges incident to v s.t. for each new edge:
P(other endpoint is node u) in-degree(u)
• Theorem: P(page has k in-links) k-3
0n
Combining preferential and uniform
• Extension of preferential attachment model:• Start with nodes• At timestep t:
– add 1 node v and
– m edges s.t. for each new edge:
P(node u is endpoint)=
• Theorem: P(page has k in-links)
tnmt
uindegree
0
1)1(
2
)(
0n
/11k
Preferential vs. uniform attachment
• always– Preferential attachment plays a
greater role in web link growth than uniform attachment
• Distribution of links to companies and newspapers close to power law
• Distribution of links to universities and scientists closer to uniform– More balanced mixture of
preferential and uniform attachment
5.0 Preferential attachment
Dataset
Companies 0.95
Newspapers 0.95
Web inlinks 0.91
Universities 0.61
Scientists 0.60
Web outlinks 0.58
E-commerce categories
Other networks
• Most social/biological networks exhibit drop-off from power law scaling at small k
• Actor collaborations, paper citations, US power grid, global web outlinks, web file sizes
Graph model summary
– Previous research: power law distribution of inlinks - “winners take all”
– Only an approximation - hides important details– Distribution varies in different categories; may be
much less biased– New model accurately accounts for the distribution
of category specific pages, the web as a whole, and other social networks
– May be used to predict degree of “winners take all” behavior
Copy model [KKRRT’99]
• At each timestep add new node u with fixed outdegree d.
• The destinations of these links are chosen:– Choose existing node v uniformly at random.– For j=1,...d, the j-th link of u points to a random
existing node with probability and to the destination of v’s j-th link with probability 1- .
• Models power law as well as large number of small bipartite cliques.
Relink model
• Hostgraph exhibits drop-off from power law scaling at small k relink model:
• With probability select a random existing node u, and with probability 1- create a new node u. Add d edges to u.
• The destinations of these links are chosen:– Choose existing node v uniformly at random and
choose d random edges with source v.– Determine destinations as in the copy model.
Relink model
Linkage between domains
com Self 1 2 3 4
com 82.9 82.9 net 6.5 org 2.6 jp 0.8 uk 0.7
cn 15.8 74.1 tw 0.4 jp 0.2 de 0.2 hk 0.1
jp 17.4 74.5 to 0.8 cn 0.6 uk 0.2 de 0.1
tw 22.0 66.0 to 1.3 au 0.6 jp 0.6 ch 0.4
ca 19.4 65.2 uk 0.6 fr 0.4 se 0.3 de 0.3
de 16.0 71.2 uk 0.8 ch 0.6 at 0.5 nl 0.2
br 17.8 69.1 uk 0.4 pt 0.4 de 0.4 ar 0.2
fr 20.9 61.9 ch 0.9 de 0.8 uk 0.7 ca 0.5
uk 34.2 33.1 de 0.6 ca 0.5 jp 0.3 se 0.3
Finding communities
Finding communities
• Identifying communities is valuable for– Focused search engines– Web directory creation– Content filtering– Analysis of communities and relationships
Recursive communities
• Several methods proposed• One link based method:• A community consists of members that have
more links within the community than outside of the community
s-t Maximum flow
• Definition: given a directed graph, G=(V,E), with edge capacities c(u,v) 0, and two vertices, s, t V, find the maximum flow that can be routed from the source, s, to the sink, t.
• Intuition: think of water pipes
• Note: maximum flow = minimum cut
• Maximum flow yields communities
Maximum flow communities
• If the source is in the community, the sink is outside of the community, and the degree of the source and sink exceeds the cut size, then maximum flow identifies the entire community.
Maximum flow communities
Maximum flow communities
SVM web community
• Seed set consisted of:– http://svm.first.gmd.de/– http://svm.research.bell-labs.com/– http://www.clrc.rhbnc.ac.uk/research/SVM/– http://www.support-vector.net/
• Four EM iterations used
• Only external links considered
• Induced graph contained over 11,000 URLs
• Identified community contained 252 URLs
Top ranked SVM pages
1. Vladimir Vapnik's home page (inventor SVMs)
2. Home page of SVM light, a popular software package
3. A hub site of SVM links4. Text categorization corpus5. SVM application list6. John Platt's SVM page (inventor of
SMO)7. Research interests of Mario
Marchand (SVM researcher)8. SVM workshop home page9. GMD First SVM publication list10. Book: Advances in Kernel Methods
- SVM Learning
11. B. Schölkopf's SVM page 12. GMD First hub page of SVM
researchers 13. Y. Li's links to SVM pages14. NIPS SVM workshop abstract
page15. GMD First SVM links 16. Learning System Group of ANU 17. NIPS*98 workshop on large margin
classifiers18. Control theory seminar (with links
to SVM material)19. ISIS SVM page20. Jonathan Howell's home page
Lowest ranked SVM pages
• Ten web pages tied for the lowest score. All were personal home pages of scientists that had at least one SVM publication.
• Other results contained researchers, students, software, books, conferences, workshops, etc.
• A few false positives: NN and data mining.
“Ronald Rivest” community summary
• One seed: http://theory.lcs.mit.edu/~rivest• Four EM iterations used• First EM iteration used internal links• Induced graph contained more than 38,000
URLs• Identified community contained 150 URLs
“Ronald Rivest” top ranked pages
1. Thomas H. Cormen’s home page 2. The Mathematical Guts of RSA
Encryption3. Charles E. Leiserson’s home page4. Famous people in the history of
Cryptography5. Cryptography sites6. Massachusetts Institute of Technology7. general cryptography links8. Spektrum der Wissenschaft -
Kryptographie9. Issues in Securing Electronic
Commerce over the Internet10. course based on “Introduction to
Algorithms”
11. Recommended Literature for Self-Study
12. Resume of Aske Plaat
13. German article on who's who of the WWW
14. People Ulrik knows
15. A course that uses ``Introduction to Algorithms''
16. Bibliography on algorithms
17. an article on encryption
18. German computer science institute
19. security links
20. International PGP FAQ
“Ronald Rivest” lowest ranked
• 23 URLs tied for the lowest ranked
• All 23 were personally related to Ronald Rivest or his research
• 11 / 23 were bibliographies of Rivest’s publications
“Rivest” community n-grams1 F.rivest 21 F.chaffing_and2 F.l_rivest 22 F.shamir3 F.ronald_l 23 F.rivest_s4 F.ronald 24 F.security5 F.cryptography 25 F.public_key6 F.rsa 26 F.algorithms7 F.ron_rivest 27 F.cormen8 T.rivest 28 F.edu_rivest9 F.lcs 29 F.adi_shamir
10 T.l_rivest 30 F.cryptography_and11 T.ronald_l 31 F.mit_edu12 F.theory_lcs 32 F.computer_science13 F.encryption 33 F.ron14 F.lcs_mit 34 F.encrypt15 F.theory 35 F.mit16 T.ronald 36 F.leiserson17 F.chaffing 37 F.adi18 F.winnowing 38 F.http_theory19 F.crypto 39 F.adleman20 F.and_winnowing 40 F.rivest_and
“Rivest” community rules
Web communities summary
• Approximate method gives promising results• Exact method should be practical as well• Both methods can be easily generalized• Applications are numerous and exciting
– Building a better web directory– Focused search engines– Filtering undesirable content
• Complements text-based methods
Focused crawling
Focused crawling
• Analyzing the web graph can help locate pages on a specific topic
• Typical crawler considers only the links on the current page
• Graph based focused crawler learns the context of the web graph where relevant pages appear
• Significant performance improvements
Focused crawling
CiteSeer
CiteSeer
• Digital library for scientific literature• Aims to improve communication and progress in science• Autonomous Citation Indexing, citation context extraction,
distributed error correction, citation graph analysis, etc. • Helps researchers obtain a better perspective and
overview of the literature with citation context and new methods of locating related research
• Lower cost, wider availability, more up-to-date than competing citation indexing services
• Faster, easier, and more complete access to the literature can speed research, better direct research activities, and minimize duplication of effort
CiteSeer
• 575,000 documents• 6 million citations• 500,000 daily requests• 50,000 daily users
• Data for research available on request• [email protected]
Distribution of articles
SCI ResearchIndex
Citations over time
Citations over time
• Conference papers and technical reports play a very important role in computer science research– Citations to very recent research are dominated by these
types of articles
• When recent journal papers are cited they are typically “in press” or “to appear”
• The most cited items tend to be journal articles and books
• Conference and technical report citations tend to be replaced with journal and book citations over time– May not be a one-to-one mapping
Online or invisible?
Online or invisible?
• Analyzed 119,924 conference articles from DBLP• Online articles cited 4.5 times more than offline
articles on average• Online articles more highly cited because
– They are easier to access and thus more visible, or– Because higher quality articles are more likely to be made
available online?
• Within venues: online articles cited 4.4 times more on average– Similar when restricted to top-tier conferences
Persistence of URLs
• Analyzed URLs referenced within articles in CiteSeer
• URLs per article increasing
• Many URLs now invalid– 1999 - 23%– 1994 - 53%
Persistence of URLs
• 2nd searcher found 80% of URLs the 1st searcher could not find
• Only 3% of URLs could not be found after 2nd searcher
How important are the “lost” URLs?
• With respect to the ability of future research to verify and/or build on the given paper
After 1st searcher After 2nd searcher
Persistence of URLs
• Many URLs now invalid• Can often relocate information• No evidence that information very important to future
research has been lost yet• Citation practices suggest more information will be
lost in the future unless these practices are improved• A widespread and easy to use web with invalid links
may be more useful than an improved system without invalid links but with added complexity or overhead
Extracting knowledge from the web
• Unprecedented opportunity for automated analysis of a large sample of interests and activity in the world
• Many methods for extracting knowledge from the web– Random sampling and analysis of pages and
hosts– Analysis of link structure and link growth
Extracting knowledge from the web
• Variety of information can be extracted– Distribution of interest and activity in different
areas– Communities related to different topics– Competition in different areas– Communication between different communities
Collaborators
• Web communities: Gary Flake, Lee Giles, Frans Coetzee
• Link growth modeling: David Pennock, Gary Flake, Lee Giles, Eric Glover
• Hostgraph modeling: Krishna Bharat, Bay-Wei Chang, Matthias Ruhl
• Web page sampling: Allan Heydon, Michael Mitzenmacher, Mark Najork
• Host sampling: Lee Giles• CiteSeer: Kurt Bollacker, Lee Giles
More information
• http://www.henzinger.com/monika/• http://www.neci.nec.com/~lawrence/• http://citeseer.org/