News Article Ranking:Leveraging the Wisdom of Bloggers

Richard McCreadie, Craig Macdonald & Iadh Ounis

Introduction

• Editorial News:• Every day newspaper editors select

articles for placement within their newspapers.

• This can be seen as a ranking problem.

• Rank articles by readership interest

We investigate how such a ranking can be approximated using evidence from the blogosphere

NewspaperEditor

FrontPage

Page2

. . .

Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions

Talk Outline

News Article Ranking

Problem Definition:• Rank news articles by their inherent

importance.• Given a day of interest dQ we wish to

score each news article a by its predicted importance, score(a,dQ) using evidence from the blogosphere.

News ArticleRanker

=29

=23

=14

=13

=4

=4

ImportanceScores

Day dQ

Idea:• The more blog posts about an article the more important the subject

must be.• Score by blog post volume

ApproachTwo Stages:

1. Score each news article a for all days d based on related blog post volume for day d.

News articles are represented by their headlines

2. Given a query day dQ rank A based on the score for each news article on day dQ, i.e. score(a, dQ)

-> a voting process

The Votes Approach

Votes Approach : Stage 1

Ranking of days for a

For eachnews article a

blog postranking

Votes

2) Select the top 1000 blog posts

for a

Stage 1: Score days for each news story

Days3) Each post votes for a day

votes = 1

2

1

3

4

votes = 2

votes = 0

votes = 2

votes = 1

2

1

3

4

votes = 2

votes = 0

votes = 2

4) Rank days by votes received

Voting Model : Count* Craig Macdonald PhD thesis 2009

Terrier

1) Use its representation

(headline) as a query

Votes Approach : Stage 2

QueryDay 2

Ranking of Articles

Stage 2: Rank news articles for day dQ

News article a1

Stage 1

votes = 1

2 votes = 2

votes = 0

votes = 24

1

3

News article a2

News article a3

votes = 3

4 votes = 6

votes = 1

votes = 23

1

2

votes = 5

1 votes = 9

votes = 0

votes = 73

2

4

2 votes = 2

votes = 12

votes = 52

votes = 3

4 votes = 6

votes = 23

1

1 votes = 9

votes = 0

votes = 73

4

votes = 1

votes = 0

votes = 24

1

3

News article a2

News article a3

Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions

Talk Outline

Hypothesis:• The volume of relevant blog posts published on a news article is a strong indicator

of that articles importance (from an editors perspective).

Research Questions:• Can the number of related blog posts to a news article published on day dQ provide

a comparative ranking to that which an editor might make?

Evaluating Votes

Setup :• TREC 2009 Blog track top news stories identification task• 100k news headlines from the New York Times to represent articles

• E.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’

• Uses blog posts from the Blogs08 blog post corpus (28 million posts)• Judgments for 50 days of interest (dQ’s)

• E.g. 2008-05-22 : headline1 headline34 headline35 headline38

Evaluation:• Mean Average Precision (MAP)

Experimental Setup

dQ Important headlines on dQ

Indexing & Retrieval:• Indexed Blogs08 using Terrier (stemming, stopwords)• Secondary index holds blog post -> day relations• Retrieve 1000 blog posts for headlines.

• DPH (DFR)• BM25

Baselines:• Random ranking : average over 10 runs• Inlinks : hyperlink evidence• TREC 2009 best systems

Experimental Setup

Votes Performance

Results:

Rando

mIn

links

uogT

rTStim

es

KLEClus

Prior

IlpsT

SExp

BM25

+Vot

es

DPH+Vot

es0

0.020.040.060.080.1

0.120.140.160.180.2

MAP

Hyperlink evidence is of less value than textual

evidence

Better performance than TREC 2009 best

systems BM25<DPH (DFR)Votes + extras

TREC 2009 Best Systems Votes Approach

Conclusions:• Blog post volume is a decent indicator of editorial importance• Can be effectively leveraged to rank news articles by their importance• However, still room for improvement (0.17 map)

Votes Performance

How can we improve Votes performance?

Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions

Talk Outline

Idea• Re-score for each news article using evidence from days before and after dQ.

Intuition• Important stories will be discussed before or after the event

E.g. Run up to an election

Temporal Promotion

1st 2nd 3rd 4th 5th 6th0

1

2

3

4

5

News Article aNews Article b

Days

NumVotes

dQBoth articles receive the same score for dQ under Votes

Hypothesis:• An article which is highly blogged about either before or after dQ should be scored

more highly than one which is not.

Approach:• Promote articles which were highly blogged before or after dQ

• Two Techniques• NDayBoost• GaussBoost

Temporal Promotion

1st 2nd 3rd 4th 5th 6th0

1

2

3

4

5

News Article aNews Article b

Approach• Linearly combines the scores for day dQ with the n days before or after dQ.

NDayBoost

Days

NumVotes

dQ

N = -2

Score=11

Score=6

Idea:• Evidence will weaken as the distance from dQ increases• NDayBoost might over-estimate the importance of days distant from dQ

Approach:• Linearly combine scores as with NDayBoost, but weight each day by its distance

from dQ using a Gaussian curve.

GaussBoost

Distance of days ∆d

Weight

GaussBoost

0 1 2 3 4 5 6 7 8 90

0.2

0.4

0.6

0.8

1

w=0.5w=1w=1.5w=2w=3

∆d

Weight

Weighting• The weight for each article is calculated as : • ∆d is the distance (in days) from dQ • w is the width of the gaussian curve

• Controls the score decay as ∆d increases

1st 2nd 3rd 4th 5th 6th0

1

2

3

4

5

News Article aNews Article b

GaussBoost

Days

NumVotes

dQ

N = -2

Score=11

ScoreGaussBoost(A,4) = (1*4)+(0.79*4)+(0.18*3)

= 7.700

ScoreGaussBoost(B,4) = (1*4)+(0.79*1)+(0.18*1) = 4.970

Score=6

Score=7.700

Score=4.970

Example:• n = -2, w = 1• Weights downward the scores for each day dependent on w.

Hypothesis:• An article which is highly blogged about either before or after dQ is more likely to

be important than one which is not.

Research Questions:• Can the promotion of articles which are highly blogged about before or after dQ

improve article ranking performance?• Does the quality of evidence decrease as distance from dQ increases?• Is historical or future (before or after dQ) blog post evidence more useful?

Research Questions

10 -9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 100.1

0.11

0.12

0.13

0.14

0.15

0.16

0.17

0.18

0.19

0.2

NDayBoostDPH+Votes

NDayBoost PerformanceFuture blog postings does provide useful evidence

Baseline DPH+Votes

Historical evidence is not useful for NDayBoost

n value (days)

MAP

-10

-9 -8 -7 -6 -5 -4 -3 -2 -1 1 2 3 4 5 6 7 8 9 100.14

0.15

0.16

0.17

0.18

0.19

0.2

GaussBoostDPH+Votes

GaussBoost PerformanceFuture blog postings provide stronger evidence than historical postings

Baseline DPH+Votes

w value (not days!)

MAP

Historical blog postings are useful for days close to dQ

• Conclusions• Both historical and future evidence is useful to improve Votes ranking performance

• Can use this evidence to generate a better ranking for editors if the data is available

• Future evidence is more powerful than historical evidence• Not too useful if we want to rank in real-time though

• NDayBoost is only effective for future evidence• GaussBoost is effective for both future and historical evidence

• The most effective of the techniques• Does not over emphasise evidence from days distant from dQ

Temporal Promotion

Introduction The News Article Ranking Problem The Votes Approach Evaluating Votes Temporal Promotion News Article Representation Conclusions

Talk Outline

Can we improve upon the news article representation?

Issue:• News articles are represented with headlines

• e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map

• Headlines are a sparse representation of an article• Many headlines are not `news-worthy’

• Editors don’t even consider these

• e.g. paid death notices

Approach:• Enrich the headlines using related terms extracted from blog posts and Wikipedia.• Prune headlines less likely to be news-worthy

Improving the Article Representation

News Article Enrichment

Idea:• Improve the news-article representation

(headline)• Add related terms (counter sparsity)

Approach:1. Select retrieve top 3 blog posts from:

• Blogs08 (query expansion , K. L. Kwok and M. S. Chan. SIGIR 1998)

• Wikipedia(collection enrichment, F. Diaz and D. Metzler. SIGIR 2006)

using DPH (DFR)2. Expand query with the top 10 terms

identified using Bo1 (G. Amati, Thesis 2003) from those documents.

Terriera

Blogs08/Wikipedia

DPH TopTerms

Bo1

Query expansion/External Query expansion/Collection Enrichment

Article Enrichment:• News headlines while being good quality representations are still ambigious• Collection enrichment helps find the blog posts that are related.

Article Improvement Performance

DPH+Votes Query Expansion Collection Enrichment0.165

0.17

0.175

0.18

0.185

0.19

0.195 Collection enrichment with Wikipedia significantly increases performance

MAP

Article Pruning

Idea:• Editors have lots of latent knowledge to

draw upon• Try simulating this within the system• Prune away articles that an editor would

not even consider

Non-stories:• Remove news articles which follow

editorially defined patterns

Noisy headlines:• Remove misleading dates• Remove uppercase category terms

• Paid Notice• Corrections for the Record• Comments of the Week• Inside the Times• Best Sellers• The Week Ahead• Movie Review

Patterns List: New York Times

• Arts Briefly• The Listings• Dance Review• Whats on Today• Critics Choice• Book of the Times• Music Review

E.g. ‘Inside the Times, November 6, 2008’

E.g. ‘N.F.L. ROUNDUP; Giants Shut Down Tyree for Season; Raiders Cut Hall’

Article Pruning:• Removing non-news-worthy articles makes the ranking of articles easier.

Article Pruning Performance

DPH+Votes Patterns Dates UpperCase All Heuristics0.16

0.1650.17

0.1750.18

0.1850.19

0.1950.2

0.205Patterns significantly increase performance over Votes alone

Dates and Uppercase further increase performance when combined.

MAP

Additive Results?

Technique MAP

DPH+Votes 0.1742

+ GaussBoost 0.1907

+ All Heuristics 0.1996

+ Collection Enrichment 0.1899

All Techniques Combined

0.2210

Idea:• Combine

• Temporal promotion (GaussBoost)• Headline pruning (All Heuristics)• Headline enrichment (Collection

Enrichment)

Results:• Significant increase in performance

over• DPH+Votes• DPH+Votes + Single techniques

Votes:• The volume of blog posts about a news story is a useful measure for the importance from an editorial

perspective• Can be used to automatically rank news stories for a newspaper editor

• The Voting model provides strong baseline ranking performace

Temporal Promotion:• Can be beneficial to look at blog post volume either before or after the day of interest• More useful to look at tomorrows blog posts than yesterdays blog posts• Evidence diminishes as we look further from the day of interest, evidence should be weighted

accordingly

Article representation Improvements• Editors hold much in the way of latent knowledge that we need to simulate

• i.e. they can disregard whole classes of articles as not being news-worthy• By pruning away such articles apriori, ranking performance is improved• Headlines are sparse representations of news articles

• Enrichment with terms from Wikipedia can help find more representative blog posts

Conclusions

TREC 2010:• Blog track top stories identification task is running again in 2010• Focus on real-time ranking of news (no future evidence)• Uses a larger news article collection from Reuters

Future Work

Questions?