text specificity and impact on quality of news summaries annie louis & ani nenkova university of...
Post on 19-Dec-2015
218 views
TRANSCRIPT
Text Specificity and Impact on Quality of
News Summaries
Annie Louis & Ani NenkovaUniversity of Pennsylvania
June 24, 2011
Texts are a mix of general and specific sentences
Recently, we have developed a classifier that can distinguish general vs. specific sentences
The notion of specificity could be useful for a number of applications In this work, we consider automatic summarization Summaries cannot include all specific content because
of the space constraint Understand the role of general/specific content in
summaries and how it impacts quality
Specificity: amount of detail
2
Seismologists said the volcano had plenty of built-up magma and even more severe eruptions could come later. [overview]
The volcano's activity -- measured by seismometers detecting slight earthquakes in its molten rock plumbing system -- is increasing in a way that suggests a large eruption is imminent, Lipman said.
[details]
Example general and specific sentences
3
Prior studies of general-specific content in summaries Humans use generalization and specification of
source sentences to create abstract sentences
One generation task is to fuse information from key (general) sentence and specific sentence on the same topic to create an abstract sentence
Subtitles of news broadcasts are often generalized compared to original text
4
[Jing & McKeown (2000)]
[Wan et al. (2008)]
[Marsi et al. (2010)]
Overview of our study Quantitative analysis of specificity in inputs and
summaries using a general/specific classifier
1. Human abstracts have much more general content than system extracts
2. Amount of specific content is related to content quality of system summaries More general ~ better
3. Preliminary study on properties of summary-worthy general sentences 5
Data: DUC 2002 Generic multidocument summarization task
59 input sets 5 to 15 news documents
3 types of summaries 200 words Manually assigned content and linguistic quality scores
1. Humanabstracts
6
2. Humanextracts
3. Systemextracts
2 assessors * 59 2 assessors * 59 9 systems * 59
General vs. specific sentence classifier: prior work
7
Sentence level
Features1. Words2. Named entities, numbers3. Likelihood under language model4. Word specificity5. Adjectives/adverbs, length of phrases6. Polar words7. Sentence length
Training Binary: General or specific Logistic regression: can get probability for a class
[Louis & Nenkova (2011)]
Classification performance
8
75% accurate Validated on human annotations On examples with high annotator agreement – 90%
The probability is indicative of annotator agreement on class Sentences with high agreement ~ high confidence
predictions
Computing specificity for a text Sentences in summary are of varying length, so
we compute a score on word level “Average specificity of words in the text”
9
S1: w12w11 …w13
S2: w22w21 …w23
S3: w32w31 …w33
Confidence for beingin specific class
0.23
0.81
0.680.68 0.68 0.68 0.68
0.23 0.23 0.23 0.23
0.81 0.81 0.81 0.81
Average score on tokens
Specificity score
Average specificity of different types of summaries
1. More general content is preferred in abstracts
2. Simply the process of extraction makes summaries more specific
3. System summaries are overly specific
10
0.7 0.80.6Inputs (0.65)
H. Abs (0.62)
S.ext (0.74)
H.ext (0.72)
specific
Is the difference related to summary quality?
general
Analysis of ‘system summaries’: specificity and quality
1. Content quality Importance of content included in the summary More general ~ better
2. Linguistic quality How well-written the summary is perceived to be More specific ~ better
3. Quality of general/specific summaries When a summary is intended to be general or specific
11
1. Specificity and content quality Coverage score: manually judged at NIST
Similarity to a human summary
Correlation with specificity -0.169 (p-value 0.0006)
More specific ~ decreased content quality
12
But the correlation is not very high Specificity is related to realization of content
Different from importance of the content
Content quality = content importance + appropriate specificity level
Content importance: ROUGE scores N-gram overlap of system summary and human summary Standard evaluation of automatic summaries
13
System summary quality: Specificity as one of the predictors Coverage score ~ ROUGE-2 (bigrams) + specificity
Linear regression
Weights for predictors in the regression model
14
Mean β Significance (hypothesis β = 0)
(Intercept) 0.212 2.3e-11
ROUGE-2 1.299 < 2.0e-16
Specificity -0.166 3.1e-05
Is the combination a better predictor than ROUGE alone?
2. Specificity and linguistic quality Used different data: TAC 2009
DUC 2002 only reported number of errors Were also specified as a range: 1-5 errors
TAC 2009 linguistic quality score Manually judged: scale 1 – 10 Combines different aspects
coherence, referential clarity, grammaticality, redundancy
15
System summaries: What is the avg specificity in different score categories?
More general ~ lower score! General content
is useful but need proper context!
16
Ling score No. summaries
Poor (1, 2) 202
Mediocre (5) 400
Best (9, 10) 79
If a summary starts as follows:“We are quite a ways from that, actually.”As ice and snow at the poles melt, …
Specificity = lowLinguistic quality = 1
Average specificity
0.71
0.72
0.77
3. Specificity and quality of general/specific summaries DUC 2005: General-specific summary task
Create general summaries for some inputs, specific summaries for others
How specificity is related to scores of these summaries?
17
System summaries: Correlation between specificity and content scores
Further hints that specificity alone is not predictive of summary quality Once a summary is general, level of generality is not
longer predictive of quality
18
Summary type
Pearson correlation
General -0.03
Specific 0.18*
Content scores were measured using the pyramid method
Analysis of general sentences in human summaries1. Generalization operation performed in human
abstracts Frequency of operations, amount of deletions
2. How general sentences are used in human extracts Position, type of sentence
19
Data for analysing generalization operation Aligned pairs of abstract and source sentences
conveying the same content Traditional data used for compression experiments
Ziff-Davis corpus 15964 sentence pairs used in Galley & McKeown, 2007 Any number of deletions, up to 7 substitutions
Only 25% abstract sentences are mapped But beneficial to observe the trends
20
Generalization operation in human abstracts
Transition
SS
SG
GG
GS
21
One-third of all transformations are specific to general
Human abstracts involve a lot of generalization
No. pairs % pairs
6371 39.9
5679 35.6
3562 22.3
352 2.2
How specific sentences get converted to general?
SG
SS
GG
GS
22
Orig. length
33.5
33.4
21.5
22.7
New/orig length
40.8
56.6
60.8
66.0
Avg. deletions(words)
21.4
16.3
9.3
8.4
Choose long sentences and compress heavily!
A measure of generality would be useful to guide compression Currently only importance and grammaticality are used
Use of general sentences in human extracts Details of Maxwell’s death were sketchy. Folksy was an understatement. “Long live democracy!” Instead it sank like the Bismarck.
Example use of a general sentence in a summary…With Tower’s qualifications for the job, the nominations
should have sailed through with flying colors. [Specific]Instead it sank like the Bismarck. [General]……
Simple categorization 75 top general sentences according to classifier
confidence
24
Type
First sentence
Last sentence
Attributions
Comparisons
General sentences are used as topic/ emphasis sentences
Proportion
6 (8%)
13 (17%)
14 (18%)
4 (5%)
Conclusion General sentences are useful content for
summaries People use them in summaries for emphasis and topic
They can improve the content quality Choosing good general sentences or generating them
will be an interesting task
But linguistic quality should also be considered General sentences difficult to understand out of context Content planning should consider the order of general
content
Thank you
26
Histogram of specificity scores