summarization and generation
DESCRIPTION
Summarization and Generation. CS 4705. What is Summarization?. Data as input (database, software trace, expert system), text summary as output Text as input (one or more articles), paragraph summary as output Multimedia in input or output - PowerPoint PPT PresentationTRANSCRIPT
1
Summarization and Summarization and GenerationGeneration
CS 4705
2
What is Summarization?What is Summarization?What is Summarization?What is Summarization? Data as input (database, software trace, expert
system), text summary as output
Text as input (one or more articles), paragraph summary as output
Multimedia in input or output
Summaries must convey maximal information in minimal space
3
Summarization is not the same Summarization is not the same as Language Generationas Language Generation Karl Malone scored 39 points Friday
night as the Utah Jazz defeated the Boston Celtics 118-94.
Karl Malone tied a season high with 39 points Friday night….
… the Utah Jazz handed the Boston Celtics their sixth straight home defeat 118-94.
Streak, Jacques Robin, 1993
4
Summarization TasksSummarization Tasks
Linguistic summarization: How to pack in as much information as possible in as short an amount of space as possible?
Streak: Jacques Robin MAGIC: James Shaw PLanDoc: Karen Kukich, James Shaw, Rebecca
Passonneau, Hongyan Jing, Vasilis Hatzivassiloglou
Conceptual summarization: What information should be included in the summary?
5
Input Data -- STREAKInput Data -- STREAKInput Data -- STREAKInput Data -- STREAK
score (Jazz, 118)score (Celtics, 94)
The Utah Jazz beat theCeltics 118 - 94.
points (Malone, 39) Karl Malone scored 39points
location(game,Boston)
It was a home gamefor the Celtics
#home-defeats(Celtics, 6)
It was the 6th straighthome defeat
6
Revision ruleRevision ruleRevision ruleRevision rule
beatbeat
JazzJazz CelticsCeltics
handhand
JazzJazz defeatdefeat CelticsCeltics
7
SUMMONS QUERY OUTPUT
Summary:
Wednesday, April 19, 1995, CNN reported that anexplosion shook a government building inOklahoma City. Reuters announced that at least 18people were killed. At 1 PM, Reuters announcedthat three males of Middle Eastern origin werepossibly responsible for the blast. Two days later,Timothy McVeigh, 27, was arrested as a suspect,U.S. attorney general Janet Reno said. As of May29, 1995, the number of victims was 166.
Image(s):
1 (okfed1.gif) (WebSeek)
Article(s):(1) Blast hits Oklahoma Citybuilding
(2) Suspects' truck said rented from Dallas
(3) At least 18 killed in bombblast - CNN
(4) DETROIT (Reuter) - A federal judgeMonday ordered James
(5) WASHINGTON (Reuter) - Asuspect in the Oklahoma Citybombing
Summons, Dragomir Radev, 1995
8
BriefingsBriefingsBriefingsBriefings
Transitional Automatically summarize series of articles Input = templates from information extraction Merge information of interest to the user from
multiple sources Show how perception changes over time Highlight agreement and contradictions
Conceptual summarization: planning operators Refinement (number of victims) Addition (Later template contains perpetrator)
9
How is summarization done?How is summarization done?How is summarization done?How is summarization done?
4 input articles parsed by information extraction system
4 sets of templates produced as output Content planner uses planning operators
to identify similarities and trends Refinement (Later template reports new # victims)
New template constructed and passed to sentence generator
10
Sample TemplateSample TemplateSample TemplateSample Template
Message ID TST-COL-0001
Secsource: source ReutersSecsource: date 26 Feb 93
Early afternoonIncident: date 26 Feb 93Incident: location World Trade CenterIncident:Type BombingHum Tgt: number At least 5
11
Document SummarizationDocument Summarization
Input: one or more text documents
Output: paragraph length summary
Sentence extraction is the standard method Using features such as key words, sentence position in document, cue phrases Identify sentences within documents that are salient Extract and string sentences together Luhn – 1950s Hovy and Lin 1990s Schiffman 2000
Machine learning for extraction Corpus of document/summary pairs Learn the features that best determine important sentences Kupiec 1995: Summarization of scientific articles
12
Summarization ProcessSummarization Process
Shallow analysis instead of information extraction
Extraction of phrases rather than sentences
Generation from surface representations in place of semantics
13
Problems with Sentence Problems with Sentence ExtractionExtraction Extraneous phrases
“The five were apprehended along Interstate 95, heading south in vehicles containing an array of gear including … ... authorities said.”
Dangling noun phrases and pronouns “The five”
MisleadingWhy would the media use this specific word
(fundamentalists), so often with relation to Muslims? *Most of them are radical Baptists, Lutheran and Presbyterian groups.
14
Cut and PasteCut and Paste in Professional in Professional SummarizationSummarization
Humans also reuse the input text to produce summaries
But they “cut and paste” the input rather than simply extract our automatic corpus analysis
300 summaries, 1,642 sentences 81% sentences were constructed by cutting
and pasting linguistic studies
15
Major Cut and Paste Major Cut and Paste OperationsOperations (1) Sentence reduction
~~~~~~~~~~~~
16
Major Cut and Paste Major Cut and Paste OperationsOperations (1) Sentence reduction
~~~~~~~~~~~~
17
Major Cut and Paste Major Cut and Paste OperationsOperations (1) Sentence reduction
(2) Sentence Combination
~~~~~~~~~~~~
~~~~~~~~~~~~~~ ~~~~~~
18
Major Cut and Paste Major Cut and Paste OperationsOperations (3) Syntactic Transformation
(4) Lexical paraphrasing
~~~~~
~~~~~~~~~~~
~~~~~
~~~
19
Summarization at ColumbiaSummarization at Columbia
News: Newsblaster, GALE Email Meetings Journal articles Open-ended question-answering
What is a Loya Jurga? Who is Mohammed Naeem Noor Khan? What do people think of welfare reform?
20
Summarization at ColumbiaSummarization at Columbia
News Single Document Multi-document
Email Meetings Journal articles Open-ended question-answering
What is a Loya Jurga? Who is Al Sadr? What do people think of welfare reform?
21
Cut and Paste Based Single Document Cut and Paste Based Single Document Summarization -- System ArchitectureSummarization -- System Architecture
Extraction
Sentence reduction
Generation
Sentence combination
Input: single document
Extracted sentences
Output: summary
Corpus
Decomposition
Lexicon
Parser
Co-reference
22
(1) Decomposition of Human-(1) Decomposition of Human-written Summary Sentenceswritten Summary Sentences
Input: a human-written summary sentence the original document
Decomposition analyzes how the summary sentence was constructed
The need for decomposition provide training and testing data for
studying cut and paste operations
23
Sample Decomposition OutputSample Decomposition OutputSummary sentence: Arthur B. Sackler, vice
president for law and public policy of Time Warner Cable Inc. and a member of the direct marketing association told the Communications Subcommittee of the Senate Commerce Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that
makes the Internet unique.
Document sentences:S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the Direct Marketing Association told a Senate panel Thursday.S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the Communications subcommittee of the SenateCommerce Committee.S5: The subcommittee is considering the Children’s Online Privacy Act, which was drafted…
24
Decomposition of human-written Decomposition of human-written summariessummaries
Summary sentence: Arthur B. Sackler, vice
president for law and public policy of Time Warner Cable Inc. and a member of the direct marketing association told the Communications Subcommittee of the Senate Commerce Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that
makes the Internet unique.
Document sentences:S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the Direct Marketing Association told a Senate panel Thursday.S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the Communications subcommittee of the SenateCommerce Committee.S5: The subcommittee is considering the Children’s Online Privacy Act, which was drafted…
A Sample Decomposition OutputA Sample Decomposition Output
25
Decomposition of human-written Decomposition of human-written summariessummaries
Summary sentence: Arthur B. Sackler, vice
president for law and public policy of Time Warner Cable Inc. and a member of the direct marketing association told the Communications Subcommittee of the Senate Commerce Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that
makes the Internet unique.
Document sentences:S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the Direct Marketing Association told a Senate panel Thursday.S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the Communications subcommittee of the SenateCommerce Committee.S5: The subcommittee is considering the Children’s Online Privacy Act, which was drafted…
A Sample Decomposition OutputA Sample Decomposition Output
26
Decomposition of human-written Decomposition of human-written summariessummaries
Summary sentence: Arthur B. Sackler, vice
president for law and public policy of Time Warner Cable Inc. and a member of the direct marketing association told the Communications Subcommittee of the Senate Commerce Committee that legislation to protect children’s privacy on-line could destroy the spondtaneous nature that
makes the Internet unique.
Document sentences:S1: A proposed new law that would require web publishers to obtain parental consent before collecting personal information from children could destroy the spontaneous nature that makes the internet unique, a member of the Direct Marketing Association told a Senate panel Thursday.S2: Arthur B. Sackler, vice president for law and public policy of Time Warner Cable Inc., said the association supported efforts to protect children on-line, but he…S3: “For example, a child’s e-mail address is necessary …,” Sackler said in testimony to the Communications subcommittee of the SenateCommerce Committee.S5: The subcommittee is considering the Children’s Online Privacy Act, which was drafted…
A Sample Decomposition OutputA Sample Decomposition Output
27
Algorithm for DecompositionAlgorithm for Decomposition A Hidden Markov Model based solution Evaluations:
Human judgements 50 summaries, 305 sentences 93.8% of the sentences were decomposed
correctly Summary sentence alignment Tested in a legal domain
Details in (Jing&McKeown-SIGIR99)
28
(2) Sentence Reduction(2) Sentence Reduction
An example:Original Sentence: When it arrives sometime next year in
new TV sets, the V-chip will give parents a new and potentially revolutionary device to block out programs they don’t want their children to see.
Reduction Program: The V-chip will give parents a new and potentially revolutionary device to block out programs they don’t want their children to see.
Professional: The V-chip will give parents a device to block out programs they don’t want their children to see.
29
Algorithm for Sentence Algorithm for Sentence ReductionReduction Preprocess: syntactic parsing Step 1: Use linguistic knowledge to decide
what phrases MUST NOT be removed Step 2: Determine what phrases are
most important in the local context Step 3: Compute the probabilities of
humans removing a certain type of phrase
Step 4: Make the final decision
30
Step 1: Use linguistic knowledge Step 1: Use linguistic knowledge to decide what MUST NOT be to decide what MUST NOT be removedremoved
Syntactic knowledge from a large-scale, reusable lexicon convince: meaning 1: NP-PP :PVAL (“of”) (E.g., “He convinced me of his innocence”)
NP-TO-INF-OC (E.g., “He convinced me to go to the party”)
meaning 2: ... Required syntactic arguments are not
removed
31
Saudi Arabia on Tuesday decided to sign… The official Saudi Press Agency reported
that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
The meeting was called in response to … the Saudi foreign minister, that the Kingdom…
An account of the Cabinet discussions and decisions at the meeting…
The agency...
Step 2: Determining context Step 2: Determining context importance based on lexical importance based on lexical linkslinks
32
Saudi Arabia on Tuesday decided to sign… The official Saudi Press Agency reported
that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
The meeting was called in response to … the Saudi foreign minister, that the Kingdom…
An account of the Cabinet discussions and decisions at the meeting…
The agency...
Step 2: Determining context Step 2: Determining context importance based on lexical importance based on lexical links links
33
Saudi Arabia on Tuesday decided to sign… The official Saudi Press Agency reported
that King Fahd made the decision during a cabinet meeting in Riyadh, the Saudi capital.
The meeting was called in response to … the Saudi foreign minister, that the Kingdom…
An account of the Cabinet discussions and decisions at the meeting…
The agency...
Step 2: Determining context Step 2: Determining context importance based on lexical importance based on lexical links links
34
Step 3: Compute probabilities Step 3: Compute probabilities of humans removing a phraseof humans removing a phrase
verb(will give)
vsubc(when)
subj(V-chip)
iobj(parents)
obj(device)
ndet(a)
adjp(and)
lconj(new)
rconj(revolutionary)
Prob(“when_clause is removed”| “v=give”)
Prob (“to_infinitive modifier is removed” | “n=device”)
35
Step 4: Make the final Step 4: Make the final decisiondecision
verb(will give)
vsubc(when)
subj(V-chip)
iobj(parents)
ndet(a)
adjp(and)
lconj(new)
rconj(revolutionary)
L Cn Pr
L Cn Pr
L Cn Pr L Cn Pr L Cn Pr
L Cn Pr
L Cn Pr L Cn Pr
obj(device)
L Cn Pr
L -- linguisticCn -- contextPr -- probabilities
36
Evaluation of ReductionEvaluation of Reduction
Success rate: 81.3% 500 sentences reduced by humans Baseline: 43.2% (remove all the clauses,
prepositional phrases, to-infinitives,…)
Reduction rate: 32.7% Professionals: 41.8%
Details in (Jing-ANLP00)
37
Multi-Document Summarization Multi-Document Summarization Research FocusResearch Focus Monitor variety of online information sources
News, multilingual Email
Gather information on events across source and time
Same day, multiple sources Across time
Summarize Highlighting similarities, new information, different
perspectives, user specified interests in real-time
38
ApproachApproach
Use a hybrid of statistical and linguistic knowledge
Statistical analysis of multiple documents Identify important new, contradictory information
Information fusion and rule-driven content selection
Generation of summary sentences By re-using phrases Automatic editing/rewriting summary
39
NewsblasterNewsblaster
http://newsblaster.cs.columbia.edu/
Clustering articles into events Categorization by broad topic Multi-document summarization Generation of summary sentences
Fusion Editing of references
40
Newsblaster Newsblaster ArchitectureArchitecture
Crawl News SitesCrawl News Sites
Form ClustersForm Clusters
CategorizeCategorize
TitleClusters
TitleClusters
SummaryRouter
SummaryRouter
SelectImagesSelectImages
EventSummary
EventSummary
BiographySummaryBiographySummary
Multi-EventMulti-Event
Convert Output to HTMLConvert Output to HTML
41
42
Fusion Fusion
43
Sentence Fusion ComputationSentence Fusion Computation
Common information identification Alignment of constituents in parsed theme sentences: only some
subtrees match Bottom-up local multi-sequence alignment Similarity depends on
Word/paraphrase similarity Tree structure similarity
Fusion lattice computation Choose a basis sentence Add subtrees from fusion not present in basis Add alternative verbalizations Remove subtrees from basis not present in fusion
Lattice linearization Generate all possible sentences from the fusion lattice Score sentences using statistical language model
44
45
46
Tracking Across DaysTracking Across Days
Users want to follow a story across time and watch it unfold
Network model for connecting clusters across days Separately cluster events from today’s news Connect new clusters with yesterday’s news Allows for forking and merging of stories
Interface for viewing connections Summaries that update a user on what’s new
Statistical metrics to identify differences between article pairs Uses learned model of features Identifies differences at clause and paragraph levels
47
48
49
50
51
Different PerspectivesDifferent Perspectives
Hierarchical clustering Each event cluster is divided into clusters by
country
Different perspectives can be viewed side by side
Experimenting with update summarizer to identify key differences between sets of stories
52
53
54
55
Multilingual SummarizationMultilingual Summarization
Given a set of documents on the same event
Some documents are in English
Some documents are translated from other languages
56
Issues for Multilingual Issues for Multilingual SummarizationSummarization Problem: Translated text is errorful
Exploit information available during summarization
Similar documents in cluster
Replace translated sentences with similar English
Edit translated text Replace named entities with extractions from similar English
57
Multilingual RedundancyMultilingual RedundancyBAGDAD. - A total of 21 prisoners has been died and a hundred more hurt by firings from mortar in the jail of Abu Gharib (to 20 kilometers to the west of Bagdad), according to has informed general into the U.S.A. Marco Kimmitt.
Bagdad in the Iraqi capital Aufstaendi attacked Bagdad on Tuesday a prison with mortars and killed after USA gifts 22 prisoners. Further 92 passengers of the Abu Ghraib prison were hurt, communicated a spokeswoman of the American armed forces.
The Iraqi being stationed US military shot on the 20th, the same day to the allied forces detention facility which is in アブグレイブ of the Baghdad west approximately 20 kilometers, mortar 12 shot and you were packed, 22 Iraqi human prisoners died, it announced that nearly 100 people were injured.
BAGHDAD, Iraq – Insurgents fired 12 mortars into Baghdad's Abu Ghraib prison Tuesday, killing 22 detainees and injuring 92, U.S. military officials said.
Germ
anG
erman
Spanish
Spanish
JapaneseJapanese
English
English
58
Multilingual RedundancyMultilingual RedundancyBAGDAD. - A total of 21 prisoners has been died and a hundred more hurt by firings from mortar in the jail of Abu Gharib (to 20 kilometers to the west of Bagdad), according to has informed general into the U.S.A. Marco Kimmitt.
Bagdad in the Iraqi capital Aufstaendi attacked Bagdad on Tuesday a prison with mortars and killed after USA gifts 22 prisoners. Further 92 passengers of the Abu Ghraib prison were hurt, communicated a spokeswoman of the American armed forces.
The Iraqi being stationed US military shot on the 20th, the same day to the allied forces detention facility which is in アブグレイブ of the Baghdad west approximately 20 kilometers, mortar 12 shot and you were packed, 22 Iraqi human prisoners died, it announced that nearly 100 people were injured.
BAGHDAD, Iraq – Insurgents fired 12 mortars into Baghdad's Abu Ghraib prison Tuesday, killing 22 detainees and injuring 92, U.S. military officials said.
Germ
anG
erman
Spanish
Spanish
JapaneseJapanese
English
English
59
Multilingual Similarity-based Multilingual Similarity-based SummarizationSummarization
ستجوا ب
القالفDeutsch
erText
日本語テキスト
EnglishTextEnglish
TextEnglishText
Related English Documents
Documents on the same Event
MachineTranslation
ArabicText
German
Text
Japanese
Text
DetectSimilar
Sentences
Replace / rewrite MT sentences
EnglishSumma
ry
Select summary sentences
Simplify English
Sentences
60
Similarity ComputationSimilarity Computation
Simfinder computes similarity between sentences based on multiple features: Proper Nouns Verb, noun, adjective WordNet (synonyms) word stem overlap
New: Noun phrase and noun phrase variant feature
(FASTR)
61
Sentence 1Iraqi President Saddam Hussein that the government of Iraq
over 24 years in a "black" near the port of the northern Iraq after nearly eight months of pursuit was considered the largest in history .
Similarity 0.27: Ousted Iraqi President Saddam Hussein is in custody following his dramatic capture by US forces in Iraq.
Similarity 0.07: Saddam Hussein, the former president of Iraq, has been captured and is being held by US forces in the country.
Similarity 0.04: Coalition authorities have said that the former Iraqi president could be tried at a war crimes tribunal, with Iraqi judges presiding and international legal experts acting as advisers.
62
Sentence SimplificationSentence Simplification
Machine translated sentences long and ungrammatical
Use sentence simplification on English sentences to reduce to approximately “one fact” per sentence
Use Arabic sentences to find most similar simple sentences
Present multiple high similarity sentences
63
Simplification ExamplesSimplification Examples 'Operation Red Dawn', which led to the capture of
Saddam Hussein, followed crucial information from a member of a family close to the former Iraqi leader. ' Operation Red Dawn' followed crucial information from a
member of a family close to the former Iraqi leader. Operation Red Dawn led to the capture of Saddam Hussein.
Saddam Hussein had been the object of intensive searches by US-led forces in Iraq but previous attempts to locate him had proved unsuccessful. Saddam Hussein had been the object of intensive searches
by US-led forces in Iraq. But previous attempts to locate him had proved unsuccessful.
64
Results on alquds.co.uk.195Results on alquds.co.uk.195
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
F-measure vs. Threshold
Full Threshold .10Full Threshold .65Simplified Threshold .10Simplified Threshold .65FASTR Simplified .10FASTR Simplified .65
Threshold
F-m
ea
sure
68
EvaluationEvaluation
DUC (Document Understanding Conference): run by NIST
Held annually Manual creation of topics (sets of documents) 2-7 human written summaries per topic How well does a system generated summary
cover the information in a human summary?
69
User Study: ObjectivesUser Study: Objectives
Does multi-document summarization help?
Do summaries help the user find information needed to perform a report writing task?
Do users use information from summaries in gathering their facts?
Do summaries increase user satisfaction with the online news system?
Do users create better quality reports with summaries? How do full multi-document summaries compare with
minimal 1-sentence summaries such as Google News?
70
User Study: DesignUser Study: Design
Four parallel news systems Source documents only; no summaries Minimal single sentence summaries (Google
News) Newsblaster summaries Human summaries
All groups write reports given four scenarios A task similar to analysts Can only use Newsblaster for research Time-restricted
71
User Study: ExecutionUser Study: Execution
4 scenarios 4 event clusters each 2 directly relevant, 2 peripherally relevant Average 10 documents/cluster
45 participants Balance between liberal arts, engineering 138 reports
Exit survey Multiple-choice and open-ended questions
Usage tracking Each click logged, on or off-site
72
““Geneva” PromptGeneva” Prompt
The conflict between Israel and the Palestinians has been difficult for government negotiators to settle. Most recently, implementation of the “road map for peace”, a diplomatic effort sponsored by ……
Who participated in the negotiations that produced the Geneva Accord?
Apart from direct participants, who supported the Geneva Accord preparations and how?
What has the response been to the Geneva Accord by the Palestinians?
73
Measuring EffectivenessMeasuring Effectiveness
Score report content and compare across summary conditions
Compare user satisfaction per summary condition
Comparing where subjects took report content from
74
75
User SatisfactionUser Satisfaction
More effective than a web search with Newsblaster
Not true with documents only or single-sentence summaries
Easier to complete the task with summaries than with documents only
Enough time with summaries than documents only
Summaries helped most 5% single sentence summaries 24% Newsblaster summaries 43% human summaries
76
User Study: ConclusionsUser Study: Conclusions
Summaries measurably improve a news browswer’s effectiveness for research
Users are more satisfied with Newsblaster summaries are better than single-sentence summaries like those of Google News
Users want search Not included in evaluation
77
Email SummarizationEmail Summarization
Cross between speech and text Elements of dialog Informal language More context explicitly repeated than speech
Wide variety of types of email Conversation to decision-making
Different reasons for summarization Browsing large quantities of email – a mailbox Catch-up: join a discussion late and participate – a
thread
78
Email Summarization: ApproachEmail Summarization: Approach
Collected and annotated multiple corpora of email Hand-written summary, categorization threads&messages Identified 3 categories of email to address:
Event planning, Scheduling, Information gathering
Developed tools: Automatic categorization of email Preliminary summarizers
Statistical extraction using email specific features Components of category specific summarization
79
Email Summarization by Email Summarization by Sentence ExtractionSentence Extraction
Use features to identify key sentences Non-email specific: e.g., similarity to centroid Email specific: e.g., following quoted material
Rule-based supervised machine learning Training on human-generated summaries
Add “wrappers” around sentences to show who said what
80
Data for Sentence ExtractionData for Sentence Extraction Columbia ACM chapter executive board mailing list Approximately 10 regular participants ~300 Threads, ~1000 Messages Threads include: scheduling and planning of meetings
and events, question and answer, general discussion and chat.
Annotated by human annotators: Hand-written summary Categorization of threads and messages Highlighting important information (such as question-answer
pairs)
81
Email Summarization by Email Summarization by Sentence ExtractionSentence Extraction
Creation of Training Data Start with human-generated summaries Use SimFinder (a trained sentence similarity measure –
Hatzivassiloglou et al 2001) to label sentences in threads as important
Learning of Sentence Extraction Rules Use Ripper (a rule learning algorithm – Cohen 1996) to learn rules
for sentence classification Use basic and email-specific features in machine learning
Creating summaries Run learned rules on unseen data Add “wrappers” around sentences to show who said what
Results Basic: .55 precision; .40 F-measure Email-specific: .61 precision; .50 F-measure
82
Sample Automatically Generated Sample Automatically Generated Summary (ACM0100)Summary (ACM0100)
Regarding "meeting tonight...", on Oct 30, 2000, David Michael Kalin wrote: Can I reschedule my C session for Wednesday night, 11/8, at 8:00?
Responding to this on Oct 30, 2000, James J Peach wrote: Are you sure you want to do it then?
Responding to this on Oct 30, 2000, Christy Lauridsen wrote: David , a reminder that your scheduled to do an MSOffice session on Nov. 14, at 7pm in 252Mudd.
83
Information Gathering Email:Information Gathering Email:The ProblemThe ProblemSummary from our rule-based sentence extractor:
Regarding "acm home/bjarney", on Apr 9, 2001, Mabel Dannon wrote:Two things: Can someone be responsible for the press releases for Stroustrup?
Responding to this on Apr 10, 2001, Tina Ferrari wrote:I think Peter, who is probably a better writer than most of us, is writing up something for dang and Dave to send out to various ACM chapters. Peter, we can just use that as our "press release", right?
In another subthread, on Apr 12, 2001, Keith Durban wrote:Are you sending out upcoming events for this week?
84
Detection of QuestionsDetection of Questions
Questions in interrogative form: inverted subject-verb order
Supervised rule induction approach, training Switchboard, test ACM corpus
Results:
Recall low because:
Questions in ACM corpus start with a declarative clause So, if you're available, do you want to come? if you don't mind, could you post this to the class bboard?
Results without declarative-initial questions:
Recall 0.56
Precision 0.96
F-measure 0.70
Recall 0.72
Precision 0.96
F-measure 0.82
85
Detection of AnswersDetection of AnswersSupervised Machine Learning Approach Use human annotated data to generate gold standard
training data Annotators were asked to highlight and associate question-answer
pairs in the ACM corpus.
Learn a classifier that predicts if a subsequent segment to a question segment answers it Represent each question and candidate answer segment
by a feature vectorLabeller 1
Labeller 2
Union
Precision 0.690 0.680 0.728
Recall 0.652 0.612 0.732
F1-Score 0.671 0.644 0.730
86
Integrating QA detection with Integrating QA detection with summarizationsummarization Use QA labels as features in sentence
extraction (F=.545) Add automatically detected answers to
questions in extractive summaries (F=.566)
Start with QA pair sentences and augmented with extracted sentences (F=.573)
87
Integrated in Microsoft OutlookIntegrated in Microsoft Outlook
88
Meeting SummarizationMeeting Summarization(joint with Berkeley, SRI, Washington)(joint with Berkeley, SRI, Washington)
Goal: automatic summarization of meetings by generating “minutes” highlighting the debate that affected each decision.
Work to date: Identification of agreement/disagreement Machine learning approach: lexical, structure,
acoustic features Use of context: who agreed with who so far?
Adressee identification Bayesian modeling of context
89
ConclusionsConclusions
Non-extractive summarization is practical today
User studies show summarization improves access to needed information
Advances and ongoing research in tracking events, multilingual summarization, perspective identification
Moves to new media (email, meetings) raise new challenges with dialog, informal language