automatic disfluency detection in multi-party conversations...
TRANSCRIPT
![Page 1: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/1.jpg)
ww
w.a
mip
roje
ct.o
rgGerman Research Center forArtificial Intelligence GmbH
FeastFeast, 30th September 2009, 30th September 2009Sebastian Sebastian GermesinGermesin
Automatic DisfluencyAutomatic DisfluencyDetection in Multi-partyDetection in Multi-party
ConversationsConversations
![Page 2: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/2.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 092
German Research Center forArtificial Intelligence GmbH
OutlineOutline• Motivation• Theoretical Background• Data (AMI Corpus)• Disfluency Detection System
• Hybrid Classification Approach• Self-arranging Modules• Experimental Results
• Conclusions & Outlook
![Page 3: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/3.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 093
German Research Center forArtificial Intelligence GmbH
MotivationMotivationExampleExample
![Page 4: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/4.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 094
German Research Center forArtificial Intelligence GmbH
MotivationMotivation• Have to detect (and clean) disfluencies
in the transcribed speech• Readability
• Transcription• Extractive Summarization
• Post-Processing• NLP-systems’ performance drop when faced with
disfluent speech
• Human detector?• Too expensive!• Too slow!
⇒Automatic Detection System!
![Page 5: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/5.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 095
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“Disfluencies are syntactical and grammatical[speech] errors that occur in spoken but notin written language.” [Besser, 2006]
DefinitionDefinition
![Page 6: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/6.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 096
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“The cat uh the dog sneaks around the corner.”
TerminologyTerminology
![Page 7: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/7.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 097
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“The cat uh the dog sneaks around the corner.”
TerminologyTerminology
Reparandum
![Page 8: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/8.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 098
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“The cat uh the dog sneaks around the corner.”
TerminologyTerminology
Reparandum
Interregnum
![Page 9: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/9.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 099
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“The cat uh the dog sneaks around the corner.”
TerminologyTerminology
Reparandum Reparans
Interregnumcomplex
![Page 10: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/10.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0910
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical Background
“The d dog sneaks around the corner.”
TerminologyTerminology
Reparandum
simple
![Page 11: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/11.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0911
German Research Center forArtificial Intelligence GmbH
Theoretical BackgroundTheoretical BackgroundAll TypesAll Types
Simple disfluencies
![Page 12: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/12.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0912
German Research Center forArtificial Intelligence GmbH
DataData
• AMI meeting corpus• 135 meetings (~ 100 hours speech)• 4 participants• task: design a remote control• freely interaction• Many annotations, e.g.:
• Transcribed speech• Dialogue acts• Gestures• ...
quantitativequantitative
![Page 13: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/13.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0913
German Research Center forArtificial Intelligence GmbH
DataData
• 45 meeting enriched with disfluencyannotation• 31,000 Disfluencies• 15.8% erroneous words• 41.5% disfluent Dialogue Acts• 80% (33) for training• 20% (12) for evaluation
quantitativequantitative
![Page 14: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/14.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0914
German Research Center forArtificial Intelligence GmbH
DataData
• Discovered a heterogeneity towards thestrictness of different disfluency types1. Some disfluencies have strict structure
• ex.: Repetition : “The cat the cat plays “2. Some other disfluencies have also strict structure but
this structure is very common in natural language• ex.: Replacement : “The dog the cat plays“• ex.: Fluent : “The dog the cat and the bird play”
3. Some other disfluencies have no obvious structure• ex.: Disruptions : “The dog the cat and“• ex.: Order : “The plays cat”
qualitativequalitative
![Page 15: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/15.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0915
German Research Center forArtificial Intelligence GmbH
Automatic SystemAutomatic SystemDesign QuestionDesign Question
• Can we leverage the heterogeneityof disfluencies for their detection?→Yes!
→ Use modules for subsets of disfluencies→ Use different feature-sets for each module
(depending on the disfluency types)→ Find “optimal” classifier for each module
![Page 16: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/16.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0916
German Research Center forArtificial Intelligence GmbH
Automatic SystemAutomatic SystemHybrid ModulesHybrid Modules
• SHS:• Stuttering, Hesitation, Slip-of-the-Tongue
• REP:• Repetition
• DNE:• Discourse Marker, Explicit Editing Term
• DEL:• Deletion
• REV:• Insertion, Replacement, Restart, Other
![Page 17: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/17.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0917
German Research Center forArtificial Intelligence GmbH
How toHow to combine the modules?combine the modules?
![Page 18: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/18.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0918
German Research Center forArtificial Intelligence GmbH
Training ProcessTraining ProcessSelf-arranging ModulesSelf-arranging Modules
• Immense search space• #(modules) * #(classifier) * placeInSystem
• Solution(s):• Old system:
• Choosen manually• Current system:
• Automatically trained1.Use greedy hill-climbing
– Use weight for errors to improve Precision!2.Reduce classifier library
– Take 10% results in maximal performance lossof 2.3% (depending on the module)
![Page 19: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/19.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0919
German Research Center forArtificial Intelligence GmbH
GroDiGroDiGreedy Hill-ClimbingGreedy Hill-Climbing
![Page 20: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/20.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0920
German Research Center forArtificial Intelligence GmbH
Training ProcessTraining ProcessSelf-arranging ModulesSelf-arranging Modules
• Immense search space• #(modules) * #(classifier) * placeInSystem
• Solution(s):• Old system:
• Choosen manually• Current system:
• Automatically trained1.Use greedy hill-climbing
– Use weight for errors to improve Precision!2.Reduce classifier library
– Take 10% results in maximal performance lossof 2.3% (depending on the module)
![Page 21: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/21.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0921
German Research Center forArtificial Intelligence GmbH
GroDiGroDiPerformance-Curve of J48Performance-Curve of J48
Best: J48 "-L -U -M 2 -A"
![Page 22: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/22.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0922
German Research Center forArtificial Intelligence GmbH
Experimental ResultsExperimental Results
93.5 %94.5 %12 m.
94.7 %95.1 %6 m.33 m.
new 0.11
94.8 %95.3 %6 m.22 m.
0.4290.5 %92.9 %6 m.22 m.old
83.3 %88.6 %12 m.0.00
85.7 %90.3 %6 m.--baseline
RT-factoravg. F1AccuracyEval.data
Train.dataSystem
![Page 23: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/23.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0923
German Research Center forArtificial Intelligence GmbH
ConclusionsConclusions• Aims:
• Development of a system that automaticallydetects a broad set of disfluencies
• Fully automatic learning process• Robust and Fast
• Achievements:• Stand-alone tool for detection of disfluencies:
GroDi - Get rid of Disfluencies• Self-arranging modules• Detection rate: 95% Accuracy• Real-time factor of 0.11
![Page 24: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/24.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0924
German Research Center forArtificial Intelligence GmbH
OutlookOutlook
• Develop module(s) for the detection ofMistake, Order, Omission
• Embed other learning approaches, e.g.:• Conditional Random Fields• HMMs
• Use other corpus like, e.g., Switchboard
![Page 25: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/25.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0925
German Research Center forArtificial Intelligence GmbH
Thank you!Thank you!
![Page 26: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/26.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0926
German Research Center forArtificial Intelligence GmbH
Demo?Demo?
![Page 27: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/27.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0927
German Research Center forArtificial Intelligence GmbH
GroDiGroDiDiff. Module ArrangementsDiff. Module Arrangements
![Page 28: Automatic Disfluency Detection in Multi-party Conversations rgfeast.coli.uni-saarland.de/slides/GermesinS300909.pdf · strictness of different disfluency types 1.Some disfluencies](https://reader033.vdocuments.us/reader033/viewer/2022060307/5f09e7167e708231d4290d80/html5/thumbnails/28.jpg)
ww
w.a
mip
roje
ct.o
rg
Sebastian Germesin September 0928
German Research Center forArtificial Intelligence GmbH
GroDiGroDi
Used technologies WEKA toolkit for machine learning Maximum Entropy classifier from Stanford NLP group CRF Tagger from http://crftagger.sourceforge.net/
Features for machine learning: Lexical: words, lexical parallelism, (POS-Tags) Prosodic: duration, pauses, pitch, energy Dynamic: disfluency types of surrounding words Speaker: age, role in meeting, native language