a framework for benchmarking entity-annotation systems
DESCRIPTION
A Framework for Benchmarking Entity-Annotation Systems. Source : WWW’ 13 Authors : Macro Cornolti , Paolo Ferragina and Massimiliano Advisor : Dr . Jia -Ling, Koh Speaker : Wei Chang. Outline. Introduction Entity Annotation Systems Measurement Experiment Conclusion. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/1.jpg)
A Framework for Benchmarking Entity-Annotation Systems
Source : WWW’13Authors : Macro Cornolti, Paolo Ferragina
and MassimilianoAdvisor : Dr. Jia-Ling, KohSpeaker : Wei Chang
![Page 2: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/2.jpg)
Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion
![Page 3: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/3.jpg)
Entity Annotation System
Knowledge Base
![Page 4: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/4.jpg)
Example
![Page 5: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/5.jpg)
Entity Annotation Systems
Many different entity annotation systems
But how to compare these systems?
![Page 6: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/6.jpg)
Goal Defining and implementing a framework for
comparing in a complete, fair and meaningful way the entity annotation systems.
![Page 7: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/7.jpg)
Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion
![Page 8: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/8.jpg)
Entity Annotation Problems
1. Disambiguate to Wikipedia (D2W)2. Annotate to Wikipedia (A2W)3. Scored-annotate to Wikipedia (Sa2W)4. Concepts to Wikipedia (C2W)5. Scored concepts to Wikipedia (Sc2W)6. Ranked-concepts to Wikipedia (Rc2W)
![Page 9: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/9.jpg)
Entity Annotation Problems
![Page 10: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/10.jpg)
Entity Annotation Problems
![Page 11: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/11.jpg)
Entity Annotation Problems
![Page 12: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/12.jpg)
Entity Annotation Problems
![Page 13: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/13.jpg)
Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion
![Page 14: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/14.jpg)
Correct MatchPresident Barack Obama issues Iran ultimatum
Barack Obama
Barack ObamaPresident Barack Obama issues Iran ultimatum
![Page 15: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/15.jpg)
Generalize Standard Evaluation
s: solution found by tested systemg: ground truthM: binary relation M which specifies the notion of correct match"
![Page 16: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/16.jpg)
Precision, Recall & F1
![Page 17: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/17.jpg)
Macro and Micro
The macro- measures are the average of the correspondingmeasure over each document in the dataset D, while the micro- measures take into account all annotations together thus giving more importance to documents with more annotations.
![Page 18: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/18.jpg)
Correct MatchIf the entity found by the system and the ground truth entity redirect to the same entity, then set the correctness true.
The mention and the entity correct must be right(strong annotation match).
The mention overlap and the entity is right(weak annotation match).
![Page 19: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/19.jpg)
Similarity between Systems
![Page 20: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/20.jpg)
Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion
![Page 21: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/21.jpg)
Dataset
![Page 22: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/22.jpg)
News
![Page 23: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/23.jpg)
Similarity between Systems
![Page 24: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/24.jpg)
Tweet
![Page 25: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/25.jpg)
Web
![Page 26: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/26.jpg)
Runtime Efficiency
![Page 27: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/27.jpg)
Outline Introduction Entity Annotation Systems Measurement Experiment Conclusion
![Page 28: A Framework for Benchmarking Entity-Annotation Systems](https://reader036.vdocuments.us/reader036/viewer/2022062323/5681628e550346895dd2fb56/html5/thumbnails/28.jpg)
Conclusion We designed, implemented and tested a
benchmarking framework to fairly and fully compare entity-annotation systems.
It is written in Java and it has been released to the public as open source code in https://github.com/marcocor/bat-framework .