Upload File

evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a...

  • Home
  • Science
  • Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)
21
Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for Doing a Better Job Marieke van Erp , Pablo Mendes, Heiko Paulheim, Filip Ilievski, Julien Plu, Giuseppe Rizzo and Joerg Waitelonis https://github.com/dbpedia-spotlight/evaluation-datasets

Upload: marieke-van-erp

Post on 16-Apr-2017

194 views

Category:

Science


0 download

Report

TRANSCRIPT

Page 1: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for Doing a Better Job

Marieke van Erp, Pablo Mendes, Heiko Paulheim, Filip Ilievski, Julien Plu, Giuseppe Rizzo and Joerg Waitelonis

https://github.com/dbpedia-spotlight/evaluation-datasets

Page 2: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Take home message

• Existing entity linking datasets:• are not interoperable• do not cover many different domains• skew towards popular and frequent entities

• We need to:• Document & Standardise• Diversify to cover different domains and the long tail

https://github.com/dbpedia-spotlight/evaluation-datasets/ 2

Page 3: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Why

• Named entity linking approaches achieve F1 scores of ~.80 on various benchmark datasets

• Are we really testing our approaches on all aspects of the entity linking task?

It’s not just us:Maud Ehrmann and Damien Nouvel and Sophie Rosset. Named Entity Resources - Overview and Outlook. LREC 2016

https://github.com/dbpedia-spotlight/evaluation-datasets/ 3

Page 4: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

This work

• Analysis of 7 entity linking benchmark datasets• Dataset characteristics (document type, domain, license etc)• Entity, surface form & mention characterisation (overlap

between datasets, confusability, prominence, dominance, types, etc)

• Annotation characteristics (nested entities, redundancy, IAA, offsets)

+ Roadmap: how can we do better

https://github.com/dbpedia-spotlight/evaluation-datasets/ 4

Page 5: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Overlap

• Number of entities present in one dataset that are also present in other datasets

AIDA-YAGO2 (5,596)NEEL2014 (2,380)NEEL2015 (2,800)OKE2015 (531)RSS500 (849)WES2015 (7,309)Wikinews (279)

https://github.com/dbpedia-spotlight/evaluation-datasets/ 5

Page 6: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Datasets

Dataset Type Domain Doc length Format Encoding License

AIDA-YAGO2 news general medium TSV ASCII Agreement

2014/2015 NEEL tweets general short TSV ASCII Open

OKE2015 encyclopaedia general long NIF/RDF UTF8 Open

RSS-500 news general medium NIF/RDF UTF8 Open

WES2015 blog science long NIF/RDF UTF8 Open

WikiNews news general medium XML UTF8 Open

https://github.com/dbpedia-spotlight/evaluation-datasets/ 6

Page 7: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Overlap

• Number of entities present in one dataset that are also present in other datasets

AIDA-YAGO2 NEEL2014 NEEL2015 OKE2015 RSS500 WES2015 Wikinews

AIDA-YAGO2 (5,596) 5.87% 8.06% 0.00% 1.26% 4.80% 1.16%NEEL2014 (2,380) 13.73% 68.49% 2.39% 2.56% 12.35% 2.82%NEEL2015 (2,800) 16.11% 58.21% 2.00% 2.54% 7.93% 2.57%OKE2015 (531) 0.00% 10.73% 10.55% 2.44% 28.06% 3.95%RSS500 (849) 8.24% 7.18% 8.36% 1.53% 3.18% 1.88%WES2015 (7,309) 3.68% 4.02% 3.04% 2.04% 0.16% 0.66%Wikinews (279) 23.30% 24.01% 25.81% 7.53% 5.73% 17.20%

https://github.com/dbpedia-spotlight/evaluation-datasets/ 7

Page 8: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Overlap

• Number of entities present in one dataset that are also present in other datasets

AIDA-YAGO2 NEEL2014 NEEL2015 OKE2015 RSS500 WES2015 Wikinews

AIDA-YAGO2 (5,596) 5.87% 8.06% 0.00% 1.26% 4.80% 1.16%NEEL2014 (2,380) 13.73% 68.49% 2.39% 2.56% 12.35% 2.82%NEEL2015 (2,800) 16.11% 58.21% 2.00% 2.54% 7.93% 2.57%OKE2015 (531) 0.00% 10.73% 10.55% 2.44% 28.06% 3.95%RSS500 (849) 8.24% 7.18% 8.36% 1.53% 3.18% 1.88%WES2015 (7,309) 3.68% 4.02% 3.04% 2.04% 0.16% 0.66%Wikinews (279) 23.30% 24.01% 25.81% 7.53% 5.73% 17.20%

https://github.com/dbpedia-spotlight/evaluation-datasets/ 8

Page 9: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Overlap

• Number of entities present in one dataset that are also present in other datasets

AIDA-YAGO2 NEEL2014 NEEL2015 OKE2015 RSS500 WES2015 Wikinews

AIDA-YAGO2 (5,596) 5.87% 8.06% 0.00% 1.26% 4.80% 1.16%NEEL2014 (2,380) 13.73% 68.49% 2.39% 2.56% 12.35% 2.82%NEEL2015 (2,800) 16.11% 58.21% 2.00% 2.54% 7.93% 2.57%OKE2015 (531) 0.00% 10.73% 10.55% 2.44% 28.06% 3.95%RSS500 (849) 8.24% 7.18% 8.36% 1.53% 3.18% 1.88%WES2015 (7,309) 3.68% 4.02% 3.04% 2.04% 0.16% 0.66%Wikinews (279) 23.30% 24.01% 25.81% 7.53% 5.73% 17.20%

https://github.com/dbpedia-spotlight/evaluation-datasets/ 9

Page 10: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Overlap

• Number of entities present in one dataset that are also present in other datasets

AIDA-YAGO2 NEEL2014 NEEL2015 OKE2015 RSS500 WES2015 Wikinews

AIDA-YAGO2 (5,596) 5.87% 8.06% 0.00% 1.26% 4.80% 1.16%NEEL2014 (2,380) 13.73% 68.49% 2.39% 2.56% 12.35% 2.82%NEEL2015 (2,800) 16.11% 58.21% 2.00% 2.54% 7.93% 2.57%OKE2015 (531) 0.00% 10.73% 10.55% 2.44% 28.06% 3.95%RSS500 (849) 8.24% 7.18% 8.36% 1.53% 3.18% 1.88%WES2015 (7,309) 3.68% 4.02% 3.04% 2.04% 0.16% 0.66%Wikinews (279) 23.30% 24.01% 25.81% 7.53% 5.73% 17.20%

https://github.com/dbpedia-spotlight/evaluation-datasets/ 10

Page 11: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Confusability

• The number of meanings a surface form (mention) can have

11

Page 12: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Confusability

Corpus Average Min Max

AIDA-YAGO2 1.08 1 13 0.37

2014 NEEL 1.02 1 3 0.16

2015 NEEL 1.05 1 4 0.25

OKE2015 1.11 1 25 1.22

RSS500 1.02 1 3 0.16

WES2015 1.06 1 6 0.30

Wikinews 1.09 1 29 1.03

https://github.com/dbpedia-spotlight/evaluation-datasets/ 12

Page 13: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Dominance

Corpus Dominance Min Max

AIDA-YAGO2 .98 1 452 0.08

2014 NEEL .99 1 47 0.06

2015 NEEL .98 1 88 0.09

OKE2015 .98 1 1 0.11

RSS500 .99 1 1 0.07

WES2015 .97 1 1 0.12

Wikinews .99 1 72 0.09

https://github.com/dbpedia-spotlight/evaluation-datasets/ 13

Page 14: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Types

https://github.com/dbpedia-spotlight/evaluation-datasets/ 14

Page 15: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Types

15

Page 16: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Entity Prominance

https://github.com/dbpedia-spotlight/evaluation-datasets/ 16

DBpedia PageRank datasets:http://people.aifb.kit.edu/ath/

Page 17: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

How can we do better?

• Document your dataset!

• Use a standardised format • Diversify both in domains and in entity distribution

https://github.com/dbpedia-spotlight/evaluation-datasets/ 17

Page 18: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Work in Progress & Future work

• Analyse more datasets • Evaluate the temporal dimension of datasets (current work

by Filip Ilievski & Marten Postma)

• Integrate and build better datasets

https://github.com/dbpedia-spotlight/evaluation-datasets/ 18

Page 19: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Want to help?

Scripts and data used here can be found at:

Contact [email protected] if you want to collaborate

https://github.com/dbpedia-spotlight/evaluation-datasets/

19

mailto:[email protected]
Page 20: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Shameless Advertising

NLP&DBpedia 2016

Workshop at ISWC2016

Submission deadline: 1 July

https://nlpdbpedia2016.wordpress.com/20

http://iswc2016.semanticweb.org/
https://nlpdbpedia2016.wordpress.com/
https://nlpdbpedia2016.wordpress.com/
Page 21: Evaluating entity linking an analysis of current benchmark datasets and a roadmap for doing a better job (3)

Acknowledgements

https://github.com/dbpedia-spotlight/evaluation-datasets/


Linking Smart Cities Datasets with Human Computation: the case of UrbanMatch

Linking Disparate Datasets of the Earth Sciences with the SemantEco Annotator

Introduction (Motivation) · GPS information etc.) Visual + Metadata. 4 Benchmark Datasets: State-of-the-art Current datasets for event detection in images low number of images (e.g.,

BigHand2.2M Benchmark: Hand Pose Dataset and State of ......BigHand2.2M automatic 2.2M 21 10 full 640×480 Table 1: Benchmark comparison. Existing datasets are limited in the number

High-level Quantum Chemistry Methods and Benchmark ... · High-level Quantum Chemistry Methods and Benchmark Datasets for Molecules MarkusSchneider FritzHaberInstituteoftheMPS,Berlin,Germany

Querying and reasoning over large scale building datasets: an outline of a performance benchmark

Open Graph Benchmark: Datasets for Machine Learning on Graphs · 2020. 12. 2. · OGB will be regularly updated and welcomes inputs from the community. OGB datasets as well as data

National longitudinal datasets: linking intervention and

252.eon, Kajiya command line Datasets profile vs ... file252.eon, Kajiya command line Datasets profile vs. Reference Dataset The following are the profiles for the 252.eon benchmark,

New Datasets and a Benchmark of Document Network …ceur-ws.org/Vol-2591/paper-03.pdf · Datasets and Benchmark of DNE Methods for Expert Finding 3 address the search for experts

What Makes a Patch Distinct?ayellet/Ps/13-MTZ.pdfWe test our method on the recently-published benchmark of Borji et al. [4]. This benchmark consists of five well-known datasets of

Towards Intelligent Visual Understanding under Minimal ...machine learning techniques, e.g., deep learning, the ... Comprehensive evaluations of three benchmark datasets and comparisons

Omics Discovery Index – Discovering and Linking Public ‘Omics’ …€¦ · ! 1! Omics Discovery Index – Discovering and Linking Public ‘Omics’ Datasets Yasset Perez-Riverol

Linking Large Datasets Why, How, and What Not To Do Bradley G Hammill Duke Clinical Research Institute

Development of Thermophysical Property Datasets, Benchmark …user.engineering.uiowa.edu/~becker/documents.dir/SFSA... · 2010-12-21 · Development of Thermophysical Property Datasets,

What's all the data about? - Linking and Profiling of Linked Datasets

Linking Disparate Datasets of the Earth Sciences with the SemantEco Annotator Session: Managing Ecological Data for Effective Use and Reuse Patrice Seyed

TAO: A Large-Scale Benchmark for Tracking Any Object arXiv ... · TAO: A Large-Scale Benchmark for Tracking Any Object 3 Table 1. Statistics of major multi-object tracking datasets

Linking Smart Cities Datasets with Human Computation { the …iswc2012.semanticweb.org/sites/default/files/76500033.pdf · 2012-10-19 · Linking Smart Cities Datasets with Human

Linking micro datasets to better service policy-making and

Exploiting kernel-based feature weighting and instance … · learning benchmark datasets. Key words: Transfer learning, unsupervised domain adaptation, feature weighting, instance

What is WideNet?M. van Erp, P. N. Mendes, H. Paulheim, F. Ilievski, J. Plu, and G. Rizzo, “Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for

Improving (Re-)Usability of Musical Datasets: an Overview ... · Improving (Re-)Usability of Musical Datasets: an ... FRBRoo, Musikwerke, semantisches Web, Datenumwandlung, Data Linking,

Quality Assurance Benchmark Datasets and Processing David Nidever SQuaRE Science Lead

Results Classification - Comparison with existing approaches on benchmark datasets Survival Analysis – International multi-centre double-blind study Survival

A BENCHMARK FOR SEMANTIC IMAGE SEGMENTATION · 2.1. Review of image segmentation datasets Several image segmentation datasets have been constructed for testing and evaluating different

Coarse-to- ne: A RNN-based hierarchical attention model for ......2019/06/04  · re-identi cation results on two challenging benchmark datasets. 4.1 Datasets and evaluation metric

A Threshold fuzzy entropy based feature selection method applied in various benchmark datasets using Ant-miner algorithm

Data-gov Wiki: Towards Linking Government Datatw.rpi.edu/media/2013/09/23/140c2/linkedai-2010-datagov.pdfData-gov Wiki: Towards Linking Government Data Li Ding, ... datasets can be

Linking Smart Cities Datasets with Human Computation - the case of UrbanMatch

Digging by Debating: linking massive datasets to specific arguments

View Adaptive Recurrent Neural Networks for High ...openaccess.thecvf.com/content_ICCV_2017/papers/Zhang_View_Adaptive... · state-of-the-art performance on three benchmark datasets

Discovery from Linking Open Data (LOD) Annotated Datasets Louiqa Raschid University of Maryland PAnG/PSL/ANAPSID/Manjal

Scaling Up Machine Learning: How to Benchmark GraphLab Create on Huge Datasets

SalientShape: Group Saliency in Image Collectionsmmcheng.net/mftp/SalObj/GroupSaliency.pdf · der larger than any existing benchmark datasets, consisting of diverse and heterogeneous