the clef initiative from 2010 to 2012 and onwards
DESCRIPTION
TRANSCRIPT
Nicola FerroInformation Management Systems (IMS) Research Group
Department of Information EngineeringUniversity of Padua, Italy
MediaEval Benchmarking Initiative for Multimedia EvaluationMediaEval 2012 Workshop, 5 October 2012, Santa Croce in Fossabanda, Pisa, Italy
I N I T I A T I V E
The CLEF InitiativeConference and Labs of the Evaluation Forum
http://www.clef- initiative.eu/
The CLEF InitiativeFrom 2010 to 2012 and Onwards
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Outline
The CLEF Initiative
From CLEF 2010 to CLEF 2012
CLEF 2013 and Beyond
2
The CLEF Initiative
I N I T I A T I V E
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
The CLEF Initiative: Mission
multilingual and multimodal system testing, tuning and evaluation;investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access;creation of reusable test collections for benchmarking;exploration of new evaluation methodologies and innovative ways of using experimental data;discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.
4
The CLEF Initiative is a self-organized body whose main mission is to promote
research, innovation, and development of information access systems with an
emphasis on multilingual and multimodal information with various levels of structure.
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
The CLEF Initiative: Structure
The CLEF Initiative is structured in two main parts:a series of Evaluation Labs, i.e. laboratories to conduct evaluation of information access systems and workshops to discuss and pilot innovative evaluation activities;a peer-reviewed Conference on a broad range of issues, including
investigation continuing the activities of the Evaluation Labs;experiments using multilingual and multimodal data; in particular, but not only, data resulting from CLEF activities;research in evaluation methodologies and challenges.
5
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
The CLEF Initiative
6
http://www.clef-initiative.eu/promise
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
The CLEF Initiative: Steering Committee
Steering Committee ChairNicola Ferro, University of Padua, Italy
Deputy Steering Committee Chair for the ConferenceJulio Gonzalo, National Distance Education University (UNED), Spain
Deputy Steering Committee Chair for the Evaluation LabsCarol Peters, ISTI, National Council of Research (CNR), Italy
MembersMartin Braschler, Zurich University of Applied Sciences, Switzerland Khalid Choukri, Evaluations and Language resources Distribution Agency (ELDA), France Paul Clough, University of Sheffield, United KingdomDonna Harman, National Institute for Standards and Technology (NIST), USA Djoerd Hiemstra, University of Twente, The NetherlandsJaana Kekäläinen, University of Tampere, FinlandEmanuele Pianta, Centre for the Evaluation of Language and Communication Technologies (CELCT), ItalyMaarten de Rijke, University of Amsterdam UvA, The NetherlandsGiuseppe Santucci, Sapienza University of Rome, ItalyAlan Smeaton, Dublin City University, IrelandChrista Womser-Hacker, University of Hildesheim, Germany
7
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF: the Name
8
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF: the Name
8
CLEF 2010
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Approach
10
Scientific and Technological Advancement in Multilingual and Multimedia Information Systems
Conference
La
bs a
nd
W
ork
sh
op
s
La
bs a
nd
W
ork
sh
op
s
La
bs a
nd
W
ork
sh
op
s
La
bs a
nd
W
ork
sh
op
s
La
bs a
nd
W
ork
sh
op
s
La
bs a
nd
W
ork
sh
op
s
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2010: 20-23 September 2010 in Padua
11
Over 140
attendees and
external sponso
rs
http://www.clef2010.org/
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2010 OrganizationHonorary Chair
Carol Peters, ISTI-CNR, Italy
General ChairsMaristella Agosti, University of Padua, ItalyMaarten de Rijke, University of Amsterdam, The Netherlands
Program ChairsNicola Ferro, University of Padua, ItalyAlan Smeaton, Dublin City University, Ireland
Lab ChairsMartin Braschler, Zurich University of Applied Sciences, SwitzerlandDonna Harman, NIST, USA
12
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2010 Conference12 papers (8 full papers and 4 short papers)
out of 21 submissions (17 full papers and 4 short papers)
Two keynote talksNorbert Fuhr, “IR Between Science and Engineering, and the Role of Experimentation”Ricardo Baeza-Yates, “Retrieval Evaluation in Practice”
Other Evaluation InitiativesEllen Voorhees for TREC; Noriko Kando for NTCIR; Prasenjit Majumder for FIRE; Jaap Kamps for INEX; Pavel Braslavski for ROMIP
Two panelsDonna Harman, Noriko Kando, Mounia Lalmas, Carol Peters, “The Four Ladies of Experimental Evaluation”“A PROMISE for Experimental Evaluation” by the partners of the project
13
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2010 Labs and Workshops5 Benchmarking activities
CLEF-IP: a benchmarking activity on intellectual propertyImageCLEF: a benchmarking activity on image retrievalPAN: a benchmarking activity on plagiarism detectionRespubliQA: A benchmarking activity on question answering using multilingual political dataWePS: A benchmarking activity on web people search
2 WorkshopsCriES: a workshop aimed at exploring the evaluation of searching for expertise in social mediaLogCLEF: a workshop aimed at exploring methodologies for studying search engine log files
14
CLEF 2011
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2011 Approach: Interleaved Sessions
16
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Over 170
attendees and
external sponso
rs
CLEF 2011: 19-22 September 2011 in Amsterdam
17
http://www.clef2011.org/http://www.clef2011.org/
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2011 Organization
General chairsJulio Gonzalo, National Distance Education University (UNED), SpainMaarten de Rijke, University of Amsterdam, The Netherlands
Program chairsJaana Kekäläinen, University of Tampere, FinlandMounia Lalmas, Yahoo! Research Barcelona, Spain
Lab chairsPaul Clough, University of Sheffield, United KingdomVivien Petras, Humboldt University, Germany
18
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2011 Conference14 papers (10 full papers and 4 short papers)
out of 23 submissions (19 full papers and 4 short papers)
Two keynote talksElaine Toms , “Would you trust your IR system to choose your date? Re-thinking IR Evaluation in the 21st Century”Omar Alonso, “Crowdsourcing for Information Retrieval Experimentation and Evaluation”
Community SessionsOther Evaluation Initiatives: Ellen Voorhees for TREC; Hideo Joho for NTCIR; Jaap Kamps for INEX; Gareth Jones for MediaEvalCHORUS+ Session: Information Retrieval from Scientific Multimedia DataInfrastructure Session: the Promise evaluation infrastructure, where Visual Analytics meets Information RetrievalFunding Opportunity Session: Intelligent Information Management - FP7 ICT Workprogramme 2011 - 2012, Strategic Objective 4.4. - General information and conditions of Call 8
19
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2011 Labs and Workshops
5 Benchmarking activitiesCLEF-IP: a benchmarking activity on intellectual propertyImageCLEF: a benchmarking activity on image retrievalLogCLEF: a benchmarking activity on Multilingual Log File Analysis: Language identification, query classification, success of a queryPAN: a benchmarking activity on plagiarism detectionQA4MRE: a benchmarking activity on the evaluation of Machine Reading systems through Question Answering and Reading Comprehension Test
2 Workshops/PilotsCHiC: a workshop aimed at moving towards a systematic and large-scale evaluation of cultural heritage digital libraries and information access systems and helping to shape a possible roadmap for itMusicCLEF: a pilot benchmarking activity on the evaluation of music search engines that are based both on audio content and on multilingual textual descriptions
20
CLEF 2012
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2012: 17-20 September 2012, Rome
22
Over 190
attendees and
external sponso
rs
http://www.clef2012.org/
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2012 Organization
General chairsTiziana Catarci, Sapienza University of Rome, ItalyDjoerd Hiemstra, University of Twente, The Netherlands
Program chairsAnselmo Peñas, National Distance Learning University, SpainGiuseppe Santucci, Sapienza University of Rome, Italy
Lab chairsJussi Karlgren, Swedish Institute of Computer Science, SwedenChrista Womser-Hacker, University of Hildesheim, Germany
23
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2012 Conference
17 papers (9 full papers, 5 short papers, and 3 posters) out of 33 submissions
Two keynote talksPeter Clark, “From Information Retrieval to Knowledgeable Machines”Tobias Schreck, “Visual Search and Analysis in Textual and Non-Textual Document Repositories - Approaches, Applications, and Research Challenges”
Community SessionsOther Evaluation Initiatives: Donna Harman for TREC and TAC; Noriko Kando for NTCIR; Pavel Braslavski for ROMIP; Prasenjit Majumder for FIRE; Gareth Jones for MediaEval (remote)Evaluation Clinic: participants had the possibility of meeting evaluation experts and discussing with them their own evaluation problems
24
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF 2012 Labs and Workshops7 Benchmarking activities
CHiC: a last-year workshop turning into a benchmarking activity for the cultural heritage domain based on Europeana collectionsCLEF-IP: a benchmarking activity on intellectual propertyImageCLEF: a benchmarking activity on image retrievalINEX: the well-known Initiative for Evaluation of XML retrieval joins efforts with CLEF to target new synergies between multilingual, multimodal and semi-structured information accessPAN: a benchmarking activity on plagiarism detectionQA4MRE: a benchmarking activity on the evaluation of Machine Reading systems through Question Answering and Reading Comprehension TestRepLab: a benchmarking activity on microblog data for online reputation management
1 WorkshopeHealth: a workshop on new evaluation issues in the health domain, related to the Louhi series of workshops on NLP in Health Informatics
25
CLEF 2013 and Beyond
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Next CLEFs
CLEF 2013 23 - 26 September 2013Technical University of Valencia, SpainMore information soon on the CLEF 2013 Web sitehttp://www.clef2013.org/
CLEF 201415 - 19 September 2014University of Sheffield, United Kingdom
CLEF 2015Bids are openSend a proposal at [email protected] by 5 April 2013 Template available at: http://www.clef-initiative.eu/documents/71612/87713/CLEF-Initiative-Template_for_bids.docx
27
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
CLEF Publication
28
We are now missing a single publication that acts as an hub for citation of all the CLEF activities (conference and labs) that happen during the yearTherefore, the Springer proceedings delivered at the conference will contain
the conference papers selected by the review process carried out by the program committeethe lab overview papers, after a review process carried out by the lab chairs
To meet the deadlines to have this publication ready by September, we need to reschedule the overall CLEF evaluation campaign period
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Conference - CLEF 2013
29
Call for Papers coming out soon, late November 2012. Don’t miss it!!!
Paper submission deadline around late April 2013
More emphasis on evaluationFocused range of hot topics
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
Lab Coordination Deadlines - CLEF 2013
1 October 2012: Call for Lab proposals15 October 2012: Deadline for presentation of proposals29 October 2012: Notification of acceptance/rejection5 November 2012: Release of first lab flyer and call for participation1 December 2012: Labs registration opens15 December 2012: Labs begin to release data15 December 2012 - 1 May 2013: Evaluation Campaign1 May 2013: Result submission1 June 2013: Results release15 June 2013: Working notes papers submitted30 June 2013: Lab overview papers submitted in Springer format30 June 2013 - 7 July 2013: Lab Chairs review lab overviews15 July 2013: Springer proceedings done
30
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
ELIAS Lab Organizers Meeting at ECIR 2013
ELIAS can support a meeting at ECIR 2013 (24-27 March 2013) in Moscow between Lab Organizers to share ideas, plans, and strategiesto coordinate
31
http://www.elias-network.eu/
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
15 June 2013: Call for Lab Proposals1 July 2013: Deadline for presentation of proposals20 July 2013: Notification of acceptance/rejectionCLEF 2014: Release of first lab flyer and call for participation1 November 2013: Labs registration opens15 November 2013: Labs begin to release data15 November 2013 - 1 May 2014: Evaluation Campaign1 May 2014: Result submission1 June 2014: Results release15 June 2014: Working notes papers submitted30 June 2014: Lab overview papers submitted in Springer format30 June 2014 - 7 July 2014: Lab Chairs review lab overviews15 July 2014: Springer proceedings done
32
Lab Coordination Deadlines - CLEF 2014
Nicola Ferro
I N I T I A T I V E
The CLEF Initiative: From 2010 to 2012 and OnwardsMediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
promisePROMISE: a supporter of CLEF
33
FP7 ICT 2009.4.3, Intelligent Information Management
WYVTPZL
Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191
PROMISEParticipative Research labOratory for Multimediaand Multilingual Information Systems Evaluation
PROMISE Retreat ReportProspects and Opportunities forInformation Access Evaluation
EditorsNicola Ferro, University of Padua, ItalyRichard Berendsen, University of Amsterdam, The NetherlandsAllan Hanbury, Vienna University of Technology, AustriaMihai Lupu, Vienna University of Technology, AustriaVivien Petras, Humboldt University Berlin, GermanyMaarten de Rijke, University of Amsterdam, The NetherlandsGianmaria Silvello, University of Padua, Italy
http://www.promise-noe.eu/