medical information retrieval and its evaluation: an overview of clef ehealth evaluation task

Medical Information Retrieval and its Evaluation: an Overview of CLEF eHealth Evaluation Task

Lorraine GoeuriotLIG – Université Grenoble Alpes (France)

lorraine.goeuriot@imag.fr

2

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

3

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

4

Medical Professionals – Web Search and Data

• Online information search on a regular basis

• Search failure for 2 patients out of 3

• PubMed search: very long (30+ minutes against 5 available)

• Knowledge production constantly growing

• More and more publications

• Varying web access

5

Medical Professionals – Web Search and Data

6

Patients and general public

• Change in the patient-physician relationship

• Patients more committed - cybercondria• How can information quality be guaranteed?

7

Patients – Web Search and Data

8

Patients – Web Search and Data

9

Patients – Web Search and Data

10

Medical Information Retrieval

• How different is medical IR from general IR? – Domain-specific search: narrowing down the

applications to improve results for categories of users– Consequences of bad performances of a medical search

system • Characteristics of medical IR:

– Data: medical/clinical reports, research papers, medical websites…

– Information need: decision support, technology/progress watch, education, daily care…

– Evaluation: relevance, readability, trustworthiness, time

11

Evaluating Information Retrieval?

Did the user find the information she needed? How many relevant documents did she get back? What is a relevant document? How many unrelevant document did she get back? How long before she found the information? Is she satisfied with the results? …

Did the user find the information she needed? How many relevant documents did she get back? What is a relevant document? How many unrelevant document did she get back? How long before she found the information? Is she satisfied with the results? …

• Creation of (artificial) datasets representing a specific search task, in order to compare various systems efficiency

• Involving human rating• Shared with the community to improve IR

12

Typical IR Evaluation Dataset

Document Collection

Topic Set Relevance Assessment

...

...

13

Existing Medical IR evaluation tasks

• Existing medical IR evaluation tasks: TREC Medical Records 2011, 2012 TREC 2000 filtering track (corpus OHSUMED) TREC genomics 2003-2007 ImageCLEFMed 2005-2013 TREC clinical decision support 2014, 2015

No patient-centered evaluation task

14

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

15

CLEF eHealth

AP: 72 yo w/ ESRD on HD, CAD, HTN, asthma, p/w significant hyperkalemia & associated arrythmias.

16

CLEF eHealth Tasks2013

• Task 1: Named entity recognition in clinical text

• Task 2: acronym normalization in clinical text

• Task 3: User-centred health IR

2014• Task 1: Visual-Interactive

Search and Exploration of eHealth Data

• Task 2: Information extraction from clinical text

• Task 3: User-centred health IR

2015• Task 1a: Clinical speech recognition from nurses handover• Task 1b: Clinical named entity recognition in French• Task 2: User-centred health IR

17

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

18

2013-2014

IR Evaluation Task Scenario

2015

19

• IR Evaluation Task over the years

2013 2014 2015

Goal Help laypersons better understand medical reports

Layperson checking their symptoms

Topics 55 EN topics built from discharge summaries

55 EN topics + translation in CZ, DE, FR

67 EN topics built from images + translation in AR, CZ, DE, FA, FR, IT, PT

Documents Medical document collection provided by Khresmoi project

Relevance assessment

Manual evaluation of relevance of documents

Manual evaluation of relevance and readability of documents

20

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

21

Document Collection

• Web crawl of health-related documents (~ 1M)• Made available through the Khresmoi project

(khresmoi.eu)• Target: general public and medical professionals• Broad range of medical topics covered• Content:• Health On the Net (HON) Foundation certified

websites (~60%)• Various well-known medical websites: DrugBank,

Diagnosia, TRIP answers, etc. (~40%)

22

Topics & context

Topics2013

Manual creation from randomly

selected annotation of disorder in the

DS (context)

2014Manual creation from manually identified main

disorders in the DS (context)

2015Manual creation from images describing

a medical problem (context)

23

Topics - Examples<topic> <id>qtest3</id><discharge_summary>02115-010823-DISCHARGE_SUMMARY.txt</discharge_summary><title>Asystolic arrest</title><desc>what is asystolic arrest</desc><narr>asystolic arrest and why does it cause death</narr><profile>A 87 year old woman with a stroke and asystolic arrest dies and the daughter wants to know about asystolic arrest and what it means.</profile></topic>

2013-2014

<topic> <id>clef2015.test.15</id><query>weird brown patches on skin</query></topic>

2015

24

Datasets - Summary

• Provided to the participants:• Document collection• Discharge summaries (optional) [2013-2014]• Training set:

– 5 queries + qrels [2013]– 5 queries (+ translation) + qrels [2014-2015]

• Test set:– 50 queries [2013]– 50 queries (+ translation) [2014]– 62 queries (+ translation) [2015]

25

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

26

Guidelines for Submissions

Submission of up to 7 runs (per language): Run 1 (mandatory) - team baseline: only title and description fields, no external resources. Runs 2-4 (optional) any experiment WITH the DS. Runs 5-7 (optional) any experiment WITHOUT the DS.

2013 - 2014

Submission of up to 10 ranked runs (per language): Run 1 (mandatory): baseline runRuns 2-10: any experiment with any external resource

2015

27

Relevance Assessment

Manual relevance assessment conducted by medical professionals and IR experts

4-point scale assessment mapped to a binary scale

– {0: non relevant, 1: on topic but unreliable} → non relevant

– {2: somewhat relevant, 3: relevant} → relevant

4-point scale for NDCG and 2-point scale for precision

[2015] Manual assessment of the readability of the documents conducted by the same assessors on a 4-point scale

28

Relevance Assessment - PoolsTraining set Test set

2013 Merged top 30 ranked documents from Vector Space Model and Okapi BM25

Merged top 10 documents from participants baseline run, the highest two priority runs with DS and highest two without DS2014

2015 Merged top 10 documents from participants three highest priority runs

29

Evaluation Metrics

• Classical TREC evaluation: P@5, P@10, NDCG@5, NDCG@10, MAP

• Ranking based on P@10

30

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation

• Conclusion

31

Participants and Runs

Monolingual IR Multilingual IR

# teams # runs # teams # runs

2013 9 48 -- --

2014 14 62 2 24

2015 12 92 1 35

32

Baselines

2013: • JSoup• Okapi stop words & Porter stemmer• Lucene BM252014: • Indri HTML parser• Okapi stop words & Krovetz stemmer• Indri BM25, tf.idf, LM

33

2013 Participants P@10 (best run)

Team-Mayo (2)Team-AEHRC (5)

Team-MEDINFO (1)Team-UOG (5)

Team-THCIB (5)Team-KC (1)

Team-UTHealth (1)Team-QUT (2)

Team-OHSU (5)

0

0.1

0.2

0.3

0.4

0.5

0.6

BM25 BM25 + PRF

34

2014 Task 3a P@10 (best run)G

RIU

M_E

N_R

un.5

SN

UM

ED

INF

O_E

N_R

un.2

KIS

TI_

EN

_Run

.2

IRLa

bDA

IIC

T_E

N_R

un.1

UIO

WA

_EN

_Run

.1

base

line.

dir

DE

MIR

_EN

_Run

.6

ReP

aLi_

EN

_Run

.5

NIJ

M_E

N_R

un.2

YO

RK

U_E

N_R

un.5

UH

U_E

N_R

un.5

CO

MP

L_E

N_R

un.5

ER

IAS

_EN

_Run

.6

mir

acl_

en_r

un.1

CU

NI_

EN

_RU

N.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

3535

Participants P@10 (2013 and 2014)

P@100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

2013

2014

BM25 2013LM Dirichlet smoothing 2014

36

Team-Mayo Team-AEHRCTeam-MEDINFO Team-UOG Team-THCIB Team-KC Team-UTHealth Team-QUT Team-OHSU0

0.1

0.2

0.3

0.4

0.5

0.6

Baseline

Best run

2013 Participants' ResultsBaseline vs best run

37

What Worked Well?

Team-Mayo:

• Markov Model Random Field to model query term dependency

• QE using external collections

• Combination of indexing techniques + re-ranking

Team-AEHRC:

• Language Models with Dirichlet smoothing

• QE with spelling correction and acronym expansion

Team-MEDINFO: Query Likelihood Model

BM25 Baseline

38

CO

MP

L

CU

NI

DE

MIR

ER

IAS

GR

IUM

IRLa

bDA

IIC

T

KIS

TI

mir

acl_

en_r

un.1

NIJ

M

ReP

aLi

SN

UM

ED

INF

O

UH

U

UIO

WA

YO

RK

U

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

BaselineBest run

2014 Participant's Results Baseline vs best run

39

What Worked Well?

Team-GRIUM:• Hybrid IR approach (text-based and concept-based)`• Language models• Query expansion based on mutual informationTeam-SNUMEDINFO:• Language Models with Dirichlet smoothing• QE with medical concepts• Google translateTeam-KISTI: • Language models• Various QE approaches

40

Task 3b Results

CS DE FR0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

CUNISNUMEDINFO

41

2013 - Use of Discharge Summaries

Team-Mayo Team-Medinfo Team-THCIB Team-KC Team-QUT0

0.1

0.2

0.3

0.4

0.5

0.6

With DS

Without DS

Baseline

42

How were DS used?

- Result re-ranking based on concepts extracted from queries, relevant documents and DS (Team-Mayo)

- Query expansion: * Filtering of non-relevant expansion terms/concepts

(Team-MEDINFO)* Expansion with all concepts from query and DS (Team-

THCIB)* Expansion with concepts identified in relevant

passages of the DS (Team-KC)* Query refinement (Team-TOPSIG)

43

2014 - Use of Discharge Summaries

IRLabDAIICT KISTI NIJM0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

DSNo DS

44

How Were DS Used?

●Query expansion:● Expansion using Metamap, with expansion

candidates filtered using the DS (Team-SNUMEDINFO)

● Expansion with abbreviations and DS combined with pseudo-relevance feedback (Team-KISTI)

● Expansion with MeSH terminology and DS (Team-IRLABDAIICT)

● Expansion with terms from the DS (Team-Nijmegen)

45

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation– Further analysis

• Conclusion

46

Medical Queries Complexity Query complexity = number of medical

concepts/entities it contains radial neck fracture and healing time facial cuts and scar tissue nausea and vomiting and hematemesis

Dataset: 50 queries from CLEF eHealth 2013 (patients queries) Runs from 9 teams

Impact of the complexity on the systems performances

47

Medical Queries Complexity

48

Presentation Overview

• Medical IR and its Evaluation• CLEF eHealth

– Context and tasks– IR tasks description– Datasets– Evaluation– Participation– Further analysis

• Conclusion

49

Conclusion

• 3 successful years running CLEF eHealth• Datasets are publicly available for research

purpose• Used for research by organizers, participants,

and other groups• Building a community – evaluation tasks,

workshop@SIGIR, special edition of JIR

50

For More Details

CLEF eHealth Lab overview: Suominen et al. (2013). Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In CLEF 2013 Proceedings.Kelly et al. (2014). Overview of the ShARe/CLEF eHealth Evaluation Lab 2014. In CLEF 2014 Proceedings.

CLEF eHealth IR task overview: Goeuriot et al. (2013). ShAReCLEF eHealth Evaluation Lab 2013, Task 3: Information Retrieval to Address Patients’ Questions when Reading Clinical Reports. In CLEF 2013 Working notes. Goeuriot et al. (2014). ShARe/CLEF eHealth Evaluation Lab 2014, Task 3: User-centred health information retrieval. In CLEF 2013 Working notes.

51

Follow us!

http://sites.google.com/site/clefehealth2015

clef-ehealth-evaluation-lab-information On Google groups

@clefehealth

Join the party in Toulouse: http://clef2015.clef-initiative.eu/CLEF2015/

conferenceRegistration.php

52

Consortium

• Lab chairs: Lorraine Goeuriot, Liadh Kelly• Task 1: Hanna Suominen, Leif Hanlen, Gareth

Jones, Liyuan Zhou, Aurélie Névéol, Cyril Grouin, Thierry Hamon, Pierre Zweigenbaum

• Task 2: Joao Palotti, Guido Zuccon, Allan Hanbury, Mihai Lupu, Pavel Pecina

53

Thank you! Questions?

54

Task 3a - Topic Generation Process (1)Discharge Medications:

1. Aspirin 81 mg Tablet, Delayed Release (E.C.) Sig: One (1) Tablet, Delayed Release (E.C.) PO DAILY (Daily). Disp:*30 Tablet, Delayed Release (E.C.)(s)* Refills:*0*

2. Docusate Sodium 100 mg Capsule Sig: One (1) Capsule PO BID (2 times a day). Disp:*60 Capsule(s)* Refills:*0*

3. Levothyroxine Sodium 200 mcg Tablet Sig: One (1) Tablet PO DAILY (Daily).

Discharge Disposition:

Extended Care

Facility:

[**Hospital 5805**] Manor - [**Location (un) 348**]

Discharge Diagnosis:

Coronary artery disease.

s/p CABG

post op atrial fibrillation

55

Task 3a - Topic Generation Process (2)Discharge Medications:

1. Aspirin 81 mg Tablet, Delayed Release (E.C.) Sig: One (1) Tablet, Delayed Release (E.C.) PO DAILY (Daily). Disp:*30 Tablet, Delayed Release (E.C.)(s)* Refills:*0*

2. Docusate Sodium 100 mg Capsule Sig: One (1) Capsule PO BID (2 times a day). Disp:*60 Capsule(s)* Refills:*0*

3. Levothyroxine Sodium 200 mcg Tablet Sig: One (1) Tablet PO DAILY (Daily).

Discharge Disposition:

Extended Care

Facility:

[**Hospital 5805**] Manor - [**Location (un) 348**]

Discharge Diagnosis:

Coronary artery disease.

s/p CABG

post op atrial fibrillation

56

Task 3a - Topic Generation Process (3)Discharge Medications:

1. Aspirin 81 mg Tablet, Delayed Release (E.C.) Sig: One (1) Tablet, Delayed Release (E.C.) PO DAILY (Daily). Disp:*30 Tablet, Delayed Release (E.C.)(s)* Refills:*0*

2. Docusate Sodium 100 mg Capsule Sig: One (1) Capsule PO BID (2 times a day). Disp:*60 Capsule(s)* Refills:*0*

3. Levothyroxine Sodium 200 mcg Tablet Sig: One (1) Tablet PO DAILY (Daily).

Discharge Disposition:

Extended Care

Facility:

[**Hospital 5805**] Manor - [**Location (un) 348**]

Discharge Diagnosis:

Coronary artery disease.

s/p CABG

post op atrial fibrillation

What is coronary heart disease?

5757

Participants Approaches