Levels of Searching

ADEPT

Glasgow

2nd October 2003

Judgingthe

Levelof

Search

Two epidemiologists are leaving a party late one night and one drops their car keys.

The other following behind offers to help scrabble around on the ground to look for them.

"Where did you drop them?" asks the second.

"Over there" says the first, pointing to a spot some feet away.

"Then why are we looking over here?"

"Because there's a street light on over here" comes the reply.

Practical Examples from...

• Rapid Evidence Retrieval to Systematic Searching

• NES/SCPMDE and BEME Collaboration (Alex Haig)

• Beatson (Annette Thain)

• Others...

• Balancing sensitivity and specificity

Sensitivity & Specificity

• Sensitivity (recall) - percentage of “gold standard”

• Specificity (precision) - positive predictive value

Searching across thehealth disciplines

• Inter-professional direction

• Search experience is extendible– same core skills for systematic and rapid

searches– core search skills are transferable

Medical Education Searching

• No filters

• Problematic study designs

• Grey literature in large quantities

• Dispersed sources– databases– grey literature– web– hand searches– expert opinion

• the dissemination of information which allows medical teachers, institutions and all concerned with medical education to make decisions on the basis of the best evidence available

• the production of appropriate systematic reviews of medical education which reflect the best evidence available and meet the needs of the user, and

• the creation of a culture of best evidence medical education amongst individual teachers, institutions and national bodies.

Topic Review Groups

• High Fidelity Simulation in Medical Education

• The impact of feedback in formative and summative assessment and its effectiveness on the improvement of knowledge, skills and attitudes of health science students during their learning process and on teaching skills of tutors.

• What are the features of faculty development that make it effective?

Topic Review Groups

• What conditions are necessary or required for assessment and feedback to enhance physician improvement?

• A systematic review of educational interventions on the communication skills of medical clinicians

• Performance-based instruments that measure medical communication competence - an update and expansion of previous reviews.

Topic Review Groups

• Predictive values of assessment measurements obtained in medical schools and future performance in medical practice

• What does early clinical experience contribute to the basic education of health professionals?

• The impact of inter-professional education on health and social care practitioners, professional practice, patient/client related health and well being, and public health and social care outcomes.

• International

• Educational

• Catholic inclusion of evidence

• Psychologists estimate that we each make up to 1,200 decisions every day.

• Increasingly, professionals are being expected to base their decisions on evidence.

“Systematic Searching in Medical Education”

September 2001, Berlin

Methods

• Topic - Feedback in assessment (BEME pilot and current topic review group, Barcelona/Tel Aviv)

• Software - Ovid [CGI version 7.8]

• Databases - Medline, Embase, ERIC

Methods - journal selected

• limited time and resources

• required a title that was most comprehensively indexed

• Academic Medicine 1996-present

Methods - strategies

• Three levels of strategy: standard (most users; limited search syntax) enhanced (some use of search syntax) expert (full use of search syntax)

• syntax includes: free-text, controlled vocabulary, term explosions, phrase lists, subheadings, sub strings, filters, proximity operators, etc...

Methods - handsearching

• “...refers to the planned searching of a journal page by page (i.e. by hand), including editorials, letters, etc., to identify all relevant items.”

• time consuming and meticulous

• produces the “gold standard” by which search efficiency can be measured

Sensitivity

• Sensitivity (recall) - percentage of “gold standard”

• Sensitivity = total retrieved by search

total of the hand-search

Gold Gold Standard = Standard = 4646

Embase Sensitivity (n)Basic 4.3% 2Enhanced 10.7% 5Expert 15.2% 7

ERIC Sensitivity (n)Basic 0% 0Enhanced 4.3% 2Expert 6.5% 3

Medline Sensitivity (n)Basic 0% 0Enhanced 10.7% 5Expert 19.6% 9

GS=46GS=46

Specificity Specificity (precision) - positive predictive value

Specificity = relevant records identified

total retrieved by search

Embase Specificity (n)Basic 40% 5Enhanced 33% 15Expert 30.4% 23

ERIC Specificity (n)Basic 0% 0Enhanced 40% 5Expert 37.5% 8

Medline Specificity (n)Basic 0% 0Enhanced 31.3% 16Expert 32.1% 28

A Note of Caution

• Academic MedicineAcademic Medicine is a journal that specialises in medical education :– more likely to be indexed for context– journal presents information for better retrieval

• Other journals will fare worse

• Other specificity scores for BEME pilots (not limited to one journal) ranged from 6 to 34%, with feedback in assessment at 17.8%

Reasons for shockingly poor performance

Incomplete coverage of journals No indexed database for medical education Existing controlled vocabularies are

inadequate for medical education

MeSH - Feedback

(Mapped Term)

1. Feedback, Biochemical

• Feedback, Biochemical: A mechanism of communication among life processes to co-ordinate development, reproduction, and homeostasis. In humans, feedback loops are especially important for communication between organs that are spatially separated. Virtually all hormones from the nervous and endocrine systems are under feedback control: by peripheral hormones, cations, metabolites, osmolarity or extracellular fluid volume.

2. Feedback

• Feedback: A mechanism of communication within a system in that the input signal generates an output response which returns to influence the continued activity or productivity of that system.

Feedback, Psychological

• Feedback, Psychological: A mechanism of information stimulus and response that may control subsequent behaviour, cognition, perception, or performance. (From APA Thesaurus of Psychological Index Terms, 8th Ed.)

Knowledge of results (Psychology)

• A principle that learning is facilitated when the learner receives immediate evaluation of learning performance. The concept also hypothesises that learning is facilitated when the learner is promptly informed whether a response is correct, and, if incorrect, of the direction of error.

AND exp education, medical/

• Feedback, biochemical - 0

• Feedback - 385

• Feedback, psychological - 10

• Knowledge of results (psychology) - 9

Results are transferable

• … to medical education searches and searches in general– lexical ambiguity of terms– inappropriate and/or inconsistent indexing– free text sacrifices specificity

Two unsatisfactory options:

• ignore the evidence • base your decisions on biased, irrelevant, or incorrect information

• hand search journals for every question

• 6 journals• 10 years• 30 minutes/issue

= 27 working days

Two (slightly less) unsatisfactory options:

• design extremely broad search strategies

• time intensive sorting through results

• rely on partial evidence from searches

• perhaps the best bet

One barrier to systematic searching:

Mortality

What is to be done?

Essentials for anyLevel of Searching:

1. Query formulation

2. Sources

3. Resources

4. Filters

5. Syntax

Define the search query

Identify and expand

concepts

Set the scope of the query

An Iterative Process:

1. Query formulation

McDonald S, Taylor L, Adams C. Searching the right database: a comparison of four databases for psychiatry journals. Health Libraries Review; 16: 151-6, 1999.

2. Sources

3. Resources

• Dedicated time

• Administrative support

• Professional support

4. Filters

• Know them

• Use them

5. Syntax

• Know it

• Use it

OSCE

• Objective structure clinical examination

• Objective structure clinical exam$

• Objective structure clinical test$

• Multiple station examination…

• Simulated patient examination...

• Standardi#ed patient examination...

• GOSCE, OSLER, etc.

Searching at the Correct Level of Detail

What has been done?• Raised Awareness

– Amongst teachers, researchers, students, deans, other practitioners in med ed

– Amongst information professionals and education technologists

• Published study guide for medical education• Creation of search filters• Completed systematic reviews• METRO

• Medical Education Taxonomy Research Organisation

• May 2002

• UK wide initiative

• LTSN-01 funding

• METRO-1 complete; METRO-2 beckons

• Annette Thain - Clinical Librarian Services

• Clinical teams had different needs for clinical librarian services

• Offered weekly search service after meetings

• SDIs

• Current Awareness Bulletins

• Often very specialised questions to help with unusual /rare presentations or complications

• Want max of 10 papers

• Prefer easy accessible as need to read paper that week before next clinic

• Would check – guidelines– Cochrane– Databases– Core/ respected journals– known authors– large trials– reviews– UK papers

Types of questions:• Often wanted info for the patient, or to answer a question the

patient had posed and the consultant needed the evidence.• Treatment options often effect of a combination• Clinical trials - often heard of trial at a meeting and wanting

any papers published and results to know if it might affect their patient.

• Want patient information on a rare topic usually tried websites of associations if usual sources failed

“Rapid” Systematic Reviews

• Policy Evaluation Unit

• to provide the evidence base for Ministers to make decisions

• team of researchers and information professionals• 85% of evidence retrieved and synthesised in 6-8

weeks• the remaining 15% is added when available

fin