10.55am Issues and Challenges around Searching the Literature

Dr Andrew Booth, Reader In Evidence Based Information Practice, School of Health and Related Research, University of Sheffield

Overview

• What is Already Known on Searching the Qualitative Research Literature

• Overview of Methodological Issues/Challenges• Recent Developments (with focus on literature

of last two years)• Outstanding Issues/Challenges

What is already known

Booth A. Chapter 3: Searching for Studies. In: Noyes J et al (editors), Supplementary Guidance for Inclusion of Qualitative Research in Cochrane

Systematic Reviews of Interventions. Version 1 (updated August 2011). Cochrane Collaboration Qualitative Methods Group, 2011.

• Retrieval of qualitative evidence syntheses or primary qualitative research studies can inform definition and refinement of Cochrane Review Question.

• For scoping, use of brief methodological filters may be sufficient. • Retrieval of qualitative trial-related evidence should not rely

merely on serendipity or chance occurrence.• Systematic approaches to identifying trial-related evidence will

include searching for process evaluations; identification of “sibling studies”, use of Related Articles features and citation searching.

• Methods exist to identify studies for a full qualitative evidence synthesis….(See Appendix A)

Appendix A – Supplementary Cochrane Guidance

1. Qualitative research methodological filters.2. Examination of references. (But may lead to bias in identification of

studies, c.p. divergent sampling). 3. Using “Related Articles” features. (But studies may be related for

various factors so this will yield both relevant and irrelevant “hits”).4. Citation searching (forward chaining and backward chaining (Bates,

1989)). ( But may introduce potential bias so avoid over-reliance on this method.

In addition the following search methods should be used to achieve the widest possible coverage of eligible studies:5. Handsearching6. Identifying theses and dissertations7. Internet searching

Predominance of “classic” systematic review model

• “When conducting exhaustive searches, topics and research questions are generally established a priori and ideally, all relevant research reports are subsequently identified and secured in a predetermined, methodical and progressive manner” (Finfgeld-Connett & Johnson 2013)

• “Linear flowcharts [e.g. PRISMA Flow Diagram] that illustrate the number of documents initially identified, excluded and retained are frequently used to illustrate these types of unidirectional search processes” (Finfgeld-Connett & Johnson 2013)

• BUT increasingly challenged – iterative, sampling – purposive and theoretical and in terms of yield

What is Already Known?• “Literature searches are seen as open-ended iterative

processes where the topic or research question of interest is honed over time as the nature of the evidence becomes more apparent” (Finfgeld-Connett & Johnson 2013)

• Number of articles not only critical factor – reports “may lack enough thick description to fully develop concepts and the interrelationships among them” (Finfgeld-Connett & Johnson 2013)

• Unpublished studies may contain particularly rich, thick description

• Goal may not be aggregative – theoretical saturation may play a part – selection of sample is crucial

Overview of Methodological Issues/Challenges

• Bias towards/Predominance of quantitative research and publication of resultant reports

• Non-optimal indexing of qualitative studies (CINAHL more evolved than MEDLINE)

• Qualitative research represents various research methodologies, including ethnography, phenomenology, grounded theory and narrative analysis, which may hinder retrieval

• Lack of informative manuscript titles and abstracts

Methods for overcoming barriers include:

• Berry picking strategies (Bates, 1989)– Electronic keyword searches– Electronic citation searches– Combing reference lists of key articles– Manually searching key journals– Searching grey literature

• Optimized search strategies (i.e. hedges, filters)

Overcoming barriers:• Searches should be as transparent as possible

without jeopardizing the creativity and complexity of the process (Finfgeld-Connett & Johnson 2013)

• Key “test” for knowledge-building/theory generating review – would small amounts of conflicting information substantially change the findings? [Theoretical Saturation/ Qualitative Sensitivity Analysis]

• Weaknesses in indexing mean that sensitivity of searches may need to be reduced to allow time for other search strategies (Pearson et al, 2011)

Simple search strategies may be aseffective as more complex ones

• E.g. in study of patients’ perceptions of living with a leg ulcer, three broad-based terms (i.e. “qualitative”, “findings” and “interviews”) were as effective as more complex search strategies in identifying relevant qualitative research reports (Flemming & Briggs 2007).

• Within time-limited context, protocol-driven, targeted, and reference-checking search strategies most effective (Pearson et al, 2011)

Intervention Searching vs Condition Searching (Lorenc et al, 2012)

• Tying search terms of SR of qualitative evidence too closely to interventions may compromise consistency of the review.

• Dilemma: Performing condition-wide searches (with no other change to strategies) would become highly over-inclusive and volumes of records impracticably large. (Suggests need for alternative sampling stategies)

Is More Necessarily Better?• In the context of knowledge-building and theory-

generating systematic reviews, “more is better only when it helps to fully explicate a concept, substantiate an interconnection between or among concepts, or build a line of argument. Simply, more of the same does not necessarily help to achieve these objectives. In fact, collecting more of the same may merely escalate the cost of a study, clutter the database and obfuscate important inferences. Concepts and the interrelationships among them can only be more fully explicated based on data that adds depth, breadth, meaning and understanding to a phenomenon” (Finfgeld-Connett & Johnson 2013).

Exhaustive versus Expansive

That is the Question!

Exhaustive versus Expansive (Finfgeld-Connett & Johnson 2013)

• Exhaustive searches recommended when conducting summative and aggregative systematic reviews

• Expansive searches recommended when conducting knowledge-building and theory-generating systematic reviews

Exhaustive versus Expansive

• Multiple databases• Comprehensive list of

terms• Assumption of

homogeneity• Other approaches are

“supplementary”

• Databases to reflect contributing disciplines

• Terminology may characterise “narrative” of each discipline

• Likelihood of divergent cases• Sibling studies (shared

context)• Cluster searching• Citation searching • “Supplementary” techniques

may be more important“Combing the Area”

“Following Up Leads”

Recent Developments

• Importance of Context• Importance of Theory• Database Coverage• Importance of Supplementary Search

Methods• Appropriate Selection of Sampling Methods • Reporting Standards (e.g. ENTREQ)

Searching for Contextual Richness

1. Requires identification of related (sibling) reports i.e. cluster searching (cluster becomes unit of analysis, not study) (Booth et al, in press)

2. Requires identification of appropriate contextual studies (source selection) (Stansfield et al, 2012)

3. Multi-context versus context-specific qualitative evidence syntheses (Hannes & Harden, 2012)

Searching for Theories

• Theory not typically reported in Abstracts• Theoretical base differs by discipline cp. HSR vs

Public Health vs Nursing vs Psychology vs Sociology• Reporting of Theory differs by discipline• Level of Theory may vary e.g. Individual versus

Society (Psychology vs Sociology)• When is a “theory” a Theory? – labelling (model,

framework, concepts ) and naming (“Health Belief Model”)

Cluster searching for “Siblings”Element Procedural Steps

Citations Backwards reference chaining

Lead Authors Author searching; Backwards reference chaining

Unpublished materials Web searches; repositories

Scholar searches Citations

Theories Backwards reference chaining

Early Examples Cited works (Forward reference chaining)

Related Projects Co-citations

BeHEMoTh (Booth et al, in process)Be - Behaviour of interest: Way population or patient interacts with health context e.g. access for a service, compliance, attitude to policy.H - Health context: i.e. the service, policy, programme or interventionE – Exclusions: To exclude non-theoretical /technical models (depends on volume).

MoTh - Models or Theories – operationalized as a generic “model* or theor* or concept* or framework*” strategy together with named models or theories if required.

Four Phase Process

1. “Trawling” using BeHEMoTH structure (dropping concepts as appropriate)

2. “Depth-charging” using Behaviour and Health context with most common theories (see next slide)

3. “Fishing” using named item searches for list of theories generated from Phase One (above) [excluding those already covered in Phase Two].

4. “Using a sprat” – citation searching (combined with topic)

Database Coverage

• CAM search - PubMed yielded 87% of relevant included qualitative studies (Franzel et al, 2013).

• Five different QES PubMed coverage values 35/44 (79.5%); 9/10 (90%); 10/11 (91%); 9/9 (100%); 7/28 (25% - Grey literature) (Booth 2012 [Unpublished])

• But (Stansfield et al, 2012) over a sixth (5/28) of studies located only through supplementary searches of three sources. 21 search sources required to locate all studies. [Explanation - Role of Grey Literature and Geographical focus – UK only]

Supplementary Strategies

More diffuse topic, more beyond electronic searching

Supplementary Strategies - 1

• Multiple search strategy more likely to identify relevant QR than relying solely on electronic searching.

• Purpose of synthesis determines appropriate sampling/search strategy. E.g. mapping out key conceptual developments – if aim not aggregative, omission of papers unlikely to have dramatic effect on results.

• Suggests max. circa 40 papers - difficult to maintain sufficient familiarity with > 40 papers (Campbell et al, 2011)

Supplementary Strategies - 2• Need ‘belt and braces’ (hand-searching; consultation with

experts)• Searching for books/theses particularly challenging (not

indexed in same way as journal papers) (Campbell et al, 2011)

• Pawson et al. (2004) recommend snowballing and consultation with experts for a realist review

• Obtaining authors' suggestions - resource-intensive process with negligible results (Pearson et al, 2011)

• Conventional search strategy = main strategy but additional search techniques essential to locate further high quality references (Papaioannou et al, 2010)

Sampling: Appropriate ≠ Comprehensive (Suri, 2011)

• 16 strategies for sampling in QES• E.g. Snowball sampling - seeking information from key

informants about other ‘information-rich cases’. – ‘The chain of recommended informants would typically

diverge initially as many possible sources are recommended, then converge as a few key names get mentioned over and over’ (Patton, 2002, p. 237).

• Identify most cited primary research reports by ‘footnote chasing’ (searching citation indices, browsing through bibliographies, previous research syntheses, primary research reports, policy documents, papers written by practitioners and papers written for practitioners).

Combination or Mixed Purposeful Sampling (Suri, 2011)

• Employ combination of two or more sampling strategies to select evidence to adequately address purpose.

• Mixed purposeful sampling can facilitate triangulation and flexibility in meeting the needs of multiple stakeholders (e.g. extensive sampling for generalisations at higher level of abstraction. Then, typical case sampling to provide readers with immediacy of typical studies that contributed towards informing more abstract generalisations).

• When selecting combination of sampling strategies, synthesists must reflect on how those strategies complement each other.

Footnote chasing (Suri, 2011)

• cp. footnote chasing for exhaustive sampling, footnote chasing for snowball sampling involves locating most cited papers.

• However, may reinforce confirmatory bias (i.e. studies agreeing with prevalent wisdom more likely to be published and cited, studies that contest conventional wisdom less likely to be published or cited)

ENTREQ & Searching (Tong et al, 2012)

3 Approach to searching

Indicate whether search was pre-planned (comprehensive search strategies to seek all available studies) or iterative (to seek all available concepts until they theoretical saturation is achieved).

4 Inclusion criteriaSpecify inclusion/exclusion criteria (e.g. in terms of population, language, year limits, type of publication, study type).

5 Data sources

Describe information sources used (e.g.electronic databases (MEDLINE, EMBASE, CINAHL, psycINFO, Econlit), grey literature databases (digital thesis, policy reports), relevant organisational websites, experts, information specialists, generic web searches (Google Scholar) hand searching, reference lists) and when searches conducted; provide rationale for using data sources.

6 Electronic Search strategy

Describe literature search (e.g. provide electronic search strategies with population terms, clinical or health topic terms, experiential or social phenomena related terms, filters for qualitative research, and search limits).

Improvement in Search Reporting

1988-2004 2005-2008

(Hannes & Macaitis, 2012)

Outstanding Issues/Challenges• How to understand publication bias in qualitative research

(not around positive/negative findings)?• How to systematise (and document) more intuitive search

approaches e.g. cluster searching and searching for theory?

• How does sampling strategy translate into search strategy?

• How should sampling frames for studies be constructed?• How to sample for diversity?• How many sources are enough?• How to retrieve rich data? How to retrieve data on theory

and context?

Publication Bias in Qualitative Research? – Part One

• “This does not mean that publication biases do not exist in….qualitative research….a bias of potentially greater proportions may threaten searches for qualitative research reports…..in some circles, qualitative research is perceived…of lesser quality and value than quantitative research……qualitative studies may be less frequently conducted, submitted for publication and/or published in high quality and easily accessible journals….raw data (i.e. research findings)….needed to conduct a qualitative systematic review may not be readily available” (Finfgeld-Connett & Johnson 2013)

Publication Bias in Qualitative Research? – Part Two

• “ignoring grey literature, such as dissertations/theses, government reports, monographs and books, on the basis that it may be of lesser quality (as with quantitative research) is empirically and logically invalid…..[such] documents may be particularly rich sources of qualitative data as page limits are not generally imposed. Also, although lengthy report formats are relatively uncommon in the health sciences, they tend to be the norm in disciplines, such as anthropology….., where context-rich data are likely to be found”. (Finfgeld-Connett & Johnson 2013)

• Qualitative researchers often choose to publish in book form (“Truncation bias”)

Conclusions

• Increasing Importance of Explanatory Sources (e.g. Context and Theory)• Need for Ongoing Investigation of Database

Coverage and Supplementary Search Techniques• Requires Exploration/Selection of

Appropriate Sampling Methods• Bottomline: Value versus Effort Trade-Off

References - 1• Bates, M. J. (1989). The design of browsing and berrypicking

techniques for the online search interface. Online Information Review, 13(5), 407-424.

• Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess 2011;15(43).

• Finfgeld Connett, D., & Johnson, E. D. (2013). Literature search ‐strategies for conducting knowledge building and theory‐ ‐generating qualitative systematic reviews. Journal of Advanced Nursing, 69(1), 194-204.

• Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing, 57(1), 95-100.

References - 2• Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B.

(2013). How to locate and appraise qualitative research in complementary and alternative medicine. BMC complementary and alternative medicine, 13(1), 125.

• Hannes, K., & Harden, A. (2011). Multi context versus context‐ ‐specific qualitative evidence syntheses: combining the best of both. Research Synthesis Methods, 2(4), 271-278.

• Hannes, K., & Macaitis, K. (2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qualitative Research, 12(4), 402-442.

• Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114-122.

References - 3• Pawson R, Greenhalgh T, Harvey G, Walshe K: Realist synthesis: an

introduction. RMP Methods Paper 2/2004. Manchester, UK: ESRC Research Methods Programme, University of Manchester; 2004.

• Pearson, M., Moxham, T., & Ashton, K. (2011). Effectiveness of search strategies for qualitative research about barriers and facilitators of program delivery. Evaluation & the Health Professions, 34(3), 297-308.

• Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people’s views: a case study in public health. BMC Medical Research Methodology, 12(1), 55.

• Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative Research Journal, 11(2), 63-75.

• Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology, 12(1), 181.