Issues and Challenges around Searching the Literature

Overview

• What is Already Known on Searching the Qualitative Research Literature

• Overview of Methodological Issues/Challenges• Recent Developments (with focus on literature

of last two years)• Outstanding Issues/Challenges

What is already known

What is Already Known? – 1 (Gallacher et al, 2013)

• Finding relevant qualitative studies arduous due to inadequate indexing (Dixon Woods et al, 2007; Ring et al 2011)

• Papers often lack abstracts or include non-informative titles, making it difficult to establish relevance (Dixon Woods et al, 2007).

• Several papers outline strategies (filters) for searching for qualitative studies (Walters et al, 2006; Wilczynski et al, 2007; Wong et al, 2004; McKibbon et al, 2006)

• Helpful techniques involve electronic or hand searching (Greenhalgh et al 2005; Bates, 1989):– Reference or footnote tracking (looking backwards at references in articles found).– Citation tracking (tracking forward subsequently citing articles).– Personal knowledge and personal contacts.– Contacting the authors of known papers or experts in the field.– Hand searching relevant journals.– Internet browsing such as berry picking (where one search leads to another and ‘clusters’ of papers are

often found together).

• Greenhalgh et al (2005) found only 30% of primary sources from predefined search strategy. 51% found by other predefined methods (i.e. reference, footnote and citation tracking).

• However this may be topic specific and may reflect a priori decisions to concentrate on non-electronic sources

What is Already Known? - 2

• “Literature searches…open-ended iterative processes where the topic or research question of interest is honed over time as the nature of the evidence becomes more apparent” (Finfgeld-Connett & Johnson 2013)

• Number of articles not only critical factor – reports “may lack enough thick description to fully develop concepts and the interrelationships among them” (Finfgeld-Connett & Johnson 2013)

• Unpublished studies may contain rich, thick description• Goal may not be aggregative – theoretical saturation

may play a part – selection of sample is crucial

References for What is Known Already - 1

• Bates MJ: The design of browsing and berrypicking techniques for on-line search interface. Online Rev 1989, 13:407-424.

• Dixon-Woods M, Bonas S, Booth A et al.: How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res 2006, 6:27-44.

• Finfgeld Connett, D., & Johnson, E. D. (2013). Literature search strategies ‐for conducting knowledge building and theory generating qualitative ‐ ‐systematic reviews. Journal of Advanced Nursing, 69(1), 194-204.

• Greenhalgh T, Peacock R: Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005, 331:1064-1065.

• Ring N, Jepson R, Ritchie K: Methods of synthesising qualitative research for health technology assessment. Int J Technol Assess Health Care 2011, 27:384-390

References for What is Known Already - 2

• Walters LA, Wilczynski NL, Haynes RB: Developing optimal search strategies for retrieving clinically relevant qualitative studies in EMBASE. Qual Health Res 2006, 16:162-168. PubMed Abstract | Publisher Full Text

• Wilczynski NL, Marks S, Haynes RB: Search strategies for identifying qualitative studies in CINAHL. Qual Health Res 2007, 17:705-710. PubMed Abstract | Publisher Full Text

• Wong SL, Wilczynski NL, Haynes RB: Developing optimal search strategies for detecting clinically relevant qualitative studies in MEDLINE. In Medinfo 2004: Proceedings of the 11th World Congress on Medical Informatics; San Francisco. Edited by Fieschi M, Coiera E, Jack Li YC. Amsterdam: IOS Press; 2004:311-314.

• McKibbon KA, Wilczynski NL, Haynes RB: Developing optimal search strategies for retrieving qualitative studies in PsycINFO. Eval Health Prof 2006, 29:440-454. PubMed Abstract | Publisher Full Text

Overview of Methodological Issues/Challenges

• Bias towards/Predominance of quantitative research and publication of resultant reports

• Non-optimal indexing of qualitative studies (CINAHL more evolved than MEDLINE)

• Qualitative research represents various research methodologies, including ethnography, phenomenology, grounded theory and narrative analysis, which may hinder retrieval

• Lack of informative manuscript titles and abstracts

Overcoming barriers:• Searches should be as transparent as possible

without jeopardizing the creativity and complexity of the process (Finfgeld-Connett & Johnson 2013)

• Key “test” for knowledge-building/theory generating review – would small amounts of conflicting information substantially change the findings? [Theoretical Saturation/ Qualitative Sensitivity Analysis]

• Weaknesses in indexing mean that sensitivity of searches may need to be reduced to allow time for other search strategies (Pearson et al, 2011)

Simple search strategies vs complex ones

• Three broad-based terms (i.e. “qualitative”, “findings” and “interviews”) as effective as more complex search strategies in identifying relevant qualitative research reports (Flemming & Briggs 2007).

• Search strategies with broad search terms (“qualitative research” or “qualitative studies” or “interviews”) [combined with CAM terms] had highest recall and precision (Franzel, 2013).

• Within time-limited context, protocol-driven, targeted, and reference-checking search strategies most effective (Pearson et al, 2011)

Intervention Searching vs Condition Searching (Lorenc et al, 2012)

• Tying search terms of SR of qualitative evidence too closely to interventions may compromise consistency of the review.

• Dilemma: Performing condition-wide searches (with no other change to strategies) would become highly over-inclusive and volumes of records impracticably large. (Suggests need for alternative sampling stategies)

Is More Necessarily Better?• For knowledge-building and theory-generating

systematic reviews, “more is better only when it helps to fully explicate a concept, substantiate an interconnection between or among concepts, or build a line of argument. Simply, more of the same does not necessarily help to achieve these objectives. In fact, collecting more of the same may merely escalate the cost of a study, clutter the database and obfuscate important inferences. Concepts and interrelationships among them can only be more fully explicated based on data that adds depth, breadth, meaning and understanding to a phenomenon” (Finfgeld-Connett & Johnson 2013).

Searching for religion and mental health studies required health, social science, and grey

literature databases (Wright et al, 2014)• PsycINFO = best performing database• ArabPsyNet, CINAHL, Dissertations and Theses, EMBASE, Global

Health, Health Management Information Consortium, MEDLINE, PsycINFO, and Sociological Abstracts essential to retrieve included references.

• Citation tracking and personal library of one of the research teams made significant contributions of unique, relevant references.

• Religion studies databases (Am Theo Lib Assoc, FRANCIS) did not provide unique, relevant references.

• Literature searches for reviews/evidence syntheses of religion and health studies should include social science, grey literature, non-Western databases, personal libraries, and citation tracking activities.

Exhaustive versus Expansive

That is the Question!

Exhaustive versus Expansive (Finfgeld-Connett & Johnson 2013)

• Exhaustive searches when conducting summative and aggregative systematic reviews

• Expansive searches when conducting knowledge-building and theory-generating systematic reviews

Exhaustive versus Expansive

• Multiple databases• Comprehensive list of

terms• Assumption of

homogeneity• Other approaches are

“supplementary”

• Databases to reflect contributing disciplines

• Terminology may characterise “narrative” of each discipline

• Likelihood of divergent cases• Sibling studies (shared

context)• Cluster searching• Citation searching • “Supplementary” techniques

may be more important“Combing the Area”

“Following Up Leads”

Outstanding challenges (Gallacher et al, 2013)

• Creating an appropriately sensitive and specific search strategy was a significant challenge….

• Adding ‘qualitative methods’ made search strategy considerably more specific while retaining sensitivity, as demonstrated by return of all key papers identified in scoping search.

• Our final results showed that 94% of papers were identified by our predefined database search.

Recent Developments

• Importance of Context/Theory• Database Coverage• Importance of Supplementary Search

Methods• Appropriate Selection of Sampling

Methods • Reporting Standards (e.g. ENTREQ)

Searching for Contextual Richness

Requires identification of related (sibling) reports i.e. cluster searching (cluster becomes unit of analysis, not study) (Booth et al, 2013)

Cluster searching for “Siblings” (Booth et al, 2013)

Element Procedural Steps

Citations Backwards reference chaining

Lead Authors Author searching; Backwards reference chaining

Unpublished materials Web searches; repositories

Scholar searches Citations

Theories Backwards reference chaining

Early Examples Cited works (Forward reference chaining)

Related Projects Co-citations

Searching for Theories

• Theory not typically reported in Abstracts• Theoretical base differs by discipline cp. HSR vs

Public Health vs Nursing vs Psychology vs Sociology• Reporting of Theory differs by discipline• Level of Theory may vary e.g. Individual versus

Society (Psychology vs Sociology)• When is a “theory” a Theory? – labelling (model,

framework, concepts ) and naming (“Health Belief Model”)

Database Coverage

• CAM search - PubMed yielded 87% of relevant included qualitative studies (Franzel et al, 2013).

• Five different QES PubMed coverage values 35/44 (79.5%); 9/10 (90%); 10/11 (91%); 9/9 (100%); 7/28 (25% - Grey literature) (Booth 2012 [Unpublished])

• But 5/28 of studies located only through supplementary searches of three sources (Stansfield et al, 2012). 21 search sources required to locate all studies. [Explanation - Role of Grey Literature and Geographical focus – UK only]

Supplementary Strategies

More diffuse topic, move beyond electronic searching

Supplementary Strategies - 1

• Multiple search strategy more likely to identify relevant QR than sole reliance on electronic searching.

• Purpose of synthesis determines appropriate sampling/search strategy. E.g. mapping out key conceptual developments – if aim not aggregative, omission of papers unlikely to have dramatic effect on results.

• Suggests max. circa 40 papers - difficult to maintain sufficient familiarity with > 40 papers (Campbell et al, 2011)

Supplementary Strategies - 2

• Need ‘belt and braces’ (hand-searching; consultation with experts)

• Searching for books/theses particularly challenging (not indexed in same way as journal papers) (Campbell et al, 2011)

• Snowballing and consultation with experts for a realist review (Pawson et al, 2004)

• Obtaining authors' suggestions - resource-intensive process with negligible results (Pearson et al, 2011)

• Additional search techniques essential to locate further high quality references (Papaioannou et al, 2010)

Sampling: Appropriate ≠ Comprehensive (Suri, 2011)

• 16 strategies for sampling in QES• E.g. Snowball sampling - seeking information from key

informants about other ‘information-rich cases’. – ‘The chain of recommended informants would typically

diverge initially as many possible sources are recommended, then converge as a few key names get mentioned over and over’ (Patton, 2002, p. 237).

• Identify most cited primary research reports by ‘footnote chasing’ (searching citation indices, browsing through bibliographies, previous research syntheses, primary research reports, policy documents, papers written by practitioners and papers written for practitioners).

Combination or Mixed Purposeful Sampling (Suri, 2011)

• Employ two or more sampling strategies to select evidence to adequately address purpose.

• Mixed purposeful sampling can facilitate triangulation and flexibility in meeting the needs of multiple stakeholders (e.g. extensive sampling for generalisations at higher level of abstraction. Typical case sampling to provide readers with immediacy of typical studies that contributed towards informing more abstract generalisations).

• When selecting combination of sampling strategies, synthesists must reflect on how strategies complement each other.

Footnote chasing (Suri, 2011)

• cp. footnote chasing for exhaustive sampling, footnote chasing for snowball sampling involves locating most cited papers.

• However, may reinforce confirmatory bias (i.e. studies agreeing with prevalent wisdom more likely to be published and cited, studies that contest conventional wisdom less likely to be published or cited)

ENTREQ & Searching (Tong et al, 2012)

3 Approach to searching

Indicate whether search was pre-planned (comprehensive search strategies to seek all available studies) or iterative (to seek all available concepts until they theoretical saturation is achieved).

4 Inclusion criteriaSpecify inclusion/exclusion criteria (e.g. in terms of population, language, year limits, type of publication, study type).

5 Data sources

Describe information sources used (e.g.electronic databases (MEDLINE, EMBASE, CINAHL, psycINFO, Econlit), grey literature databases (digital thesis, policy reports), relevant organisational websites, experts, information specialists, generic web searches (Google Scholar) hand searching, reference lists) and when searches conducted; provide rationale for using data sources.

6 Electronic Search strategy

Describe literature search (e.g. provide electronic search strategies with population terms, clinical or health topic terms, experiential or social phenomena related terms, filters for qualitative research, and search limits).

Improvement in Search Reporting

1988-2004 2005-2008

(Hannes & Macaitis, 2012)

Outstanding Issues/Challenges

• How to understand publication bias in qualitative research (not around positive/negative findings)?

• How to systematise (and document) more intuitive search approaches e.g. cluster searching and searching for theory?

• How does sampling strategy translate into search strategy?

• How to construct sampling frames for studies?• How to sample for diversity?• How many sources are enough?• How to retrieve rich data? How to retrieve data on

theory and context?

Publication Bias in Qualitative Research? – Part One

• “This does not mean that publication biases do not exist in….qualitative research….a bias of potentially greater proportions may threaten searches for qualitative research reports…..in some circles, qualitative research is perceived…of lesser quality and value than quantitative research……qualitative studies may be less frequently conducted, submitted for publication and/or published in high quality and easily accessible journals….raw data (i.e. research findings)….needed to conduct a qualitative systematic review may not be readily available” (Finfgeld-Connett & Johnson 2013)

Publication Bias in Qualitative Research? – Part Two

• “ignoring grey literature, such as dissertations/theses, government reports, monographs and books, on the basis that it may be of lesser quality (as with quantitative research) is empirically and logically invalid…..[such] documents may be particularly rich sources of qualitative data as page limits are not generally imposed. Also, although lengthy report formats are relatively uncommon in the health sciences, they tend to be the norm in disciplines, such as anthropology….., where context-rich data are likely to be found”. (Finfgeld-Connett & Johnson 2013)

• Qualitative researchers often choose to publish in book form (“Truncation bias”)

Conclusions

• Increasing Importance of Explanatory Sources (e.g. Context and Theory)• Need for Ongoing Investigation of Database

Coverage and Supplementary Search Techniques• Requires Exploration/Selection of

Appropriate Sampling Methods• Bottomline: Value versus Effort Trade-Off

References - 1• Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E. Towards a

methodology for cluster searching to provide conceptual and contextual "richness" for systematic reviews of complex interventions: case study (CLUSTER). BMC Med Res Methodol. 2013 Sep 28;13:118.

• Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill R, et al. Evaluating meta-ethnography: systematic analysis and synthesis of qualitative research. Health Technol Assess 2011;15(43).

• Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: evaluation of three strategies. Journal of Advanced Nursing, 57(1), 95-100.

• Gallacher K, Jani B, Morrison D, Macdonald S, Blane D, Erwin P, May CR, Montori VM, Eton DT, Smith F, Batty GD, Mair FS; International Minimally Disruptive Medicine Workgroup. Qualitative systematic reviews of treatment burden in stroke, heart failure and diabetes - methodological challenges and solutions. BMC Med Res Methodol. 2013 Jan 28;13:10.

References - 2• Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B. (2013).

How to locate and appraise qualitative research in complementary and alternative medicine. BMC complementary and alternative medicine, 13(1), 125.

• Guise, J. M., Chang, C., Viswanathan, M., Glick, S., Treadwell, J., Umscheid, C. A., ... & Trikalinos, T. (2014). Systematic Reviews of Complex Multicomponent Health Care Interventions [Internet]. http://www.ncbi.nlm.nih.gov/books/NBK194851/

• Hannes, K., & Macaitis, K. (2012). A move to more systematic and transparent approaches in qualitative evidence synthesis: update on a review of published papers. Qualitative Research, 12(4), 402-442.

• Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010). Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Information & Libraries Journal, 27(2), 114-122.

References - 3• Pearson, M., Moxham, T., & Ashton, K. (2011). Effectiveness of search strategies

for qualitative research about barriers and facilitators of program delivery. Evaluation & the Health Professions, 34(3), 297-308.

• Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The selection of search sources influences the findings of a systematic review of people’s views: a case study in public health. BMC Medical Research Methodology, 12(1), 55.

• Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative Research Journal, 11(2), 63-75.

• Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC medical research methodology, 12(1), 181.

• Suri, H. (2013). Towards Methodologically Inclusive Research Syntheses: Expanding Possibilities. Routledge.

• Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies required health, social science, and grey literature databases. J Clin Epidemiol. 2014 Jul;67(7):800-10.