The Meta-Narrative ReviewSystematic Reviewing Across DifferentParadigms

Henry W. W. Potts

Centre for Health Informatics & MultiprofessionalEducation (CHIME),

Institute of Epidemiology & Health Care, UCL

With thanks to Trish Greenhalgh, Geoff Wong & others

From Taylor & Potts (2008),Eur J Cancer 44(6):798-807cancer detection rate

• Systematic reviewing has evolved over time

• Meta-analysis for quantitative outcomes

• Some degree of methodological heterogeneitycan be handled with sub-group analyses

• Various ‘mixed methods’ approaches developedto combine qualitative and quantitative studies

Not just heterogeneity,

not just mixed methods,

but incommensurability

Problems of heterogeneity multiply with more complex questions, with multipleoutcomes, varying systems and different methodologies – different paradigms

Various approaches developed to review broad methods…

Moran-Ellis et al. (Qual Res 2006;6(1):45-59):

“Researchers who advocate the use of multiple methods often write interchangeablyabout ‘integrating’, ‘combining’ and ‘mixing’ methods […] [This] obscures the differencebetween (a) the processes by which methods (or data) are brought into relationship witheach other (combined, integrated, mixed) and (b) the claims made for the epistemologicalstatus of the resulting knowledge.”

Yardley & Bishop (In The SAGE Handbook of Qualitative Research in Psychology,2007: pp. 352-67):

‘Composite analysis’: retain integrity of each method – integrate findings rather than‘mixing methods’

Noblit & Hare (Meta-ethnography: Synthesising Qualitative Studies, 1988):

Distinction between integrative and interpretive reviews

Lewis & Grimes (Acad Manage Rev 1999;24:672-90):

Meta-triangulation: building theory from multiple paradigms

Meta-narrative review – key citations

1st: Greenhalgh, Robert, Macfarlane et al., MilbankQ 2004;82:581-629 / expanded as Diffusion of Innovations in

Health Service Organisations: A Systematic Literature Review,Blackwell BMJ Books

Methods paper: Greenhalgh, Robert, Macfarlane etal., Soc Sci Med 2005;61:417-30

2nd(ish): Greenhalgh, Potts, Wong et al., Milbank Q2009;87:729-88.

Publication standards: Wong, Greenhalgh,Westhorp et al., BMC Med 2013;11:20

Meta-narrative review – key principles

Use a historical and philosophical perspective as a pragmaticway of making sense of a diverse literature

• Pragmatism

• Pluralism

• Historicity

• Contestation

• Peer review

Key questions (from Kuhn, “The structureof scientific revolutions”)

• What research teams have researched this area?

• How did they CONCEPTUALISE the problem?

• What THEORIES did they use to link problem withpotential causes and impacts

• What METHODS did they define as ‘rigorous’ and‘valid’?

Application more post-Kuhnian than Kuhnian

Explore the literature

Open-ended question

Meta-narrative review (how to get started)

Research tradition C

Evaluate, summarise

Qualitycriteria

Theoreticalbasis

Research tradition B

Evaluate, summarise

Qualitycriteria

Theoreticalbasis

Research tradition A

Evaluate, summarise

Qualitycriteria

Theoreticalbasis

Meta-narrative map of underpinning traditions

Researchtradition

Disciplinaryroots

Definition &scope

General formatof researchquestion

EPRconceptualisedas...

EPR userconceptualisedas...

Contextconceptualisedas...

Healthinformationsystems

(Evidence-based)medicine,computerscience

Study ofstorage,computation& transmissionof clinical data.Focus often onbenefits ofEPRs andhow to achievethem

What is impactof technology X(EPR, DSS,etc.) onprocess Y (e.g.clinicianperformance)and outcomeZ?

Container forinformationabout patient;tool foraggregatingclinical datafor secondaryuses

Rationaldecision-makerwhosecognitive abilitysets limits towhat can beachievedwithoutcomputers

Potentialconfounderwhich can be‘controlledfor’ if right studydesign used

Changemanage-ment(withinhealthservicesresearch)

(Evidence-based)medicine,socialpsychology,management

Study ofachievingorganisation-level change inHealthcare

How can weimprovedelivery ofhealthcare andsustainimprovement?

Innovation that,if implementedwidely andconsistently,will improveprocess andoutcome ofcare

‘Resistant’agent whomust be trainedandincentivised toadopt newtechnologiesand ways ofworking

External milieuof interactingvariables thatserve as barriersor facilitators tochange efforts

Informationsystems(positivist)

Businessstudies,psychology,computerscience

Study of howorganisationsdo or do noradopt &assimilateinformationsystems

What factors(independentvariables)account forsuccess orfailure(dependentvariable) ofinformationsystem X inorganisation Y?

Unwelcomechange,likely to beresisted, andwhich mayfit poorly withorganisationalstructures &systems

Potentialadopter whomay engagewith or resistchange;member ofgroup whosepower basemay beenhanced orthreatened

External milieuof interactingvariables thatmediate ormoderate therelationshipbetween inputand outputvariables

Researchtradition

Disciplinaryroots

Definition &scope

General formatof researchquestion

EPRconceptualisedas...

EPR userconceptualisedas...

Contextconceptualisedas...

Informationsystems(interpret-ivist)

Management,sociology,socialpsychology,anthropology

Study of howorganisationalmembers makesense ofinformationsystems &therebyassimilate them

What meaningsdoesinformationsystem X holdfor members oforganization Y?How to achieveaccommoda-tion betweendifferent views?

Socio-technicalchange thatholds differentmeanings fordifferentindividuals andgroups

Stakeholderwhose ‘framing’of the EPR iscrucial to itsassimilation.Agent whosecreativity canbe drawn uponin this effort

Scene & settingfor an unfoldingstory; webs ofmeaning inwhichorganisationalactors aresuspended

Informationsystems(technology-in-practice)

Organizationalsociology,socialpsychology,philosophy

Study of howsocialstructuresrecursivelyshape & areshaped byhuman agency,& role oftechnology inthis

What is therelationshipbetweenorganisationalactors,technology X,and theorganisation –and how doesthis changeover time?

Itinerary andorganiserwhose physical& technicalpropertiesstructure &supportcollaborativeclinicalwork

Knowledgeablecreative agentfor whom socialstructures bothcreatepossibilities &limit thepossible

Generated &regeneratedthrough interplayof action &structure. Doesnot study‘technologies’ &‘contexts’separately buttechnologies-in-use

Computersupportedcooperativework

Computerscience,softwareengineering,psychology,sociology

Study of howgroups ofpeople workcollaboratively,supported byinformationtechnology

How cantechnologiessupport thework of multipleinteractingpeople?

Contextualizedartefact

Agent whoworks to localgoals incollaborationwith others &creativelyovercomeslimitations offormal tools

External milieuor emergentproperty ofaction(constituted by& inextricablefrom an activityinvolving people& technologies)

Researchtradition

Disciplinaryroots

Definition &scope

General formatof researchquestion

EPRconceptualisedas...

EPR userconceptualisedas...

Contextconceptualisedas...

Criticalsociology

Sociology,philosophy

Study ofrelationshipbetweenpeople & socialorder, & role oftechnologies inthis

What socialstructures& powerimbalances areembedded intechnology X, &what impactdoes this haveon social roles/relationships?

Implicated inmicro & macropowerdynamics(because of linkbetweenknowledge& power)

Constrained bydominant socialStructures,which may bebuilt intotechnologies bydesigners

Social & materialconditions intowhich theunequal socialorder isinscribed;more or lessstable structureof macro socialrelations

Empiricalphilosophy(actornetworkcasestudies)

Philosophy,sociology,linguistics

Study ofsociotechnicalnetworks:considers howrelationships &power shiftwithin network

How hasnetwork,with its variousrelationships,work practices& risks,changed as aresult oftechnology X?

Actor in anetwork

Actor in anetwork

EPR & itscontexttogether formthe network; theone cannot bestudied withoutthe other

Systemsapproaches

Systems &managementresearch,drawing oncognitivepsychology,CSCW & ANT

Systemsperspective

What role doesthe EPR playwithin acomplexhealthcaresystem?

Component ofcomplex socio-technicalsystem whosefeatures &properties maycome togetherinunpredictableways

Component ofcomplex socio-technicalsystem whosefeatures &properties maycome togetherinunpredictableways

Complex,changingenvironment

Synthesis phase

Highlight similarities and differences in the findingsfrom different traditions

Contestation between the disciplines is data (andleads to higher order constructs)

Offer conclusions of the general format “incircumstances such as X, don’t forget to thinkabout Y”

Summary• Techno-utopianism

– Promoting (health informatics) or challenging (technology-in-practice, CSCW) it

• Recursivity• Different affordances of paper and electronic

– Health informatics stresses advantages of electronic; HCI/CSCWand technology structuration stress paper has advantages too

• Records support work / nature of co-operative work– Different participants’ view of others’ work / hidden work (feminist

critiques of hidden work) and changed visibility– Different people do different things & EPRs help or hinder people

differently– Impacts on power relationships

• EPRs are not an agreed and agreeable commonaccount, but communicative, boundary objects

con

text

for

all

tech

no

logy

stu

die

sm

ate

ria

lity

na

ture

of

me

dic

alw

ork

na

ture

of

EH

Rs

Crisis

Thomas Kuhn“The Structure of Scientific Revolutions” (1962)

Para

dig

msh

ift

Normal science

time

Pre-science Normal science

Thomas Kuhn“The Structure of Scientific Revolutions” (1962)

A discipline sees a repeated cycle of ‘crises’,leading to ‘paradigm shifts’, out of which emerges‘normal science’.

Greenhalgh, Robert, Macfarlane et al.“Diffusion of Innovations in Service Organizations: SystematicReview and Recommendations” (2004)

Different disciplines separately develop aparadigm and conduct ‘normal science’.

Rise and fall of diffusion research in rural sociology

0

5

10

15

20

25

30

35

40

41 44 47 50 53 56 59 62 65 68 71 74 77 80

Nu

mb

ero

fp

ub

licat

ion

s

DEVELOPED NATIONS DEVELOPING NATIONS

Rise and fall of diffusion research in health related fields

0

20

40

60

80

100

120

140

160

1975-1976

1977-1978

1979-1980

1981-1982

1983-1984

1985-1986

1987-1988

1989-1990

1991-1992

1993-1994

1995-1996

1997-1998

1999-2000

2001-2002

Nu

mbe

ro

fpu

blic

atio

ns

NURSING MEDICAL EDUCATION

EBM OR GUIDELINES DELIVERY OF HEALTH CARE

Greenhalgh, Potts, Wong et al.“Tensions and Paradoxes in Electronic Patient Record Research: ASystematic Literature Review Using the Meta narrative Method” (2009)

Reflections

• The piles are subjective (but let’s not pretend‘traditional’ systematic reviewing isn’t)

• Synthesis difficult

• Very different picture to traditional Cochrane/EBMapproach

• Rich array of theories and methods

• Systematic, but interpretive

End of talk – turn offthe computer.

Thank you for yourattention.

Ask me questions.

Henry Potts,h.potts@ucl.ac.uk

Cite as… Potts HWW (2013). “The Meta-narrative Review: Systematic ReviewingAcross Different Paradigms.” At Mixed Method Evidence Synthesis:How to Combine Quantitative and Qualitative Evidence in Systematic Reviewsworkshop, University of Manchester/NICE Evidence Synthesis Network,Manchester, 12 Mar 2013.

References:

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O (2004).Diffusion of innovations in service organisations: Systematic literaturereview and recommendations for future research. Milbank Quarterly, 82,581-629.

Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R(2005). Storylines of research in diffusion of innovation: A meta-narrativeapproach to systematic review. Social Science & Medicine, 61, 417-30.

Greenhalgh T, Potts HWW, Wong G, Bark P, Swinglehurst D (2009).Tensions and paradoxes in electronic patient record research: Asystematic literature review using the meta-narrative method. MilbankQuarterly, 87(4), 729-88.

Wong G, Greenhalgh T, Westhorp G, Buckingham J, Pawson R (2013).RAMESES publication standards: Meta-narrative reviews. BMC Medicine,11, 20.