Issues and challenges around integrating qualitative and quantitative data

Angela Harden, Professor of Community and Family Health

InCQuuiRES, 8th September, 2011

• A centre for research, innovation and community engagement to promote health and wellbeing.

• 15 members of staff from a range of health and social science disciplines.

• Includes collaborative programme of research with Newham University Hospital NHS Trust (NUHT) focused on women, children and young people

INSTITUTE FOR HEALTH AND HUMAN DEVELOPMENT

Overview of current SR projectsTitle Funder Main collaborating

institutions

The effects of schools and school-environment interventions on health

NIHR LSHTM, UEL, IOE

Can specific approaches to community engagement help to reduce inequalities in health?

NIHR IOE, UEL, LSE

A meta-narrative review of conceptualisations and meanings of ‘community’ within and across research traditions

AHRC UEL

The acceptability, feasibility and transferability of interventions to improve early uptake of antenatal care

NIHR UEL, IOE

Acknowledgements

• Colleagues at the EPPI-Centre, Social Science Research Unit, Institute of Education

• Co-convenors of the Cochrane Qualitative Research Methods Group

Outline

• Drivers and challenges of integration

• Review of approaches and methods

• Comparing methods

• Future challenges and issues

Drivers for integration• Greater recognition of the value of qualitative research in evidence-

based policy• ‘Empty’ reviews• Public perspectives and experiences • Systematic reviews of complex interventions• Issues of process and implementation• Extension of evidence-based health care to other areas of public

policy• Growing tradition for integration in primary research (mixed

methods research)• Dedicated research funding for methodological research• Establishment of dedicated methods groups (e.g. Cochrane

Qualitative Methods Research Group)

Travelling east from Westminster, every two tube stops represent over one year of life expectancy lost –Data revised to 2004-08

Westminster

Waterloo

Southwark

London Bridge

BermondseyCanada

Water

CanaryWharf

NorthGreenwich

Canning Town

London Underground Jubilee Line

Differences in Male Life Expectancy within a small area in London

Electoral wards just a few miles apart geographically have lifeexpectancy spans varying by years. For instance, there are eight stops between Westminster and Canning Townon the Jubilee Line – so as one travels east, every two stops, onaverage, mark over a year of shortened lifespan. 1

River Thames

1 Source: Analysis by London Health Observatory of ONS and GLA data for 2004-08. Diagram produced by Department of Health

Male Life Expectancy73.6 (CI 71.9-75.2)

Male LifeExpectancy78.5 (CI 75.5-81.6)

Mega-events and urban regeneration

• Examples of complex interventions with multiple anticipated outcomes

• Regeneration - a set of activities to reverse economic, social and physical decline

• Housing stock, service improvements, physical environment, creation of socially mixed population

McCartney et al. BMJ 2010; 340:c2369

McCarthy et al. (2010) conclude…….

“There is little evidence that major multi-sport events held between 1978 and 2008 delivered health or socio-economic benefits for the population of the host

country…..”

“Future events such as the 2012 Olympic Games and Paralympic Games, or the 2014 Commonwealth Games, cannot be expected to automatically provide benefits.

Until decision makers include robust, long term evaluations as part of their design and implementation of

events, it is unclear how the costs of major multi-sport events can be justified in terms of benefits to the host

population.”

Unanswered questions

• What happens during implementation?

• How do the various actors involved experience mega-sporting events and the associated regeneration?

• What are the pathways to impact?

• Do variations in the above relate to outcomes?

Challenges of integration

• New and evolving field• Scale of task• Paradigm wars• Lack of training and expertise• Few worked examples• Infrastructure in development

Approaches and methods (1)• Narrative summary• Thematic analysis

• Grounded theory• Meta-ethnography• Meta-study• Miles and Huberman's data

analysis techniques• Content analysis• Case survey• Qualitative comparative

analysis • Bayesian meta-analysis

2004

Approaches and methods ( 2 )

Meta-narrative mapping

Critical Interpretive

synthesis

Realist synthesis

Mixed methodssynthesis

Narrative synthesis

Integration of qualitative

and quantitative

research

Bayesian meta-

analysis

Origins of approaches and methodsMethod Developed by/Exemplars Context and purpose

Narrative synthesis Popay et al. (2005) Cochrane remit – to examine issues of process, implementation and experience

Mixed methods synthesis

Thomas et al. (2004)Harden and Thomas (2005)

Informing policy to promote children’s health

Bayesian synthesis Roberts et al. (2002) Factors that affect the uptake of childhood immunisation

Critical interpretive synthesis

Dixon-Woods et al. (2006) Access to healthcare for vulnerable groups

Meta-narrative Greenhalgh et al. (2005) To review research on diffusion of innovation to inform healthcare policy

Realist synthesis Pawson (2006) To develop and test theories of change underpinning complex policy interventions

Comparing approaches and methodsMethod Idealist –

realist continuum

Deconstruct body of literature?

Mixed methods lens Other characteristics

Narrative synthesis

Realist No Complementary strengths stance

Little iteration in methods

Synthetic product aims to directly address policy

Mixed methods synthesis

Realist No Complementary strengths stance andDialectical stance

Bayesian synthesis

Realist No Alternative or single paradigm stance

Critical interpretive synthesis

Idealist Yes Alternative or single paradigm stance Iterative approach

key

Synthetic product requires interpretation

Meta-narrative Idealist Yes Dialectical stance

Realist synthesis Idealist No Alternative or single paradigm stance

In focus: Mixed methods synthesis

• Aim is to generate and test theory from diverse body of literature

• Exhaustive search, review questions, inclusion criteria and quality assessment largely specified a priori

• Each review typically has three syntheses: 1. Statistical meta-analysis2. Thematic synthesis3. Cross-study synthesis

Recent examples

Teenage pregnancy and social disadvantage (Harden et al., 2009)

Knowledge and information needs of young people with epilepsy (Lewis

et al., 2010)Behavioural interventions for weight

management in pregnancy (Campbell et al., 2011)

The effects of schools and school environment interventions (Bonell

et al., 2011)

Mixed methods systematic reviews

Working definition

Combining the findings of ‘qualitative’ and ‘quantitative’ studies within a single systematic

review, in order to address the same, overlapping or complementary review

questions(Harden and Thomas, 2010)

Three ways in which reviews are mixed….

1. The types of studies included and hence the type of findings to be synthesised (i.e. ‘qualitative/ textual and quantitative/numerical)

2. The types of synthesis method used (e.g. statistical meta-analysis and qualitative synthesis)

3. The mode of analysis: theory testing AND theory building

Review questions•What is known about the barriers to, and facilitators of, healthy eating amongst children?

•Do interventions promote healthy eating amongst children?

•What are children’s perspectives on healthy eating?

•What are the implications of the above for intervention development?

REVIEW PROCESS

Searching, screening and mapping

Synthesis 1: Trials (n=33)1. Quality assessment

2. Data extraction3. Statistical meta-analysis

Synthesis 2: Qualitative studies (n=8)

1. Quality assessment2. Data extraction

3. Thematic synthesis

Synthesis 3: Trials and qualitative studies

Focus narrowed to ‘fruit &veg’

Children’s views Trials

Recommendation for interventions

Good quality

Other

Do not promote fruit and vegetables in the same way

0 0

Brand fruit and vegetables as an ‘exciting’ or child-relevant product, as well as a ‘tasty’ one

5 5

Reduce health emphasis in messages to promote fruit and vegetables particularly those which concern future health

5 6

Synthesis 3: Across studies

Increase (standardised portions per day) in vegetable intake across trials

0

0.1

0.2

0.3

0.4

0.5

0.6

War

dle

Liquo

ri

Henry

Ander

son

Reyno

lds Auld

Auld (b

)

Baran

owsk

i

Perry

Study

Po

rtio

ns

Little or no emphasis on health messages

Synthesis 3: Across studies

In focus: Bayesian synthesis

• Aim is to test theory• Findings from qualitative and

quantitative research are ‘fused’• Only three worked examples to

date• Variation in weight given to the

qualitative evidence• Synthesis product is a set of

weighted factors associated with/predicting the phenomenon under review

Recent examples

Factors affecting the uptake of childhood immunisation

(Roberts et al., 2002)

Factors that influence adherence to HIV

medication regimes (Vollis et al., 2009)

(Crandell et al., 2011)

Factors that influence adherence to HIV medication (Crandell et al., 2011)

In focus: Critical Interpretive Synthesis

• Aim is to generate theory from large and diverse body of literature

• Literature itself is an object of scrutiny (critical)

• Comprehensive search to identify sampling frame

• Purposive and theoretical sampling

• Analysis leads to generation of synthetic constructs and a synthesising argument

Recent examples

Access to health care for vulnerable groups (Dixon-

Woods et al. 2006)

Use of morphine to treat cancer related pain (Flemming, 2009)

Nurses response to suicide and suicidal patients (Talseth and

Gilje, 2011

Access to health care for vulnerable groups (Dixon-Woods et al. 2006)

• Based on 119 papers

• Interpretive qualitative analysis of diverse types of studies

• Conceptual and methodological problems with measures of health service utilisation

• Synthesising argument organised around a set of central concepts (e.g. navigation, adjudications) with the synthetic construct of ‘candidacy’ at the core

Access to health care for vulnerable groups (Dixon-Woods et al. 2006)

Core construct – candidacy

“Candidacy describes the ways in which people's eligibility for

medical attention and intervention is jointly negotiated between

individuals and health services…….candidacy is a dynamic and contingent

process, constantly being defined and redefined through

interactions between individuals and professionals,

including how "cases" are constructed.”

Central concepts

• Identification of candidacy• Navigation

• Permeability of services• Appearances at health services

• Adjudications• Offers and resistance

• Operating conditions and local production of candidacy

In focus: Meta-narrative review

• Aim is to make sense of and understand diverse bodies of literature and their findings

• Literature itself is an object of scrutiny (critical)

• Searching is iterative, ‘snowballing’ a key technique

• Analysis leads to production of a set of meta-narratives (‘storylines of research’)

Recent examples

Spread and sustainability of innovations in health service

delivery and organisation (Greenhalgh et al., 2005)

Understanding the use of electronic patient records in health care

organisations (Greenhalgh et al., 2009)

See also the:Realist and meta-narrative evidence

synthesis evolving standards project (RAMASES) (Greenhalgh et

al., 2011)

A meta-narrative approach

• The influence of Kuhn’s ‘paradigms’ (1962) and the makings of the first meta-narrative

• The essential technique is interpretive synthesis exploring distinct research traditions, each with its own meta-narrative

• Methods of ‘unpacking’ the meta-narrative: exploratory methods; expert consultations; snowballing; database searching

Stages of a meta-narrative review (from Greenhalgh et al., 2009)

Meta-narratives identified in the electronic patient records review (from Greenhalgh et al., 2009)

Comparing approaches and methodsMethod Idealist –

realist continuum

Deconstruct body of literature?

Mixed methods lens Other characteristics

Mixed methods synthesis

Realist No Complementary strengths stance andDialectical stance Little iteration in

methods

Synthetic product aims to directly address policy

Bayesian synthesis

Realist No Alternative or single paradigm stance

Critical interpretive synthesis

Idealist Yes Alternative or single paradigm stance Iterative approach

key

Synthetic product requires interpretation

Meta-narrative Idealist Yes Dialectical stance

Future challenges• More worked examples are key• Focus on methods and tools for the actual

integration– Enhancing transparency– Establishing rigour

• Further conceptual work to illuminate points of difference, strengths and weaknesses, fit for purpose

• Learning from, and contributing to, the mixed methods literature for primary research.

Thank you!a.harden@uel.ac.uk

Forthcoming book

Gough D, Oliver S, Thomas J (Eds) (forthcoming, Feb 2012) An Introduction to Systematic Reviews. London: Sage