How qualitative research contributes to evaluation

Professor Alicia O’CathainScHARR University of Sheffield

22 June 2015

What is qualitative research?

• Normal in evaluation• Understanding not

measuring • Set of methods– Focus groups– Semi-structured or in

depth interviews– Non-participant

observation– Diaries

What is evaluation?

• Researcher led evaluation

• Policy evaluation

Researcher-led

• MRC Framework developing an evaluating complex interventions

• ACTIF programme – 5 years– RCT– Qualitative at each phase

O’Cathain A, Thomas KJ, Drabble SJ, Rudolph A, Hewison J. What can qualitative research do for randomised controlled trials? A

systematic mapping review. BMJ Open 2013;3:e002889

Intervention Trial design and conduct Outcomes

Measures Health conditions

Intervention n=254Develop

n=48

Describe it n=10

Understand how it works

n=23

Value and benefits

n=42Acceptability in principle

n=32

Feasibility and acceptability

n=83

Fidelity, reach and dose n=12

Implementation in real world n=4

Trial design and conduct n=54

Recruitment

Diversity

Participation in trials

Acceptability in principle

Acceptability in practice

Ethics/informed consent

Adapting to local circumstances

Impact on staff, researchers, patients

Potential value Potential valueBias Avoidance of measurement biasEfficiency Faster recruitment Saves money Ethics Trials sensitive to human beings

Improved informed consent

Implementation Facilitates replicability of intervention in the real world

Facilitates transferability of findings in the real world

Interpretation Explains trial findingsRelevance Ensures interventions meet the needs of health

professionals and patients

Success Makes a trial successful, feasible, viableValidity Improves internal validity Improves external validity

Maximising value…

1. Do it early– 28% pre-trial• Intervention development 100%• Acceptability of intervention in principle 25%• Acceptability of intervention in practice 24%• Recruitment 18%• Breadth of outcomes 0%

…otherwise its about future trials

2. Publish learning for specific trial or future trials3. Think beyond interviews: non-participant observation4. Try iterative or dynamic or participatory approaches at feasibility phase5. Not just complex interventions

• 38% of 104 data extracted were drugs or devices

6. Think about the range of work

Problems with quantitative only

• Null RCTs….explain findings (context, mechanisms of action, implementation)

• Failed trials….prevent this at pilot stage

• It works but what is ‘it’?

…..qualitative can fix

Policy evaluation

• Learning from early adopters (feasibility)• Stakeholder reception (acceptability)• Service delivered (implementation, workforce)

Useful but challenges remain

– Fast evaluation– When to evaluate– Moving target– Replacement of difficult-to-measure outcomes

with understanding of processes

Conclusions

• Useful contribution no matter what type of evaluation – essential due to complexity

• Can help to fix problems faced in researcher-led evaluation

• Challenges in policy evaluation need reflection