m ixed methods and the credibility of evidence in evaluation : l earnings from p hysician a ssistant...
TRANSCRIPT
![Page 1: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/1.jpg)
MIXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION: LEARNINGS FROM PHYSICIAN ASSISTANTDEMONSTRATIONS
Synergia Ltd 6th June 2015
Dr Sarah Appleton
Dr Adrian Field
![Page 2: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/2.jpg)
2
This presentation
• Identifies the context of the evaluation• Summarises our approach to evaluating the physician assistant
demonstrations• Presents learnings from applying a mixed methods approach• Identifies key considerations for evaluation practice and theory
![Page 3: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/3.jpg)
3
EVALUATING HEALTH WORKFORCE DEMONSTRATIONS
![Page 4: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/4.jpg)
4
Use of demonstrations in the health sector
• Demonstration– A practical exhibition and explanation of how something works or is
performed
• Not an RCT or pilot study but an:– “assessment of workforce changes in specific settings” (Health Workforce New
Zealand, 2012)
• Time limited• Smaller samples
![Page 5: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/5.jpg)
5
Challenges of demonstrations
• Generalisability • Attribution• Adjustment and learning• Diversity of contexts and approaches
![Page 6: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/6.jpg)
6
Context of evidence in health sector
• Power of positivist constructs – hierarchy of methods
![Page 7: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/7.jpg)
7
THE PHYSICIAN ASSISTANT DEMONSTRATIONS
![Page 8: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/8.jpg)
8
Background
• Extension of physician role, working under supervision• Four demonstration sites trialling role of Physician Assistant in primary
health care and rural ED settings• Established role in US, emerging elsewhere (UK, Australia, Netherlands)• Assess value and contribution of PA role to health workforce, in sites and
implications more widely• Mixed methods approach
![Page 9: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/9.jpg)
9
Rationale and key drivers for demonstrations
Site and PA selection
Mix of settings: urban, rural, Maori and Pacific populations, SES, primary care, ED, IFHC
Intended for PAs to have 2-3 years experience with leadership qualities and a good fit for settings
For demonstration sitesImproving patient
throughputCost effective
solutionsImprove continuity
(vs locums)Trial new models of
workingAddress workforce
shortages
For HWNZMedical workforce supply, particularly distributional
(e.g. rural, high dep areas) Fiscally sustainable solutions
![Page 10: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/10.jpg)
10
Site settings and roles
Radius Group (Hamilton)• 3 PAs at 3 sites• Mixture of high needs populations (Davies Corner and K’Aute) and more affluent (Rototuna)• Mix of patient types, some focus on acute• Mix of fee paying/non-fee paying (K’Aute)
Tokoroa Medical Centre• 2 PAs• Acute and women’s health; fewer long-term conditions patients• Transition in 2014 to integrated model with 2 other practices and NP
Gore Hospital• 1 PA• Key point of contact in ED• Limited ward work (follow-up)
Commonalities• Largely drop-in clinics; relatively few appointments (exception Tokoroa Medical Centre)• Extension of physician role• Tend to focus on acute rather than long-term conditions
![Page 11: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/11.jpg)
11
A contested terrain
PA as a disruptive innovation
Established professional practices and boundaries
Cultural fit
Regulatory and
educational frameworks
Paolo Ucello, The Battle of San Romano, 15th century
![Page 12: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/12.jpg)
12
Contested terrain
“The physician assistant trial is probably the best example, creating a minor storm when two US-trained physician assistants were employed at Middlemore Hospital. Nurses were upset they weren't considered for the role; junior doctors worried they would be sidelined somehow. Some groups wanted the trial halted, others had reservations about introducing yet another entity into the health workforce. And, while the trial has so far been a success, scepticism remains”
(New Zealand Doctor, June 2011)
![Page 13: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/13.jpg)
13
OUR EVALUATION APPROACH
![Page 14: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/14.jpg)
15
Role introduction theory
Rummler G, Brache A. 1995. Improving Performance: How to Manage the White Space in the Organization Chart. San Francisco: Jossey Bass.
FEASIBILITY CLARITY
FEEDBACKKNOWLEDGE/SKILLS
INPUT OUTPUT CONSEQUENCES
FEEDBACK
Frontline staff
CONSEQUENCES
![Page 15: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/15.jpg)
16
Realistic evaluation
![Page 16: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/16.jpg)
Mixed methods: The idea
“Doing our work better, generating understandings that are broader, deeper, more inclusive and that more centrally honour the complexity and contingency of human phenomenon”
(Greene, 2007, p. 98).
![Page 17: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/17.jpg)
18
Data DomainsEvaluation Questions
Evaluation questions and data domains
1. How have PAs integrated with practice activities and service models?
2. What was the impact and contributory value of the PA role for patient outcomes, service quality and business models at the demonstration sites; within this, have the PAs extended or changed the practice model?
3. What factors supported or challenged the integration of the PA role into local practices and with specific professional groups?
4. What are the implications and/or risks for the fit and applicability of the PA role within New Zealand, arising from the evaluation findings?
5. What issues arise from the demonstrations for the potential establishment, transferability and sustainability of the PA role in New Zealand?
Patient experience and impact
Clinical contribution
Workforce impact
PA integration and development
Financial and business impact
Contextual contributors
Evaluation questions Data domains
![Page 18: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/18.jpg)
19
KEY LEARNINGS FROM OUR APPROACH
![Page 19: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/19.jpg)
20
Data collection opportunities and challenges
Patient experience and impact
Clinical contribution
Workforce impact
PA integration and development
Financial and business impact
Contextual contributors
Patient management
systemClinical notes Administration
dataStakeholder interviews
Staff/patient surveys
![Page 20: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/20.jpg)
21
Credibility of quantitative data
• Perceived limitations:– Proxy indicators of change– Small sample size
• Credibility enhanced by:– Support from qualitative data– Depth of insight
![Page 21: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/21.jpg)
22
Credibility of qualitative data
Stakeholder: “it’s only anecdotal…..”
Evaluator: “But these findings are also reflected in the quantitative data.”
Moving beyond methods:
![Page 22: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/22.jpg)
23
VALUE OF A MIXED METHODS APPROACH
![Page 23: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/23.jpg)
24
WHAT VALUE DO YOU SEE IN A MIXED METHODS APPROACH?
Rapid reflections:
![Page 24: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/24.jpg)
25
Value of mixed methods in the PA evaluation
Strengths•Comprehensive•Multiple sources•High engagement•Integration with service data•Insights for other settings
Limitations•Coverage•Some source data•Linking databases•Pre- and post •Contexts
![Page 25: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/25.jpg)
26
Our thoughts on value
• Interpreting outcomes in context • Understanding the typical and the unique case• Comprehensive insight
![Page 26: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/26.jpg)
27
Sharing more and hearing more
• Multiple perspectives of credible evidence• Inclusive and respectful of different ways of knowing and valuing
![Page 27: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/27.jpg)
28
Key considerations for practice
• What do you value?• What do you know?• When and how will you deploy methods?• Time and resources • Limited guidance on write-up and analysis• Maintains focus of the evaluation
![Page 28: M IXED METHODS AND THE CREDIBILITY OF EVIDENCE IN EVALUATION : L EARNINGS FROM P HYSICIAN A SSISTANT DEMONSTRATIONS Synergia Ltd 6 th June 2015 Dr Sarah](https://reader031.vdocuments.us/reader031/viewer/2022020721/56649ddc5503460f94ad41b5/html5/thumbnails/28.jpg)
29
Fit for purpose
“Premise is that using multiple and diverse methods is a good idea, but is not automatically good science. Rather, just as survey research, quasi-experimentation, panel studies, and case studies require careful planning and thoughtful decisions, so do mixed method studies.”
Mertens (2013). Mixed Methods and Credibility of Evidence in Evaluation: New Directions for Evaluation, Number 138