enhanced feedback from perioperative quality indicators: studying the impact of a complex qi...
TRANSCRIPT
Enhanced feedback from perioperative quality indicators: Studying the impact of a complex QI intervention
Jonathan BennCentre for Patient Safety and Service QualityImperial College London
Glenn ArnoldImperial College Healthcare NHS Trust
Research Group:Danielle D’LimaJoanna MooreIgor WeiAlan PootsAlex BottleStephen Brett
Declaration of funding and conflicts of interest
Project funding:NIHR CLAHRC Northwest LondonNIHR HS&DR Research Programme
Conference attendance funded by:NIHR & Imperial College London
Conflicts of Interest:None: No payment received for presentation
QI concept:Provision of real-time feedback on quality of anaesthetic care (for anaesthetists)
Anaesthetists rarely receive systematic, routine feedback on the quality of anaesthetic care delivered (and as experienced by the patient) in post-operative recovery
Review of quality indicators in anaesthesia (2009)
Perioperative morbidity and
mortality data lacks the
sensitivity and specificity
necessary for analysis of
variation in quality of
anaesthesia.
Few validated indicators
incorporating the patient's
perspective on quality of
anaesthetic care.
Survey of use of quality indicators in perioperative units (2012)
Local data collection driven by theatre productivity and external reporting requirements
Patient satisfaction with anaesthesia infrequently monitored
Post-op patient temperature, pain and nausea data is not reliably monitored and utilised at local level, in the majority of perioperative units
Concept for a quality monitoring and feedback initiative
A continuous control loop representing learning at the individual and micro-system
levels:
Concept for a quality monitoring and feedback initiative
A continuous control loop representing learning at the individual and micro-system
levels:
Research questions for improvement science: • Can we conceptualise “data feedback” as the core of a quality
improvement intervention?• Under what conditions are “data feedback initiatives” effective in
improving care?
Contributions from improvement science
Continuous process monitoring - an industrial model:
Provides a continuous signal, representing variation over time, rather than a snapshot view of standards at one point in time
Emphasises reliability rather than the extent of specific deviations Supports open and objective discussion about variations in
performance and learning from best practice examples Supports rapid detection and correction of problems in near real-
time Effects of QI interventions are observable, iterations are systematic
and guided by empirical evidence Disaggregates data onto a level that is meaningful for users
Fosters local ownership of data and responsibility for improvement Data collection is integrated within routine operations
Metrics are stable and reliable
Research basis for data feedback interventions
Systematic reviews of the effects of feedback on professional practice typically show small to moderate positive effects (e.g. Jamdtvedt, 2005)
Adding elements (such as education & quality improvement methods) to basic data feedback reports enhance their effectiveness (van der Veer, 2010; de Vos, 2009)
Qualitative research suggests that effective data feedback for quality
improvement has a number of characteristics (Bradley, 2004)
Timeliness Specific to the local context Originates from credible/respected sources Is non-punitive Is sustained over time
IMPAQT (CLAHRC project): Anaesthetic quality monitoring & feedback at St Mary’s, London
CLAHRC improvement model:
Iterative change (PDSA)
Focus upon local multidisciplinary engagement
Supported by continuous measurement and evaluation (SPC)
Quality monitoring in PACU:
Temperature on arrival in recovery (NICE Guideline)
Quality of recovery/anaesthetic: Patient reported Quality of Recovery (QoR) score (Myles, 1999) Post Operative Nausea and Vomiting (PONV) (Categorical) Pain scales (Categorical and continuous scales)
Patient transfer efficiency (Ward Wait Time)
Additional data is routinely compiled from the theatre and patient administration systems.
St Mary’s Main Theatres: Data process
PACU
Surgicalwards
Pre & Intra-operative care
DatabaseExcel
templates
Anaes.feedback
report
PACUdata
posting
Wardfeedback
report
Intra-operativecare pathway
Datavalidation
& cleansing
Quality of Recovery, PONV,
Pain, Temp, Patient transfer delays
Feedback anaesthetic quality indicators (personal level data)
Feedback quality of recovery and transfer efficiency metrics
Feedback patient transfer efficiency metrics (ward level data)
Monthly PACU & Ward FeedbackData posted in recovery Surgical ward reports
Personalised feedback for anaesthetists(Version 1: Sep 2010)
Enhanced feedback reports (Version 3: Feb-July 2012)
Developed based on interviews with end-users
Programme of active, trust-wide engagement and work with specialty sub-groups
Enhanced monthly report features:
Inclusion of multi-site data Comparative perspective:
individual vs peer group Longitudinal view on variation
in personal and group practice Identification and description
of statistical outlying cases to support case-based learning
Specialty-specific reporting of Pain scores (to better account for case mix)
Mixed-methods evaluation of anaesthetics QI initiative (NIHR HS&DR)
Evaluation of effects upon perioperative process and outcome indicators
Interrupted time series analysis of quality indicators dataset merged at case level with hospital administrative data
Semi-structured investigation of implementation context and perceived acceptability of the initiative
Theoretically-informed qualitative research interviews with consultant anaesthetists and perioperative service leads
2 rounds of interviews: 1) formative, 2) evaluative
End-user evaluation
Survey data collected at multiple time points Baseline (pre-feedback) Multiple post-implementation follow-ups at three hospital sites
Effects of implementation of feedback on perioperative warming
Main anaesthetist cohort, all St Mary’s surgical cases Mar 2010 - Sep 2013
No Feedback Basic feedback Enhanced feedback
Effect of introduction of enhanced feedback(multi-site data)
Proportion of patients with temp below 36 degrees: Stepwise decrease of 9% with introduction of enhanced feedback
(p<0.01)
Proportion of patients reporting no pain or mild pain (compared to moderate or severe):
Stepwise increase of 8% with introduction of enhanced feedback (p<0.01)
Proportion of patients free from nausea: Small improvement in rate of change over time following introduction
of enhanced feedback (p<0.01)
No significant effect of feedback on Surgical Site Infection rate
No significant effect of feedback on 30 day mortality
Qualitative investigation: Anaesthetists’ views on feedback
“I know that I’m able to immediately affect the outcome of these measures, so I can do things to make these measures different.”
“I thought: ‘My goodness, I do quite a lot of patients’; ‘my goodness, oh, some of them are in more pain than I thought they would be in’. So I did some things to change it.”
“For me to improve my practice I would need to first have my own data over a month or over a year.....and also how does my data compare to other anaesthetists that do exactly the same thing”
“I think having departmental level data is important, data for the department that identifies areas where the department as a whole needs to improve or is performing adequately.”
“I don’t think we’re particularly adversarial here, and I think we generally, discuss things and we’re quite open with each other about our data and about how we do things.”
Comparison of pre and post feedback implementation
Longitudinal survey evaluation: Usefulness of locally available data for QI
Scale:
1 “Completely inadequate” to 8 “Excellent”
Item descriptions
Level of analysis: Relevance of data to personal practice
Timeliness: Adequate frequency for monitoring variation
Communication: Effectiveness of channel and method of dissemination
Data presentation: Clarity and usefulness of graphical formats
Credibility: Perception of trustworthiness and freedom from bias
Complexity & challenges in evaluation
• Multiple feedback iterations, serial and cumulative effects – need time-sensitive approach to analysis (ITSA & SPC)
1. Complex intervention timeline
• Need to account for hard and soft outcomes of feedback, using mixed methods
2. Socio-technical effects
• Need to understand how the reaction to feedback may be influenced by local context (e.g. “open”, “non-punitive” local unit climate)
3. Interactions with context
• Data feedback is potentially a passive intervention, it does not specify the mechanism of learning or change – need to investigate and describe “process”
4. Risk of under-specification
• Rapid, responsive development sometimes undermines quasi-experimental intent - Impose some discipline on development; accept that dataset won’t be perfect!
5. Tension between “science” and “service”
