uva-dare (digital academic repository) understanding and ... · appendix [30]...

55
UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl) UvA-DARE (Digital Academic Repository) Understanding and optimising electronic audit and feedback to improve quality of care Gude, W.T. Link to publication Creative Commons License (see https://creativecommons.org/use-remix/cc-licenses): Other Citation for published version (APA): Gude, W. T. (2019). Understanding and optimising electronic audit and feedback to improve quality of care. General rights It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons). Disclaimer/Complaints regulations If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible. Download date: 02 Aug 2020

Upload: others

Post on 06-Jul-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

UvA-DARE is a service provided by the library of the University of Amsterdam (http://dare.uva.nl)

UvA-DARE (Digital Academic Repository)

Understanding and optimising electronic audit and feedback to improve quality of care

Gude, W.T.

Link to publication

Creative Commons License (see https://creativecommons.org/use-remix/cc-licenses):Other

Citation for published version (APA):Gude, W. T. (2019). Understanding and optimising electronic audit and feedback to improve quality of care.

General rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s),other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, statingyour reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Askthe Library: https://uba.uva.nl/en/contact, or a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam,The Netherlands. You will be contacted as soon as possible.

Download date: 02 Aug 2020

References

169

Appendix

References

[1] Green LW, Glasgow RE. Evaluating the Relevance,Generalization, and Applicability of Research. Evalu-ation & the Health Professions. 2006 Mar;29(1):126–153.

[2] Rothwell PM. External validity of randomised con-trolled trials: “To whom do the results of this trialapply?”. The Lancet. 2005 Jan;365(9453):82–93.

[3] Morris ZS, Wooding S, Grant J. The answer is 17 years,what is the question: Understanding time lags intranslational research. Journal of the Royal Societyof Medicine. 2011;104(12):510–520.

[4] Kessler R, Glasgow RE. A Proposal to Speed Transla-tion of Healthcare Research Into Practice. AmericanJournal of Preventive Medicine. 2011 Jun;40(6):637–644.

[5] Shekelle PG, Ortiz E, Rhodes S, Morton SC, Eccles MP,Grimshaw JM, et al. Validity of the Agency for Health-care Research and Quality Clinical Practice Guide-lines. Journal of the American Medical Association.2001 Sep;286(12):1461.

[6] Grol R. Successes and failures in the implementa-tion of evidence-based guidelines for clinical prac-tice. Medical care. 2001 Aug;39(8 Suppl 2):II46–54.

[7] McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, De-Cristofaro A, et al. The Quality of Health Care De-livered to Adults in the United States. New EnglandJournal of Medicine. 2003 Jun;348(26):2635–2645.

[8] Schuster MA, McGlynn EA, Brook RH. How good isthe quality of health care in the United States? 1998.The Milbank quarterly. 2005;83(4):843–95.

[9] Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE.Knowledge translation of research findings. Imple-mentation Science. 2012 Jan;7(1):50.

[10] Marshall M, ??vretveit J. Can we save money byimproving quality? BMJ Quality and Safety. 2011Apr;20(4):293–296.

[11] Oxman AD, Thomson MA, Davis DA, Haynes B. Nomagic bullets: a systematic review of 102 trials of in-terventions to improve professional practice. CMAJ.1995;153(10):1423–1431.

[12] Donabedian A. Evaluating the quality of medicalcare. The Milbank Quarterly. 2005;83(4):691–729.

[13] Davis DA, Mazmanian PE, Fordis M, Van Harrison R,Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures ofcompetence: A systematic review. Journal of theAmerican Medical Association. 2006;296(9):1094–1102.

[14] Karraker RJ. Knowledge of results and incorrect re-call of plausible multiple-choice alternatives. Jour-nal of Educational Psychology. 1967;58(1):11–14.

[15] Locke EA. Motivational effects of knowledge of re-sults: Knowledge or goal setting? Journal of AppliedPsychology. 1967;51(4, Pt.1):324–329.

[16] Shortliffe EH, Cimino JJ. Biomedical Informatics. 4thed. London: Springer London; 2014.

[17] Friedman CP. A ”Fundamental Theorem” of Biomed-ical Informatics. Journal of the American MedicalInformatics Association. 2009;16(2):169–170.

[18] Damberg CL, Sorbero ME, Lovejoy SL, Lauderdale K,Wertheimer S, Smith A, et al. An Evaluation of theUse of Performance Measures in Health Care; 2011.

[19] Groene O, Skau JKH, Frølich A. An international re-view of projects on hospital performance assess-ment. International Journal for Quality in HealthCare. 2008;20:162–171.

[20] Marshall MN, Shekelle PG, Leatherman S, BrookRH. The public release of performance data:what do we expect to gain? A review of the evi-dence. Journal of the American Medical Association.2000;283(14):1866–74.

[21] Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: ef-fects on professional practice and healthcare out-comes. The Cochrane database of systematic re-views. 2012 Jan;6:CD000259.

[22] Weed LL. Medical records, patient care, and med-ical education. Irish Journal of Medical Science.1964;39(6):271–282.

[23] Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scov-ille R, et al. Can electronic medical record systemstransform health care? Potential health benefits,savings, and costs. Health Affairs. 2005;24(5):1103–1117.

[24] Prokosch HU, Ganslandt T. Perspectives for medicalinformatics. Reusing the electronic medical recordfor clinical research. Methods of information inmedicine. 2009;48(1):38–44.

[25] Jensen PB, Jensen LJ, Brunak S. Mining electronichealth records: towards better research applica-tions and clinical care. Nature reviews Genetics.2012 May;13(6):395–405.

[26] Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S,O’Brien MA, French SD, et al. Growing Litera-ture, Stagnant Science? Systematic Review, Meta-Regression and Cumulative Analysis of Audit andFeedback Interventions in Health Care. Journal ofGeneral Internal Medicine. 2014;29(11):1534–1541.

[27] Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM,Baker R. What do we know about how to do auditand feedback? Pitfalls in applying evidence froma systematic review. BMC health services research.2005 Jan;5:50.

[28] Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Fran-cis JJ, et al. No more ’business as usual’ with auditand feedback interventions: Towards an agenda fora reinvigorated intervention. Implementation Sci-ence. 2014;9(1):14.

[29] Hysong SJ, Kell HJ, Petersen LA, Campbell BA, Traut-ner BW. Theory-based and evidence-based designof audit and feedback programmes: Examples fromtwo clinical intervention studies. BMJ Quality andSafety. 2017 Apr;26(4):323–334.

171

Appendix

[30] Michie S, Abraham C. Interventions to change healthbehaviours: evidence-based or evidence-inspired?Psychology & Health. 2004 Feb;19(1):29–49.

[31] Colquhoun HL, Brehaut JC, Sales A, Ivers N, GrimshawJ, Michie S, et al. A systematic review of the use oftheory in randomized controlled trials of audit andfeedback. Implementation Science. 2013;8(1):66.

[32] Goudie AC, Sutton AJ, Jones DR, Donald A. Empiricalassessment suggests that existing evidence couldbe used more fully in designing randomized con-trolled trials. Journal of clinical epidemiology. 2010Sep;63(9):983–91.

[33] Colquhoun H, Michie S, Sales A, Ivers N, GrimshawJM, Carroll K, et al. Reporting and design elementsof audit and feedback interventions: A secondaryreview. BMJ Quality and Safety. 2017 Jan;26(1):54–60.

[34] Roland M, Torgerson DJ. Understanding controlledtrials: What are pragmatic trials? BMJ. 1998Jan;316(7127):285–285.

[35] Auerbach AD, Landefeld CS, Shojania KG. The ten-sion between needing to improve care and knowinghow to do it. The New England journal of medicine.2007 Aug;357(6):608–13.

[36] Macleod MR, Michie S, Roberts I, Dirnagl U, ChalmersI, Ioannidis JPA, et al. Biomedical research: increas-ing value, reducing waste. Lancet (London, England).2014 Jan;383(9912):101–4.

[37] van der Veer SN, de Keizer NF, Ravelli ACJ, TenkinkS, Jager KJ. Improving quality of care. A systematicreview on how medical registries provide informa-tion feedback to health care providers. Internationaljournal of medical informatics. 2010 May;79(5):305–23.

[38] Gardner B, Whittington C, McAteer J, Eccles MP,Michie S. Using theory to synthesise evidence frombehaviour change interventions: The example ofaudit and feedback. Social Science and Medicine.2010 May;70(10):1618–1625.

[39] Hysong SJ. Meta-analysis: audit and feedback fea-tures impact effectiveness on care quality. Medicalcare. 2009 Mar;47(3):356–63.

[40] Collins LM, Murphy SA, Nair VN, Strecher VJ. A strat-egy for optimizing and evaluating behavioral in-terventions. Annals of Behavioral Medicine. 2005Aug;30(1):65–73.

[41] Chokshi DA, Avorn J, Kesselheim AS. Designingcomparative effectiveness research on prescriptiondrugs: lessons from the clinical trial literature.Health affairs (Project Hope). 2010 Oct;29(10):1842–8.

[42] Kiefe CI, Allison JJ, Williams OD, Person SD, WeaverMT, Weissman NW. Improving quality improve-ment using achievable benchmarks for physicianfeedback: a randomized controlled trial. Jour-nal of the American Medical Association. 2001Jun;285(22):2871–9.

[43] Elouafkaoui P, Young L, Newlands R, Duncan EM,Elders A, Clarkson JE, et al. An Audit and Feed-back Intervention for Reducing Antibiotic Prescrib-ing in General Dental Practice: The RAPiD ClusterRandomised Controlled Trial. PLOS Medicine. 2016Aug;13(8):e1002115.

[44] Patel S, Rajkomar A, Harrison JD, Prasad PA, Va-lencia V, Ranji SR, et al. Next-generation auditand feedback for inpatient quality improvement us-ing electronic health record data: A cluster ran-domised controlled trial. BMJ Quality and Safety.2018 Sep;27(9):691–699.

[45] Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, IversN, Michie S, et al. Advancing the literature on de-signing audit and feedback interventions: Identify-ing theory-informed hypotheses. ImplementationScience. 2017 Sep;12(1):117.

[46] Gude WT, van der Veer SN, van Engen-Verheul MM,de Keizer NF, Peek N. Inside the Black Box ofAudit and Feedback: a Laboratory Study to Ex-plore Determinants of Improvement Target Selec-tion by Healthcare Professionals in Cardiac Reha-bilitation. Studies in health technology and infor-matics. 2015;216:424–8.

[47] Craig P, Dieppe P, Macintyre S, Michie S, NazarethI, Petticrew M. Developing and evaluating complexinterventions: the new Medical Research Councilguidance. BMJ. 2008 Jan;337(sep29_1):a1655.

[48] Hayward J, Thomson F, Milne H, Buckingham S,Sheikh A, Fernando B, et al. ’Too much, toolate’: mixed methods multi-channel video record-ing study of computerized decision support systemsand GP prescribing. Journal of the American MedicalInformatics Association. 2013 Jun;20(e1):e76–84.

[49] Gude WT, van der Veer SN, de Keizer NF, Coiera E,Peek N. Optimizing Digital Health Informatics Inter-ventions Through Unobtrusive Quantitative ProcessEvaluations. Studies in health technology and infor-matics. 2016;228:594–8.

[50] Scott PJ, Rigby M, Ammenwerth E, McNair JB, Geor-giou A, Hyppönen H, et al. Evaluation Considerationsfor Secondary Uses of Clinical Data: Principles for anEvidence-based Approach to Policy and Implemen-tation of Secondary Analysis. Yearbook of MedicalInformatics. 2017 Sep;26(01):59–67.

[51] Coiera E. Designing and evaluating information andcommunication systems. In: Guide to Health Infor-matics. 3rd ed. Boca Raton: CRC Press; 2015. p. 151–154.

[52] Hysong SJ, Knox MK, Haidet P. Examining clini-cal performance feedback in patient-aligned careteams. Journal of General Internal Medicine.2014;29(SUPPL. 2).

[53] Borgert M, Binnekade J, Paulus F, Goossens A,Vroom M, Dongelmans D. Timely individual au-dit and feedback significantly improves transfusionbundle compliance—a comparative study. Inter-

172

Appendix

References

national Journal for Quality in Health Care. 2016Oct;28(5):601–607.

[54] Gude WT, Van Engen-Verheul MM, Van Der Veer SN,De Keizer NF, Peek N. How does audit and feed-back influence intentions of health professionalsto improve practice? A laboratory experiment andfield study in cardiac rehabilitation. BMJ Quality andSafety. 2017 Apr;26(4):279–287.

[55] Gude WT, Roos-Blom MJ, van der Veer SN, Dongel-mans DA, de Jonge E, Francis JJ, et al. Health pro-fessionals’ perceptions about their clinical perfor-mance and the influence of audit and feedback ontheir intentions to improve practice: a theory-basedstudy in Dutch intensive care units. ImplementationScience. 2018;13(1):33.

[56] Piepoli MF, Corrà U, Adamopoulos S, Benzer W,Bjarnason-Wehrens B, Cupples M, et al. Sec-ondary prevention in the clinical management ofpatients with cardiovascular diseases. Core compo-nents, standards and outcomemeasures for referraland delivery: A Policy Statement from the CardiacRehabilitation Section of the European Associationfor. European journal of preventive cardiology. 2014Jun;21(6):664–81.

[57] Heran BS, Chen JM, Ebrahim S, Moxham T, Oldridge N,Rees K, et al. Exercise-based cardiac rehabilitationfor coronary heart disease. The Cochrane databaseof systematic reviews. 2011 Jul;(7):CD001800.

[58] Receipt of Outpatient Cardiac Rehabilitation AmongHeart Attack Survivors—United States, 2005. Jour-nal of the American Medical Association. 2008Apr;299(13):1534.

[59] Ades PA. Cardiac Rehabilitation and Secondary Pre-vention of Coronary Heart Disease. New EnglandJournal of Medicine. 2001 Sep;345(12):892–902.

[60] Bethell H, Lewin R, Dalal H. Cardiac rehabilitation inthe United Kingdom. Heart. 2008 Dev;95(4):271–275.

[61] Thomas RJ, King M, Lui K, Oldridge N, Piña IL, SpertusJ, et al. AACVPR/ACC/AHA 2007 Performance Mea-sures on Cardiac Rehabilitation for Referral to andDelivery of Cardiac Rehabilitation/Secondary Pre-vention Services. Journal of the American Collegeof Cardiology. 2007 Oct;50(14):1400–1433.

[62] De Vries H, Kemps HMC, Van Engen Verheul MM,Kraaijenhagen RA, Peek N. Cardiac rehabilitationand survival in a large representative communitycohort of Dutch patients. European Heart Journal.2015;36(24):1519–1528.

[63] Papadakis S, Oldridge NB, Coyle D, Mayhew A, ReidRD, Beaton L, et al. Economic evaluation of cardiacrehabilitation: A systematic review. European Jour-nal of Cardiovascular Prevention and Rehabilitation.2005;12(6):513–520.

[64] Suaya JA, Shepard DS, Normand SLT, Ades PA, Prot-tas J, Stason WB. Use of cardiac rehabilitationby Medicare beneficiaries after myocardial infarc-

tion or coronary bypass surgery. Circulation. 2007Oct;116(15):1653–62.

[65] Martin BJ, Hauer T, Arena R, Austford LD, GalbraithPD, Lewin AM, et al. Cardiac rehabilitation atten-dance and outcomes in coronary artery disease pa-tients. Circulation. 2012 Aug;126(6):677–687.

[66] Arena R, Williams M, Forman DE, Cahalin LP, Coke L,Myers J, et al. Increasing referral and participationrates to outpatient cardiac rehabilitation: The valu-able role of healthcare professionals in the inpa-tient and home health settings: A science advisoryfrom the american heart association. Circulation.2012;125(10):1321–1329.

[67] Piepoli MF, Corrà U, Benzer W, Bjarnason-WehrensB, Dendale P, Gaita D, et al. Secondary preven-tion through cardiac rehabilitation: from knowledgeto implementation. A position paper from the Car-diac Rehabilitation Section of the European Associ-ation of Cardiovascular Prevention and Rehabilita-tion. European journal of cardiovascular preventionand rehabilitation: official journal of the EuropeanSociety of Cardiology, Working Groups on Epidemi-ology & Prevention and Cardiac Rehabilitation andExercise Physiology. 2010;17:1–17.

[68] Rehabilitation Committee. Netherlands Society forCardiology (NVVC) and Netherlands Heart Founda-tion (NHS) (both Guidelines on Cardiac Rehabilita-tion 2004) and Working Group PAAHR (partial revi-sion 2011). Multidisciplinary guidelines for cardiacrehabilitation (in Dutch). Utrecht: Netherlands So-ciety for Cardiology (NVVC); 2011.

[69] Gielen S, Mezzani A, Hambrecht R, Saner H. CardiacRehabilitation. In: The ESC Textbook of Cardiovascu-lar Medicine. Oxford University Press; 2009. p. 919–954.

[70] Jones K, Saxon L, Cunningham W, Adams P. Sec-ondary prevention for patients after a myocardialinfarction: summary of updated NICE guidance. BMJ.2013 Nov;347(nov13 3):f6544–f6544.

[71] Van Engen-Verheul MM, Kemps HMC, De Keizer NF,Hellemans IM, Goud R, Kraaijenhagen RA, et al. Revi-sion of the Dutch clinical algorithm for assessing pa-tient needs in cardiac rehabilitation based on iden-tified implementation problems. European Journalof Preventive Cardiology. 2012;19(3):504–514.

[72] Mazzini MJ, Stevens GR, Whalen D, Ozonoff A, BaladyGJ. Effect of an American Heart Association Get Withthe Guidelines Program-Based Clinical Pathway onReferral and Enrollment Into Cardiac RehabilitationAfter Acute Myocardial Infarction. The AmericanJournal of Cardiology. 2008 Apr;101(8):1084–1087.

[73] Short R. Access to cardiac rehabilitationvaries widely across Europe. BMJ. 2008May;336(7653):1095.11–1095.

[74] van Engen-Verheul M, de Vries H, Kemps H, Kraai-jenhagen R, de Keizer N, Peek N. Cardiac rehabili-tation uptake and its determinants in the Nether-

173

Appendix

lands. European journal of preventive cardiology.2013;20(2):349–356.

[75] Vromen T, Spee RF, Kraal JJ, Peek N, van Engen-Verheul MM, Kraaijenhagen RA, et al. Exercise train-ing programs in Dutch cardiac rehabilitation cen-tres. Netherlands Heart Journal. 2013 Mar;21(3):138–143.

[76] Goud R, van Engen-Verheul M, de Keizer NF, Bal R,Hasman A, Hellemans IM, et al. The effect of com-puterized decision support on barriers to guidelineimplementation: A qualitative study in outpatientcardiac rehabilitation. International Journal of Med-ical Informatics. 2010 Jun;79(6):430–437.

[77] van de Klundert N, Holman R, Dongelmans DA,de Keizer NF. Data Resource Profile: the Dutch Na-tional Intensive Care Evaluation (NICE) Registry ofAdmissions to Adult Intensive Care Units. Interna-tional journal of epidemiology. 2015 Dev;44(6):1850–1850h.

[78] Bennett D, Bion J. ABC of intensive care: organisa-tion of intensive care. BMJ. 1999 May;318(7196):1468–70.

[79] Higgins AM, Pettila V, Bellomo R, Harris AH, NicholAD, Morrison SS. Expensive care - a rationalefor economic evaluations in intensive care. Crit-ical care and resuscitation : journal of the Aus-tralasian Academy of Critical Care Medicine. 2010Mar;12(1):62–6.

[80] Rotondi AJ, Chelluri L, Sirio C, Mendelsohn A, SchulzR, Belle S, et al. Patients’ recollections of stressfulexperiences while receiving prolonged mechanicalventilation in an intensive care unit*. Critical caremedicine. 2013;30(4):746–752.

[81] Carroll KC, Atkins PJ, Herold GR, Mlcek CA, ShivelyM, Clopton P, et al. Pain assessment and manage-ment in critically ill postoperative and trauma pa-tients: a multisite study. American journal of criticalcare : an official publication, American Associationof Critical-Care Nurses. 1999;8(2):105–117.

[82] Stanik-Hutt JA, Soeken KL, Belcher AE, Fontaine DK,Gift AG. Pain experiences of traumatically injuredpatients in a critical care setting. American Journalof Critical Care. 2001;10(4):252–259.

[83] Puntillo KA, Wild LR, Morris AB, Stanik-Hutt J, Thomp-son CL, White C. Practices and predictors of anal-gesic interventions for adults undergoing painfulprocedures. American Journal of Critical Care.2002;11(5):415–431.

[84] Chanques MDG, Sebbane MDM, Barbotte MDE, VielMDPDE, Eledjam MDPDJJ, Jaber MDPDS. A Prospec-tive Study of Pain at Rest: Incidence and Character-istics of an Unrecognized Symptom in Surgical andTrauma versus Medical Intensive Care Unit Patients.Anesthesiology. 2007;107(5):858–860.

[85] Jacobi J, Fraser GL, Coursin DB, Riker RR, FontaineD, Wittbrodt ET, et al. Clinical practice guidelinesfor the sustained use of sedatives and analgesics in

the critically ill adult. Critical care medicine. 2002Jan;30(1):119–41.

[86] Barr J, Fraser GL, Puntillo K, Ely EW, Gélinas C, DastaJF, et al. Clinical practice guidelines for the man-agement of pain, agitation, and delirium in adultpatients in the intensive care unit. Critical caremedicine. 2013 Jan;41(1):263–306.

[87] Payen JF, Bosson JL, Chanques G, Mantz J, Labarere J.Pain assessment is associated with decreased dura-tion of mechanical ventilation in the intensive careunit: A post hoc analysis of the dolorea study. Anes-thesiology. 2009 Dev;111(6):1308–1316.

[88] Schelling G, Stoll C, Haller M, Briegel J, Manert W,Hummel T, et al. Health-related quality of life andposttraumatic stress disorder in survivors of theacute respiratory distress syndrome. Critical CareMedicine. 1998;26(4):651–659.

[89] Jones C, Bäckman C, Capuzzo M, Flaatten H, Rylan-der C, Griffiths RD. Precipitants of post-traumaticstress disorder following intensive care: A hypoth-esis generating study of diversity in care. IntensiveCare Medicine. 2007;33(6):978–985.

[90] van derWoudeMCE, Bormans L, Hofhuis JGM, SpronkPE. Current Use of Pain Scores in Dutch Inten-sive Care Units: A Postal Survey in the Netherlands.Anesthesia and analgesia. 2016 Feb;122(2):456–61.

[91] Tabah A, De Waele J, Lipman J, Zahar JR, Cotta MO,Barton G, et al. The ADMIN-ICU survey: a surveyon antimicrobial dosing and monitoring in ICUs.The Journal of antimicrobial chemotherapy. 2015Sep;70(9):2671–7.

[92] Murphy DJ, Pronovost PJ, Lehmann CU, Gurses AP,Whitman GJR, Needham DM, et al. Red bloodcell transfusion practices in two surgical intensivecare units: a mixed methods assessment of barri-ers to evidence-based practice. Transfusion. 2014Oct;54(10 Pt 2):2658–67.

[93] Neto AS, Simonis FD, Barbas CSV, Biehl M, Deter-mann RM, Elmer J, et al. Lung-Protective Ventila-tion With Low Tidal Volumes and the Occurrence ofPulmonary Complications in Patients Without AcuteRespiratory Distress Syndrome: A Systematic Reviewand Individual Patient Data Analysis. Critical caremedicine. 2015 Oct;43(10):2155–63.

[94] Ahlers SJGM, van Gulik L, van der Veen AM, van Don-gen HPA, Bruins P, Belitser SV, et al. Comparisonof different pain scoring systems in critically ill pa-tients in a general ICU. Critical Care. 2008;12(1).

[95] Pasero C, Puntillo K, Li D, Mularski RA, Grap MJ, Er-stad BL, et al. Structured approaches to pain man-agement in the ICU. Chest. 2009;135(6):1665–1672.

[96] Donaldson LJ, Kelley ET, Dhingra-Kumar N, Kieny MP,Sheikh A. Medication Without Harm: WHO’s ThirdGlobal Patient Safety Challenge. The Lancet. 2017Apr;389(10080):1680–1681.

174

Appendix

References

[97] Aitken M, Gorokhovich L. Advancing the ResponsibleUse of Medicines: Applying Levers for Change. SSRNElectronic Journal. Sep; 2012.

[98] Elliott RA, Camacho E. Prevalence and economicburden of medication errors in the NHS England;2018.

[99] Stocks SJ, Kontopantelis E, Akbarov A, Rodgers S, Av-ery AJ, Ashcroft DM. Examining variations in pre-scribing safety in UK general practice: cross sec-tional study using the Clinical Practice ResearchDatalink. BMJ. 2015 Nov;p. h5501.

[100] Carver CS, Scheier MF. Control theory: a usefulconceptual framework for personality-social, clini-cal, and health psychology. Psychological bulletin.1982;92:111–135.

[101] Wiener N. Cybernetics: Control and communicationin the animal and the machine. Cambridge, MA: MITPress; 1948.

[102] Ammons RB. Effects of Knowledge of Performance:A Survey and Tentative Theoretical Formulation. TheJournal of General Psychology. 1956;54(2):279–299.

[103] Powers WT. Feedback: Beyond Behaviorism. Sci-ence. 1973;179(4071):351–356.

[104] Kluger AN, DeNisi A. The effects of feedbackinterventions on performance: A historical re-view, a meta-analysis, and a preliminary feed-back intervention theory. Psychological Bulletin.1996;119(2):254–284.

[105] Locke EA, Latham GP. Building a practically usefultheory of goal setting and task motivation. A 35-yearodyssey. The American psychologist. 2002;57(9):705–717.

[106] Bandura A, Cliffs NJ. Social foundations of thoughtand action: A social cognitive theory (pp. 376). En-glewood . Prentice-Hall; 1986.

[107] de Keizer NF, Peute L, van der Zwan E, Jaspers M,de Jonge E. NICE Online; a web-based tool for mon-itoring performance measures in intensive care.Netherlands Journal of Critical Care. 2011;15(3):131–136.

[108] Tuti T, Nzinga J, Njoroge M, Brown B, Peek N, En-glish M, et al. A systematic review of electronic auditand feedback: intervention effectiveness and use ofbehaviour change theory. Implementation Science.2017 Dev;12(1):61.

[109] James C, Denise O, Susan M. Validation of the the-oretical domains framework for use in behaviourchange and implementation research. Implemen-tation Science. 2012;7(1):37.

[110] Goldstein MK, Hoffman BB, Coleman RW, Musen MA,Tu SW, Advani A, et al. Implementing clinical prac-tice guidelines while taking account of changing ev-idence: ATHENA DSS, an easily modifiable decision-support system for managing hypertension in pri-mary care. Proceedings AMIA Symposium. 2000;p.300–4.

[111] Simon ACR, Holleman F, Gude WT, Hoekstra JBL,Peute LW, Jaspers MWM, et al. Safety and usabil-ity evaluation of a web-based insulin self-titrationsystem for patients with type 2 diabetes mellitus.Artificial Intelligence in Medicine. 2013;59(1):23–31.

[112] Fagan L, Kunz J. Extensions to the Rule-BasedFormalism for a Monitoring Task. In: BuchananBG, Shortliffe EH, ”Rule-Based Expert Systems: TheMYCIN Experiments of the Stanford Heuristic Pro-gramming Project. Addison-Wesley; 1984. p. 397–423.

[113] Attneave F, Hebb DO. The Organization of Behavior;A Neuropsychological Theory. The American Journalof Psychology. 2006 Oct;63(4):633.

[114] Ypma TJ. Historical Development of the Newton–Raphson Method. SIAM Review. 1995;37(4):531–551.

[115] Friedman JH. Greedy function approximation: Agradient boosting machine. Annals of Statistics.2001;29(5):1189–1232.

[116] Goodfellow I, Bengio Y, Courville A. Deep Learning.MIT Press; 2016.

[117] Jamtvedt G, Young JM, Kristoffersen DT, O’BrienMA, Oxman AD. Audit and feedback: effects onprofessional practice and health care outcomes.Cochrane Database of Systematic Reviews. 2006Apr;(2):CD000259.

[118] Gould NJ, Lorencatto F, Stanworth SJ, Michie S, PriorME, Glidewell L, et al. Application of theory to en-hance audit and feedback interventions to increasethe uptake of evidence-based transfusion practice:An intervention development protocol. Implemen-tation Science. 2014 Jan;9(1):92.

[119] Michie S, Richardson M, Johnston M, Abraham C,Francis J, Hardeman W, et al. The behavior changetechnique taxonomy (v1) of 93 hierarchically clus-tered techniques: Building an international con-sensus for the reporting of behavior change in-terventions. Annals of Behavioral Medicine. 2013Aug;46(1):81–95.

[120] Gude WT, van Engen-Verheul MM, van der Veer SN,Kemps HMC, Jaspers MWM, de Keizer NF, et al. Ef-fect of a web-based audit and feedback interventionwith outreach visits on the clinical performance ofmultidisciplinary teams: a cluster-randomized trialin cardiac rehabilitation. Implementation Science.2016 Dev;11(1):160.

[121] Gude WT, Roos-BlomMJ, van der Veer SN, de Jonge E,Peek N, Dongelmans DA, et al. Electronic audit andfeedback intervention with action implementationtoolbox to improve pain management in intensivecare: Protocol for a laboratory experiment and clus-ter randomised trial. Implementation Science. 2017May;12(1):68.

[122] Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A,Michie S, et al. Practice Feedback Interventions: 15Suggestions for Optimizing Effectiveness. Annals ofinternal medicine. 2016 Mar;164(6):435–41.

175

Appendix

[123] Eva KW, Armson H, Holmboe E, Lockyer J, LoneyE, Mann K, et al. Factors influencing responsive-ness to feedback: on the interplay between fear,confidence, and reasoning processes. Advancesin health sciences education: theory and practice.2012 Mar;17(1):15–26.

[124] Landis-Lewis Z, Brehaut JC, Hochheiser H, Dou-glas GP, Jacobson RS. Computer-supported feed-back message tailoring: Theory-informed adapta-tion of clinical audit and feedback for learning andbehavior change. Implementation Science. 2015Jan;10(1):12.

[125] Kluger AN, Van Dijk D. Feedback, the various tasks ofthe doctor, and the feedforward alternative. MedicalEducation. 2010;44:1166–1174.

[126] Booth A, Carroll C. Systematic searching for theoryto inform systematic reviews: is it feasible? Is it de-sirable? Health information and libraries journal.2015 Sep;32(3):220–35.

[127] Brown B, Gude WT, Blakeman T, van der Veer SN,Ivers N, Francis JJ, et al. Clinical Performance Feed-back Intervention Theory (CP-FIT): a new theory fordesigning, implementing, and evaluating feedbackin health care based on a systematic review andmeta-synthesis of qualitative research. Implemen-tation Science. 2019 Dec;14(1):40.

[128] Ritchie J, Spencer L. Qualitative data analysis forapplied policy research. Analyzing qualitative data.2010;p. 173–194.

[129] Pawson R, Tilley N. Realistic Evaluation. SAGE Pub-lications Ltd; 2007.

[130] Pawson R. Evidence-based policy: A realist perspec-tive. London: SAGE Publications; 2006.

[131] Brehaut JC, Eva KW. Building theories of knowl-edge translation interventions: Use the entiremenu of constructs. Implementation Science. 2012Dev;7(1):114.

[132] Byng R, Norman I, Redfern S. Using Realistic Eval-uation to Evaluate a Practice-level Intervention toImprove Primary Healthcare for Patients with Long-term Mental Illness. Evaluation. 2005 Jan;11(1):69–93.

[133] Ilgen DR, Fisher CD, Taylor MS. Consequences of indi-vidual feedback on behavior in organizations. Jour-nal of Applied Psychology. 1979;64(4):349–371.

[134] Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH,Abboud PAC, et al. Why Don’t Physicians Follow Clin-ical Practice Guidelines? Journal of the AmericanMedical Association. 1999;282(15):1458.

[135] Cialdini RB. Influence: The Psychology of Persua-sion. New York, NY: HarperCollins Publishers Inc;1993.

[136] Festinger L. A Theory Of Cognitive Dissonance. Stan-ford, CA: Stanford University Press; 1957.

[137] Steele CM. The Psychology of Self-Affirmation: Sus-taining the Integrity of the Self. Advances in Exper-imental Social Psychology. 1988;21(C):261–302.

[138] Johnson RE, Howe M, Chang CHD. The impor-tance of velocity, or why speed may matter morethan distance. Organizational Psychology Review.2013;3(1):62–85.

[139] Festinger L. A theory of social comparison pro-cesses. Human Relations. 1954;7:117–140.

[140] Merton RK. Contributions to the Theory of ReferenceGroup Behavior. Social Theory and Social Structure.1968;p. 279–334.

[141] Berkowitz AD. The Social Norms Approach: The-ory, Research, and Annotated Bibliography; 2004.August.

[142] Dixon-Woods M, Redwood S, Leslie M, Minion J, Mar-tin GP, Coleman JJ. Improving quality and safety ofcare using technovigilance: An ethnographic casestudy of secondary use of data from an electronicprescribing and decision support system. MilbankQuarterly. 2013;91(3):424–454.

[143] Guldberg TL, Vedsted P, Lauritzen T, Zoffmann V.Suboptimal quality of type 2 diabetes care discov-ered through electronic feedback led to increasednurse-GP cooperation. A qualitative study. PrimaryCare Diabetes. 2010;4(1):33–39.

[144] Yi SG, Wray NP, Jones SL, Bass BL, Nishioka J, Brann S,et al. Surgeon-specific performance reports in gen-eral surgery: An observational study of initial imple-mentation and adoption. Journal of the AmericanCollege of Surgeons. 2013;217(4).

[145] Eldh AC, Fredriksson M, Halford C, Wallin L,Dahlström T, Vengberg S, et al. Facilitators and bar-riers to applying a national quality registry for qual-ity improvement in stroke care. BMC Health ServicesResearch. 2014;14(1).

[146] Jeffs L, Doran D, Hayes L, Mainville C, VanDeVelde-Coke S, Lamont L, et al. Implementation of the Na-tional Nursing Quality Report Initiative in Canada:Insights from Pilot Participants. Journal of NursingCare Quality. 2015;30(4):E9–E16.

[147] Ross JS, Williams L, Damush TM, Matthias M. Physi-cian and other healthcare personnel responses tohospital stroke quality of care performance feed-back: A qualitative study. BMJ Quality and Safety.2016;25(6):441–447.

[148] Taylor A, Neuburger J, Walker K, Cromwell D, GroeneO. How is feedback from national clinical auditsused? Views from English National Health Servicetrust audit leads. Journal of Health Services Re-search and Policy. 2016;21(2):91–100.

[149] Lloyd M, Watmough S, O’Brien S, Furlong N, HardyK. Formalized prescribing error feedback from hos-pital pharmacists: doctors’ attitudes and opinions.British Journal of Hospital Medicine. 2015;76(12):713–718.

[150] Lippert ML, Kousgaard MB, Bjerrum L. General prac-titioners uses and perceptions of voluntary elec-tronic feedback on treatment outcomes – a quali-tative study. BMC Family Practice. 2014 Dev;15(1):193.

176

Appendix

References

[151] Wilkinson EK, McColl A, Exworthy M, Roderick P,Smith H, Moore M, et al. Reactions to the use ofevidence-based performance indicators in primarycare: A qualitative study. Quality and Safety inHealth Care. 2000;9(3):166–174.

[152] Johnston S, Green M, Thille P, Savage C, Roberts L,Russell G, et al. Performance feedback: An ex-ploratory study to examine the acceptability andimpact for interdisciplinary primary care teams.BMC Family Practice. 2011;12.

[153] Mannion R, Goddard M. Impact of published clinicaloutcomes data: case study in NHS hospital trusts.BMJ. 2001 Aug;323(7307):260–263.

[154] Palmer C, Bycroft J, Healey K, Field A, Ghafel M. Canformal collaborative methodologies improve qual-ity in primary health care in New Zealand? insightsfrom the EQUIPPED Auckland Collaborative. Journalof Primary Health Care. 2012;4(4):328–336.

[155] Vachon B, Désorcy B, Camirand M, Rodrigue J, Ques-nel L, Guimond C, et al. Engaging primary care prac-titioners in quality improvement: Making explicitthe program theory of an interprofessional educa-tion intervention. BMC Health Services Research.2013;13(1).

[156] Paskins Z, John H, Hassell A, Rowe I. The per-ceived advantages and disadvantages of regionalaudit: A qualitative study. Clinical Governance.2010;15(3):200–209.

[157] Payne VL, Hysong SJ. Model depicting aspects ofaudit and feedback that impact physicians’ accep-tance of clinical performance feedback. BMC HealthServices Research. 2016;16(1).

[158] Eldh AC, Fredriksson M, Vengberg S, Halford C, WallinL, Dahlström T, et al. Depicting the interplay be-tween organisational tiers in the use of a nationalquality registry to develop quality of care in Sweden.BMC Health Services Research. 2015;15(1).

[159] Chadwick LM, Macphail A, Ibrahim JE, Mcauliffe L,Koch S, Wells Y. Senior staff perspectives of aquality indicator program in public sector residen-tial aged care services: A qualitative cross-sectionalstudy in Victoria, Australia. Australian Health Re-view. 2016;40(1):54–62.

[160] De Vos MLG, Van Der Veer SN, Graafmans WC, DeKeizer NF, Jager KJ, Westert GP, et al. Process evalu-ation of a tailored multifaceted feedback programto improve the quality of intensive care by usingquality indicators. BMJ Quality and Safety. 2013Mar;22(3):233–241.

[161] Wones RG. Failure of low-cost audits with feedbackto reduce laboratory test utilization. Medical Care.1987;25(1):78–82.

[162] Søndergaard J, Andersen M, Vach K, Kragstrup J,Maclure M, Gram LF. Detailed postal feedback aboutprescribing to asthma patients combined with aguideline statement showed no impact: a ran-

domised controlled trial. European journal of clini-cal pharmacology. 2002 May;58(2):127–32.

[163] Seip B, Frich JC, Hoff G. Doctors’ experiences with aquality assurance programme. Clinical Governance.2012;17(4):297–306.

[164] Shepherd N, Meehan TJ, Davidson F, Stedman T. Anevaluation of a benchmarking initiative in extendedtreatment mental health services. Australian HealthReview. 2010;34(3):328–333.

[165] McLellan L, Dornan T, Newton P, Williams SD, LewisP, Steinke D, et al. Pharmacist-led feedback work-shops increase appropriate prescribing of antimi-crobials. Journal of Antimicrobial Chemotherapy.2016;71(5):1415–1425.

[166] Powell AA, White KM, Partin MR, Halek K, Hysong SJ,Zarling E, et al. More than a score: A qualitativestudy of ancillary benefits of performancemeasure-ment. BMJ Quality and Safety. 2014;23(8):651–658.

[167] Boyce MB, Browne JP, Greenhalgh J. Surgeon’s ex-periences of receiving peer benchmarked feedbackusing patient-reported outcome measures: A qual-itative study. Implementation Science. 2014;9(1).

[168] Schectman JM, Kanwal NK, Scott Schroth W, ElinskyEG. The effect of an education and feedback inter-vention on group-model and network-model healthmaintenance organization physician prescribing be-havior. Medical Care. 1995;33(2):139–144.

[169] Cameron M, Penney G, MacLennan G, McLeer S,Walker A. Impact on maternity professionals ofnovel approaches to clinical audit feedback. Eval-uation and the Health Professions. 2007;30(1):75–95.

[170] Søndergaard J, Andersen M, Kragstrup J, Hansen H,Freng Gram L. Why has postal prescriber feedbackno substantial impact on general practitioners’ pre-scribing practice? A qualitative study. EuropeanJournal of Clinical Pharmacology. 2002;58(2):133–136.

[171] Dixon-Woods M, Leslie M, Bion J, Tarrant C. Whatcounts? An ethnographic study of infection data re-ported to a patient safety program. Milbank Quar-terly. 2012;90(3):548–591.

[172] Grando V, Rantz M, Maas M. Nursing home staff’sviews on quality improvement interventions: a fol-low up study. Journal of gerontological nursing.2007;33(1):40–47.

[173] Schneider A, Wensing M, Biessecker K, Quinzler R,Kaufmann-Kolle P, Szecsenyi J. Impact of quality cir-cles for improvement of asthma care: results of arandomized controlled trial. Journal of evaluationin clinical practice. 2008 Apr;14(2):185–90.

[174] Ferguson TB, Peterson ED, Coombs LP, Eiken MC,Carey ML, Grover FL, et al. Use of Continuous Qual-ity Improvement to Increase Use of Process Mea-sures in Patients Undergoing Coronary Artery By-pass Graft Surgery: A Randomized Controlled Trial.Journal of the American Medical Association. 2003Jul;290(1):49–56.

177

Appendix

[175] McFadyen C, Lankshear S, Divaris D, Berry M, HunterA, Srigley J, et al. Physician level reporting of surgicaland pathology performance indicators: A regionalstudy to assess feasibility and impact on quality.Canadian Journal of Surgery. 2015;58(1):31–40.

[176] Kirschner K, Braspenning J, Jacobs JEA, Grol R. Expe-riences of general practices with a participatory pay-for-performance program: A qualitative study in pri-mary care. Australian Journal of Primary Health.2013;19(2):102–106.

[177] Groene O, Klazinga N, Kazandjian V, Lombrail P, Bar-tels P. The World Health Organization PerformanceAssessment Tool for quality improvement in hospi-tals (PATH): An analysis of the pilot implementationin 37 hospitals. International Journal for Quality inHealth Care. 2008;20(3):155–161.

[178] Henri Maurice Veillard J, Louise Schiøtz M, GuissetAL, Davidson Brown A, S Klazinga N. The PATH projectin eight European countries: an evaluation. Interna-tional Journal of Health Care Quality Assurance. 2013Sep;26(8):703–713.

[179] Morrell C, Harvey G, Kitson A. Practitioner basedquality improvement: a review of the Royal Collegeof Nursing’s dynamic standard setting system. Qual-ity in health care : QHC. 1997;6(1):29–34.

[180] Siddiqi K, Volz A, Armas L, Otero L, Ugaz R, OchoaE, et al. Could clinical audit improve the diagnosisof pulmonary tuberculosis in Cuba, Peru and Bo-livia? Tropical Medicine and International Health.2008;13(4):566–578.

[181] Nessim C, Bensimon CM, Hales B, Laflamme C,Fenech D, Smith A. Surgical Site Infection Preven-tion: A Qualitative Analysis of an Individualized Au-dit and Feedback Model. Journal of the AmericanCollege of Surgeons. 2012 Dev;215(6):850–857.

[182] Gort M, Broekhuis M, Regts G. How teams use in-dicators for quality improvement - A multiple-casestudy on the use of multiple indicators in multidis-ciplinary breast cancer teams. Social Science andMedicine. 2013;96:69–77.

[183] Jeffs L, Beswick S, Lo J, Lai Y, Chhun A, CampbellH. Insights from staff nurses and managers onunit-specific nursing performance dashboards: Aqualitative study. BMJ Quality and Safety. 2014Sep;23(12):1001–1006.

[184] Grant AM, Guthrie B, Dreischulte T. Develop-ing a complex intervention to improve prescrib-ing safety in primary care: Mixed methods feasi-bility and optimisation pilot study. BMJ Open. 2014Jan;4(1):e004153.

[185] Damschroder LJ, Robinson CH, Francis J, Bentley DR,Krein SL, Rosland AM, et al. Effects of PerformanceMeasure Implementation on Clinical Manager andProvider Motivation. Journal of General InternalMedicine. 2014;29(4):877–884.

[186] Simms RA, Ping H, Yelland A, Beringer AJ, Fox R,Draycott TJ. Development of maternity dashboards

across a UK health region; Current practice, contin-uing problems. European Journal of Obstetrics Gy-necology and Reproductive Biology. 2013;170(1):119–124.

[187] Kristensen H, Hounsgaard L. Evaluating the impactof audits and feedback as methods for implemen-tation of evidence in stroke rehabilitation. BritishJournal of Occupational Therapy. 2014;77(5):251–259.

[188] Kiefe CI, Weissman NW, Allison JJ, Farmer R, WeaverM, Dale Williams O. Identifying achievable bench-marks of care: Concepts and methodology. In-ternational Journal for Quality in Health Care. 1998Oct;10(5):443–447.

[189] Baker R, Smith JF, Lambert PC. Randomised con-trolled trial of the effectiveness of feedback in im-proving test ordering in general practice. Scandina-vian Journal of Primary Health Care. 2003;21(4):219–223.

[190] Filardo G, Nicewander D, Herrin J, Edwards J, Galim-bertti P, Tietze M, et al. A hospital-randomized con-trolled trial of a formal quality improvement educa-tional program in rural and small community Texashospitals: One year results. International Journalfor Quality in Health Care. 2009;21(4):225–232.

[191] Foster JM, Hoskins G, Smith B, Lee AJ, Price D, Pin-nock H. Practice development plans to improve theprimary care management of acute asthma: ran-domised controlled trial. BMC Family Practice. 2007Dev;8(1):23.

[192] Rantz MJ, Popejoy L, Petroski GF, Madsen RW, MehrDR, Zwygart-Stauffacher M, et al. Randomizedclinical trial of a quality improvement interven-tion in nursing homes. The Gerontologist. 2001Aug;41(4):525–38.

[193] Sommers LS, Sholtz R, Shepherd RM, StarkweatherDB. Physician involvement in quality assurance.Medical Care. 1984;22(12):1115–1138.

[194] Goff DC, Gu L, Cantley LK, Sheedy DJ, Cohen SJ. Qual-ity of care for secondary prevention for patientswith coronary heart disease: Results of the Has-tening the Effective Application of Research throughTechnology (HEART) trial. American Heart Journal.2003;146(6):1045–1051.

[195] Fairbrother G, Hanson KL, Friedman S, Butts GC.The impact of physician bonuses, enhanced fees,and feedback on childhood immunization cover-age rates. American Journal of Public Health.1999;89(2):171–175.

[196] Curran E, Harper P, Loveday H, Gilmour H, Jones S,Benneyan J, et al. Results of a multicentre ran-domised controlled trial of statistical process con-trol charts and structured diagnostic tools to re-duce ward-acquired meticillin-resistant Staphylo-coccus aureus: the CHART Project. Journal of Hospi-tal Infection. 2008;70(2):127–135.

[197] Reeves D, Doran T, Valderas JM, Kontopantelis E,Trueman P, Sutton M, et al. How to identify when a

178

Appendix

References

performance indicator has run its course. BMJ. 2010Apr;340:c1717.

[198] authors listed N. Medical audit in general prac-tice. I: Effects on doctors’ clinical behaviour for com-mon childhood conditions. North of England Studyof Standards and Performance in General Practice.BMJ. 1992 Jun;304(6840):1480–4.

[199] Nasser M, Oxman AD, Paulsen E, Fedorowicz Z. Localconsensus processes: effects on professional prac-tice and health care outcomes. Cochrane Databaseof Systematic Reviews. 2017 Dev;4.

[200] Ivers NM, Desveaux L, Presseau J, Reis C, WittemanHO, Taljaard MK, et al. Testing feedback messageframing and comparators to address prescribing ofhigh-risk medications in nursing homes: Protocolfor a pragmatic, factorial, cluster-randomized trial.Implementation Science. 2017 Jul;12(1):86.

[201] Hoffmann TC, Glasziou PP, Boutron I, Milne R, Per-era R, Moher D, et al. Better reporting of inter-ventions: template for intervention description andreplication (TIDieR) checklist and guide. BMJ. 2014Mar;348(mar07 3):g1687–g1687.

[202] Dowding D, Randell R, Gardner P, Fitzpatrick G, DykesP, Favela J, et al. Dashboards for improving patientcare: Review of the literature. International Journalof Medical Informatics. 2015 Feb;84(2):87–100.

[203] Foy R, MacLennan G, Grimshaw J, Penney G, Camp-bell M, Grol R. Attributes of clinical recommenda-tions that influence change in practice following au-dit and feedback. Journal of clinical epidemiology.2002 Jul;55(7):717–22.

[204] Sales A, Smith J, Curran G, Kochevar L. Models,Strategies, and Tools. Journal of General InternalMedicine. 2006 Feb;21(S2):S43–S49.

[205] van Engen-Verheul MM, de Keizer NF, van der VeerSN, Kemps HMC, op Reimer WJS, Jaspers MWM, et al.Evaluating the effect of a web-based quality im-provement system with feedback and outreach vis-its on guideline concordance in the field of cardiacrehabilitation: rationale and study protocol. Imple-mentation Science. 2014 Dev;9(1):780.

[206] Van Engen-Verheul MM, Van Der Veer SN, De KeizerNF, Tjon Sjoe Sjoe W, Van Der Zwan EPA, PeekN. A Web-based system to facilitate local, sys-tematic quality improvement by multidisciplinarycare teams: Development and first experiences ofCARDSS online. In: Studies in Health Technologyand Informatics. vol. 192; 2013. p. 248–252.

[207] Van Engen-Verheul M, Kemps H, Kraaijenhagen R, DeKeizer N, Peek N. Modified rand method to derivequality indicators: A case study in cardiac rehabili-tation. In: Studies in Health Technology and Infor-matics. vol. 169; 2011. p. 88–92.

[208] Jager KJ, Zoccali C, Macleod A, Dekker FW. Confound-ing: what it is and how to deal with it. Kidney inter-national. 2008 Feb;73(3):256–60.

[209] Nakagawa S, Schielzeth H. A general and simplemethod for obtaining R 2 from generalized linearmixed-effects models. Methods in Ecology and Evo-lution. 2013 Feb;4(2):133–142.

[210] Workman M. An experimental assessment of se-mantic apprehension of graphical linguistics. Com-puters in Human Behavior. 2008 Sep;24(6):2578–2596.

[211] Elting LS, Martin CG, Cantor SB, Rubenstein EB. In-fluence of data display formats on physician inves-tigators’ decisions to stop clinical trials: prospec-tive trial with repeated measures. BMJ. 1999Jun;318(7197):1527–1531.

[212] Presseau J, Sniehotta FF, Francis JJ, Campbell NC.Multiple goals and time constraints: Perceivedimpact on physicians’ performance of evidence-based behaviours. Implementation Science. 2009Jan;4(1):77.

[213] Pannick S, Sevdalis N, Athanasiou T. Beyond clini-cal engagement: A pragmatic model for quality im-provement interventions, aligning clinical and man-agerial priorities. BMJ Quality and Safety. 2016Dev;25(9):716–725.

[214] Grol R, Grimshaw J. From best evidence to best prac-tice: Effective implementation of change in patients’care. The Lancet. 2003 Oct;362(9391):1225–1230.

[215] Lilford R, Mohammed MA, Spiegelhalter D, Thom-son R. Use and misuse of process and outcomedata in managing performance of acute medicalcare: Avoiding institutional stigma. The Lancet.2004;363(9415):1147–1154.

[216] Bradley EH, Holmboe ES, Mattera JA, Roumanis SA,Radford MJ, Krumholz HM. Data feedback ef-forts in quality improvement: Lessons learned fromUS hospitals. Quality and Safety in Health Care.2004;13(1):26–31.

[217] Daley K, Richardson J, James I, Chambers A, CorbettD. Clinical dashboard: use in older adult mentalhealth wards. The Psychiatrist. 2013 Mar;37(3):85–88.

[218] Rogers EM. Diffusion of Innovations. 5th ed. Simonand Schuster; 2003.

[219] Lawler E. Control systems in organizations. In:Handbook of industrial and organizational psychol-ogy. American Psychological Association; 1976. p.1247–1291.

[220] Brown B, Jameson D, Daker-White G, Buchan I, IversN, Peek N, et al. A meta-synthesis of findings fromqualitative studies of audit and feedback interven-tions. PROSPERO. 2015;CRD4201501.

[221] Wolff JL, Starfield B, Anderson G. Prevalence, expen-ditures, and complications of multiple chronic con-ditions in the elderly. Archives of internal medicine.2002;162:2269–2276.

[222] Van Oostrom SH, Picavet HSJ, Van Gelder BM, Lem-mens LC, Hoeymans N, Van Dijk CE, et al. Multi-morbidity and comorbidity in the Dutch population-

179

Appendix

data from general practices. BMC Public Health.2012;12(1):715.

[223] Friedman CP, Rubin J, Brown J, Buntin M, Corn M,Etheredge L, et al. Toward a science of learning sys-tems: A research agenda for the high-functioningLearning Health System. Journal of the AmericanMedical Informatics Association. 2015;22(1):43–50.

[224] Lawler PR, Filion KB, Eisenberg MJ. Efficacyof exercise-based cardiac rehabilitation post-myocardial infarction: A systematic review andmeta-analysis of randomized controlled trials.American Heart Journal. 2011;162(4).

[225] Bjarnason-Wehrens B, McGee H, Zwisler AD, PiepoliMF, Benzer W, Schmid JP, et al. Cardiac rehabilita-tion in Europe: results from the European CardiacRehabilitation Inventory Survey. European Journalof Cardiovascular Prevention & Rehabilitation. 2010Aug;17(4):410–418.

[226] Goud R, de Keizer NF, ter Riet G, Wyatt JC, HasmanA, Hellemans IM, et al. Effect of guideline basedcomputerised decision support on decision makingof multidisciplinary teams: cluster randomised trialin cardiac rehabilitation. BMJ. 2009 Apr;338(apr272):b1440–b1440.

[227] Leon AS, Franklin BA, Costa F, Balady GJ, BerraKA, Stewart KJ, et al. Cardiac rehabilitation andsecondary prevention of coronary heart disease:an American Heart Association scientific statementfrom the Council on Clinical Cardiology (Subcom-mittee on Exercise, Cardiac Rehabilitation, and Pre-vention) and the Council on Nut. Circulation. 2005Jan;111(3):369–76.

[228] Langley G, Nolan K, Nolan T, Norman C, Provost L.The Improvement Guide: A Practical Approach to En-hancing Organizational Performance. 2nd ed. SanFrancisco: Jossey-Bass Publishers; 2009.

[229] Goud R, Hasman A, Peek N. Development of aguideline-based decision support system with ex-planation facilities for outpatient therapy. Com-puter Methods and Programs in Biomedicine.2008;91(2):145–153.

[230] Pinheiro J, Bates D. Mixed effects models in S andS-plus. New York: Springer; 2000.

[231] White IR, Royston P, Wood AM. Multiple imputationusing chained equations: Issues and guidance forpractice. Statistics in Medicine. 2011;30(4):377–399.

[232] Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D,Reed JE. Systematic review of the application ofthe plan-do-study-act method to improve quality inhealthcare. BMJ Quality and Safety. 2014;23(4):290–298.

[233] Sinuff T, Muscedere J, Rozmovits L, Dale CM, ScalesDC. A qualitative study of the variable effects of Au-dit and feedback in the ICU. BMJ Quality and Safety.2015;24(6):393–399.

[234] Gardner KL, Sibthorpe B, Longstaff D. National qual-ity and performance system for divisions of general

practice: Early reflections on a system under devel-opment. Australia and New Zealand Health Policy.2008;5.

[235] Ivers N, Barnsley J, Upshur R, Tu K, Shah B, GrimshawJ, et al. My approach to this job is. One person at atime: Perceived discordance between population-Level quality targets and patient-Centred care.Canadian Family Physician. 2014;60:258–266.

[236] Wahlström R, Kounnavong S, Sisounthone B,Phanyanouvong A, Southammavong T, Eriksson B,et al. Effectiveness of feedback for improving casemanagement of malaria, diarrhoea and pneumonia- A randomized controlled trial at provincial hospi-tals in Lao PDR. Tropical Medicine and InternationalHealth. 2003;8(10):901–909.

[237] Raasch BA, Hays R, Buettner PG. An educational in-tervention to improve diagnosis and managementof suspicious skin lesions. The Journal of continuingeducation in the health professions. 2000;20(1):39–51.

[238] van Engen-Verheul MM, Peek N, Haafkens JA, JoukesE, Vromen T, Jaspers MWM, et al. What is needed toimplement a web-based audit and feedback inter-vention with outreach visits to improve care quality:A concept mapping study among cardiac rehabilita-tion teams. International Journal of Medical Infor-matics. 2017 Jan;97:76–85.

[239] Ancker JS, Kern LM, Abramson E, Kaushal R. The Tri-angle Model for evaluating the effect of health infor-mation technology on healthcare quality and safety.Journal of the American Medical Informatics Associ-ation. 2012;19(1):61–65.

[240] King HB, Battles J, Baker DP, Alonso A, Salas E, Web-ster J, et al. TeamSTEPPS™: Team Strategies andTools to Enhance Performance and Patient Safety.January; 2008.

[241] Carayon P, Schoofs Hundt A, Karsh BT, Gurses AP,Alvarado CJ, Smith M, et al. Work system design forpatient safety: The SEIPS model. Quality and Safetyin Health Care. 2006 Dev;15(SUPPL. 1):i50–8.

[242] Sittig DF, Kahol K, Singh H. Sociotechnical evalua-tion of the safety and effectiveness of point-of-caremobile computing devices: A case study conductedin india. Electronic Health Records: Challenges inDesign and Implementation. 2013;p. 115–131.

[243] van der Veer SN, de Vos MLG, van der Voort PHJ, PeekN, Abu-Hanna A, Westert GP, et al. Effect of a mul-tifaceted performance feedback strategy on lengthof stay compared with benchmark reports alone: acluster randomized trial in intensive care*. Criticalcare medicine. 2013 Aug;41(8):1893–904.

[244] de Vos MLG, Veer SN, Wouterse B, Graafmans WC,Peek N, de Keizer NF, et al. A multifaceted feedbackstrategy alone does not improve the adherence toorganizational guideline-based standards: A clusterrandomized trial in intensive care. ImplementationScience. 2015;10(1):95.

180

Appendix

References

[245] Ivers NM, Tu K, Young J, Francis JJ, Barnsley J, Shah BR,et al. Feedback GAP: Pragmatic, cluster-randomizedtrial of goal setting and action plans to increasethe effectiveness of audit and feedback interven-tions in primary care. Implementation Science. 2013Jan;8(1):142.

[246] O’Brien MA, Rogers S, Jamtvedt G, Oxman A,Odgaard-Jensen J, Kristoffersen D, et al. Educationaloutreach visits: effects on professional practice andhealth care outcomes. Cochrane Database Syst Rev.2008;p. CD000409.

[247] Ukoumunne OC, Guilliford MC, Chin S, Serne JAC,Burney PGJ, Donner A. Methods in health service re-search: Evaluation of health interventions at areaand organisation level. . British Medical Journal.1999;319:376–379.

[248] Dawson L, Zarin DA, Emanuel EJ, Friedman LM,Chaudhari B, Goodman SN. Considering usual med-ical care in clinical trial design. PLOS Medicine.2009;6(9).

[249] Eysenbach G. The law of attrition. Journal of MedicalInternet Research. 2005;7(1).

[250] Moher D, Hopewell S, Schulz KF, Montori V, GøtzschePC, Devereaux PJ, et al. CONSORT 2010 explanationand elaboration: Updated guidelines for reportingparallel group randomised trials. International Jour-nal of Surgery. 2012;10(1):28–55.

[251] Campbell MK, Elbourne DR, Altman DG. CONSORTstatement: extension to cluster randomised trials.BMJ. 2004;328(7441):702–708.

[252] Zwarentein M, Treweek S, Gagnier JJ, Altman DG, Tu-nis S, Haynes B, et al. Improving the reportingof pragmatic trials: An extension of the CONSORTstatement. Journal of Chinese Integrative Medicine.2009;7(4):392–397.

[253] Arts DGT, De Keizer NF, Scheffer GJ. Defining andimproving data quality in medical registries: A lit-erature review, case study, and generic framework.Journal of the American Medical Informatics Asso-ciation. 2002;9(6):600–611.

[254] Flodgren G, Parmelli E, Doumit G, Gattellari M,O’Brien MA, Grimshaw J, et al. Local opinion leaders:effects on professional practice and health care out-comes. Cochrane Database of Systematic Reviews.2011 Aug;(8):CD000125.

[255] Arts D, De Keizer N, Scheffer GJ, De Jonge E. Qualityof data collected for severity of illness scores in theDutch National Intensive Care Evaluation (NICE) reg-istry. Intensive Care Medicine. 2002 May;28(5):656–659.

[256] Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalleJR, Lazaro P, et al. The RAND/UCLA AppropriatenessMethod User’s Manual. Transformation. 2001;p. 109.

[257] Brown B, Balatsoukas P, Williams R, Sperrin M,Buchan I. Interface design recommendations forcomputerised clinical audit and feedback: Hybrid

usability evidence from a research-led system. In-ternational Journal of Medical Informatics. 2016Oct;94:191–206.

[258] Ajzen I, Netemeyer R, Ryn MV, Ajzen I. The The-ory of Planned Behaviour. Organ Behav Hum Dec.1991;50(2):179–211.

[259] Roberts C, Torgerson DJ. Baseline imbalance in ran-domised controlled trials. BMJ. 1999;319(7203):185.

[260] Hysong SJ, Best RG, Pugh JA. Audit and feedbackand clinical practice guideline adherence: Mak-ing feedback actionable. Implementation Science.2006;1(1):9.

[261] Ivers NM, Halperin IJ, Barnsley J, Grimshaw JM, ShahBR, Tu K, et al. Allocation techniques for balanceat baseline in cluster randomized trials: a method-ological review. Trials. 2012 Dev;13(1):120.

[262] Roos-Blom MJ, Dongelmans D, Arbous MS, de JongeE, de Keizer N. How to Assist Intensive Care Units inImproving Healthcare Quality. Development of Ac-tionable Quality Indicators on Blood use. Studies inhealth technology and informatics. 2015;210:429–33.

[263] van der Veer SN, de Vos ML, Jager KJ, van der VoortPH, Peek N, Westert GP, et al. Evaluating the ef-fectiveness of a tailored multifaceted performancefeedback intervention to improve the quality of care:protocol for a cluster randomized trial in intensivecare. Implementation Science. 2011 Dev;6(1):119.

[264] Eva KW, Armson H, Holmboe E, Lockyer J, LoneyE, Mann K, et al. Factors influencing responsive-ness to feedback: on the interplay between fear,confidence, and reasoning processes. Advances inhealth sciences education : theory and practice.2012 Mar;17(1):15–26.

[265] Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO,Beyer F, et al. Do self- reported intentions predictclinicians’ behaviour: A systematic review. Imple-mentation Science. 2006 Nov;1(1):28.

[266] and Hartley S, Foy R, Walwyn REA, Cicero R, Far-rin AJ, Francis JJ, et al. The evaluation of en-hanced feedback interventions to reduce unneces-sary blood transfusions (AFFINITIE): protocol for twolinked cluster randomised factorial controlled trials.Implementation Science. 2017 Jul;12(1):84.

[267] Ivers NM, Grimshaw JM. Reducing research wastewith implementation laboratories. Lancet (London,England). 2016 Aug;388(10044):547–8.

[268] van der Sluijs AF, van Slobbe-Bijlsma ER, Chick SE,Vroom MB, Dongelmans DA, Vlaar APJ. The impactof changes in intensive care organization on patientoutcome and cost-effectiveness—a narrative review.Journal of Intensive Care. 2017 Dev;5(1):13.

[269] Puntillo KA, Max A, Timsit JF, Vignoud L, Chanques G,Robleda G, et al. Determinants of procedural painintensity in the intensive care unit. The Europain®study. American journal of respiratory and criticalcare medicine. 2014 Jan;189(1):39–47.

181

Appendix

[270] Chanques G, Viel E, Constantin JM, Jung B, de LattreS, Carr J, et al. The measurement of pain in inten-sive care unit: Comparison of 5 self-report intensityscales. Pain. 2010 Dev;151(3):711–721.

[271] Gélinas C, Harel F, Fillion L, Puntillo KA, Johnston CC.Sensitivity and Specificity of the Critical-Care PainObservation Tool for the Detection of Pain in Intu-bated Adults After Cardiac Surgery. Journal of Painand Symptom Management. 2009 Jan;37(1):58–67.

[272] Payen JF, Bru O, Bosson JL, Lagrasta A, Novel E, De-schaux I, et al. Assessing pain in critically ill sedatedpatients by using a behavioral pain scale. CriticalCare Medicine. 2001;29(12):2258–63.

[273] McMillan TR, Hyzy RC. Bringing quality improvementinto the intensive care unit. Critical Care Medicine.2007 Feb;35(Suppl):S59–S65.

[274] Scales DC, Dainty K, Hales B, Pinto R, Fowler RA, Ad-hikari NKJ, et al. A multifaceted intervention forquality improvement in a network of intensive careunits: A cluster randomized trial. Journal of theAmerican Medical Association. 2011;305(4):363–372.

[275] Campbell MK, Piaggio G, Elbourne DR, Altman DG.Consort 2010 statement: extension to cluster ran-domised trials. BMJ. 2012 Sep;345(sep04 1):e5661–e5661.

[276] Roos-BlomMJ, GudeWT, Spijkstra JJ, de Jonge E, Don-gelmans D, de Keizer NF. Measuring quality indica-tors to improve pain management in critically ill pa-tients. Journal of critical care. 2018 Oct;49:136–142.

[277] Dutch National Intensive Care Evaluation (NICE)foundation. Data dictionary; 2018.

[278] Roos-Blom MJ, Gude WT, de Jonge E, Spijkstra JJ,van der Veer SN, Dongelmans DA, et al. Developmentof a Web-Based Quality Dashboard Including a Tool-box to Improve Pain Management in Dutch IntensiveCare. Studies in health technology and informatics.2017;235:584–588.

[279] Flottorp SA, Oxman AD, Krause J, Musila NR, WensingM, Godycki-Cwirko M, et al. A checklist for identifyingdeterminants of practice: A systematic review andsynthesis of frameworks and taxonomies of factorsthat prevent or enable improvements in healthcareprofessional practice. Implementation Science. 2013Dev;8(1):35.

[280] Chanques G, Jaber S, Barbotte E, Violet S, SebbaneM, Perrigault PF, et al. Impact of systematic evalua-tion of pain and agitation in an intensive care unit.Critical Care Medicine. 2006 Jun;34(6):1691–1699.

[281] Glynn RJ, Brookhart MA, Stedman M, Avorn J,Solomon DH. Design of Cluster-Randomized Tri-als of Quality Improvement Interventions Aimed atMedical Care Providers. Medical Care. 2009;45(Suppl2):S38–S43.

[282] Roos-Blom MJ, Gude WT, de Jonge E, Spijkstra JJ,van der Veer SN, Peek N, et al. Effect of audit andfeedback with action implementation toolbox on

pain management in intensive care: a cluster ran-domised trial. In submission; 2019.

[283] Erdek MA. Improving assessment and treatment ofpain in the critically ill. International Journal forQuality in Health Care. 2004 Feb;16(1):59–64.

[284] Birt L, Scott S, Cavers D, Campbell C, Walter F. Mem-ber Checking: A Tool to Enhance Trustworthiness orMerely a Nod to Validation? Qualitative Health Re-search. 2016;26(13):1802–1811.

[285] Tierney S, Kislov R, Deaton C. A qualitative study ofa primary-care based intervention to improve themanagement of patients with heart failure: the dy-namic relationship between facilitation and context.BMC Family Practice. 2014;15(1):153.

[286] Guidet B, González-Romá V. Climate and cul-tural aspects in intensive care units. Critical Care.2011;15(6):312.

[287] Reszel J, Dunn SI, Sprague AE, Graham ID, GrimshawJM, Peterson WE, et al. Use of a maternal newbornaudit and feedback system in Ontario: A collectivecase study. BMJ Quality and Safety. 2019;Epub aheadof print.

[288] Orlikowski WJ, Scott SV. Sociomateriality: Challeng-ing the Separation of Technology, Work and Orga-nization. Academy of Management Annals. 2008Jan;2(1):433–474.

[289] Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B.Process evaluations for cluster-randomised trials ofcomplex interventions: a proposed framework fordesign and reporting. Trials. 2013 Jan;14:15.

[290] Avery AJ, Ghaleb M, Barber N, Dean Franklin B, Arm-strong SJ, Serumaga B, et al. The prevalence and na-ture of prescribing and monitoring errors in Englishgeneral practice: A retrospective case note review.British Journal of General Practice. 2013;63(613).

[291] Akbarov A, Kontopantelis E, Sperrin M, Stocks SJ,Williams R, Rodgers S, et al. Primary Care Medica-tion Safety Surveillance with Integrated Primary andSecondary Care Electronic Health Records: A Cross-Sectional Study. Drug Safety. 2015;38(7):671–682.

[292] Agrawal A. Medication errors_ prevention using in-formation technology systems. British Journal ofClinical Pharmacology. 2009;67(6):681–686.

[293] Lainer M, Mann E, Sönnichsen A. Information tech-nology interventions to improve medication safetyin primary care: A systematic review. InternationalJournal for Quality in Health Care. 2013;25(5):590–598.

[294] Shojania KG, Jennings A, Mayhew A, Ramsay C, Ec-cles M, Grimshaw J. Effect of point-of-care computerreminders on physician behaviour: A systematic re-view. Cmaj. 2010 Mar;182(5):E216–25.

[295] van der Sijs H, Aarts J, van Gelder T, Berg M, VultoA. Turning Off Frequently Overridden Drug Alerts:Limited Opportunities for Doing It Safely. Journal

182

Appendix

References

of the American Medical Informatics Association.2008;15(4):439–448.

[296] Van Der Sijs H, Aarts J, Vulto A, Berg M. Overriding ofdrug safety alerts in computerized physician orderentry. Journal of the American Medical InformaticsAssociation. 2006;13(2):138–147.

[297] Avery AJ, Rodgers S, Cantrill JA, Armstrong S, Cress-well K, Eden M, et al. A pharmacist-led informationtechnology intervention for medication errors (PIN-CER): A multicentre, cluster randomised, controlledtrial and cost-eff ectiveness analysis. The Lancet.2012;379(9823):1310–1319.

[298] Brown B, Peek N, Buchan I. The Case for Concep-tual and Computable Cross-Fertilization BetweenAudit and Feedback and Clinical Decision Support.Studies in health technology and informatics. 2015Jan;216:419–23.

[299] Sadler S, Rodgers S, Howard R, Morris CJ, Avery AJ.Training pharmacists to deliver a complex infor-mation technology intervention (PINCER) using theprinciples of educational outreach and root causeanalysis. International Journal of Pharmacy Prac-tice. 2014;22(1):47–58.

[300] Cresswell KM, Sadler S, Rodgers S, Avery A, Cantrill J,Murray SA, et al. An embedded longitudinal multi-faceted qualitative evaluation of a complex clusterrandomized controlled trial aiming to reduce clini-cally important errors in medicines management ingeneral practice. Trials. 2012;13.

[301] Williams R, Keers R, Gude WT, Jeffries M, Davies C,Brown B, et al. SMASH! The Salfordmedication safetydashboard. Journal of Innovation in Health Infor-matics. 2018 Oct;25(3):183.

[302] Jeffries M, Keers RN, Phipps DL, Williams R, BrownB, Avery AJ, et al. Developing a learning health sys-tem: Insights from a qualitative process evaluationof a pharmacist-led electronic audit and feedbackintervention to improve medication safety in pri-mary care. PLOS ONE. 2018 Oct;13(10):e0205419.

[303] Walshe K. Understanding what works—and why—in quality improvement: the need for theory-drivenevaluation. International Journal for Quality inHealth Care. 2007 Apr;19(2):57–59.

[304] Moore GF, Audrey S, Barker M, Bond L, Bonell C,Hardeman W, et al. Process evaluation of complexinterventions: Medical Research Council guidance.BMJ. 2015 Jan;350(mar19_6):h1258.

[305] Hawe P, Shiell A, Riley T. Complex interventions: how“out of control” can a randomised controlled trialbe? BMJ. 2004 Jun;328(7455):1561–1563.

[306] Aarts J. A Sociotechnical Perspective of ElectronicPrescribing. Safety in Medication Use. 2015;p. 101–108.

[307] Greenhalgh T, Shaw S, Wherton J, Hughes G, Lynch J,A’Court C, et al. SCALS: a fourth-generation study ofassisted living technologies in their organisational,

social, political and policy context. BMJ Open. 2016Feb;6(2):e010208.

[308] Cresswell K, Worth A, Sheikh A. Implementing andadopting electronic health record systems. Clin-ical Governance: An International Journal. 2011Oct;16(4):320–336.

[309] Campbell NC, Murray E, Darbyshire J, Emery J, FarmerA, Griffiths F, et al. Designing and evaluating com-plex interventions to improve health care. BMJ. 2007Mar;334(7591):455–9.

[310] McMullen H, Griffiths C, Leber W, Greenhalgh T. Ex-plaining high and low performers in complex inter-vention trials: A new model based on diffusion ofinnovations theory. Trials. 2015;16(1).

[311] O’Cathain A, Murphy E, Nicholl J. Why, and how,mixed methods research is undertaken in healthservices research in England: A mixed methodsstudy. BMC Health Services Research. 2007;7.

[312] Blackwood B, O’Halloran P, Porter S. On the prob-lems of mixing RCTs with qualitative research: Thecase of the MRC framework for the evaluation ofcomplex healthcare interventions. Journal of Re-search in Nursing. 2010;15(6):511–521.

[313] O’Cathain A, Murphy E, Nicholl J. Three techniquesfor integrating data in mixed methods studies. BMJ(Online). 2010;341(7783):1147–1150.

[314] Fetters MD, Curry LA, Creswell JW. Achieving in-tegration in mixed methods designs - Principlesand practices. Health Services Research. 2013;48(6PART2):2134–2156.

[315] King N. Doing Template Analysis. In: QualitativeOrganizational Research: Core Methods and CurrentChallenges. London: SAGE Publications, Inc.; 2017. p.426–450.

[316] Greenhalgh T, Swinglehurst D, Stones R. Rethink-ing resistance to ‘big IT’: a sociological study of whyand when healthcare staff do not use nationallymandated information and communication tech-nologies. Health Services and Delivery Research.2014;2(39):1–86.

[317] Jeffries M, Phipps DL, Howard RL, Avery AJ, RodgersS, Ashcroft DM. Understanding the implementa-tion and adoption of a technological interventionto improve medication safety in primary care: a re-alist evaluation. BMC Health Services Research. 2017Dev;17(1):196.

[318] Crowe S, Tully MP, Cantrill JA. Information in gen-eral medical practices: The information processingmodel. Family Practice. 2009;27(2):230–236.

[319] Petrakaki D, Klecun E, Cornford T. Changes in health-care professional work afforded by technology: Theintroduction of a national electronic patient recordin an English hospital. Organization. 2016;23(2):206–226.

183

Appendix

[320] Klecun E. Transforming healthcare: policy dis-courses of IT and patient-centred care. EuropeanJournal of Information Systems. 2016 Jan;25(1):64–76.

[321] Grant A, Dreischulte T, Guthrie B. Process evaluationof the Data-driven Quality Improvement in PrimaryCare (DQIP) trial: Case study evaluation of adoptionand maintenance of a complex intervention to re-duce high-risk primary care prescribing. BMJ Open.2017;7(3).

[322] Dixon-Woods M, McNicol S, Martin G. Ten chal-lenges in improving quality in healthcare: Lessonsfrom the Health Foundation’s programme evalua-tions and relevant literature. BMJ Quality and Safety.2012;21(10):876–884.

[323] Department for Health and Social Care. Medicinesafety: indicators for safer prescribing; 2018.

[324] Bates DW, Gawande AA. Improving Safety with In-formation Technology. New England Journal ofMedicine. 2003;348(25):2526–2534.

[325] Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Red-den LM, et al. Overrides of medication-related clin-ical decision support alerts in outpatients. Jour-nal of the American Medical Informatics Associa-tion. 2014;21(3):487–491.

[326] Des Jarlais DC, Lyles C, Crepaz N. Improving theReporting Quality of Nonrandomized Evaluations ofBehavioral and Public Health Interventions: TheTREND Statement. American Journal of PublicHealth. 2004;94(3):361–366.

[327] MacDonald TM, Morant SV, Robinson GC, Shield MJ,McGilchrist MM, Murray FE, et al. Association of up-per gastrointestinal toxicity of non-steroidal anti-inflammatory drugs with continued exposure: co-hort study. BMJ. 1997;315(7119):1333–1337.

[328] Hallas J, Dall M, Andries A, Andersen BS, AalykkeC, Hansen JM, et al. Use of single and com-bined antithrombotic therapy and risk of seri-ous upper gastrointestinal bleeding: Populationbased case-control study. British Medical Journal.2006;333(7571):726–728.

[329] Hawkey CJ, Langman MJS. Non-steroidal anti-inflammatory drugs: Overall risks and management.Complementary roles for COX-2 inhibitors and pro-ton pump inhibitors. Gut. 2003;52(4):600–608.

[330] Gooch K, Culleton BF, Manns BJ, Zhang J, AlfonsoH, Tonelli M, et al. NSAID Use and Progressionof Chronic Kidney Disease. American Journal ofMedicine. 2007;120(3).

[331] Huerta C, Castellsague J, Varas-Lorenzo C, García Ro-dríguez LA. Nonsteroidal anti-inflammatory drugsand risk of ARF in the general population. AmericanJournal of Kidney Diseases. 2005;45(3):531–539.

[332] Dreischulte T, Morales DR, Bell S, Guthrie B. Com-bined use of nonsteroidal anti-inflammatory drugswith diuretics and/or renin-angiotensin system in-hibitors in the community increases the risk of acute

kidney injury. Kidney International. 2015;88(2):396–403.

[333] Thomas MC. Diuretics, ACE inhibitors and NSAIDs- The triple whammy. Medical Journal of Australia.2000;172(4):184–185.

[334] Loboz KK, Shenfield GM. Drug combinations and im-paired renal function - The ’triple whammy’. BritishJournal of Clinical Pharmacology. 2005;59(2):239–243.

[335] Gislason GH, Rasmussen JN, Abildstrom SZ,Schramm TK, Hansen ML, Fosbøl EL, et al. Increasedmortality and cardiovascular morbidity associatedwith use of nonsteroidal anti-inflammatory drugs inchronic heart failure. Archives of Internal Medicine.2009;169(2):141–149.

[336] New JP, Leather D, Bakerly ND, McCrae J, Gibson JM.Putting patients in control of data from electronichealth records. BMJ. 2018 Jan;360:j5554.

[337] StataCorp. Stata Statistical Software: Release 15.College Station, TX: StataCorp LLC; 2017.

[338] Linden A. Conducting interrupted time-series anal-ysis for single- and multiple-group comparisons.Stata Journal. 2015;15(2):480–500.

[339] Stevens S, Valderas JM, Doran T, Perera R, Kontopan-telis E. Analysing indicators of performance, satis-faction, or safety using empirical logit transforma-tion. BMJ. 2016 Mar;p. i1114.

[340] Kontopantelis E, Doran T, Springate DA, Buchan I,Reeves D. Regression based quasi-experimental ap-proach when randomisation is not an option: inter-rupted time series analysis. BMJ. 2015 Jun;350(jun095):h2750–h2750.

[341] Petropoulou M, Mavridis D. A comparison of 20heterogeneity variance estimators in statistical syn-thesis of results from studies: a simulation study.Statistics in medicine. 2017 Nov;36(27):4266–4280.

[342] Kontopantelis E, Springate DA, Reeves D. A Re-Analysis of the Cochrane Library Data: The Dan-gers of Unobserved Heterogeneity in Meta-Analyses.PLOS ONE. 2013 Jul;8(7):e69930.

[343] Kontopantelis E, Reeves D. Metaan: Random-effectsmeta-analysis. Stata Journal. 2010;10(3):395–407.

[344] Higgins JPT, Thompson SG. Quantifying hetero-geneity in a meta-analysis. Statistics in Medicine.2002;21(11):1539–1558.

[345] Keers R, Phipps D, Ojeleye L, Hassan L, Chuter A,Mann F, et al. Improving medication safety in pri-mary care: developing a stakeholder-centred elec-tronic prescribing safety indicator dashboard. Phar-macoepidemiology and Drug Safety;2015.

[346] Iacobucci G. General practices achieve 95% of QOFpoints. BMJ. 2016 Oct;p. i5834.

[347] Charlson ME, Pompei P, Ales KL, MacKenzie CR. Anew method of classifying prognostic comorbidityin longitudinal studies: development and valida-tion. Journal of chronic diseases. 1987;40(5):373–83.

184

Appendix

References

[348] Jeffries M, Gude WT, Keers RN, Phipps DL, Williams R,Kontopantelis E, et al. Understanding the utilisationof a novel interactive electronic medication safetydashboard in general practice: a mixed-methodsstudy. In submission; 2019.

[349] NHS. Investment and evolution: A five-year frame-work for GP contract reform to implement The NHSLong Term Plan; 2019.

[350] Dreischulte T, Donnan P, Grant A, Hapca A, McCowanC, Guthrie B. Safer Prescribing — A Trial of Education,Informatics, and Financial Incentives. New EnglandJournal of Medicine. 2016 Mar;374(11):1053–1064.

[351] MacBride-Stewart S, Marwick C, Houston N, Watt I,Patton A, Guthrie B. Evaluation of a complex in-tervention to improve primary care prescribing: aphase IV segmented regression interrupted time se-ries analysis. British Journal of General Practice.2017 May;67(658):e352–e360.

[352] Friedman CP, Rubin JC, Sullivan KJ. Toward anInformation Infrastructure for Global Health Im-provement. Yearbook of Medical Informatics. 2017Sep;26(01):16–23.

[353] Campbell M, Fitzpatrick R, Haines A, Kinmonth AL,Sandercock P, Spiegelhalter D, et al. Framework fordesign and evaluation of complex interventions toimprove health. BMJ. 2000 Sep;321(7262):694–6.

[354] Medical Research Council. A framework for devel-opment and evaluation of RCTs for complex inter-ventions to improve health; 2000.

[355] Eccles M, Grimshaw J, Walker A, Johnston M, PittsN. Changing the behavior of healthcare profession-als: the use of theory in promoting the uptake ofresearch findings. Journal of Clinical Epidemiology.2005 Feb;58(2):107–112.

[356] Davies P, Walker AE, Grimshaw J. Theories of be-haviour change in studies of guideline implemen-tation. Proceedings of the British Psychological So-ciety. 2003;11:120.

[357] Noar SM, Zimmerman RS. Health Behavior Theoryand cumulative knowledge regarding health behav-iors: are we moving in the right direction? Healtheducation research. 2005 Jun;20(3):275–90.

[358] Lippke S, Ziegelmann JP. Theory-Based Health Be-havior Change: Developing, Testing, and ApplyingTheories for Evidence-Based Interventions. AppliedPsychology. 2008 Oct;57(4):698–716.

[359] Ashford A. Behavioural change in profesionnal prac-tice: supporting the development of effective im-plementation strategies. University of Newcastleupon Tyne, Centre for Heal; 1998.

[360] Ferlie EB, Shortell SM. Improving the Quality ofHealth Care in the United Kingdom and the UnitedStates: A Framework for Change. The Milbank Quar-terly. 2001 May;79(2):281–315.

[361] Grol RPTM, Bosch MC, Hulscher MEJL, Eccles MP,Wensing M. Planning and studying improvement

in patient care: The use of theoretical perspectives.Milbank Quarterly. 2007 Mar;85(1):93–138.

[362] Designing theoretically-informed implementationinterventions. Implementation Science. 2006Dev;1(1):4.

[363] Miller G, Galanter E, Pribram I. Plans and the struc-ture of behavior. New York: Henry Holt; 1960.

[364] Locke EA. Goal theory vs. control theory: Contrast-ing approaches to understanding work motivation.Motivation and Emotion. 1991 Mar;15(1):9–28.

[365] Gude WT, Brown B, van der Veer SN, Colquhoun HL,Ivers NM, Brehaut JC, et al. Clinical performancecomparators in audit and feedback: a review of the-ory and evidence. Implementation Science. 2019Dec;14(1):39.

[366] Peek N, Gude WT, Keers RN, Kontopantelis E, Jef-fries M, Phipps DL, et al. Impact of a pharmacist-led, actionable audit and feedback intervention toimprove medication safety in primary care: inter-rupted time series analysis. In submission; 2019.

[367] Barac R, Stein S, Bruce B, Barwick M. Scoping re-view of toolkits as a knowledge translation strategyin health. BMC Medical Informatics and DecisionMaking. 2014 Dev;14(1):121.

[368] Yamada J, Shorkey A, Barwick M, Widger K, StevensBJ. The effectiveness of toolkits as knowledge trans-lation strategies for integrating evidence into clin-ical care: a systematic review. BMJ Open. 2015Apr;5(4):e006808–e006808.

[369] Forsetlund L, Bjørndal A, Rashidian A, JamtvedtG, O’Brien MA, Wolf F, et al. Continuing ed-ucation meetings and workshops: Effects onprofessional practice and health care outcomes.Cochrane Database of Systematic Reviews. 2009Apr;(2):CD003030.

[370] Alagoz E, Chih MY, Hitchcock M, Brown R, QuanbeckA. The use of external change agents to promotequality improvement and organizational change inhealthcare organizations: a systematic review. BMCHealth Services Research. 2018 Dev;18(1):42.

[371] Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexan-der JA, Lowery JC. Fostering implementation ofhealth services research findings into practice: Aconsolidated framework for advancing implemen-tation science. Implementation Science. 2009;4(1).

[372] Hibbard JH, Peters E, Slovic P, Finucane ML, TuslerM. Making health care quality reports easier to use.The Joint Commission journal on quality improve-ment. 2001 Nov;27(11):591–604.

[373] Sweller J, van Merriënboer J, Paas F. Cognitive Archi-tecture and Instructional Design. Educational Psy-chology Review. 1998;10(3):251–296.

[374] Tierney WM, Hui SL, McDonald CJ. Delayed feed-back of physician performance versus immedi-ate reminders to perform preventive care. Effects

185

Appendix

on physician compliance. Medical care. 1986Aug;24(8):659–66.

[375] Geisser S. Predictive Inference: An Introduction.Routledge; 1993.

[376] Sebo P, Maisonneuve H, Fournier JP, Senn N, HallerDM. General practitioners’ views and preferencesabout quality improvement feedback in preventivecare: a cross-sectional study in Switzerland andFrance. Implementation Science. 2017 Dev;12(1):95.

[377] Pedersen MS, Landheim A, Møller M, Lien L. Act-ing on audit & feedback: a qualitative instrumentalcase study in mental health services in Norway. BMCHealth Services Research. 2018 Dev;18(1):71.

[378] Gotlib Conn L, Hoeft C, Neal M, Nathens A. Use ofperformance reports among trauma medical direc-tors and programme managers in the American Col-lege of Surgeons’ Trauma Quality Improvement Pro-gram: a qualitative analysis. BMJ Quality and Safety.2019 Mar;Epub ahead of print.

[379] Shaw T, Janssen A, Crampton R, O’Leary F, Hoyle P,Jones A, et al. Attitudes of health professionalsto using routinely collected clinical data for per-formance feedback and personalised professionaldevelopment. Medical Journal of Australia. 2019Apr;210(S6):S17–S21.

[380] Gude WT, Roos-Blom MJ, van der Veer SN, Dongel-mans DA, de Jonge E, Peek N, et al. How to facilitateaction planning within audit and feedback interven-tions? A mixed-methods process evaluation of an

action implementation toolbox in intensive care. Insubmission; 2019.

[381] Dijkstra R, Wensing M, Thomas R, Akkermans R,Braspenning J, Grimshaw J, et al. The relationshipbetween organisational characteristics and the ef-fects of clinical guidelines on medical performancein hospitals, a meta-analysis. BMC Health ServicesResearch. 2006 Dev;6(1):53.

[382] Sales AE, Schalm C. Data for improvement and clin-ical excellence: Protocol for an audit with feedbackintervention in long-term care. Implementation Sci-ence. 2010;5(1).

[383] Hutchinson AM, Sales AE, Brotto V, Bucknall TK. Im-plementation of an audit with feedback knowledgetranslation intervention to promote medication er-ror reporting in health care: a protocol. Implemen-tation Science. 2015 Dev;10(1):70.

[384] Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ,Cheater F, Flottorp S, et al. Tailored interven-tions to overcome identified barriers to change: ef-fects on professional practice and health care out-comes. Cochrane Database of Systematic Reviews.2010 Mar;(3):CD005470.

[385] Phekoo K, Clements J, Bell D. National Clinical AuditQuality Assessment; 2014.

[386] Grimshaw JM, Ivers N, Linklater S, Foy R, Francis JJ,Gude WT, et al. Reinvigorating stagnant science: Im-plementation laboratories and ameta-laboratory toefficiently advance the science of audit and feed-back. BMJ Quality and Safety. 2019 May;28(5):416–423.

186

Summary

187

Appendix

Summary

Audit and feedback (A&F) is one of the most commonly used quality improvement strate-gies in healthcare. In its most basic form, it provides individual health professionals andcare teams with summaries of their clinical performance, as measured by a set of qualityindicators, over specified periods of time. Although A&F interventions are generally effec-tive at improving quality of care, there is substantial variation in the observed effects, witha quarter of randomised trials of A&F finding little to no effect. So far, there has been littleprogress in increasing A&F effectiveness because interventions have been scarcely guided byextant theory and evidence, and because our understanding of how A&F may lead to qualityimprovement is limited.

In this thesis we aimed to advance this stagnant science of A&F through better under-standing and optimising its ability to improve quality of care. We formulated the followingresearch questions:

1. What is the theory and evidence underpinning A&F and the use of clinical performancecomparators in A&F interventions?

2. What is the impact of three state-of-the-art, theory and evidence-based electronic A&Finterventions across different clinical settings on the quality of care?

3. What are the mechanisms through which A&F affects health professionals’ clinical per-formance perceptions, improvement intentions, behaviour, and, ultimately, quality ofcare?

This thesis consists of three parts, which we summarise below. We also provide an overallconclusion.

Part I. Theory and evidence of audit and feedbackChapter 2 explains A&F on the basis of Control Theory, and discusses the theory’s role in thefield of health informatics in general and in the design and evaluation of A&F interventionsin particular. Control Theory proposes that behaviour is regulated by a negative feedbackloop, in which a self-regulating agent compares the perception of its current state against agoal state and will strive to reduce perceived discrepancies by modifying its behaviour. Theclassical example of this feedback loop is the thermostat. In this system, the thermostatcontinuously measures current air temperature and compares it to its desired temperaturesetting. If it detects a discrepancy between these two values, the thermostat “behaves” byturning on the heater which begins to bring warm air into the room. It continues to do thisuntil the room has warmed up enough so that the discrepancy is no longer observed. Whentranslated to healthcare settings, health professionals who perceive discrepancies betweencurrent clinical performance and their target will develop intentions and subsequently un-dertake action (i.e. behaviour) to improve practice, and will continue to do so until the targetis achieved. Control Theory has been used to synthesise evidence of A&F interventions, en-hance their designs, explain why interventions were or were not successful, and to generatehypotheses about how feedback mechanisms work in practice.

A&F often includes clinical performance comparators to help health professionals iden-tify discrepancies between desired and actual practice. In Chapter 3 we described currentchoices for performance comparators and their associated mechanisms that might have im-plications for effective A&F. We extracted these from theories and empirical studies previouslyidentified by other reviews. We found across 146 randomised trials that feedback recipients’

189

Appendix

performance was most frequently compared against the performance of others (benchmarks;60%). Other comparators included recipients’ own performance over time (trends; 10%) andtarget standards (explicit targets; 11%), and 13% of trials used a combination of these options.In studies featuring benchmarks, 42% compared against mean performance. We distilled 20behaviour change mechanisms of each comparator from 12 theories, 5 randomised trials,and 42 qualitative A&F studies. Clinical performance comparators in published literaturewere poorly informed by theory and did not explicitly account for mechanisms reported inqualitative studies. Based on our review we argued that there is considerable opportunity toimprove the design of performance comparators by (1) providing tailored comparisons ratherthan benchmarking everyone against the mean, (2) limiting the amount of comparators beingdisplayed while providing more comparative information upon request to balance the feed-back’s credibility and actionability, (3) providing performance trends but not trends alone,and (4) encouraging feedback recipients to set personal, explicit targets guided by relevantinformation.

Part II. Cardiac rehabilitation

The studies in this part all involve an electronic A&F intervention to improve cardiac rehabil-itation (CR) in the Netherlands. Feedback was provided on a set of 18 quality indicators (CRprocesses and outcomes) with benchmark comparisons (low, intermediate, or high) throughan online dashboard. During quarterly outreach visits, multidisciplinary CR teams used thedashboard to discuss the feedback, select indicators for improvement into their structuredaction plans and set specific improvement goals.

Chapter 4 presents a laboratory experiment with 41 individual CR professionals, and a fieldstudy with the 18 CR teams to investigate the factors that influenced intentions to improvepractice; both took place alongside a cluster randomised controlled trial that evaluated theeffectiveness of the intervention. In the laboratory experiment, we presented the 41 indi-vidual participants with two feedback reports which were randomly selected from the 50reports generated during the trial. From the reports, we asked them to select indicators forimprovement into their action plan. Selections that were at odds with Control Theory (e.g.not selecting an indicator for which performance was below the benchmark) prompted aquestion asking participants to explain their choice. In the field study, we analysed whichindicators were selected into the action plans by the 18 CR teams during the trial. Both thelaboratory experiment and field study showed that indicators with lower performance scoreswere more likely to be selected for improvement. Also, performance being benchmarked aslow and intermediate increased this probability in laboratory settings. Participants ignoredthe benchmarks in 34% (laboratory experiment) and 48% (field study) of their selections, be-cause they disagreed with benchmarks, deemed improvement unfeasible or did not considerthe indicator an essential aspect of care quality.

Chapter 5 describes the results of the cluster randomised controlled trial where 18 CRcentres (14,847 patients) were randomly assigned to receive feedback about either psychoso-cial or physical rehabilitation, for at least 1 year. The primary outcome was performance oncare processes and outcomes measured at patient-level. Secondary outcomes were guide-line concordance with respect to prescribing CR therapies, and data completeness. Exceptfor a modest improvement in data completeness, we did not find any improvements on theprimary or secondary outcome measures. Post-hoc analysis revealed that teams had defined233 improvement actions, but only completed 49 (21%) of them during the study period.

190

Appendix

Summary

Part III. Intensive care pain managementThe chapters in this part concern an electronic A&F intervention to improve painmanagementin Dutch intensive care units (ICUs). The central component of the intervention was an on-line dashboard which provided ICU teams with two functionalities: (1) monthly performancefeedback on four newly developed quality indicators with comparisons to trends, two bench-marks (median and top 10%), and explicit targets that ICU teams could set themselves, and(2) developing and managing structured action plans. In addition, an action implementationtoolbox, containing predefined lists of potential barriers in the care process and suggestedactions for improvement, was designed to facilitate ICUs’ action planning processes.

Chapter 6 describes the study protocol, consisting of a laboratory experiment to investi-gate how A&F affects improvement intentions, and a head-to-head cluster randomised con-trolled trial with a mixed-methods process evaluation to assess whether and how the toolboxfacilitated ICU professionals in translating their improvement intentions into action.

Chapter 7 reports on the laboratory experiment with 72 ICU professionals from 21 units,which took place approximately one month before units enrolled into the trial. We collectedprofessionals’ perceptions about their clinical performance; peer performance; targets; andimprovement intentions before and after receiving first-time feedback. We found that ICU pro-fessionals often overestimated their clinical performance, on average by 23%, and seldom un-derestimated it. They similarly overestimated peer performance, and set targets much higherthan the top performance benchmarks. Sixty-eight percent of improvement intentions wereconsistent with actual gaps in performance, even before professionals had received feedback;the consistency increased to 80% after receiving feedback. However, in 56% professionals stillwanted to improve care aspects at which they were already top performers. Alternatively, in8% they lacked improvement intentions because they did not consider indicators important;did not trust the data; or deemed benchmarks unrealistic.

Chapter 8 presents the cluster randomised controlled trial in 21 ICUs (25,234 patient admis-sions) which all received feedback and a blank structured action plan for six to nine months,but were randomly assigned to receive the intervention with or without the action implemen-tation toolbox. The primary outcome was a composite of four pain management indicatorsreflecting the proportion of patient-shift observations in which patients’ pain was measuredand acceptable, or unacceptable and normalised within one hour. Secondary outcomes wereeach of the individual indicators. We found an absolute improvement on adequate pain man-agement of 14.8% (95% confidence interval (CI), 14.0 to 15.5) in ICUs with toolbox and 4.8% (95%CI, 4.2 to 5.5) in ICUs without. Improvement was higher in the toolbox group (p=0.049). Thefound effects were significant for pain management processes but not outcomes.

Chapter 9 is the mixed-methods process evaluation in which we extracted all individualactions from the action plans developed by ICUs for six months. We also conducted semi-structured interviews with participants during the trial. Our analysis was guided by Clini-cal Performance Feedback Intervention Theory. We have recently published this new theorywhich builds upon 30 pre-existing theories, including Control Theory, and a meta-synthesisof 65 qualitative evaluation studies of A&F (not included in this thesis). We compared thenumber and type of planned (intention) and completed (behaviour) actions between studygroups, and explored perceived barriers and facilitators to effective action planning. ICUswith the toolbox planned more actions directly aimed at improving practice (p=0.037) and tar-geted a wider range of practice determinants compared to ICUs without the toolbox. ICUs withtoolbox also completed more actions during the study period, but not significantly (p=0.142).ICUs without toolbox tended to focus more on feedback verification and exploring solutions

191

Appendix

without developing intentions for actual change. Regardless of the toolbox, all ICUs still expe-rienced barriers relating to the feedback (low controllability and accuracy) and organisationalcontext (competing priorities, limited resources, and cost) that inhibited eventual completionof actions.

Part IV. Primary care medication safetyThe context of the chapters in this part is a pharmacist-led, electronic A&F intervention to im-prove primary care medication safety in Salford (Greater Manchester), United Kingdom. Feed-back was provided through an actionable, interactive dashboard that listed the patients whowere potentially exposed to one or multiple types of hazardous prescribing or inadequateblood-test monitoring based on 12 safety indicators. Trained clinical pharmacists deliveredthe intervention and worked in partnership with general practice staff to review individualpatients that triggered the indicators, and initiated remedial actions (e.g. discontinuing med-ication; ordering laboratory tests) or advised practice staff on doing so.

Chapter 10 explored how pharmacists and general practice staff used the dashboard toidentify potential medication safety hazards and their workflow to resolve identified hazards.We used a mixed-methods study design involving quantitative data from dashboard userinteraction logs from 43 general practices during the first year of receiving the intervention,and qualitative data from semi-structured interviews with 22 pharmacists and physiciansfrom 18 practices. Practices interacted with the dashboard a median of 12 (interquartile range,5 to 15) times per month during the first quarter of use. Once they observed a potentialhazard, pharmacists and practice staff worked together to resolve that in a sequence of steps:(1) verifying the dashboard information, (2) reviewing the patient’s clinical records, and (3)deciding potential changes to the patient’s medicines. Dashboard use transitioned over timetowards regular but less frequent (median of 5.5 [3.5 to 7.9] times per month) checks to identifyand resolve new cases. The frequency of dashboard use was higher in practices with a largernumber of at-risk patients. In 24 (56%) practices the dashboard was used only by pharmacists;in 12 (28%) use by other practice staff increased as pharmacist use declined after the initialintervention period; and in 7 (16%) there was mixed use by both pharmacists and practicestaff over time.

Chapter 11 reports the impact of the intervention using an interrupted time series analysison the rates of potentially hazardous prescribing and inadequate blood-test monitoring, com-paring observed rates post-intervention to extrapolations from a 24-month pre-interventiontrend. We found that the prevalence of potentially hazardous prescribing reduced by 27.9%(95% confidence interval, 20.3% to 36.8%) at six months and by 40.7% (95% CI, 29.1% to 54.2%)at twelve months after introduction of the intervention. The rate of inadequate blood-testmonitoring reduced by 22.0% (95% CI, 0.2% to 50.7%) at six months and by 23.5% (95% CI, -4.5% to 61.6%) at 12 months. At baseline, 95% of practices had rates of high-risk prescribingbetween 0.88% and 6.19%. After 12 months this ranged from 0.74% to 3.02%.

ConclusionThe figure on the next page summarises the conclusions drawn from this thesis. It showshow we used Control Theory as a generic framework to explain and explore the mecha-nisms through which audit and feedback (A&F) leads to better care. A&F helps because itguides health professionals to work on those quality aspects for which improvement is rec-ommended, particularly through communicating low performance scores and making explicit

192

Appendix

Summary

performance comparisons to benchmarks, trends, or targets. However, professionals may notaccept feedback, causing the feedback cycle to stop, if they: perceive the accuracy of the un-derlying feedback data to be low; disagree with benchmarks; or do not consider the clinicaltopic a relevant aspect of care quality. Health professionals who are confronted with per-formance feedback nevertheless often intend to reduce any indicated gaps between actualand recommended practice, but lack knowledge about improvement actions they could take,or they experience organisational barriers which inhibit translation of their intentions intoactual change in clinical practice. By identifying these mechanisms we have contributed tobetter understanding of A&F. The findings in this thesis furthermore suggest that, to optimisethe effectiveness of A&F interventions, providers and researchers should (1) make explicituse of theory and evidence to guide their design and evaluation, (2) tailor performance com-parators, (3) provide suggested actions for improvement, (4) target a limited set of indica-tors about both care processes and outcomes, (5) support recipients to achieve and sustainorganisation-level change, (6) focus on addressing barriers to practice change rather than onhealth professionals’ motivation to change, (7) facilitate rapid verification of the underlyingfeedback data, and (8) deploy laboratory experiments and quantitative process evaluationsin future A&F interventions to tailor feedback to recipients’ individual needs and preferencesrather than providing “one-size-fits-all” feedback.

Target

Audit and Feedback

Intention to improve practice

Health professional

Action / Behaviour

Action implementation

toolbox

Perceived clinical performance

Clinical performance

Comparator choice influences thefeedback message and response

by its recipients (Chapter 3)

Professionals ignore up to half of feedback-recommended

targets (Chapter 4, 7)

Care processes are more actionable than patientoutcomes (Chapter 4, 9)

Professionals prioritiseimproving data quality instead

of care (Chapter 4, 9, 10)

Low performers interact more frequently with feedback thanhigh performers (Chapter 10)

Toolbox increases effectiveness of feedback and diversity of

improvement intentions (Chapter 8, 9)

Actions at organisation-level, compared topatient-level, are more sustainable but more

difficult to complete (Chapter 5, 9, 10, 11)

Intentions are driven by low performance and benchmark comparisons (Chapter 4, 7)

Performance comparators (benchmarks, trends,

explicit targets)

Professionals oftenoverestimate their

performance (Chapter 7)

Professionals each have their own opinion about what

constitutes quality (Chapter 4)

Routinely collected clinical data

Measured clinical performance

Conclusions drawn from this thesis, mapped onto the feedback cycle (adapted from Control Theory), toadvance the science of audit and feedback.

193

Nederlandse samenvatting

Appendix

Nederlandse samenvatting

Audit en feedback (A&F) is een van de meest gebruikte strategieën voor kwaliteitsverbeteringin de gezondheidszorg. In de meest elementaire vorm biedt het individuele zorgprofessio-nals en zorgteams samenvattingen van hun klinische prestaties, gemeten aan de hand vaneen reeks kwaliteitsindicatoren, gedurende een bepaalde periode. Hoewel A&F-interventiesover het algemeen effectief zijn in het verbeteren van de kwaliteit van zorg, is er een aan-zienlijke variatie in de geobserveerde effecten, waarbij een kwart van de gerandomiseerdeA&F onderzoeken weinig tot geen effect vindt. Tot nu toe is er weinig vooruitgang geboektbij het vergroten van de effectiviteit van A&F, omdat interventies nauwelijks gebaseerd warenop bestaande theorie en wetenschappelijk bewijs, en omdat ons begrip over hoe A&F kanleiden tot kwaliteitsverbetering beperkt is.

In dit proefschrift hebben we ernaar gestreefd om deze gestagneerde wetenschap te be-vorderen door ons begrip over A&F uit te breiden, en het vermogen van A&F om de kwaliteitvan zorg te verbeteren te optimaliseren. We hebben de volgende onderzoeksvragen gefor-muleerd:

1. Wat is de theorie en het wetenschappelijk bewijs voor A&F en het gebruik van klinischeprestatievergelijkers bij A&F-interventies?

2. Wat is de impact van drie geavanceerde, op theorie en evidence-based elektronischeA&F-interventies in verschillende klinische domeinen op de kwaliteit van zorg?

3. Wat zijn de mechanismen waarmee A&F van invloed is op zorgprofessionals’ perceptiesover hun klinische prestaties, verbeterintenties, gedrag en, uiteindelijk, de kwaliteit vande zorg?

Dit proefschrift bestaat uit drie delen, die we hieronder samenvatten, gevolgd door een al-gemene conclusie.

Deel I. Theorie en wetenschappelijk bewijs voor audit en feedbackHoofdstuk 2 legt A&F uit aan de hand van Control Theory, en bespreekt de rol van deze theoriebinnen de medische informatiekunde in het algemeen, en binnen het ontwerp en de evalu-atie van A&F-interventies in het bijzonder. Control Theory stelt dat gedrag wordt gereguleerddoor een feedbackcyclus, waarbij een zelfregulerend systeem (zoals een machine of mens)de perceptie van zijn huidige toestand vergelijkt met een doeltoestand, en ernaar streeftom waargenomen discrepanties te verminderen door zijn gedrag te wijzigen. Het klassiekevoorbeeld van deze feedbackcyclus is de thermostaat. In dit systeem meet de thermostaatcontinu de huidige luchttemperatuur en vergelijkt deze met de gewenste temperatuurinstel-ling. Als de thermostaat een discrepantie tussen deze twee waarden detecteert, “gedraagt”deze zich door de verwarming in te schakelen, die warme lucht de kamer binnen brengt, enblijft dit doen totdat de kamer genoeg is opgewarmd zodat het verschil niet langer wordtwaargenomen. Als we dit vertalen naar de zorg, betekent het dat zorgprofessionals die dis-crepanties tussen de huidige klinische prestaties en hun doel waarnemen, intenties ontwik-kelen en vervolgens actie ondernemen (d.w.z. gedrag vertonen) om de praktijk te verbeteren,en dit blijven doen tot het doel is bereikt. Control Theory is gebruikt om wetenschappelijkbewijs voor A&F-interventies te synthetiseren, hun ontwerpen te verbeteren, uit te leggenwaarom interventies wel of niet succesvol waren, en om hypothesen te genereren over hoefeedbackmechanismen in de praktijk werken.

A&F gebruikt vaak klinische prestatievergelijkers om zorgprofessionals te helpen bij hetidentificeren van verschillen tussen de gewenste en daadwerkelijke praktijk. In Hoofdstuk 3

197

Appendix

beschrijven we de huidige keuzes voor prestatievergelijkers en de bijbehorende mechanis-men die mogelijk van invloed zijn op de effectiviteit van A&F. We haalden deze uit theorieënen empirische studies die eerder werden geïdentificeerd door andere reviews. We vonden in146 gerandomiseerde studies dat de prestaties van feedbackontvangers het vaakst vergelekenwerden met de prestaties van anderen (benchmarks, 60%). Andere vergelijkers omvatten deeigen historische prestaties van de ontvangers zelf (trends, 10%) en streefnormen (explicietedoelen, 11%). In 13% van de studies gebruikte men een combinatie van deze opties. In stu-dies met benchmarks werd 42% vergeleken met het groepsgemiddelde. We destilleerden 20gedragsveranderingsmechanismen van elke vergelijker uit 12 theorieën, 5 gerandomiseerdestudies en 42 kwalitatieve A&F onderzoeken. Klinische prestatievergelijkers in gepubliceerdeliteratuur hielden niet expliciet rekening met de theorie en de mechanismen die gerappor-teerd werden in kwalitatieve studies. Op basis van onze review stelden wij dat er aanzienlijkemogelijkheden zijn om het ontwerp van prestatievergelijkers te verbeteren door (1) op maatgemaakte vergelijkingen te bieden in plaats van iedereen te vergelijken met het gemiddelde,(2) het aantal vergelijkers dat wordt weergegeven te beperken en op verzoek meer vergelij-kende informatie te bieden om de geloofwaardigheid en bruikbaarheid van de feedback tebalanceren, (3) prestatietrends te bieden, maar niet alléén trends, en (4) feedbackontvangersaan te moedigen om persoonlijke, expliciete doelen te stellen aan de hand van relevanteinformatie.

Deel II. HartrevalidatieDe studies in dit deel omvatten allemaal een elektronische A&F-interventie om hartrevali-datie in Nederland te verbeteren. Feedback werd gegeven over een reeks van 18 kwaliteits-indicatoren (hartrevalidatieprocessen en -uitkomsten) met benchmarkvergelijkingen (laag,gemiddeld of hoog) via een online dashboard. Tijdens driemaandelijkse bezoeken waarbijeen onderzoeker de centra bezocht, gebruikten lokale multidisciplinaire hartrevalidatieteamshet dashboard om de feedback te bespreken, indicatoren voor verbetering te selecteren inhun gestructureerde actieplan, en specifieke verbeteringsdoelen vast te stellen.

Hoofdstuk 4 presenteert een labstudie met 41 individuele hartrevalidatieprofessionals eneen veldstudie met 18 hartrevalidatieteams om de factoren te onderzoeken die van invloedzijn op de intenties om de praktijk te verbeteren; beide vonden plaats naast een cluster-gerandomiseerde trial die de effectiviteit van de interventie evalueerde. In het labstudiepresenteerden we de 41 individuele deelnemers twee feedbackrapporten die willekeurig wer-den geselecteerd uit de 50 rapporten die tijdens de trial werden gegenereerd. We vroegenhen om vanuit de rapporten indicatoren voor verbetering te selecteren in hun actieplan. Bijselecties die niet in overeenstemming waren met de Control Theory (bijvoorbeeld het nietselecteren van een indicator waarvoor de prestatie beneden de benchmark was), vroegenwe de deelnemers om hun keuze toe te lichten. In de veldstudie hebben we geanalyseerdwelke indicatoren tijdens de trial door de 18 hartrevalidatieteams in de actieplannen zijn ge-selecteerd. Zowel het labstudie als de veldstudie toonden aan dat indicatoren met lagereprestatiescores eerder voor verbetering werden geselecteerd. Prestaties die als laag of ge-middeld waren gebenchmarkt verhoogden deze waarschijnlijkheid in het lab. Deelnemersnegeerden de benchmarks in 34% (labstudie) en 48% (veldstudie) van hun selecties, omdatze het niet eens waren met benchmarks, de verbetering onhaalbaar achtten, of de indicatorgeen essentieel aspect van zorgkwaliteit vonden.

Hoofdstuk 5 beschrijft de resultaten van een cluster-gerandomiseerde trial waarbij 18 hart-revalidatiecentra (14.847 patiënten) willekeurig werden toegewezen om feedback te ontvangen

198

Appendix

Nederlandse samenvatting

over hetzij psychosociale of fysieke revalidatie, gedurende minstens één jaar. De primaireuitkomstmaat was de prestatie op zorgprocessen en de uitkomsten gemeten op patiëntni-veau. Secundaire uitkomsten waren richtlijnadherentie met betrekking tot het voorschrijvenvan hartrevalidatietherapieën en de volledigheid van gegevensverzameling. Afgezien van eenbescheiden verbetering van de volledigheid van de gegevens, hebben we geen verbeterin-gen gevonden in de primaire of secundaire uitkomstmaten. Post-hoc analyse liet zien datteams 233 verbeteringsacties hadden gedefinieerd, maar slechts 49 (21%) daarvan tijdens destudieperiode hadden voltooid.

Deel III. Pijnmanagement op de intensive careDe hoofdstukken in dit deel betreffen een elektronische A&F-interventie om pijnmanagementop Nederlandse intensive cares te verbeteren. Het centrale onderdeel van de interventie waseen online dashboard dat intensive care (IC) teams twee functionaliteiten bood: (1) maande-lijkse prestatiefeedback op vier nieuw ontwikkelde kwaliteitsindicatoren met vergelijkingenmet trends, twee benchmarks (mediaan en top 10%) en expliciete doelen die IC teams zichzelfkonden stellen, en (2) het ontwikkelen en beheren van gestructureerde actieplannen. Daar-naast hadden we een “actie implementatie toolbox” ontwikkeld, bestaande uit een voorafgedefinieerde lijst van potentiële knelpunten in het zorgproces en voorgestelde verbeterac-ties, om IC’s te ondersteunen in hun verbeterprocessen.

Hoofdstuk 6 beschrijft het studieprotocol. Deze bestond uit een labstudie om te onder-zoeken hoe A&F verbeterintenties beïnvloedt, en een cluster-gerandomiseerde trial met eenmixed-methods procesevaluatie om te bepalen of en hoe de toolbox IC professionals helptbij het omzetten van hun verbeterintenties naar acties.

Hoofdstuk 7 rapporteert over de labstudie met 72 professionals uit 21 IC’s die ongeveer eenmaand voorafgaand aan hun deelname aan de trial plaatsvond. We verzamelden de percep-ties van professionals over hun eigen klinische prestaties, prestatie van anderen, doelen, enverbeteringsintenties vóór en na het voor de eerste keer ontvangen van feedback. We ontdek-ten dat IC professionals hun prestaties vaak overschatten, met gemiddeld 23%, en het zeldenonderschatten. Ze overschatten ook de prestaties van anderen en stelden doelen veel hogerdan de benchmarks voor topprestaties. In 68% van de gevallen waren hun verbeterintentieswas consistent met de feedbackaanbevelingen, nog vóórdat professionals feedback haddenontvangen. Die consistentie nam toe tot 80% na het ontvangen van feedback. In 56% vande gevallen wilden professionals echter nog steeds verbeteren op zorgaspecten waar ze altoppresteerders waren. Daarentegen ontbrak in 8% van de gevallen de verbeterintentie om-dat ze indicatoren niet belangrijk vonden; de gegevens niet vertrouwden; of de benchmarksonrealistisch achtten.

Hoofdstuk 8 presenteert de cluster-gerandomiseerde trial in 21 IC’s (25.234 patiënten) dieallemaal gedurende zes tot negen maanden feedback ontvingen met een leeg gestructureerdactieplan, maar willekeurig werden toegewezen tot een groep die de interventie mét of zónderde actie implementatie toolbox zou ontvangen. De primaire uitkomstmaat was een samen-stelling van vier pijnmanagementindicatoren die het percentage patiënt-dienst observatiesweerspiegelden waarin de pijn van de patiënt was gemeten en acceptabel was, of onaccepta-bel en genormaliseerd binnen één uur. Secundaire uitkomsten waren elk van de individueleindicatoren. We vonden een absolute verbetering op adequaat pijnmanagement van 14,8%(95% betrouwbaarheidsinterval; 14,0 tot 15,5) in IC’s met toolbox en 4,8% (4,2 tot 5,5) in IC’szonder toolbox. De verbetering was significant hoger in de groep met toolbox (p=0,049), maarbleef beperkt tot de pijnmanagementprocessen en niet de uitkomsten.

199

Appendix

Hoofdstuk 9 is de mixed-methods procesevaluatie waarin we alle individuele acties heb-ben geëxtraheerd uit de actieplannen die IC’s hadden ontwikkeld. We hebben ook semige-structureerde interviewsmet deelnemers uitgevoerd tijdens de trial. In onze analysemaaktenwe gebruik van Clinical Performance Feedback Intervention Theory. Dit is een nieuwe theo-rie die hebben we onlangs gepubliceerd, en voortbouwt op 30 reeds bestaande theorieën,waaronder de Control Theory, en een metasynthese van 65 kwalitatieve evaluatiestudies vanA&F (niet opgenomen in dit proefschrift). We vergeleken het aantal en het type geplande(intentie) en voltooide (gedrag) acties tussen studiegroepen, en verkenden welke barrièresen facilitators voor effectieve verbetering werden ervaren. IC’s met de toolbox planden mééracties die rechtstreeks gericht waren op het verbeteren van de praktijk (p=0,037) en richt-ten zich op een breder scala van praktijkdeterminanten in vergelijking met IC’s zonder detoolbox. IC’s met toolbox voltooiden ook meer acties tijdens de studieperiode, maar nietsignificant meer (p=0,142). IC’s zonder toolbox hadden de neiging om zich meer te richten ophet verifiëren van de onderliggende feedbackgegevens en het verkennen van verbetermo-gelijkheden, zonder intenties voor daadwerkelijke verandering te ontwikkelen. Ongeacht detoolbox ondervonden alle IC’s nog steeds barrières met betrekking tot de feedback (lage con-troleerbaarheid en nauwkeurigheid) en organisatorische context (concurrerende prioriteiten,beperkte middelen, en kosten) die de uiteindelijke voltooiing van acties belemmerden.

Deel IV. Medicatieveiligheid binnen de eerstelijnszorgDe context van de hoofdstukken in dit deel is een elektronische A&F-interventie ter verbete-ring van de medicatieveiligheid binnen de eerstelijnszorg in Salford (Greater Manchester), inhet Verenigd Koninkrijk. Feedback werd gegeven via een actiegericht, interactief dashboarddat inzicht bood in welke patiënten mogelijk waren blootgesteld aan potentieel gevaarlijkemedicatie of ontoereikende controle op bloedwaarden, op basis van 12 veiligheidsindicato-ren. Als deel van de interventie, werden getrainde klinische apothekers ingezet. Deze werktensamen met huisartsenpraktijken om individuele patiënten die waren blootgesteld te beoor-delen, en startten corrigerende maatregelen (bijvoorbeeld medicatie staken, of een labora-toriumtest aanvragen) of adviseerden huisartspersoneel om dit te doen.

In Hoofdstuk 10 werd onderzocht hoe apothekers en huisartspersoneel het dashboardgebruikten om mogelijke gevaren voor medicatieveiligheid te identificeren en wat hun werk-proces was om geïdentificeerde medicatierisico’s op te lossen. We gebruikten een mixed-methods onderzoeksopzet dat gebruik maakte van kwantitatieve gegevens uit de interactie-logs van het dashboard uit 43 huisartspraktijken gedurende het eerste jaar na ontvangst vande interventie, en kwalitatieve gegevens van semigestructureerde interviews met 22 apothe-kers en huisartsen uit 18 praktijken. Praktijken gebruikten het dashboard gemiddeld 12 keerper maand in het eerste kwartaal (interkwartielafstand, 5 tot 15). Na het identificeren van eenpotentieel medicatierisico, werkten apothekers en huisartspersoneel samen om deze op telossen in een reeks stappen: (1) het verifiëren van de dashboardinformatie, (2) het bekijkenvan de klinische gegevens van de patiënt, en (3) het beslissen over mogelijke wijzigingen inde geneesmiddelen van de patiënt. Het dashboard werd in de loop van de tijd regelmatigmaar minder frequent (5,5 [3,5 tot 7,9] keer per maand) gebruikt voor controles om nieuwegevallen te identificeren en op te lossen. De frequentie waarmee het dashboard werd ge-bruikt was hoger in praktijken met een groter aantal risicopatiënten. In 24 praktijken (56%)werd het dashboard alleen door apothekers gebruikt; in 12 (28%) nam het gebruik door anderhuisartspersoneel toe naarmate het gebruik van de apotheker na de beginperiode afnam; enin 7 (16%) werd het dashboard zowel door apothekers als door huisartspersoneel gebruikt.

200

Appendix

Nederlandse samenvatting

Hoofdstuk 11 rapporteert de impact van de interventie aan de hand van een tijdreeksana-lyse (interrupted time-series analysis) van de percentages risicovol voorschrijven van me-dicatie en ontoereikende controle op bloedwaarden. Hierbij zijn geobserveerde percenta-ges na introductie van de interventie vergeleken met extrapolaties van een tweejarige pre-interventietrend. We vonden dat de prevalentie van risicovol voorschrijven afnam met 27,9%(95% betrouwbaarheidsinterval; 20,3% tot 36,8%) na zes maanden en met 40,7% (29,1% tot54,2%) na twaalf maanden. Het percentage ontoereikende controle op bloedwaarden ver-minderde met 22,0% (0,2% tot 50,7%) na zes maanden en met 23,5% (-4,5% tot 61,6%) na 12maanden. Bij aanvang had 95% van de praktijken een hoog-risico voorschrijfpercentage tus-sen 0,88% en 6,19%. Na 12 maanden varieerde dit van 0,74% tot 3,02%.

ConclusieDe figuur op de volgende pagina vat de conclusies uit dit proefschrift samen. Het laat zienhoe we Control Theory als een generiek raamwerk hebben gebruikt om de mechanismen uitte leggen en te verkennen waardoor audit en feedback (A&F) tot betere zorg leidt. A&F helptomdat het zorgprofessionals begeleidt bij het werken aan de kwaliteitsaspecten waarvoorverbetering wordt aanbevolen, met name door het communiceren van lage prestatiescoresen het maken van expliciete prestatievergelijkingen met benchmarks, trends, of doelen. Pro-fessionals accepteren feedback echter niet, waardoor de feedbackcyclus stopt, als ze: denauwkeurigheid van de onderliggende feedbackgegevens als laag ervaren, het niet eens zijnmet benchmarks, of het klinische onderwerp niet als een relevant aspect van zorgkwaliteitbeschouwen. Zorgprofessionals die worden geconfronteerd met prestatiefeedback hebbendesalniettemin vaak de intentie om aangegeven verschillen tussen de daadwerkelijke enaanbevolen praktijk te verkleinen, maar missen kennis over welke verbeteracties ze kun-nen uitvoeren, of ervaren organisatorische barrières die de vertaling van hun intenties naardaadwerkelijke verandering in de klinische praktijk belemmeren. Door deze mechanismente identificeren hebben we bijgedragen aan een beter begrip van A&F. De bevindingen uit ditproefschrift kunnen gebruikt worden om de effectiviteit van A&F-interventies te optimalise-ren. Onze aanbevelingen zijn dat A&F verleners en onderzoekers: (1) expliciet gebruik makenvan theorie en wetenschappelijk bewijs om het ontwerp en de evaluatie te sturen, (2) pres-tatievergelijkers op maat kiezen, (3) voorgestelde verbeteracties aanbieden, (4) zich richtenop een beperkte reeks indicatoren over zowel zorgprocessen als uitkomsten, (5) ontvangersondersteunen om organisatorische veranderingen te bereiken en in stand te houden, (6) zichrichten op het aanpakken van barrières tot praktijkverandering in plaats van op zorgprofes-sionals’ motivatie om te veranderen, (7) snelle verificatie van de onderliggende feedbackge-gevens mogelijk maken, en (8) labstudies en kwantitatieve procesevaluaties toepassen bijtoekomstige A&F-interventies om feedback aan de individuele behoeften en voorkeuren vanontvangers aan te passen in plaats van “one-size-fits-all” feedback te bieden.

201

Appendix

Audit en Feedback

Intentie om de praktijk te verbeteren

Zorg-professional

Actie / Gedrag

Actieimplementatie

toolbox

Doel

Klinische prestatie

Keuze van vergelijker beïnvloedt de feedbackinhoud en reactie door ontvangers (Hoofdstuk 3)

Professionals negeren tot de helft van aanbevolen doelen

(Hoofdstuk 4, 7)

Professionals overschatten vaak hun prestatie (Hoofdstuk 7)

Zorgprocessen zijn meer actiegericht dan patiëntuitkomsten

(Hoofdstuk 4, 9)

Professionals prioriteren het verbeteren van de kwaliteit van

data i.p.v. zorg (Hoofdstuk 4, 9, 10)

Laag-presteerders interacteren méér met feedback dan hoog-presteerders (Hoofdstuk 10)

Toolbox verhoogt effectiviteit van feedback en diversiteit van

verbeterintenties (Hoofdstuk 8, 9)

Acties op organisatieniveau, t.o.v. patiëntniveau, zijn duurzamer maar

moeilijker te voltooien (Hoofdstuk 5, 9, 10, 11)

Professionals hebben elk huneigen mening over wat

kwaliteit omvat (Hoofdstuk 4)

Routinematig verzameldeklinische data

Intenties worden gedreven door lage prestatie en benchmarkvergelijkingen

(Hoofdstuk 4, 7)

Waargenomen klinischeprestatie

Gemeten klinischeprestatie

Prestatievergelijkers(benchmarks, trends,

expliciete doelen)

Conclusies uit dit proefschrift, weergegeven op de feedbackcyclus (aangepast van Control Theory), terbevordering van de wetenschap rondom audit en feedback.

202

Dankwoord

Acknowledgements

203

Appendix

Dankwoord

Ik wil graag iedereen bedanken die heeft bijgedragen aan de totstandkoming van dit proef-schrift. Ik durf met zekerheid te zeggen dat ik mijn promotie niet op deze manier zou hebbenvolbracht zonder al jullie ondersteuning, gelach, mede- en tegenwerking, en andere hulp inde afgelopen vier jaar. Een aantal mensen wil ik in het bijzonder noemen.

Allereerst mijn promotieteam: Nicolet, Niels, en Sabine. Jullie fantastische begeleidingheeft me geholpen om een prachtig proefschrift te maken waar ik trots op ben. Ik had mewerkelijk geen beter team kunnen wensen. Mijn eeuwige dank diep vanuit mijn hart.

Nicolet, je hebt voor mij een warm nest gemaakt en gaf mij de ruimte om mijn eigen gangte gaan, nieuwe samenwerkingen te starten, en creatief te zijn. Hierbij gaf je mij altijd goedrichting zodat ik niet verdwaald raakte in deze vrijheid wanneer ik te veel hooi op de vorknam. Tegelijkertijd motiveerde je me ook om tot het uiterste te gaan door de hooivork altijdgoed gevuld te houden. Dat heeft mijn promotie zowel uitdagend als leerzaam gemaakt.Jouw talent om overzicht te creëren, prioriteren, en de terugkomende vraag “zie je het nogzitten?” gaven mij telkens de energie die ik nodig had. Ik ben je ook dankbaar voor jouwtoegankelijkheid, vrijgevigheid, en zorgzaamheid.

Niels, ik weet al sinds mijn bachelorstage dat wij een perfecte match zijn. Tijdens onzeoverleggen was je direct en duidelijk, had je altijd voorbeelden en methoden paraat, en washet bovenal gezellig. Dat werkte meer dan goed voor mij. Jouw “we kunnen het ook helemaalanders doen” was soms frustrerend maar kwam altijd de wetenschap ten goede. Jouw enSabine’s vertrek naar Manchester bleek een blessing in disguise, want het gaf mij de kans ommet jullie en andere fantastische mensen in het VK samen te werken. Dat was heel bijzonderen leerzaam, en ik koester de herinneringen. Uiteraard ook bedankt voor mijn twee exen.

Sabine, mijn eeuwige steun en toeverlaat, in het bijzonder als de professoren weer eens tedruk waren. Je kon mij altijd goed wijzen op mijn veel te nauwkeurige beschrijvingen en langezinnen, die door overmatig kommagebruik (en nota bene veel tussen de haakjes), ondanksdat ze feitelijk correct en volledig waren, enigszins complex werden. Daarnaast hielp jouwonuitputtelijke creativiteit, rappe een-tweetjes, en jouw “even een klein stapje terug” telkensweer om ons onderzoek helder, BS-vrij, aantrekkelijk én onweerlegbaar op te schrijven.

Commissieleden prof. dr. M.C. de Bruijne, prof. dr. N. van Dijk, prof. dr. R. Foy, prof. dr. K.J.Jager, prof. dr. N.S. Klazinga, en prof. dr. D.W. de Lange. Hartelijk dank voor het kritisch lezenen beoordelen van mijn proefschrift. Special thanks to prof. dr. Foy for making his way acrossthe channel, back into the EU, to participate in my doctorate committee.

Al mijn collega’s van de afdeling Klinische Informatiekunde. De docenten die ik al kenvanaf mijn studie, jullie hebben mij fantastisch klaargestoomd voor mijn promotietraject.Bedankt voor alle kansen die jullie mij hebben gegeven. Natuurlijk gaat mijn grote dank uitnaar de Koele KIKkers met wie ik al het lief en leed rondom promoveren heb kunnen delen. Inrecentere tijden, zijn dit: Birgit, Charlotte, Esmée, Femke, Gaby, Hatem, Hugo, Joanna, Leonie,Manon, Martijn, Nick, Philip, Rianne², Tinka, en Sophie. Maar zeker ook degenen die er inhet begin bij waren: Airin, Ellen, Eva, Jos, Lilian, Marjan, Mariëtte, Maurice, Sabine, Sandra, enTom. Ik heb ontzettend genoten van alle koffierondjes, borrels, congressen, lunch runs, officeolympics, curling, en andere activiteiten. Dank aan Richard voor het accepteren vanmijn OCD-trekjes in het ontwikkelproces van een prachtig kwaliteitsdashboard voor ons onderzoek. Ilse,jij was altijd de ‘sociale brug’ naar de andere kant van de gang, en bracht daarmee de afdelingdichter bij elkaar. Daarnaast heb ik veel bewondering voor jou, omdat jij als het (l)even tegen-zit er sterk uitkomt. Marie-José, we hebben lang en fijn met elkaar samengewerkt. Gelukkig,want ons project bleek (meer dan) a two man job! Je hebt mij geleerd samenwerken, endaarvoor ben ik je heel dankbaar. Ronald, hands down de beste docent tijdens mijn studie,

205

Appendix

gangmaker van de afdeling, kennisbank, en het in algemeen gewoon een toffe peer. Ik dank jevoor alles wat we samen hebben gedaan. Hardlopen, racefietsen, muziek maken, promotie-en oratieliedjes schrijven, concert en cabaret bezoeken… eigenlijk alles behalve onderzoek.

Erik, ondanks dat onze promoties inhoudelijk niets met elkaar te maken hadden, heb jijontzettend veel bijgedragen. Vanaf het eerste moment hebben wij tegenover elkaar gezetenen dat kwam mijn promotie echt ten goede. In de eerste plaats door de gezelligheid, maarniet in mindere mate door het (gelukkig wederzijds) wegwijs maken in R, InDesign, Photoshop,LATEX, UvA-regels, AMC-regels, AMR-regels, het omzeilen van ingewikkelde procedures, en debalans vinden tussen doorpakken en afremmen. Ik heb altijd veel respect gehad voor jouwruime kennis van feiten, citaten, sketches, moppen, en logo’s van vrouwenmerken. Bedanktvoor alle paaseieren, (peper)noten, kerstmuziek met YouTube-haardvuur en choco de luxe,soundboardgeweld, kattenposters, en dat je mijn paranimf bent op 20 september.

All the wonderful people I worked with at the Health e-Research Centre in Manchester.Darren, Evan, Mark, Markel, and Richard², it was truly wonderful to be part of your researchteam. Ben, I enjoyed our time working together, reviewing thousands of papers, agreeing todisagree, going back and forth on ideas, but eventually writing up beautiful science. I alsowant to thank Alex, Alexia, Glen, Hannah, Iain, Leonnie, Miguel, Paolo, Ruth, Skevi, Stephen,Vicky, and Will for making me feel welcome each and every time.

The Audit & Feedback Meta-Laboratory. Special thanks to Jeremy, Jill, Noah, Robbie, Ste-fanie, and Sylvia for inviting me to join the steering group of this great international researchcommunity. It has been more than valuable connecting all these experts to create sharedlearning and advance the science in our field. The Meta-Lab has givenme some great researchideas, new connections, and a lot of fun. It has also undoubtedly improved this thesis.

De geweldige bestuursleden van Jong AMC. Amy, Annieke, Donja, Erne, Frederique, Joëlle,Judith, Kelly, Mara, Marieke, Nadine, en Sven: ik kijk terug op een ontzettend leuke tijd. Wehebben heel veel leuke events voor onze jonge AMC collega’s georganiseerd, en dat maakteen grote organisatie als de onze een heerlijke plek om te werken.

Malou, Eva, en Marc, terwijl we op totaal verschillende plekken werkten, bracht hardlopenons bij elkaar. Het was heerlijk om na werk of tijdens de lunch met jullie de prachtige natuurin de omgeving van onze betonnen bunker te verkennen. Ik houd jullie in de gaten op Strava!

Mijn vrienden, de MIKkers, de Geneeskunde kletsgroep, en natuurlijk de ePisa groep. Rinke,Thijs, en Allard, ik weet niet meer zeker of we uiteindelijk consensus hebben bereikt overde definitie van e-health, maar ik heb altijd erg genoten van onze vergaderingen hierover.Hopelijk blijven we dit doen. Met goed eten en slechte films, uiteraard.

Sonia, grazie per avermi migliorato. Sarai sempre nel mio cuore. Aurora, sei veramente lacuoca più brava del mondo. Don Vito, ti ringrazio per farmi vedere tutte le belle cose Italiane.

Tot slot, mijn lieve familie. Pa en ma, jullie hebben me gemaakt tot wie ik ben. Jullieonvoorwaardelijke steun en liefde hebben me erg geholpen in de afgelopen jaren. Mijn pro-motieonderzoek nog wel eens werd verward met ‘studeren’ en ‘stagelopen’, of samengevat als‘iets met computers en dokters’. Ik hoop dat het lezen van (de samenvatting van) dit proef-schrift een en ander duidelijker maakt. Edwin, Jeroen, en Nico, ik kan me geen fijnere broerswensen. Ik kijk uit naar nog veel broederavonden vol spanning en sensatie. Ik ben trots endankbaar, Nico, dat jij als paranimf naast mij staat in de Agnietenkapel op 20 september.Lotte en Jette, ik ben ontzettend blij dat ik jullie mijn schoonzussen mag noemen. En natu-urlijk mijn allerliefste nichtjes Cato en Julie, en neefje Mees. Waarschijnlijk is de inhoud vandit proefschrift tegen de tijd dat jullie het (kunnen) lezen al achterhaald door nieuw, hieropvoortbouwend onderzoek. Maar dat is precies waar je als wetenschapper op hoopt.

206

Curriculum vitae

207

Appendix

Curriculum vitae

Wouter Thomas Gude was born on April 25th 1990 in Naarden, where he grew up being theyoungest of twins and two other brothers. Wouter attended secondary school (gymnasiumlevel) at the St. Vituscollege in Bussum from which he graduated in 2009. In the same year hestarted his study Medical Informatics at the Academic Medical Center - University of Amster-dam. He graduated from the bachelor with honours in 2012 and the master with distinction(cum laude) in 2014. His master thesis entitled “How do health care professionals select tar-gets for improving their care when confronted with performance feedback? A laboratory andclinical study in cardiac rehabilitation” was nominated for the university’s Best Thesis Award.

After a month’s holiday Down Under, his mother’s birth country, he continued his researchin a PhD position in the department of Medical Informatics at the Academic Medical Center(now Amsterdam UMC) under the supervision of prof. dr. Nicolette de Keizer, prof. dr. NielsPeek, and dr. Sabine van der Veer. He conducted his research working in various teams at theCArdiac Rehabilitation Decision Support System (CARDSS) research group and National Inten-sive Care Evaluation (NICE) foundation at Amsterdam UMC, and the Health e-Research Centreand NIHR Greater Manchester Patient Safety Translational Research Centre at The Universityof Manchester. Next to his PhD Wouter was active as a board member of Jong AMC – a networkconnecting around 1300 young AMC employees through social and educational activities, andon the steering group of the Audit & Feedback Meta-Laboratory – an international collabora-tion of audit and feedback researchers and providers to create shared learning across auditand feedback (A&F) laboratories to increase the effectiveness of A&F.

Since May 2019, Wouter has started improving the quality of care from the “other side”,working as a healthcare consultant at PwC Nederland. Wouter lives in Amsterdam, and enjoysrunning, cycling, playing music, photography, and food.

209

Portfolio

211

Appendix

Portfolio

PhD candidate: Wouter T. GudePeriod: September 2014 to March 2019Supervisors: Prof. dr. Nicolette F. de Keizer

Prof. dr. Niels PeekCo-supervisor: Dr. Sabine N. van der Veer

PhD training Year ECTS

General coursesAMC World of Science 2014 0.7Project Management 2016 0.6

Specific coursesPractical Biostatistics 2015 1.1Observational Epidemiology 2015 0.6Advanced Topics in Biostatistics 2016 2.1

Seminars, workshops and master classesThe Science of Learning Health Systems tutorial (Manchester, UK) 2016 0.2Introduction to Machine Learning in Health and Process Analytics for Care Path-ways tutorial (Manchester, UK)

2016 0.2

Big Data Analytics master class (Taipei, Taiwan) 2014 1.6Amsterdam Public Health (APH) research institute annual meeting (Amsterdam) 2016 - 2018 0.6APH research institute Quality of Care symposium (Amsterdam) 2016 0.2APH research institute Methodology symposium (Amsterdam) 2017 0.2Promovendidag (Breukelen; Amsterdam) 2015 - 2019 1.25NICE discussion meeting (Nieuwegein) 2015 - 2018 0.8

PresentationsOral ”Using Control Theory to investigate clinical performance perceptions andthe influence of feedback on improvement intentions”, 3rd International Audit &Feedback meeting (Toronto, Canada)

2018 0.5

Panel ”Optimizing electronic audit and feedback through unobtrusive quantita-tive process evaluations”, 3rd International Audit & Feedback meeting (Toronto,Canada)

2018 0.5

Seminar ”Increasing the effectiveness and understanding of health informaticsinterventions through theory-informed, unobtrusive quantitative process evalua-tions”, Australian Institute of Health Innovation seminar series (Sydney, Australia)

2017 1

Workshop ”Electronic audit and feedback interventions to enhance patientsafety”, NIHR Patient Safety Translational Research Centre conference on SaferPrimary Care: A shared responsibility for system-wide learning (Manchester, UK)

2017 0.5

213

Appendix

Oral ”Associations between medication safety and use of an electronic medi-cation safety dashboard in primary care”, Medical Informatics Europe (MIE) 2017conference (Manchester, UK)

2017 0.5

Oral ”Optimizing electronic audit and feedback through unobtrusive quantitativeprocess evaluations”, 2nd International Audit & Feedback meeting (Leeds, UK)

2017 0.5

Oral ”(Hoe) werkt feedback via het nieuwe kwaliteitsdashboard?”, NICE discussionmeeting (Nieuwegein)

2017 0.5

Science Slam ”The quality of quantitative process evaluations”, Medical Informat-ics Europe (MIE) 2016 (Munich, Germany)

2016 0.5

Oral ”Optimizing digital health informatics interventions through unobtrusivequantitative process evaluations”, Medical Informatics Europe (MIE) 2016 (Mu-nich, Germany)

2016 0.5

Workshop ”Dashboard actiegerichte indicatoren en tool voor systematische pro-cesverbetering”, NICE discussion meeting (Nieuwegein)

2015 0.5

Oral ”Inside the Black Box of Audit and Feedback: a Laboratory Study to Ex-plore Determinants of Improvement Target Selection by Healthcare Professionalsin Cardiac Rehabilitation”, MEDINFO 2015 - 15th World Congress on Health andBiomedical Informatics (São Paulo, Brazil)

2015 0.5

Oral ”Zo weinig tijd en zoveel te doen! Hoe helpt feedback op indicatorenhartrevalidatieteams om doelen voor kwaliteitsverbetering te kiezen?”, NVHVVCarVasZ-congres (Utrecht)

2014 0.5

Conferences3rd International Audit & Feedback Meta-Lab meeting (Toronto, Canada) 2018 0.75Medical Informatics Europe (MIE) 2017 (Manchester, UK) 2017 0.752nd International Audit & Feedback Meta-Lab meeting (Leeds, UK) 2017 0.5Medical Informatics Europe (MIE) 2016 (Munich, Germany) 2016 1MEDINFO 2015 - 15th World Congress on Health and Biomedical Informatics (SãoPaulo, Brazil)

2015 0.75

NVHVV CarVasZ-congres (Utrecht) 2014 0.25

OtherJong AMC board 2016 - 2019 3Audit & Feedback Meta-Laboratory steering group 2017, 2018 1Peer review for BMJ Quality & Safety and Implementation Science 2017 0.5Organisation Promovendidag ”So You Think You Can PhD?” 2015 0.2Organisation Science Slam at Medical Informatics Europe (MIE) 2017 2017 0.2Biweekly research meeting 2014 - 2019 2

214

Appendix

Portfolio

Teaching Year ECTS

LecturingDatabases en programmeren, bachelor Medische informatiekunde 2014 - 2017 3Databases, pre-master Medical Informatics 2015 - 2017 1.5Eenmalige registratie, meervoudig gebruik, master Medicine 2016 0.2HEAL+ project, development of master program in Health Informatics in the Mid-dle East and North Africa region

2017 0.2

Tutoring, mentoringEvaluation of Health IT, master Medical Informatics 2016 - 2018 0.2

Supervising5-month bachelor Medische Informatiekunde internship, Shabnam: Influence ofcontextual factors on A&F effectiveness in Dutch ICUs

2018 2

8-month master Medical Informatics internship, Macy: Process evaluation of anA&F intervention in Dutch ICUs

2016, 2017 2

1-month master Medical Informatics internship, Dyantha: Social and organisa-tional barriers and facilitators to implementing A&F

2016 0.5

1-month master Medical Informatics internship, Joost: Goal setting and actionplanning strategies in A&F

2015 0.5

OtherDeveloping e-learning module Quality Registries and Indicators for the executivemaster Health Informatics

2016 2

UvA matching, voorlichting, proefstudeerdag 2014 - 2018 2.5

Awards and prizes Year

First prize Science Slam presentation, MIE 2016 (Munich, Germany) 2016Nomination Best Paper award, MIE 2016 (Munich, Germany) 2016

215

List of publications

217

Appendix

List of publications

Publications in this thesis

1 Impact of a pharmacist-led, actionable audit and feedback intervention to improve medi-cation safety in primary care: interrupted time series analysis.Peek N, GudeWT, Keers RN, Williams R, Kontopantelis E, Jeffries M, Phipps DL, Brown B, AveryAJ, Ashcroft DM.Submitted.

2019

2 How to facilitate action planning within audit and feedback interventions? A mixed-methods process evaluation of an action implementation toolbox in intensive care.Gude WT, Roos-Blom MJ, van der Veer SN, Dongelmans DA, de Jonge E, Peek N, de Keizer NF.Submitted.

2019

3 Effect of audit and feedback with action implementation toolbox on pain management inintensive care: a cluster randomised trial.GudeWT*, Roos-Blom MJ*, de Jonge E, Spijkstra JJ, van der Veer SN, Peek N, Dongelmans DA,de Keizer NF.Submitted.

2019

4 Understanding the utilisation of a novel interactive electronic medication safety dashboardby pharmacists and clinicians in general practice: a mixed methods study.Gude WT*, Jeffries M*, Keers RN, Phipps DL, Williams R, Kontopantelis E, Brown B, Avery AJ,Peek N, Ashcroft DM.Submitted.

2019

5 Clinical performance comparators in audit and feedback: a review of theory and evidence.Gude WT, Brown B, van der Veer SN, Colquhoun HL, Ivers NM, Brehaut JC, Landis-Lewis Z,Armitage CJ, de Keizer NF, Peek N.Implement Sci. 2019 Apr 24;14(1):39. doi: 10.1186/s13012-019-0887-1.

2019

6 Control Theory to design and evaluate audit and feedback interventions.Gude WT, Peek N.In: Scott P, de Keizer NF, Georgiou A, eds. Applied interdisciplinary theory in health infor-matics: a knowledge base for practitioners. Amsterdam: IOS Press; 2019.

2019

7 Health professionals’ perceptions about their clinical performance and the influence ofaudit and feedback on their intentions to improve practice: a theory-based study in Dutchintensive care units.Gude WT, Roos-Blom MJ, van der Veer SN, Dongelmans DA, de Jonge E, Francis JJ, Peek N, deKeizer NF.Implement Sci. 2018 Feb 17;13(1):33. doi: 10.1186/s13012-018-0727-8.

2018

8 Electronic audit and feedback intervention with action implementation toolbox to improvepain management in intensive care: protocol for a laboratory experiment and cluster ran-domised trial.Gude WT*, Roos-Blom MJ*, van der Veer SN, de Jonge E, Peek N, Dongelmans DA, de KeizerNF.Implement Sci. 2017 May 25;12(1):68. doi: 10.1186/s13012-017-0594-8.

2017

9 Effect of a web-based audit and feedback intervention with outreach visits on the clinicalperformance ofmultidisciplinary teams: a cluster-randomized trial in cardiac rehabilitation.Gude WT*, van Engen-Verheul MM*, van der Veer SN, Kemps HM, Jaspers MW, de Keizer NF,Peek N.Implement Sci. 2016 Dec 9;11(1):160. doi: 10.1186/s13012-016-0516-1.

2016

219

Appendix

10 How does audit and feedback influence intentions of health professionals to improve prac-tice? A laboratory experiment and field study in cardiac rehabilitation.Gude WT, van Engen-Verheul MM, van der Veer SN, de Keizer NF, Peek N.BMJ Qual Saf. 2017 Apr;26(4):279-287. doi: 10.1136/bmjqs-2015-004795.

2015

* Equal contributors.

Other publications

11 Clinical Performance Feedback Intervention Theory (CP-FIT): A New Theory for Designing,Implementing, and Evaluating Feedback in Health Care Based on a Systematic Review andMeta-synthesis of Qualitative Research.Brown B, Gude WT, Blakeman T, van der Veer SN, Ivers NM, Francis JJ, Lorencatto F, PresseauJ, Peek N, Daker-White G.Implement Sci. 2019 Apr 26;14(1):40. doi: 10.1186/s13012-019-0883-5.

2019

12 Reinvigorating stagnant science: implementation laboratories and a meta-laboratory toefficiently advance the science of audit and feedback.Grimshaw JM, Ivers NM, Linklater S, Foy RC, Francis JJ, Gude WT, Hysong SJ, on behalf of theAudit & Feedback MetaLab.BMJ Qual Saf. 2019 Mar 9. doi: 10.1136/bmjqs-2018-008355.

2019

13 Development of actionable quality indicators and an action implementation toolbox forappropriate antibiotic use at intensive care units: A modified-RAND Delphi study.Kallen MC, Roos-Blom MJ, Dongelmans DA, Schouten JA, Gude WT, de Jonge E, Prins JM, deKeizer NF.PLOS ONE. 2018 Nov 29;13(11):e0207991. doi: 10.1371/journal.pone.0207991.

2018

14 Measuring quality indicators to improve pain management in critically ill patients.Roos-Blom MJ, Gude WT, Spijkstra JJ, de Jonge E, Dongelmans D, de Keizer NF.J Crit Care. 2019 Feb;49:136-142. doi: 10.1016/j.jcrc.2018.10.027.

2018

15 SMASH! The Salford medication safety dashboard.Williams R, Keers R,GudeWT, Jeffries M, Davies C, Brown B, Kontopantelis E, Avery AJ, AshcroftDM, Peek N.J Innov Health Inform. 2018 Oct 18;25(3):183-193. doi: 10.14236/jhi.v25i3.1015.

2018

16 Informatics for Health 2017: Advancing both science and practice.Scott PJ, Cornet R, McCowan C, Peek N, Fraccaro P, Geifman N, GudeWT, Hulme W, Martin GP,Williams R.J Innov Health Inform. 2017 Apr 21;24(1):1-185. doi: 10.14236/jhi.v24i1.939.

2017

17 Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis.Scott PJ, Rigby M, Ammenwerth E, McNair JB, Georgiou A, Hyppönen H, de Keizer N, MagrabiF, Nykänen P, Gude WT, Hackl W.Yearb Med Inform. 2017 Aug;26(1):59-67. doi: 10.15265/IY-2017-010.

2017

18 Development of a Web-Based Quality Dashboard Including a Toolbox to Improve Pain Man-agement in Dutch Intensive Care.Roos-Blom MJ, GudeWT, de Jonge E, Spijkstra JJ, van der Veer SN, Dongelmans DA, de KeizerNF.Stud Health Technol Inform. 2017;235:584-588. doi: 10.3233/978-1-61499-753-5-584.

2017

220

Appendix

List of publications

19 Optimizing Digital Health Informatics Interventions Through Unobtrusive Quantitative Pro-cess Evaluations.Gude WT, van der Veer SN, de Keizer NF, Coiera E, Peek N.Stud Health Technol Inform. 2016;228:594-8. doi: 10.3233/978-1-61499-678-1-594.

2016

20 Inside the Black Box of Audit and Feedback: a Laboratory Study to Explore Determinants ofImprovement Target Selection by Healthcare Professionals in Cardiac Rehabilitation.Gude WT, van der Veer SN, van Engen-Verheul MM, de Keizer NF, Peek N.Stud Health Technol Inform. 2015;216:424-8. doi: 10.3233/978-1-61499-564-7-424.

2015

21 Improving guideline concordance in multidisciplinary teams: preliminary results of acluster-randomized trial evaluating the effect of a web-based audit and feedback inter-vention with outreach visits.van Engen-Verheul MM, Gude WT, van der Veer SN, Kemps HM, Jaspers MM, de Keizer NF,Peek N.AMIA Annu Symp Proc. 2015 Nov 5;2015:2101-10. PMID: 26958310.

2015

22 Diabetes patients’ experiences with the implementation of insulin therapy and their per-ceptions of computer-assisted self-management systems for insulin therapy.Simon AC, Gude WT, Holleman F, Hoekstra JB, Peek N.J Med Internet Res. 2014 Oct 23;16(10):e235. doi: 10.2196/jmir.3198.

2014

23 Safety and usability evaluation of a web-based insulin self-titration system for patientswith type 2 diabetes mellitus.Simon AC, Holleman F, Gude WT, Hoekstra JB, Peute LW, Jaspers MW, Peek N.Artif Intell Med. 2013 Sep;59(1):23-31. doi: 10.1016/j.artmed.2013.04.009.

2013

24 Safety of a web-based insulin titration system for patients with type 2 diabetes mellitus -pilot study.Simon AC, Holleman F, Gude WT, Hoekstra JB, Peek N.Stud Health Technol Inform. 2012;180:731-5. doi: 10.3233/978-1-61499-101-4-731.

2012

25 Formative usability evaluation of a web-based insulin self-titration system: preliminaryresults.Gude WT, Simon AC, Peute LW, Holleman F, Hoekstra JB, Peek N, Jaspers MW.Stud Health Technol Inform. 2012;180:1209-11. doi: 10.3233/978-1-61499-101-4-1209.

2012

221

Uitnodiging

voor het bijwonen van de openbare verdediging van het proefschrift

Understanding and optimising electronic audit and feedback

to improve quality of care

door

Wouter Thomas Gude

ter verkrijging van de graad van doctor aan de

Universiteit van Amsterdam

op 20 september 2019 om 14.00 uur

in de AgnietenkapelOudezijds Voorburgwal 231

te Amsterdam

aansluitend op de verdediging bent u van harte welkom op de receptie ter plaatse

Paranimfen:

Nico [email protected]

Erik Joukes [email protected]

Understanding and optimising electronic audit and feedback to im

prove quality of care

Depicted is the thermostat of room J1b-127 at Amsterdam UMC, location AMC.

The thermostat is the classical example of a simple feedback system. It continuously measures room temperature and compares it to its desired temperature setting. If it detects a discrepancy between the two values, the thermostat will respond by bringing warm (or cold) air into the room, until the discrepancy is no longer observed.

Unfortunately, similar to providing doctors with clinical performance feedback to improve quality of care, the thermostat in this room often did not work optimally. And we didn’t really understand why. It was through carefully crafted experimentation, based on the theoretical mechanisms of the thermostat, that we managed to improve the quality of our working environment. In this thesis, we did the same to optimise the ability of clinical performance feedback to improve quality of care. Wouter T. Gude

Understanding and optimising electronic audit and feedback to improve quality of care

Wouter T. Gude