PublicreportingandNICORupdate

MarkdeBelderTheJamesCookUniversityHospital

Middlesbrough

NOCONFLICTOFINTERESTTODECLARE

PublicreportinginInterventionalCardiology.Thechallengesahead

DehmerGJ,JACCCardiovascInt2013;6(6):631-3

Decisionsaboutyourhealtharesomeofthemostimportantdecisionsyouwillmake,somanynowadvocatethatinformationaboutphysiciansandhealthcarefacilitiesbeavailableforalltosee.

…Publicreportingisverycomplexandmakesmanyinhealthcarenervous.Theirsecretwishisforittodisappearlikeabaddream,butpublicreportingisheretostay.Thechallengesaheadarehowbesttodeveloppublicreportingforagoodpurposeandyetavoidunintendedconsequencessuchastherisk-adversebehavior.

Rationaleforpublicreporting

• Buildpublictrust– Publicinformation(“righttoknow”)– Publicreassurance

• Encourageadoptionofbestpractice– Allowsyoutocompareyourresultswithyourpeers

– Focusesattentiononprocessandqualityofcare(Hawthorneeffect):Improvedqualityofcare

Driverstowardstransparency

TheKennedyReport2001 TheFrancisReport2013

Driverstowardstransparency

August2013

TheBerwickReport2013

“Transparencyshouldbecomplete,timelyandunequivocal.Alldataonqualityandsafety,whetherassembledbygovernment,organisations,orprofessionalsocieties,shouldbesharedinatimelyfashionwithallpartieswhowantit,including,inaccessibleform,withthepublic.”

RationaleforAudit

• AmI(arewe)doingagoodjob?• Ifmy(our)resultsaren’tasgoodastheyshouldbe,howcanI(we)improve?

• Ifmy(our)resultsseemOK,canI(we)getevenbetter?

Whatwouldmyreactionreally beifmyresultsweren’tasgoodasIhadexpectedthemtobe?

WhatwouldIwantmycolleagues’reactiontobe?

SirBruceKeoghApril2013:

Organisationscannotknowtheyareprovidingsafeoreffectivecareunlesstheyaremeasuringandmonitoringtheirservices

AppliedtoPCI

• WhatisthespectrumofdiseasethatItreat(mycasemix)?

• Aremydecisionsalwaysappropriate?– DoItreatsomepatientswhenit’snotjustified?– AmInottreatingsomepatientswhenIshouldbe?– Applicationof,butalsorecognisingthelimitationsof,“the

evidencebase”.

• Aremyproceduralcomplicationswithinacceptablelimits?• Aremypatients’outcomeswithinacceptablelimits?

Considerationsforriskadjustment

• Basicdemographics(age,sex,etc)• Riskfactors• PreviousCVhistory• Co-morbidities• Clinicalpresentation• Haemodynamicstatus• Angiographiccriteria• Methodused• Adjunctivepharmacology• After-careandsecondaryprevention

Butdatabasescan’tcoverevery“whatif?”scenario

Needfordata

collection

BCISAuditReports1992-2013

TrustsandConsultants’OutcomesReports2013onwards

DatamanagementandvalidationSomeissuestoconsider

• Caseascertainment• Datadefinitionsordatacollectionrules(dissemination)• Dataprocessingandcleansing• Self-collectedversusindependentlycollecteddata• Datacompleteness• Methodsofimputationorresettingtodefaultreadings• Datavalidity(useof“independent”auditteams)• Rangechecksandotherchecksforinconsistencies• Methodstodealwithmultipleprocedures(especiallyinsameadmission)• Requeststoparticipants/institutionstochecktheir“riskfactor”data,

especiallyiftheyappearasanoutlier(whichmethod?)• Requestsforchecksshouldbeindependentofoutcomesreporting• Selectionofriskmodel• Methodofpresentationofdata

Processese.g.Door-to-Balloontimes

Majoradversecardiovascularandcerebrovascularevents(MACCE)10/01/14to10/01/15N=1315

UsespublishedandvalidatedNWQIPriskmodel

IndividualConsultantOutcomes

PatientArea

IndividualConsultantOutcomes

Viewsourceinformation

IndividualConsultantOutcomesBCISwebsiteNHSChoices

17

Potentialforunintendedconsequences

• Riskaversebehaviour– Mistrustofriskmodels

• Withdrawalfromqualitycontrolprocess• Gaming• Misinterpretationormisrepresentationofdata

• Misattributionoftheoutcometotheprocedure

RiskaversionandpublicreportingResnicFS,WeltFGP,JACC2009;53:825-30

47%reduction

Improvedriskmodels

• Differentriskmodelsfordifferentcohorts?– e.g.MassachusettsMass-DACSOS(STEMIorShock)vsnon-SOS

• Exclusionofgroupswhereriskadjustmentisdifficult?

• Hardindependentlycollectedend-points?

Riskmodels

MACCEmodelProMeasuresmostseriousin-hospitalCVoutcomesConReliesonself-reportingofeventsNotpossibletovalidate>90,000casesayear

MortalitymodelProMeasureshardend-pointthatcanbeexternallymeasured(ONS)ConMortalityisbutoneend-pointMortalitymaynotreflectwhathappensduringtheprocedureStilldependentonaccurateriskprofiles

GraysonADetal,Heart2006;92:658-63 McAllisterKSLetal,2015(submitted)

Neitherprovidelonger-termoutcomesorrecordpatientexperiences

ImprovingriskmodelsEllisSetal,JACCInt2011;4:442-8

AdditionofAldrete andsimpleneurologicalscorestoACC-NCDRmodel

..butdidn’tperformaswellinsalvage(C-stat0.754)vs non-salvage(C-stat0.902)cohorts

PCIinOHCA:RecommendationsfromtheAHA

PeberdyMAetal,Circulation2013;128:

Options1. ExcludeOHCAfrominstitutionaland

IndividualOutcomesreports2. Includewhenappropriateriskmodel

isdeveloped(isthislikelygivendiversityofpatients?)

3. TrackOHCAcasesbutnottobeusedinperformanceranking(Recommended)

Inclusionof“CompassionateUseCriteria”ResnicFSetal,JACC2011;57(8):904-11

CompassionateUse(CU)criteria:ComaonpresentationforemergencyPCI;useofpercutaneousventricularassistorbypass(plannedoremergency);andCPresuscitationatthestartofPCIAUCimprovedfrom0.87– 0.90(p<0.01)

Publishingindividualsurgeon’sdeathratespromptsriskaversebehaviour

WestabyS,BMJ2014;349:g5026doi:10.1136/bmj.g502612th August2014

Highlightsthattheoperationitselfisonlypartofacomplexprocessthatdeterminesoutcome

Whathappenstopatientsnotacceptedforsurgery?

MeasuresofqualityRelationshiptoRevalidation

• Access(egwaitingtimes)• Patientfeed-back

– Out-patient– In-patient

• Colleaguefeed-back• Proceduresundertakenandcase-mixanalysis• Measuresofprocess• Measuresofoutcomes

– Short-termvslong-term– Mortalityvsfreedomfromangina,freedomfromadverseevents

• Compliancewithmorbidityandmortalityprogrammes

UpdatefromBCISDMAGandNICOR

• DecisiontoexcludeOHCAcasesfornow(buttrackthem)• Newriskmodelsubmittedforpublication• LessonslearnedacrossallthenationalCVaudits

– Processes,datacompletenessandaccuracy,timetable,confidentialityissues– Externalfeedbackonmethodologytootherauditsshouldbeshared– NICORgovernancereview:RecommendationsforDMAG

• On-goingdiscussionswithHQIP• FurtherdiscussionplannedwithSirBruceKeoghandNHSEngland• ITchanges• Newtechnologyregistries(CtE)• OHCAproposal

Conclusions

• Qualityofcarecan’tbemeasuredbymortalityoradverseoutcomesalone,butthesehavetobeessentialcomponentsofaqualityprogramme

• Measuringprocessesisasimportantasmeasuringmortality• Riskmodelsneedtoaccountforhighestriskcasesandneedupdatingtoaccommodatechangesinevidenceandclinicalchange

Conclusions

• Publicreportingaroundtreatmentsisunlikelytogoaway

• Weneedamorein-depthdebatewiththepublicandthosewhorepresentthepublic(politicians,governmentagencies)aboutwhatistobemeasuredandtheresourcesneeded

• Weneedtodevelopapackageofmeasuresthatwillgainbuy-infromtheprofession,publicandcommissioners,avoidingunintendedconsequences

ImprovingsystemsBerwickDM,BMJ1996;312:619-22

• Concentrateonmeetingtheneedsofpatientsratherthantheneedsoforganisations

• Measurementisbestusedforlearningratherthanforselection,rewardorpunishment

Finalthoughts• Operators

– Provideanhonestaccountofthepatientandyourprocedure– Don’tgame– it’sdishonestandthereforeunprofessional– Fillinthedatabaseandensureend-pointstohospitaldischargeare

collectedaccurately– Regularlylookatyourresultsandyourteam’sresults– Youareresponsibleforthecompletemanagementofyourpatient,not

justtheprocedure– Workwithcolleaguesasateamandsupporteachother

• Trusts– Provideappropriateauditresourcestotheclinicalteams– Encourageanenvironmentoflearningfromproblems– Examinecauseswhenproblemsarise

Finalthoughts• Commissioners

– Don’tmakeitimpossibleforTrustsandclinicianstoengagesatisfactorilywiththeQualityImprovementagenda

– Workwithnationalsocietiestoaidprocessandensureappropriateresponsetoresults

• BCIS/NICOR– Shouldcontinuethenationalaudit– Shouldworkwithacademicgroupstoensureappropriateriskmodelsare

developedorre-calibrated– ShouldcontinuetofeedbacktoTrustsandindividualconsultant

operators– Shoulddebatethewiderissuessurroundingqualityimprovement– ShouldworkwithDoHagenciesonissuesoftransparency– Shoulddowhatitcantoensureappropriateresponsestooutliers