clinical kpis and partnership optimization - cbinet.com1).pdf · the median time from subject visit...
Post on 11-Aug-2019
214 Views
Preview:
TRANSCRIPT
© Copyright 2017 TriRadial Solutions
Key Performance Indicators (KPIs) and Sponsor/CRO Partnership Optimization
February 23, 2017CBI Clinical Trial
Budgeting & Forecasting
Geoff GarabedianTriRadial Solutions
© Copyright 2017 TriRadial Solutions
Agenda
2
Ø Outsourcing Trends and Partnership Dynamics
Ø Lessons Learned –Sponsor and CRO Perspectives
Ø Metrics to Help Optimize Partnership Collaboration
© Copyright 2017 TriRadial Solutions
Outsourcing is increasing, with more focus on partnerships, but progress is slow
3
Bene
fits
PreferredProviders
StrategicPartners
AlliancesandJointVentures
DegreeofIntimacy
Tactical/Transactional
1
2
3
4
Accordingtoarecentsurvey…• 90%ofsponsors
plantoincreasetheiruseofCROs
• 98%indicateCROoversightisatopapriority
• 34%feeltheyachievedtheirplannedCROgoals
• Lessthan20%believetheirrelationshipsareproductive
Surveysource:ComprehendSystems,Inc.CROOversightBenchmarkReport,2016
reality
target
© Copyright 2017 TriRadial Solutions
A trust deficit in most partnerships limits potential speed, quality and efficiency
4
3KeyFactors:
Lackofaccesstoreal-timeinsightstoinvestigateandresolveissuesquickly
Ø DatatransparencymostoftencitedasakeyrequirementforimprovingCROoversightandcollaboration
1
Metricsnotalignedwithcommongoalsacrossthepartnership(s),aretooreactive,anddonotpromoteinnovation
3
UnnecessaryoverlapandsharedresponsibilitiesbetweenSponsorsandCROs2
CurrentState TargetState
© Copyright 2017 TriRadial Solutions
Effective CRO governance is based on a common vision and driven by metrics
MissionVisionValues
Program&Partnership
Goals
Structureand
ResourcesProcesses
TechnologyandData
Metrics
ContinuousImprovement
5
Ensureeachcomponentmaximizessuccessandreturnsforboth parties
© Copyright 2017 TriRadial Solutions 6
Lessons Learned
© Copyright 2017 TriRadial Solutions
Sponsor Perspectives
• Most sponsors do not have enough visibility to issues as they arise, let alone a way to predict when issues are looming
• The governance plan, including issue escalation and quality agreements, is developed in a silo or not agreed to upfront
• CROs often more focused on “the relationship” and follow-on work• Penalty clauses alone do not typically drive better performance;
incentives hold more promise and can drive more long-term value• Overlapping roles & responsibilities cause inefficiencies and confusion• Metrics are sometimes too narrowly focused or confining• Site performance not necessarily a good indicator of CRO performance/
quality• Many companies do not engage sites regarding metrics or feedback
7
© Copyright 2017 TriRadial Solutions
CRO Perspectives
• Sponsors create unnecessary oversight layers and want visibility to day-to-day details that do not necessarily impact trial success
• Little opportunity to get involved in design activities• Sponsors talk about strategic partnerships, but actions demonstrate
more tactical outsourcing of specific activities (but at ”preferred pricing”)• Overlapping roles & responsibilities cause inefficiencies and confusion• Sometimes not a “unified voice” to the customer• Often little or no incentive to being more efficient or innovative• Vision and strategy not always communicated throughout the
partnership; employees are not motivated to go above and beyond in support of the strategy
8
© Copyright 2017 TriRadial Solutions 9
Development of KPIs
© Copyright 2017 TriRadial Solutions
SourceDataSystems
SourceDataSystems
Just because you can measure something, doesn’t mean you should
10
Metrics (and Other Information
Requirements)
Questions (for Assessing Goal Performance)
Information
Goals (and Other Performance Expectations)
AssessedBy ...
AnsweredBy ...
Comprising ... Provide DataPoints for ...
To Answer ...
To AssessPerformanceRelated to ...
InternalandExternal
Stakeholders
Takeasystematicapproachtodefiningmetricsthatalignwithbusinessandpartnershipgoals
SourceDataSystems
Thereshouldbeoneversionofthetruth,visibleacrossthepartnership
© Copyright 2017 TriRadial Solutions
Considerations for Metrics Development
• Develop measures along multiple dimensions
• Measure process performance, not people performance
• Align metrics with the quality management system
• Don’t optimize separately for each partner
11
• Leverage benchmarks and historical data to set thresholds
• Incorporate the site perspective to help evaluate CRO performance
• Use 360o survey tool across the partnership to assess ongoing collaboration/value
• Develop role-based dashboards that allow management by exception
© Copyright 2017 TriRadial Solutions
Balance the types of metrics being used
12
Metric Type Description Example Metric
Leading Indicator
Provides information that you can act on immediately to get the trial/ process back on track.
The proportion of sites activated versus expected would be a leading indicator for whether subject enrollment is likely to be on track.
Lagging Indicator
Provides information that you can use for future trials or for baselining for process improvement efforts.
The time taken from “last subject last visit” to database lock.
Cycle-time Measures the time taken to complete a task.
The median time from subject visit to data entry into an electronic data capture system.
Timeliness Measures whether a particular milestone has been met.
The number of days between planned and actual dates of the first site activated.
Efficiency Measures the amount of resource required to complete a task or set of tasks versus that expected.
The difference between the actual final total contract value and the initial baseline contract value for a CRO running a clinical trial.
Quality Measures how well an output from a process meets the requirements of the customer of that process.
The proportion of expedited safety reports that are received by regulatory authorities within the required timelines gives an indication of the quality of the pharmacovigilance reporting process.
© Copyright 2017 TriRadial Solutions
StudyStart-Up StudyConduct StudyCloseout
A balanced set of measures is critical for long-term success
13
Ø Initial KPIs will depend on data available
Ø Incorporate satisfaction surveys and other feedback mechanisms
Ø Actively review and discuss measures jointly during governance meetings
Time
Cost
Quality
Compliance
Planning/ Performance
Partnership Health
© Copyright 2017 TriRadial Solutions 14
Dimension Metric Definition
Time
AwardtoExecutedContract Cycletimefromprojectofficiallyawarded tofullyexecutedWorkOrder(calendardays)
DataSystemsAvailable %DatabasemanagementandDataCaptureSystemsgo-livewithintheagreedtimelines
SiteActivationTime–contractsandessential docs
Cycle-timepersitefrom initialcontact withInvestigatorto signedcontractandallessentialdocumentsreceived&IRBapproval
MajorMilestonesAchieved %keystudymilestonesmetorexceededvs.baseline(e.g.,ProtocolApproved,eCRFApproved,FPI, LPI,LPLV,DBL,FinalCSR)
COTurnaroundTime DaysfromCOnotification toCOexecution
%DBLDelays PercentDBLsthatdonotoccur ontimeduetosponsororCROconstraintsvs.totaldelays
QueryResponsetime Averageresponsetimetoqueryresolution
IssueClosureTime Averagetimetocloseopenissues
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Dimension Metric Definition
Cost
%BudgetVariance(Plannedvs.Actual)
Costvariancefromoriginalworkorderforstudyor portfolioofstudies.Calculatedbycategory(labor,pass-throughcostsandinvestigators)andfortotalstudycosts
FinancialPlans/ToolsInPlace %oftrialswithallpartnershipfinancialtoolsinplacebyFPI(ifrequired)
ChangeOrderImpact %IncreaseinbudgetduetoChangeOrders
TimelinessofPayments Average timefrominvoicedatetopayment
OversightFTERatio MeasureofsponsorandCROemployeesresponsibleforstudydelivery/managementtoindicatelevelofoversight(andredundancy)
LaborUnitCosts(e.g.,pervisit,persite,permonth, perunitofdatacleaned)
Costofrelevantstudycomponents dividedbyapplicablevariable(e.g.,monitoringbudgetand#ofenrolledsubjects).Canalsoincludevariancestoplan
15
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Dimension Metric Definition
Quality
%ofTrialQualityManagementPlansinPlacePriortoFPI
%oftrialswithsignedQualityManagementPlansinPlacePriortoFPI
PatientRecruitment %ofsitesrecruitingmorethat50%ofplannedsiteenrollment
PatientRetention %patientsretained:patientsrandomized/dosedineachstudywhoarenotlosttofollowupanddonotwithdrawconsent
DocumentQuality Qualityofdocumentation, assessedbysponsor
CRAQuality Qualityofmonitoringstaff, assessedbysite
#ofCriticalSiteAudit/InspectionObservations
Totalnumberofsiteauditand/orinspectionfindingsthatarecritical
# ofDatabaseUnlocks #ofunplanned databaseunlocks(i.e.,DBunlockstosolvedataissuesdetectedafterlock,requiringre-transferandre-analysis)
16
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Dimension Metric Definition
Compliance
ProjectManagementPlans inPlace
%oftrialswithPMPsignedandin placepriortoFPI(ormutuallyagreeddate)
SAEReportingCompliance %ofreportedSAEswithinregulatorytimelinesvsthetotalnumberofreportedSAEs
SUSARReportingCompliance %ofreportedSUSARswithinregulatorytimelines vsthetotalnumberofreportedSUSARs
ProtocolDeviations/SiteCompliancetoProtocol
%ofenrolledsubjectswithprotocolcritical/majordeviations/issuesdefinedaspervariouscategories(e.g., eligibility/entrycriteria,IP,ICF,sourcedoccriteria,SAE)
PartnershipTrainingCompliance
%compliance(trainingcompletedvs.required)forpartnership-specificprocessesandSOPs
17
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Dimension Metric Definition
Planning/Performance
SiteActivation – drugsupply %ofsitesreadytoenrollwithdrugreceivedatsite
TimelineShifts #oftimestimelinesshiftedperstudy
Plannedvs.ActualSitesOpen(readytoenroll)
%oftrialsmeetingorexceedingplannedvs.actualsitesopen(readytoenrollsubjects)
EnrollmentPlanAccuracy(plan vs.actual)
%oftrialsmeetingorexceedingplanned vs.actualenrollmenttargets(randomization)
Non-EnrollingSites % ofsiteswhohavenotenrolledasubject(siteinitiationvisitdate>100days)vs.totalnumberofsitesinitiated
ProtocolAmendment Rate Totalnumberofprotocolamendmentsdividedbynumberofactiveprotocolsduringthesameperiod
18
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Dimension Metric Definition
PartnershipHealth
Appointment ofStafftoKeyRoles
Keystaffinplaceasagreedpercontractandtimelines
StaffTurnoverRate %ofsponsororCROstaffdedicatedtoaPartnershipstudyatthebeginningofthequarterbutnolongerworkingonthesamestudyattheendofthequarter
Innovation NumberofCRO-generated ideasthathaveapositiveimpactonthedesign,qualityorperformanceofatrial
360PartnershipSatisfactionSurvey
Periodic(e.g.,quarterly) scoresfromstandardsurveytoindicateemployeeexperienceandengagementinPartnership (opentoallsponsorandCROpersonnelinvolvedinpartnership)
ExternalInvestigator/SiteSatisfactionSurvey
Scoresfromsurveyofstudysite personnel(e.g.,investigator,nurses,studycoordinators,etc.)
19
Example Metrics and Definitions
© Copyright 2017 TriRadial Solutions
Detailed Indicator Definitions
20
Eachmetricshouldhaveadetaileddefinitionthatgoesbeyondadescriptionandformula
Metric#:32a-1Name:MeanStudyStart-upTime
KPICategory:CycleTime
Sub-category:ClinicalStudyCycleTime
Description:Daysfromprotocolapprovaltofirstsubjectrandomized,forallsitesinstudy
UnitofMeasure:CalendardaysFormula/Definition:Median#calendardaysfromCSPtoFSI
Calc.Frequency:EndofstudyDataSource:CTMS
ReportingLevel(s):Study,Portfolio
Target:90Days
Leading/Lagging:LaggingRelatedKPIs:
BusinessGoalLinkage:
AdditionalAnalysisDrill-down:
Comments:
© Copyright 2017 TriRadial Solutions
Summary
Ø It’s not all about metrics – Spend time on up-front collaboration to develop a shared vision, define common goals and select appropriate KPIs to measure those goals
Ø Take a holistic view – Agree on a governance plan that proactively manages the relationship and provides effective oversight across several areas (cost, quality, performance, issue escalation, etc.)
Ø Find a Balance – Identify a mix of activity-based and outcomes-based measures, along multiple dimensions, that align with goals
Ø Simplify – Focus on a manageable set of KPIs that incent the right behaviors
Ø Visualize – Develop a set of role-specific dashboards that help improve performance through data-driven decision making
21
© Copyright 2017 TriRadial Solutions 22
Geoff GarabedianM: 201.218.8827
geoffgarabedian@triradialsolutions.com
top related