Quality Metrics for IRBs

Annette R. Anderson, MS, RHIA, CIP

VA Central IRB Administrator

Program for Research Integrity Development and Education (PRIDE)

Overview

• What are quality metrics and how can they be used?

• What are the current problems in using these concepts to measure the quality and effectiveness of IRBs?

• How can we use metrics in our local programs to measure and improve the performance of our IRB?

• What kind of metrics should be used in the future to measure the quality and effectiveness of IRBs and/or are different types of tools needed?

2

Background

• “Quality” and “Metrics” – definitions differ depending on upon context, industry, and viewpoint

• Used most in business, engineering, and manufacturing

• Adopted by many other sectors (e.g., the government and health care industry)

• Applying the business principles underlying “Quality Metrics” to IRBs can be problematic

3

What is the Meaning of “Quality”

• Definitions:• An inherentinherent or distinguishing characteristiccharacteristic; a property or attributeattribute

• A personal trait, especially a character trait

• Essential character; nature

• Superiority of kindSuperiority of kind

• Degree or grade of excellenceDegree or grade of excellence

• ISO 9000: Degree to which a set of inherent characteristics fulfills requirements

• A perceptual, conditional, and somewhat subjective attributeattribute understood differently by different people

4

Two Common Quality-Related Functions

• Quality Control - the process of reviewing the quality of all factors involved in the production of goods or services • Emphasizes testing and verification

• Use of audits and checklists to uncover defects or deficiencies

• Quality Assurance – systematic monitoring and evaluation of the various aspects of a process, service, or facility to maximize the probability that minimum standards of quality are being met• Emphasizes validation

• Uses metricsmetrics to prevent deficiencies and measure progress toward set goals

5

Definition of “Metrics”

• Verifiable measures Verifiable measures stated in either quantitative or qualitative terms that:• Capture performance in terms of how something is being done

relative to a standard or target

• Allow and encourage comparison

• Support the goals and objectives of the organization

• Performance metrics (i.e., Key Performance Indicators) often represent how far a metric is above or below a specified target

6

Use of Quality Metrics

• EffectiveEffective quality metrics must:• Have a support system to perform ongoing monitoring

• Be used to inform decision making

• Focused on seeking corrective actions that build a foundation for continuous process improvement; just collecting and reporting the data is not enough

• Quality metrics can be used to:• Spot trends in performance (good and bad)

• Adjust processes per institutional goals and objectives

• Compare alternative processes

• Compare to both inside and outside benchmarks (best practices)

• Predict performance

7

Quality Metrics Applied to IRBs - Do IRBs Protect Human Subjects?

•Researchers, Institutions, IRB members, and IRB staff have all complained about IRB mission creep, excessive paperwork, and attention to inconsequential details

• Studies have focused on structure, administrative efficiency, and procedural compliance with federal regulations and local policies

• Data suggest a need for more efficiency and less variation among IRBs

•Little is known about whether or not IRBs achieve their goalgoal of enhancing human subject protections in researchenhancing human subject protections in research

8

Other Problems in Developing and Using Quality IRB Metrics

• No agreed upon metrics or outcome measures No agreed upon metrics or outcome measures for evaluating IRB effectiveness

• Each Institution often has its own way of calculating how a metric is defined and measured (e.g., turnaround time)

• Practices and decisions vary from IRB to IRB and even within the same IRB

• Who is the customer (e.g., Institution, investigators, or human subjects?)

9

No Generic “One Size Fits All” IRB Metrics – Organizational Factors to Consider• Commitment from facility and research leadership to:

• Allocate resources for collection and analysis of metrics data

• Use the data in their decision making

• Communicate and coordinate with all stakeholders; transparency is key

• Fit organizational culture, goals and objectives

• Take into account Institution’s existing structure • Size and complexity of the Institution and research program

• Type of IRB arrangement

• Available resources

10

Additional Factors to Consider

• Top down drivers Top down drivers – what is most crucial to the achievement of your organizational goals and objectives?

• Set attainable targets and/or select benchmarks (ensure they are compatible) or establish internal comparisons

• Are you already collecting the metric in some form or will you need to implement a new process?

• Balanced Approach• How many metrics are too much or too little

• How often should a particular metric be collected

• How often should it be analyzed and/or reported?

11

Data Analysis

• CrucialCrucial for monitoring progress and identifying trends

• Tell what is going on and what is needed to improve

• Justify or explain outliers • Present data with and without outliers

• Present mean, median, and mode

• Gather additional data if needed to better diagnose and resolve problem areas

• Prepare and share draft reports to get additional input

12

Take Appropriate Action

• Authority to take action should be at lowest level possible• Change policy or procedures

• Add additional resources (increase staff/use of technology)

• Obtain additional training or consultant services

• Re-measure and perform ongoing monitoring to ensure problem resolved and/or does not recur

• Publicize results and emphasize any improvements or areas that need attention

• Adjust targets based on performance; are they too low?

• Re-evaluate appropriateness and effectiveness of metrics; should be fluid, not static

13

IRB Metrics Examples – Turnaround Time

• Must define what this means and set target(s):

• Submission date to final approval date or approval contingent on minor modifications?

• Includes investigator response time?

• When does clock start?

• Does clock start again if investigator does not respond within defined timeframe?

• Separate measurements at different process points?

• Separate measures for expedited and exempt reviews?

• How measured? (e.g., business or calendar days?)

14

IRB Metrics Examples – IRB Staff Workload and Performance

• Define how workload will be measured and set target

• Cradle to grave system or functional system?

• Factors to consider:• Number of studies overseen per staff member by type of study

• Number of actions overseen by staff member by type of action

• Complexity, risk, and duration of the study

• Timeliness and completeness of administrative pre-reviews and drafting of determination letters

• Other duties assigned that have impact on ratio

• Training and experience (i.e., # of CIPs; training opportunities given per time frame)

• Turnover metric

15

IRB Metrics Examples – IRB Member Workload and Performance

• Quantitative Factors to Consider

• IRB size and member qualifications

• Number of studies per reviewer by study type

• Number of actions reviewed by IRB member by type of action

• Complexity, risk, and duration of the study

• Timeliness of Reviews

• Attendance at meetings and training events

• Frequency of use of Alternate Reviewers and Consultants

16

IRB Metrics Examples – IRB Member Workload and Performance (Cont.)

• Qualitative Factors to Consider

• Depth and thoroughness of reviews

• Adequate completion of checklists

• Logical and coherent motions

• Give individual feedbackGive individual feedback

17

IRB Metrics Examples– IRB Meeting Management

• Factors to Consider:• Number and type of IRBs – local or affiliate

• Frequency and length of meetings

• Number of items on agenda

• How often is quorum lost and why?

• How often items tabled to next meeting and why?

• Did all members receive all required materials in time

• Minutes:• How soon after the meeting are minutes available for review?

• How many corrections must be made to minutes?

• Are all expedited and exempt actions appropriately reported?

18

Other IRB Metrics

• Attaining and maintaining accreditation

• Monitoring of audit reports to ensure compliance with regulatory requirements

• Internal monitoring process for review of IRB records

• Number of complaints (i.e., investigator, participants)

• Quality and completeness of IRB communications

• Feedback from:• IRB Members

• IRB Staff

• Investigators

• Other organizational stakeholders

19

A Little Bit of Quality Metrics Philosophy

From a presentation by Dr. Stephen Rosenfeld, former From a presentation by Dr. Stephen Rosenfeld, former President and CEO of Western Institutional Review President and CEO of Western Institutional Review Board, given on June 21, 2010Board, given on June 21, 2010

• Metrics rarely tell us something we don’t already know, but when they do, it’s important

• They force us to make our assumptions explicit, and sometimes force us to abandon our assumptions

20

Quality Metrics Appear to Be Here to Stay

• Decision making is increasingly data drivendata driven

• Successful organizations are shifting to evidence-evidence-basedbased management

• Data is used to back-up decisions as a way to manage risk

21

The Future for IRBs – A Back to Basics Movement

• Research has changed dramatically since IRBs were created

• Current metrics on IRB workload continue to be important; however, they do not necessarily measure the effectiveness of the IRB in enhancing human subjects protections

• Growth of centralized IRBs

• Symposia, conferences, and papers on alternative models

22

The Future of IRB Quality Metrics

• Movement to base metrics on IRB approval criteriaIRB approval criteria

• Challenge: Converting these concepts into valid, reliable, and verifiable metrics

• The Clinical Research Group of the NIH-funded Clinical and Translational Science Awards Consortium has begun an initiative delineating measures of IRB quality

23

Recommended Reading

• JAMA Commentary, September 8, 2010 – Vol 304, No. 10, by Dr. Christine Grady, Department of Bioethics, National Institutes of Health Clinical Center, Bethesda, Maryland

• Holly A. Taylor, “Moving Beyond Compliance: Measuring Ethical Quality to Enhance the Oversight of Human Subjects Research,” IRB Ethics & Human Research 29, no. 5

24

Questions and Contact Information

Annette Anderson, MS, RHIA, CIPVA Central IRB Administrator202-461-1813; annette.anderson3@va.gov

Websitehttp://www.research.va.gov/programs/pride/cirb/default.cfm

Toll free number 1-877-254-3130Email va.central.irb@va.gov

25