Patient-Centered Outcomes Research Institute

What is it and what does it mean for health services researchers?

Patrick S. Romano, MD MPHUC Davis Center for Healthcare Policy and Research

March 28, 2012

Speaker’s Verbal Disclosure Statement:

Have you (or your spouse/partner) had a personal financial relationship in the last 12 months with the manufacturer of the products or services that will be discussed in this CME activity?

_x_ Yes

___ No

(If yes, please state disclosures and resolutions)

I received a $200 honorarium and reimbursement of travel

expenses for participating in a PCORI grant review meeting.

Educational objectives for this seminar:

Please type between two and four educational objectives here.

To describe the origin and mission of PCORI

To differentiate Patient Centered Outcomes Research from Comparative Effectiveness Research (if possible?)

To explain the PCORI grant review process and discuss future opportunities for UC Davis/CHPR research support

What is CER?

• Comparative effectiveness research is the conduct and synthesis of systematic research comparing different interventions and strategies to prevent, diagnose, treat and monitor health conditions. The purpose of this research is to inform patients, providers, and decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances. To provide this information, comparative effectiveness research must assess a comprehensive array of health-related outcomes for diverse patient populations…– Federal Coordinating Council for Comparative

Effectiveness Research, definition; [cited 28 July 2010].

What is PCORI?

• Prior to several years ago, no coordinated effort for comparative effectiveness research (CER)

• 2009 stimulus package provided $1.1 billion for CER (most went to AHRQ and NIH)

• 2010 Affordable Care Act created Patient Centered Outcomes Research Institute

• Funded by tax on government and private insurers for each covered life (~$500 million per year)

What is PCORI?• Independent, nonprofit, quasi-governmental

organization– AHRQ almost eliminated in 1990s after published guideline

that surgery for back pain was unnecessary– Provide political protection – PCORI funding does not depend on annual Congressional

appropriation

• No infrastructure for determining funding priorities, issuing opportunities, reviewing and administering grants– Outsourcing (NIH, AHRQ, CMS)

• Peer review, contract management• May receive $ from PCORI to originate grants

– PCORI will be its own entity

1. Identifying national priorities for research.2. Creating a research agenda based on identified priorities.3. Funding research consistent with these priorities and agenda.4. Providing patients and their caregivers with useful research information.

PCORI Board of Directors• Director of AHRQ (Carolyn Clancy)• Director of NIH (Francis Collins)• 17 members appointed by the Comptroller General of the

United States as follows:– 3 members representing patients and health care consumers– 5 members representing physicians and providers, including at least 1

surgeon, nurse, State-licensed integrative practitioner, and hospital– 3 members representing private payers, including at least 1 health

insurance issuer and at least 1 self-insured employer– 3 members representing pharmaceutical, device, and diagnostic

manufacturers or developers– 1 member representing quality improvement or independent health

service researchers (Harlan Krumholz or Arnold Epstein)– 2 members representing the Federal Government or the States,

including at least 1 representing a Federal health program or agency

PCORI Leadership• Eugene Washington, MD, MSc, (chair), Vice

Chancellor of UCLA Health Sciences and Dean of the David Geffen School of Medicine

• Steven Lipstein, MHA (vice chair), President and Chief Executive Officer of BJC HealthCare

• Joe Selby, MD, MPHExecutive Director

• Anne C. Beal, MD, MPHChief Operating Officer, abeal@pcori.org

PCORI Mission

PCORI helps people make informed health care decisions and improves health care delivery and outcomes by producing and promoting high integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader health care community.

PCOR vs. CER

• Why PCORI, not CERI?• Political concerns that determining which treatments

better → “rationing” – Restrict patients’ ability to choose inferior treatment

• Compromise focus on patient-centered outcomes research

• Legislation barred PCORI from – Developing or using QALYs– Considering cost in making coverage recommendations– In practice, PCORI will not fund cost-effectiveness analyses

Patient-Centered Outcomes Research

• Not just a semantic difference

• Focus on PCOR essential role in shaping funding priorities and “rules of the game” for getting funding

• Can’t just replace placebo control with active comparison group, do everything else the same

• Distinct elements – Outcomes should be patient-centered– Effectiveness, not efficacy – Heterogeneity of treatment effects– Involvement of stakeholders

Patient-Centered Outcomes Research

• Outcomes should be patient-centered– “…outcomes that people notice and care about

such as survival, function, symptoms, and health-related quality of life”

– Not biomarkers (BP, serum cholesterol, etc.)

Patient-Centered Outcomes Research

• Effectiveness– Research done in clinically relevant populations– Representative of patients– Representative of settings

• Heterogeneity of treatment effects– “Given my personal characteristics, conditions and

preferences, what should I expect will happen to me?”– “What are my options and what are the benefits and

harms of those options?”

Role of stakeholders

• Government payors and regulators; health plans; hospitals and other care providers; industry

• Consider how results will shape policy and practice through dissemination

• Stakeholders should be intimately involved in the study design process to identify important outcomes– Trials of anti-seizure drugs in children– Traditional measure – seizure frequency– Patient-centered measure – parent want to know how

drug will affect school performance– Drug ↓ seizures but ↑ sedation may not be preferable

Implications for research design• RCTs

– Pragmatic - real-world settings; large N’s to assess subgroups

– Focus on effect size, not just superiority– Bayesian and adaptive approaches to design, accounting for

prior probabilities• Observational studies

– Causal inference modeling– Tension for studies using pure administrative data –

powerful and low-cost; lack high-quality patient-centric data• Cost and cost-effectiveness

– PCORI not allowed to evaluate cost in making coverage recommendations, won’t fund “pure” CEA

– Cost can be studied and used by decision-makers

Proposed National Priorities• Assessment of Prevention, Diagnosis, and Treatment Options. The

research goal is to determine which option(s) work best for distinct populations with specific health problems.

• Improving Healthcare Systems. Focuses on ways to improve healthcare services, such as the coordination of care for patients with multiple chronic conditions.

• Communication and Dissemination. Looks at ways to provide information to patients so that they, in turn, can make informed healthcare decisions with clinicians.

• Addressing Disparities. Assures that research addresses the healthcare needs of all patient populations. This is needed as treatments may not work equally well for everyone.

• Accelerating Patient-Centered and Methodological Research. Includes patients and caregivers in the design of research that is quick, safe, and efficient.

PCORI Funding Opportunities

• Developmental• RFPs (Sept 2011)

• ~$30-$100K each• Methods for setting research priorities

• Pilot Projects Grant Program (Dec 2011)• $13 million in year 1 for ~40 projects• Help develop national research priorities for PCOR• Develop ways to engage patients and bring stakeholders together• Develop measurement tools and data collection methods for

novel patient-centered outcomes

PCORI Funding Opportunities

• $150 million in current FY; ultimately perhaps $500 million/year

• Mechanisms unknown, priorities under discussion– Likely NOT “traditional” research– Patient-centered outcomes

• Including developing methods

– Real-world populations– Heterogeneity of treatment effects– Involvement of stakeholders

Other Funding Opportunities

• Existence of PCORI does not preclude government agencies or foundations from funding CER

• AHRQ has Effective Healthcare Program – Large evidence-synthesis and dissemination projects– EBPCs, DEcIDE network, CERTS– Does not support investigator-initiated projects

• Areas of CER explicitly outside PCORI’s mandate– CER with “traditional” outcomes – Cost-effectiveness analyses

• Remains to be seen

PCORI Pilot Research Projects

• $13 million in 2012• Approximately 40 awards• 2 years maximum support

– 2nd year of funding must be well justified and will be evaluated for successful completion of the first year goals

• $250,000 per year (plus indirect costs)– Travel to annual PCORI meeting is an allowable expense

• Requires stakeholder involvement as co-investigators– unless not feasible as for an application on analytic

methods

Principles of grant review

• Scientific and technical merit• Fair, competent and thorough• Conflict of interest• Confidentiality• Difference between overall impact and significance

– Overall impact: Probability of whether the research will exert a sustained, powerful influence on the research field.

– Significance: Does the project address an important problem or a critical barrier to progress in the field? If the aims are achieved, how will scientific knowledge, technical capability, and/or clinical practice be improved?

Structure of application• Signed Face Page, Table of Contents• Project Summary: single paragraph using plain language to summarize the

– Broad objectives– Specific aims– Relevance to PCORI Objectives– Research Design and methods for achieving the goals

• Project/Performance Site(s)• Key Personnel and Other Significant Contributors• Biographical Sketches–Principal Investigator, Other Biographical Sketches

– Personal statement allows each person to explain their role and goals for the project• Budget and Justification• Resources

– Provide information on the facilities to be used in the proposed research project– Describe how the research environment contributes to the probability of success– Describe how project will benefit unique features of the research environment or

community involvement or allow for useful collaborative arrangements– Describe institutional and community investment in the success of the research

• Checklist• Addendum Form: Checklist to identify PCORI Areas of Interest

Research Plan

• Specific Aims (1 page)• Research Strategy (10 pages)• References Cited• Protection of Human Subjects• Consortium/Contractual• Letters of Support

• No appendices are allowed and will be removed by PCORI before sending to reviewers

PCORI Review Criteria

• Significance (Research Plan)• Patient/Stakeholder Engagement (Research Plan)• Investigator (Biosketch)• Innovation (Research Plan)• Approach (Research Plan)• Environment (Project/ Performance Sites)

PCORI Review Criteria -Significance

• Does the project address an important problem or critical barrier to PCOR?

• Will the result produce new knowledge that can advance PCOR methods or infrastructure?

• Does the investigator demonstrate thorough knowledge of previous and ongoing work related to their proposed topic?

PCORI Review Criteria -Patient/Stakeholder Engagement

• Will the project make a unique contribution to learning about engagement of patients and stakeholders?

• Does the research team demonstrate authentic, feasible partnerships with the patients/stakeholders?

• Is there evidence that the stakeholders were involved in the preparation of the research proposal?

• Is the research team well suited to the project?• Is there appropriate scientific expertise?• Does at least one member of research team have

experience in patient/stakeholder engagement?• Will the research team be able to achieve the study

aims?• Does the research team have complementary and

integrated expertise? Is the approach and structure of the research team appropriate for the project?

PCORI Review Criteria -Investigator

PCORI Review Criteria -Innovation

• Does the project address a new method or approach for the field of PCOR?

• Does the project apply a proven method or approach in a novel way to the field of PCOR?

• Is there a new application or refinement of theoretical concepts, approaches, methodologies, instrumentation or interventions proposed?

PCORI Review Criteria -Approach

• Are the overall strategy, methodology, and analyses appropriate to accomplish the specific aims?

• Are potential problems, alternative strategies, and benchmarks for success presented?

• Will the strategy establish feasibility and will particular risky aspects be managed

• Is the budget and timeframe appropriate for the research plan?

• Are there plans for dissemination among key PCOR stakeholders in education, practice and policy?

PCORI Review Criteria -Environment

• Will the environment contribute to the probability of success?

• Are the institutional support, equipment and other physical resources adequate for the project?

• Will the project benefit from unique features of the research environment, community involvement, patient populations, or non-traditional stakeholder collaborative arrangements?

33

PCORI ReviewTemplate Based

OVERALL IMPACT SECTION

SCORED REVIEW CRITERIA SECTIONS: The six review criteria, Significance, Patient/Stakeholder Engagement, Investigators, Innovation, Approach, Environment, with a separate score for the first five criteria.

Overall Impact

1. Significance

Strengths

Weaknesses

When Writing CritiquesOverall Impact paragraph –

– requires a summary narrative (paragraph) of the strengths and weaknesses. Do not cut and paste.

– Focus on major strengths and weaknesses that impacted your overall rating of the application

– Overall impact score is not an average of the criterion scores; weights can differ across reviewers

•Use bulleted points to make succinct, focused comments (Be careful of being too brief – the reader needs to understand your comment)

• Any criterion score between 3 to 9 needs to have weaknesses listed

3434

Scoring

9-point score scale is used to provide:• Criterion Scores for each of the core review criteria• Overall Impact/Priority Score based on but not an average of the

core criterion scores plus additional criteria

All applications receive scores:• Not discussed applications (bottom half by pre-meeting scores) will

receive only initial criterion scores from the 3 assigned reviewers.• Discussed applications also receive an averaged overall impact

score from eligible panel members (i.e., those without conflicts of interest).

• These scores will be averaged to one decimal place, and multiplied by 10. The 81 possible priority scores will thus range from 10-90.

9-Point Score Scale Descriptors

Impact Score Descriptor Additional Guidance

High

1 Exceptional Exceptionally strong with essentially no weaknesses

2 Outstanding Extremely strong with negligible weaknesses

3 Excellent Very strong with only some minor weaknesses

Medium

4 Very Good Strong but with numerous minor weaknesses

5 Good Strong but with at least one moderate weakness

6 Satisfactory Some strengths but also some moderate weaknesses

Low

7 Fair Some strengths but with at least one major weakness

8 Marginal A few strengths and a few major weaknesses

9 Poor Very few strengths and numerous major weaknesses

Process During the Meeting• Chair calls for preliminary overall impact scores• Primary reviewer

– Brief description– Discuss major/ score-driving strengths and weaknesses

• Second and third reviewers add only major/score-driving strengths and weaknesses

• Open discussion among all reviewers• Human subjects issues• Final scores from assigned reviewers• Unassigned reviewers may vote outside the score range• Record scores (in IAR and on hard copy)• Budget (does not influence score!)• 10-15 minutes per application

Reviewer Voter Sheet (available only during the meeting)

Acknowledgments

• Michael Steinman MD, UCSF CTSI Director of Comparative Effectiveness Research

• Jonathan Arias, PhD, NIH Center for Scientific Review

• All of you