Michael S. Lauer, MD, FACC, FAHADirector, Division of Cardiovascular Sciences

National Heart, Lung, and Blood Institute

CSR Advisory Council MeetingPooks Hill Marriott, Bethesda, MD

October 20, 2014

Perspectives from the NHLBI on Portfolio Prediction and Performance

2 Langer J. Science 2012; 388:171

“The U.S. system for funding research was designed to function well in times of growth. It is failing now because the system as a whole is contracting. What is to be done? The funding agencies must admit that it is not humanly possible to predict, with high accuracy, which research projects ultimately will have the most impact. When there are too many of these, as at present, the agencies must find other ways to decide which to support. That will be hard.”

Predicting Performance? …

Cochrane 2008; DOI: 10.1002/14651858.MR000003.pub2.

“Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce.”

We (Largely) Rely on Peer Review

Fang F, Casadevall A. Infect Immun 2009;77:929-32

“The imperfect science of grant reviewing: For a system that determines the fate of scientific proposals, peer review is remarkably unscientific. Analyses have revealed that the NIH peer review system is statistically weak, imprecise, and prone to bias. … Almost no scientific investigation has been performed to examine the predictive accuracy of study section peer review.”

-- Ferric Fang & Arturo Casadevall

Strong Feelings Out There …

Ioannidis JP. Nature 2011;477:529-31

“The imperfections of peer review mean that as many as one-third of current grants are effectively being awarded at random. This situation will only worsen as falling acceptance rates encourage investigators to bombard agencies with proposals, leaving fewer qualified reviewers to judge each one.”

Not Only a Peer Review Issue …

Rand Report (July 2013) MG-1217-AAMC

“Characteristics that are scalable, quantitative, transparent, comparable, free from judgment and suitable for high frequency, longitudinal use.”

Measuring Impact is Hard

Ioannidis JP, Khoury M. JAMA 2014 (June 9 2014 online)

Productivity Number of publications in top tier % of citations

for the scientific field and year* > 1 published reports of the main results*

Quality Need to select standards, limit to top-cited articles

Reproducibility* Sharing of data and other resources Translational influence of research

The PQRST Approach

Danthi N et al. Circ Res 2014;114:600-606; Gordon D et al. N Engl J Med 2013;369:1926-34Kaltman J et al. Circ Res 2014;115:617-624

Three Reports from the (NHLBI) Field

Danthi N et al. Circ Res 2014;114:600-606; Kaltman J et al. Circ Res 2014;115:617-624

Investigator-initiated de novo R01’s that: Were funded by NHLBI between 2001-2008 Were assigned to cardiovascular division(s) Received a percentile ranking Received at least two years of funding

Outcomes Number of publications in top 10 % of citations for

the scientific field and year Normalized citation impact (for field and year)

Percentile Ranking and Citation Impact

Kaltman J et al. Circ Res 2014;115:617-624

No Association with Percentile Ranking

Kaltman J et al. Circ Res 2014;115:617-624

Association with Prior Productivity

No gold standard Citations reflect only one perspective Limited scope

Cardiovascular R01 handled within one division Limited time-span

“Multiple-grant problem” Only analyzed outcomes for funded grants

Limitations

Many thanks to Jamie Doyle, Yancy Bodenheim, and Tom Insel

Replication for NIMH R01 Grants

Kahneman D. Thinking Fast and Slow. New York: Farrar, Strauss, and Giroux, 2011 (220)

“People who attempt to predict the future make many errors; that goes without saying. Errors of prediction are inevitable because the world is unpredictable. High subjective confidence is not to be trusted as an indicator of accuracy. Behavior and achievements can be predicted with fair accuracy from previous behavior and achievements.”

-- Daniel Kahneman

Should We Be Surprised?

15

Taking the Powerball Approach to Funding Medical ResearchWinning a government grant is already a crapshoot. Making it official by running a lottery would be an improvement

By Ferric C. Fang And Arturo CasadevallApril 14, 2014

“And NIH's peer review has been criticized. The latest critique is a landmark study just published by Michael Lauer and colleagues at the NHLBI. Other studies have reported that NIH peer review lacks statistical rigor … We suggest another approach …”

Reactions

Gordon D et al. N Engl J Med 2013;369:1926-34

Different Case …

Gordon D et al. N Engl J Med 2013;369:1926-34

Predictors of time to publication were a focus on clinical events and higher costs up to approximately $5 million. Above $5 million, the cost of the trial was no longer a significant determinant of the time to publication.

What Factors Predict? Endpoints & Investments

Gordon D et al. N Engl J Med 2013;369:1926-34

… And Not So Much What is Often Claimed

Devereaux PJ, Yusuf S. N Engl J Med 2013;369:962-

“Change in priorities of funders, an appreciation from review panels of the importance of [large trials focused on clinical endpoints], and avoidance of inefficient and unimportant aspects of trial design…”

Reactions …

Kaiser J. Science July 25, 2014

NIH Considering Alternate Models

Nature 2010;465:864-6 and Nature 2013;495:437-40

“A second criticism is that the very idea of quantifying scientific impact is misguided. This really will not do. We scientists routinely search out numerical data; we cannot insist that our cogitations are uniquely exempt. The ultimate judge of scientific quality is the scientific community; its judgments are expressed in actions and these actions may be measured.”

Acknowledge Discomfort and Grow

Azoulay P. Nature 2012;483:31-2

“The current system already gambles with scientific careers, just in a haphazard way. … These experiments could be rolled out with minimal disruption. If [we] could test even a small number of hypotheses, the system-wide benefits would dwarf [the] modest investment.”

Should We Consider …

Mervis J. Science 2014;343:596-598

Final Thoughts …

Gary Gibbons Susan Shurin Richard Hodes Nancy Miller Josie Briggs Jim Kiley Keith Hoots Donna DiMichele Sean Coady Susan Pucie Terrie Wheeler Lisa Federer Phil Lee John Ioannidis Mona Puggal Jon Kaltman Monica Shah Josephine Boyington Steve Mockrin

Simhan Danthi Colin Wu Peibei Shi Frank Evans David Gordon Peter Kaufmann Melissa Antman Alice Mascette Wendy Taddei-Peters Ebyan Addou Zorina Galis Gina Wei Caroline Fox Richard Fabsitz Teri Gosselin Gail Pearson Denis Buxton Jodie Black Meena Hiremath

Thank You!