center for scientific review national institutes of health department of health and human services...
TRANSCRIPT
![Page 1: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/1.jpg)
Center for Scientific ReviewNational Institutes of HealthDepartment of Health and Human Services
Toni Scarpa
NIH Peer Review: Continuity and Change
APS/Cardiovascular Section
Salt Francisco, April 2006
![Page 2: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/2.jpg)
Peer Review: An N.I.H. “Conception”
• Is the heart and soul of NIH• Has produced an effective partnership between the
federal government and research institutions• Has created the best academic medical centers, the best
biomedical/behavioral research and biotechnology• Has made possible the best cures and the best prevention• Has been admired and imitated here and abroad• Has protected NIH against outside influence
![Page 3: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/3.jpg)
Center for Scientific Review
![Page 4: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/4.jpg)
Applications received for all of NIH and applications referred for CSR review, FY 1998-2004
0
20,000
40,000
60,000
80,000
1998 2000 2002 2004Fiscal year
Nu
mb
er o
f ap
pli
cati
on
s
Applications received for all of NIH
Applications assigned for review by CSR
![Page 5: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/5.jpg)
CSR 4 Review Divisions & 23 IRGs
Division of Biologic Basis of Disease
Elliot Postow, Ph.D.
AIDS and RelatedResearch IRG (AARR)
Ranga V. Srinivas, Ph.D.
Endocrinology, Metabolism, Nutrition, and Reproductive
Sciences (EMNR)Sooja Kim, Ph.D.
Immunological SciencesIRG (IMM)
Calbert Laing, Ph.D.
Oncological SciencesIRG (ONC)
Syed Quadri, Ph.D.
Infectious Diseases andMicrobiology IRG (IDM)
Alex Politis, Ph.D.
Division of Physiology and Pathology
Michael Martin, Ph.D.
Cardiovascular SciencesIRG (CVS)
Joyce Gibson, D.Sc.
Integrative, Functionaland Cognitive Neuroscience
IRG (IFCN)Christine Melchior, Ph.D.
Renal and Urological Sciences IRG (RUS)
Daniel McDonald, Ph.D.
HematologyIRG (HEME)
Joyce Gibson, D.Sc.
Digestive Sciences IRG (DIG)
Mushtaq Khan, Ph.D., DVM
Division of Clinical andPopulation-Based Studies
Anita Miller Sostek, Ph.D
Surgery, Radiology, andBioengineering IRG (SRB)
Eileen Bradley, D.Sc.
Health of the Population IRG (HOP)
Robert Weller, Ph.D.
Risk, Prevention, and HealthBehavior IRG (RPHB)Michael Micklin, Ph.D.
Brain Disorders and ClinicalNeuroscience IRG (BDCN)
Dana Plude, Ph.D.
Behavioral & BiobehavioralProcesses IRG (BBBP)Karen Sirocco, Ph.D.
Division of Molecular and Cellular Mechanisms
Donald Schneider, Ph.D.
Bioengineering Sciences and Technologies IRG (BST)
Sally Amero, Ph.D.
Biology of Development and and Aging (BDA)
Sherry Dupere, Ph.D.
Biological Chemistry and Macromolecular Biophysics
IRG (BCMB)John Bowers, Ph.D.
Cell Biology IRG (CB)Marcia Steinberg, Ph.D.
Genes, Genomes, and Genetics IRG (GGG)
Richard Panniers, Ph.D.
Molecular, Cellular and Developmental
Neuroscience IRG (MDCN)Carole Jelsema, Ph.D.
Respiratory Sciences IRG (RES)
Mushtaq Khan, Ph.D., DVM
Musculoskeletal, Oral, and Skin Sciences IRG (MOSS)
Daniel McDonald, Ph.D.
![Page 6: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/6.jpg)
CSR Mission Statement
To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.
![Page 7: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/7.jpg)
Time
C
om
ple
xity
an
d I
mp
act
CSR Operations
C
urrent S
ystems
New Systems?
Necessary Changes in CSR Peer Review Operations
![Page 8: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/8.jpg)
• Increase communications between CSR, the ICs, our reviewers and applicants
• Increase uniformity
• Increase efficiency
• Facilitate work of IC program staff
Changes in CSR Operations
![Page 9: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/9.jpg)
Potential of Knowledge Management Tools for Peer Review
Collexis Software or Others
• Knowledge management solutions• Fingerprinting and text retrieving• Disease coding
Benefits for Peer Review
• Assigning applications to Integrated Review Groups or Study Sections
• Selecting reviewers (one application, multiple applications)
Nine pilots are underway to begin to assess these benefits
![Page 10: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/10.jpg)
Study Section Realignment
• Review of one IRG every month• Total review every 2 years
![Page 11: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/11.jpg)
Required Changes in Current Systems
• Shorten the review cycle
![Page 12: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/12.jpg)
This is Not an Ford Assembly Line
Receipt ReferEvaluate Scientific
Merit of Applications
![Page 13: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/13.jpg)
Shortening the NIH Review Cycle, Initial Steps
For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)
We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)
![Page 14: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/14.jpg)
Desirable Changes in CSR Review
• Shorten the review cycle
• Address concern that clinical research is not properly evaluated
• Improve the assessment of innovative, high- risk/high-reward research
![Page 15: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/15.jpg)
“The judging of grants has become a charade.”
The American Society for Cell Biology
“The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.”
Zena WerbPresident, ASCB
Newsletter August 2005
![Page 16: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/16.jpg)
Possible Changes in Current Systems
• Shorten the review cycle
• Address concern that clinical research is not properly evaluated
• Improve the assessment of innovative, high- risk/high-reward research
• Do more to recruit and retain more high-quality reviewers
![Page 17: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/17.jpg)
Expanding Peer Review’s Platforms
Electronic Reviews• Telephone Enhanced Discussions• Video Enhanced Discussions• Asynchronous Electronic Discussions
Study Sections
Necessity ● Clinical reviewers Preference ● Physicists, computational biologists
New Opportunities ● Fogarty, International Reviewers
![Page 18: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/18.jpg)
The First NIH Study Section
1945
The Last NIH Study Section
2005
![Page 19: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/19.jpg)
If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?
Possible New Systems
![Page 20: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/20.jpg)
![Page 21: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/21.jpg)
Questions—Applications
• Should we have 3 or 365 deadlines for most applications?
• Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size?
• Is there more value in having 2-3 reviewers reading 25-page applications or 10-15 reviewers reading 5-page applications?
![Page 22: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/22.jpg)
Questions—Study Section Meetings
• What is the ideal number of members to have serve on a study section?
• Is one study section with 50-70 reviewers efficient?
• What is the intellectual contribution of individual reviewers in large study sections?
![Page 23: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/23.jpg)
Questions—Scoring
• Is it proper or desirable to have 50-70 reviewers voting on priority scores for each application referred to their study section?
• Is consensus always good? Or should we focus on score variance?
![Page 24: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/24.jpg)
Applications Received for All of NIH FY 1998-2004
0
20,000
40,000
60,000
80,000
1998 2000 2002 2004Fiscal year
Nu
mb
er o
f ap
pli
cati
on
s
![Page 25: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/25.jpg)
Number of Research Grant Applications/Applicant
1.05
1.1
1.15
1.2
1.25
1.3
1.35
1.4
1.45
1980
1982
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
![Page 26: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/26.jpg)
CSR Applications Reviewed, Regular and SEP May Council Only
0
2000
4000
6000
8000
10000
12000
14000
16000
1999 2000 2001 2002 2003 2004 2005
Applications Regular
Applications SEP
![Page 27: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/27.jpg)
Study Section Application/Reviewer Ratio October Council Only
0
0.5
1
1.5
2
2.5
3
3.5
4
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Regular
SEP
![Page 28: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/28.jpg)
Two groups of challenges/opportunities
• Reviewers– Decrease the number of reviewers and increase the
quality– Increase the number of applications reviewed
without extra workload– Recruit and retain the best reviewers
• SRA– Increase efficiency– Recruit and train
![Page 29: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/29.jpg)
Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR
• Replace Many SEPs with Smaller Parallel Study Sections
• Enlarge Study Section Membership and Decrease Frequency of Participation
• Pre Meeting Streamlining• Various Review Platforms• Hybrid Review Platforms• Staggering Application Deadlines• 2 instead of 3 reviews• Shorten Applications• More Structured Applications and Reviews
![Page 30: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/30.jpg)
Size of Grant Applications
• RO1 • Will increase number of applications reviewed by
reviewers• Will decrease the number of reviewers in a study section• May be combined with a change in format of the
application, more consonant to review criteria• May be combined with scoring individual criteria (i.e.
relevance, innovation, etc• Strong support by councils and scientific leadership• May result in more innovation
![Page 31: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/31.jpg)
This is CSR
![Page 32: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/32.jpg)
Coronary Heart DiseaseAge-Adjusted Death Rates in U.S.: Actual (blue) vs Expected (yellow)
500
400
300
200
100
01950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000
Dea
ths
per
100
,000
Year
~ 514,000 Actual Deathsin 2000
~ 1,329,000 Projected Deaths in 2000
815,000 Deaths Prevented in 2000
![Page 33: Center for Scientific Review National Institutes of Health Department of Health and Human Services Toni Scarpa NIH Peer Review: Continuity and Change APS/Cardiovascular](https://reader035.vdocuments.us/reader035/viewer/2022081516/5514b194550346ea6e8b61b9/html5/thumbnails/33.jpg)
The True Value of Peer Review