the new nih review system: reviewer’s perspective liz madigan, fpb school of nursing
TRANSCRIPT
![Page 1: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/1.jpg)
The New NIH Review System: Reviewer’s perspective
Liz Madigan, FPB School of Nursing
![Page 2: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/2.jpg)
• NIH information/guidance
• Advantages/disadvantages as a reviewer
• Recommendations for further improvement
![Page 3: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/3.jpg)
Relationship of Old vs. New Scores
![Page 4: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/4.jpg)
NIH Guidance
• The NIH grant application scoring system uses a 9-point rating for the impact/priority score
• Assigned reviewers also provide ratings for each review criterion (Significance, Investigator(s), Innovation, Approach, Environment) using the same 9-point scale. – These criterion ratings are provided in the summary statement of
all applications, both discussed and undiscussed.– Criterion ratings should be considered in determining the overall
impact/priority score, but because the relative importance of each criterion to the overall impact/priority score differs for each application, reviewers should not apply a formula of unweighted or weighted criterion scores across applications.
![Page 5: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/5.jpg)
NIH Guidance
• Reviewers are strongly encouraged to utilize the full range of the rating scale in determining ratings. Optimally, scores will be normally distributed with very few 1’s and 9’s and a majority of scores in the middle of the range (4-6).
• Discussed applications will receive impact/priority scores from all eligible (not in conflict) reviewers, and these scores will be averaged and multiplied by 10 to determine the final impact priority score (range of 10 to 90).
• Because the relative importance of each individual criterion to the overall score differs for each application, reviewers should not use a formula of weighted or unweighted averages across applications to determine the overall impact/priority score.
![Page 6: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/6.jpg)
Score Descriptor Additional Guidance on Strengths/Weaknesses
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
4 Very Good Strong but with numerous minor weaknesses
5 Good Strong but with at least one moderate weakness
6 Satisfactory Some strengths but also some moderate weaknesses
7 Fair Some strengths but with at least one major weakness
8 Marginal A few strengths and a few major weaknesses
9 Poor Very few strengths and numerous major weaknesses
Minor: An easily addressable weakness that does not substantially lessen the impact the project Moderate: A weakness that lessens the impact of the projectMajor: A weakness that is severely limits the impact of the project
![Page 7: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/7.jpg)
Score Descriptor Significance Descriptors
1 ExceptionalAchieving the proposed aims is likely to advance the research field in profound
and lasting ways
2 OutstandingAchieving the proposed aims is likely to advance the research field in critically
important ways
3 ExcellentAchieving the proposed aims is likely to advance the research field in important
ways
4 Very GoodAchieving the proposed aims is likely to contribute substantially to the current
knowledge base of the research field
5 GoodAchieving the proposed aims is likely to contribute meaningfully to the current
knowledge base of the research field
6 SatisfactoryAchieving the proposed aims is likely to contribute somewhat to the current
knowledge base of the research field
7 FairAchieving the proposed aims is likely to contribute only incrementally to the
current knowledge base of the research field
8 MarginalAchieving the proposed aims is likely to contribute only minimally to the current
knowledge base of the research field
9 PoorAchieving the proposed aims is unlikely to contribute in any way to the current
knowledge base of the research field
![Page 8: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/8.jpg)
Score Descriptor Investigator Descriptors
1 Exceptional
The investigators are extremely well qualified to achieve the proposed aims.2 Outstanding
3 Excellent
4 Very Good
The investigators are qualified to achieve the proposed aims.5 Good
6 Satisfactory
7 Fair
The investigators do not appear to have adequate qualifications to achieve the proposed aims.
8 Marginal
9 Poor
![Page 9: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/9.jpg)
NIH-provided Word Template
• Specified template provided that reviewers were to use
• “Limit text to ¼ page” for each of the criterion areas– Significance– Investigators– Innovation– Approach– Environment
![Page 10: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/10.jpg)
Reviewer Advantages
• Broader range of scores and more descriptions made it somewhat easier to rank applications within each criterion
• Identifying strengths and weaknesses made the review more focused
• Review was easier to write in some ways—bullet points of strengths and weaknesses
![Page 11: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/11.jpg)
Reviewer Disadvantages
• Impact/priority score was still difficult for some applications (e.g. very experienced team, well funded in the past, application extended the work somewhat but not in a very exciting way OR application from relatively new investigator that may move field forward but approach was not precise or defined)
• Worries about the scope and extent of comments being given with the suggested limitations in the review length
![Page 12: The New NIH Review System: Reviewer’s perspective Liz Madigan, FPB School of Nursing](https://reader035.vdocuments.us/reader035/viewer/2022081007/56649d145503460f949e902c/html5/thumbnails/12.jpg)
Reviewer Recommendations
• Need to indicate for the applicants which weaknesses are minor, moderate and major so the applicants can revise accordingly—we were not forced to do this in the review so I worry that the applicants are getting unprioritized comments
• Recommendations for ¼ page of text is difficult and may not result in higher quality reviews, applications and revisions