community conversations research grant review the review...•external reviewer/staff reviewer...

Community Conversations Research Grant Review

Live Orientation LaunchApril 26th 2018

Agenda

• Preparing for the Review

• Panel Information

• Review Participant Roles

• Review Form

• Sample Timeline

• Expectations for Comments and Clarification

• Next Steps

• Q&A

Preparing for the Review

• All Reviewer resources are posted at:

– https://cnsgov.sharepoint.com/sites/extranet/OGPO/2018communityconversations/SitePages/Home.aspx

– Reviewers will have four mandatory trainings (calls/webinars)

• Launching the Review (current training)

• Reviewing the Applications-Part I

• Reviewing the Applications-Part II

• Ensuring an Equitable Review (External Reviewers only)

• Conflicts of Interest LMS training (Staff Reviewers only)

– Read competition documents:

• Notice of Funding

– Application Instructions

– FAQs

Preparing for the Review

• Reading Materials

• Sample Application

• Review Materials:

– Review Form & Rubric

• Other reviewer resources and administrative forms on Reviewer Resource Webpage:

– Writing Meaningful Comments document

– Conflict of Interest and Participation Agreement Forms for External Reviewers

– Conflict of Interest Guidance and Participation Guidance for CNCS Staff Reviewers

Panel Introduction emails will be sent Monday, April 30th

Approximately 10 Applications per panel

1 External Reviewer

1 CNCS Staff Reviewer

1 Program Office Liaison

1 GARP Liaison

The GARP Liaisons and Program Office Liaisons will each have multiple panels.

Review Roles

• External Reviewer/Staff Reviewer

– Responsible for reading and assessing applications, engaging in panel discussion, addressing feedback, to complete high-quality review forms

• Program Office Liaison (POL)

– Serve as a resource to the panel on programmatic elements and answers content-specific questions

• GARP Liaison

– Manages review logistics for panels, ensures reviewers have completed requirements for orientation and access to materials, verifies final reviewer documents have been completed

Review Roles

• Review Coordinators

– Program Office Lead: Andrea Robles, R&E

• POLs will be contacted as needed if specific selection criteria questions arise, programmatic questions about ratings or aspects on a particular criteria

– Office of Grants Policy and Operations Lead: Nikki Hayden and Susan Cohn, OGPO

• Manages review process

• Confirm successful review form submission

Reviewer Responsibilities

• Review and score applications against selection

criteria and other considerations in the Notice

• Produce high quality assessments of the

strengths and weaknesses of the applications

• Help inform decision makers on quality of

applications to aid in their funding decisions

IRF Development Process

Use Review Form to guide you through the criteria for assessing the application

AFTER initial IRF is completed, then meet with panel partner for Panel Discussion

Return to IRF to revise as necessary. Save IRF using Naming Convention, then send to GARP Liaison

Receive GARP Liaison feedback, revise & resubmit (if necessary) until approval is received

OGPO informs the Reviewer when Approval is granted on each application

Panel Discussions

Consensus is not required of the panel discussion.

Reviewers are asked to engage in discussion

about the Criteria and consider the assessments

and finding of their fellow panel members. The

discussion should cover each of the relevant

elements of the application.

Panel Discussion Report

Panel Discussion Naming Convention

All review forms should follow the naming convention:

Legal Applicant Name. Application ID. PDR. Reviewer First Initial Last Name. Panel

Number.

Example:

UniversityofToronto.RX123456.PDR.ASmith.1

Review Form Naming Conventions

All review forms should follow the naming convention:

Legal Applicant Name. Application ID. IRF. Reviewer First Initial Last Name. Panel

Number.

Example:

UniversityofToronto.RX123456.IRF.ASmith.1

Submission of Review Materials

Both Panel Discussion Reports as well as Review Forms should be submitted to:

PeerReviewers@cns.gov

Required Timeline Dates

Review Begins:

Monday, April 30th

Review Ends:

Monday, May 21st

All IRFs must be approved as final by OGPO by 5:00pm EST

Week 1 Sample Timeline

• Monday, April 30th:

– All applications are received

– Reviewers have completed all orientations

– Begin reading applications for COI and follow up with GARP Liaison as needed

– Reach out via email to fellow panel member to schedule Panel Intro Call

Week 1 Cont. Sample Timeline

• Tuesday, May 1st:

–Hold Panel Intro Call to establish order of reading applications and introduce panel members (this may also be covered via email exchange)

–Begin reading and completing IRF for three applications

• By Fri, May 4th

–First Panel Discussion: discuss 1-3 applications

–Send first IRF’s to GARP Liaison

Week 2 Sample Timeline

• Wed, May 9th

– Receive feedback from GARP Liaison on first IRF

– Work with liaison to ensure first set IRF is approved

– Continue reading and reviewing applications 4-6

• Thu, May 10th:

– Second Panel Discussion: Discuss applications 4-6

• Fri, May 11th:

– Send IRFs for applications 4-6 to for GL review, and revise as necessary

– Continue reading and reviewing applications 7-10

Week 3 Sample Timeline

• Tue, May 15th:

– Receive feedback from GL on applications 4-6

– Final Panel Discussion for applications 7-10

• Wed, May 16th:

– Send remaining IRFs to GL for review

• Fri, May 18th

– Receive feedback from GL on 7-10

• Mon, May 21:

– All IRFs must be accepted as final by 5:00 pm EST

Tips for Producing High-Quality Review Forms

• Address only the Selection Criteria. Do not compare with other applications

• Address only the information within the application

• Use evaluative, grammatically correct language

• Comments and ratings should be aligned

Writing Strong Comments

• Comments should align with/support Ratings

– Fair or Does Not Meet should result in a Weakness comment

– Good or Excellent should result in a Strength comment

• Comments should relate directly to the selection criteria

• Do not just reiterate the criteria or repeat summary from the application, explain how the applicant’s response to the criteria is strong or weak

• Use complete sentences

• Significant Strengths and Weaknesses are used as feedback to applicants

Writing Items for Clarification

• Allows applicants to explain aspects of the application that were not clear, not intended as an option to provide additional information that should have been included in the initial application

• Allows applicants to adjust budget or other items that must be resolved prior to award

• Reviewers should always write Clarification as a clearly articulated question or statement requesting a response

Next Steps: • Complete all Orientation Sessions/Readings

by Mon, April 30th –Verify completion at PeerReviewers@cns.gov

• Receive Panel Assignment email – (Monday, April 30th)

• Panel Intro Call/email – (Tue, May 1st )

• Review Check-In Q&A Calls – Fri, May 4th 1:00p.m. Eastern

– Fri, May 11th 1:00 p.m. Eastern

Questions?