PowerPoint Presentation

Panel Reviewer TrainingOverview1

{Greeting from the Commissioner}

We want to thank you for expressing an interest in being a part of ANAs 2013 Panel Reviews, and we look forward to working with you during this application review season. This online pre-requisite training is one of the initial steps to becoming an ANA panel reviewer for 2013.

In addition to submitting an application, we would like to make sure that all reviewers have a basic understanding about the panel review process for this year. This training will give you an overview of ANAs funding process, the roles and responsibilities of those involved in panel review, technological requirements, time commitments, and expectations.

This year, ANA will be holding a completely offsite panel review for all of the program areas. Even if you have participated in ANAs panel review process before, there will be some new expectations for this year, due to this change.

There will be a short survey at the end of this training that will address your role in Panel Review.

Thank you again for expressing an interest in becoming an ANA panel reviewer!ANA Objective Panel Review ProcessEach year, ANA convenes panels of experts to objectively analyze and score eligible grant applications. These scores are used to rank eligible applications which help ANA decide which eligible tribes or native organizations will receive funds. Being a panel reviewer is an integral part of ANAs funding process and affords participants the opportunity to have a significant impact in native communities.

2

Each year, ANA convenes panels of experts to objectively analyze and score eligible grant applications. These scores are used to rank eligible applications which help ANA decide which eligible tribes or native organizations will receive funds. Being a panel reviewer is an integral part of ANAs funding process and affords participants the opportunity to have a significant impact in native communities.

2Produced by the Administration for Native AmericansANA awards grants The Funding ProcessProvide panel commentsto applicantsOffer Technical Assistance to unfunded applicantsANA screens applicationsANA reviews applications in funding rangeReviewers analyze and score applications, facilitators summarize comments.3ANA Publishes Funding Opportunity Announcement

3In order to demonstrate how it all fits together, well start by giving you an overview of the funding process.

The process begins with the publication of the Funding Opportunity Announcement, or FOA. The FOAs include: details about the funding that is available, the program areas for which funding is being offered, how a proposal should be put together, and the criteria that will be used to evaluate the proposal.

Once applications are received, ANA screens all applications in order to ensure they are qualified to be entered in to the funding competition.

Applications are then distributed to the panel participants who analyze, discuss, and score applications based on evaluation criteria in the FOA.

Applications are then ranked according to the scores given by the panels. This is what makes panel review so critical.

The contribution of the panel reviewers does not end with funding. Panel scores and comments are provided to unfunded applicants along with the opportunity to work with ANA technical assistance providers to enhance their application for potential future competitions. The comments provided from the panel are a critical part of the technical assistance provided, and provide the foundation for the applications potential enhancements.The Review ProcessApplication Review Individual review and analysis of all assigned applicationsComments and scores entered into ARMPanel Meetings (Conference Calls)Group discussion focused on noted differences in scoring and analysisPanel Summary ReportsConsolidated comments and scores are approved by ANAProduced by the Administration for Native Americans

4

Now that you have an idea about how panel review fits into the overall funding process, lets talk about the panel review process. All panel reviews for this year will be conducted off-site.

Applications will be assigned to panels, and will be available in the Application Review Module, or ARM system. Each panel will consist of 3 reviewers and one facilitator who are all assigned the same set of applications. All panel members will read each assigned application. The panel reviewers will provide a score for each application, and develop comments that support their score and demonstrate their analysis. Comments and scores are then submitted to the facilitator in the ARM system.

Once individual panel reviewers have scored and commented on an application, the panel members will participate on scheduled conference calls to discuss their analysis and scores. Panel conference calls will be led by the facilitator, who will focus the discussion on any disparities in scores and analysis in order to prevent contradictory feedback to the applicant. During panel discussions it will be important to listen to each other and stay flexible to ensure the best review of applications.

A panel summary report will be created by the facilitator for each application. This document will identify each panel reviewers score and the average for the application. Additionally it will include a summary of comments, compiled from individual reviewers comments, to demonstrate the analysis and support the score.

Panel summary reports are sent from the facilitator to the Subject Area Manager or SAM and the Program Area Manager or PAM for review and approval. ResponsibilitiesProduced by the Administration for Native Americans

5Reviewer-Read and understand FOA for assigned program area-Analyze the applications based on ANA Evaluation Criteria. Score applicationsSupport score by commenting on strengths and weaknesses in applicationParticipate in panel review discussions.FacilitatorRead and understand FOA for assigned program area Review applications Develop and maintain application review schedule. Facilitate application discussion, ensure participation and guide discussion. Create panel summary report Facilitate feedback and communication between SAM and panel reviewers. PAMReview applicationsApprove panel summary report- Approve Final Scores- Provide overall guidance and feedback to SAMSAMReview applicationsProvide FOA guidance to the Facilitator and reviewers- Review panel summary reportsProvide overall guidance and feedback to facilitatorFacilitate feedback and communication between PAM and facilitators

Now that we have explained the panel review process, lets go over in more detail the roles and responsibilities. There are four key roles in the Panel Review process. Each panel has three reviewers, one Facilitator, one SAM and one PAM. Each SAM will participate on two panels simultaneously, and PAMS will participate on four panels. There are many key players that make panel review a success and again we are grateful for your assistance this year.

Essential tasks for the reviewer include; reading and analyzing all assigned applications , providing initial scores and comments based on the evaluation criteria, participating in discussions during panel review conference calls, and preparing revisions to the comments and scores based on the panel team feedback.

The facilitator on each panel will read applications but will not provide scores or analysis. Responsibilities of the facilitator include being a resource to panel reviewers, reviewing each panel reviewers comments prior to the panel review conference call, providing feedback to each reviewer, facilitating the panel review conference call to ensure each reviewer is heard and discrepancies in analysis, scoring, and comments are addressed, as well as creating the panel summary report. Facilitators will also be the liaison between the reviewers and the SAMs (Sub Area Managers).

The SAM responsibilities include reviewing and providing feedback to the facilitators on the panel summary report to ensure all summaries include constructive comments reflective of the reviewers analysis and score. The SAM also serves as a resource to the facilitators.

PAM responsibilities include reviewing applications, providing feedback and approving the panel summary reports. The PAM also serves as a resource to the SAMs.

Roles6PAMSAMReviewerFacilitatorProduced by the Administration for Native Americans

As you go through the panel review process it is important to remember your point of contact for panel review. As a reviewer, your facilitator is your direct point of contact for questions or concerns. For facilitators, your point of contact is your SAM. For SAMs the point of contact is your PAM.

6ExpectationsAll reviewers and facilitators are expected to take leave from work to dedicate a full week to this assignment. The responsibility could range from 8-12 hours per day, and may include some reading over the weekend.Read and understand ANAs FOA.Complete all scheduled Panel Review and ARM virtual training in advance of the panel sessionRead all applications and submit comments based on the schedule provided by the Facilitator.Evaluate and score all applications according to the criteria provided.Participate in panel discussions for all applications.Maintain objectivity during review of applications.Respond in a timely manner to all e-mail and phone correspondence from the Facilitator, SAM or PAM.Be respectful and considerate of fellow panel reviewers and facilitatorProduced by the Administration for Native Americans

7

As we continue, lets now discuss the expectations for all reviewers and facilitators.

Due to the amount of work expected during the week, it is crucial that all reviewers and facilitators take leave from work to dedicate a full week to this assignment. Although you will be working from home as this is an off-site responsibility, the workload and commitment are the same.

Prior to reviewing applications, all reviewers must thoroughly read the Funding Opportunity Announcements (FOAs). The FOAs can be accessed on the ANA website.

In order to be utilized as a reviewer or facilitator, each consultant must participate in the required training offered by ANA. This will be discussed in more detail in a few minutes.

Panel Reviewers are expected to read and score all assigned applications based on the conference call schedule provided by the Facilitator for your panel. These conference calls will provide an opportunity to discuss scores and comments according to the evaluation criteria provided for each FOA. Each application will require approximately 6-8 hours of reading and scoring time to provide an accurate and complete review.

During this time you will be expected to attend all panel sessions via conference call and participate constructively in all panel discussions. All panel reviewers are expected to be responsive to their facilitators, and facilitators are expected to be responsive to their SAMS.

It is also crucial that reviewers maintain objectivity when reviewing applications.

What is it really likeTrainings get everyone geared up.First application is exciting.The week gets busy!Long phone conversations.Learning and capacity building.There will be revisions.The process can be intense, but is rewarding.

8

Now that you have a basic introduction to ANAs funding and panel review process, it is important you also truly understand what it is like to participate in panel review.

ANA strives to support all reviewers and facilitators to be successful during panel review. Trainings are developed with the reviewers and facilitators in mind and are meant to inspire and motivate everyone.

With each application, you will have an opportunity to read about the exciting projects being proposed. There will be long phone conversations and revisions to your comments, but view this as an opportunity for learning and capacity building. At the end of the week, reviewers will have the rewarding experience of supporting ANA and the native communities being served through these grant opportunities. 8Technology RequirementsInternet connectionComputerPrinterScanning abilityTelephone and/or VoIP capabilities (conference calls may last 1-2 hours per application. Unlimited calling plans are preferred if using mobile phone)

9

Produced by the Administration for Native Americans

In order to participate in Panel Review there are technology requirements. You are required to have an internet connection and computer as the entire panel review process is completed using the Application Review Module which is an online system. Reviewers and facilitators will utilize this system to enter their scores and comments and to complete the panel summary reports. At the end of the panel review session, you will be required to print out the panel summary report signature page to sign, scan and then email to ANA so you must have printing and scanning capabilities to complete this task. Each panel will be required to conduct discussions on each application they have read and as this panel review will be conducted off-site, it is important you have telephone access or Voice over IP capabilities such as Skype or Google Plus in order to participate in these discussions. Conference calls may last 1-2 hours per application. If you are using a mobile phone, it is recommended you have an unlimited calling plan. ANA will not reimburse panel review participants for telephone costs.

These items will be necessary to fulfill your duties as a panel reviewer. Again, if you dont have any one of the items please let ANA know immediately.

9Four Steps to become an ANA Reviewer or FacilitatorStep 1- SelectionSubmit a reviewer profile form , resume and writing sample.Complete required online pre-requisite training. Complete the Pre-Requisite Training Commitment survey ANA staff review profiles and writing samples. Confirm completion of the pre-requisite training and that the potential reviewer has submitted the Expectations document. ANA determines the number of applications submitted to confirm the number of reviewers required for each panel review session. Step 2 -NotificationANA will send out notifications letting you know if you have or have not been selected as a reviewer. Those selected will be asked to participate in additional required training in preparation for the start of panel review.

Produced by the Administration for Native Americans

10

ANA is committed to selecting reviewers and facilitators who have the knowledge and experience working in Native communities as these perspectives are invaluable in the review of applications received by ANA. ANAs Panel Reviews could not be completed without your commitment and participation.

The scores provided by the panel determine the priority ANA will use to award the limited funding that is available each year. The ranking list arranges applications from highest to lowest, based on panel reviewer scores. ANA staff then begin at the top of the list and review applications for eligibility and allowability of funds. Applications are reviewed individually, in order of their scoring rank until all available funds have been awarded. Because of the importance your scores and comments have in determining the applications ANA will fund, it is important ANA chooses the most qualified reviewers and facilitators for each panel review session.

The first step in the process is the selection of reviewers and facilitators. Any person interested in becoming a reviewer or facilitator is required to submit a reviewer profile form, a resume, and a writing sample. You are then also required to take this training. Once you have completed these steps, ANA staff then review all of the profiles, resumes, and writing samples submitted to develop the initial list of recommended reviewers and facilitators. This list is then submitted to the Director and the Commissioner for their approval. Before a person can be added to the list of potential reviewers and facilitators, they must also complete the Pre-requisite Training Commitment survey at the end of this training confirming their understanding of their role and the expectations ANA has for them.

ANA then notifies reviewers and facilitators if they have or have not been selected to participate in the panel review sessions. Those selected will be asked to complete additional required training prior to the start of panel review. ANA does not know how many applications will be submitted for each panel review session. Based on historical trends, ANA has estimated an anticipated number of applications that will be submitted and this number ANA determines the anticipated number of reviewers and facilitators that will be needed. Because of this, not every reviewer and facilitator originally chosen is selected for panel review. This does not mean you were not qualified but that ANA ended up needing less reviewers and facilitators due to the number of applications that were received.

Step 3 Required trainingProduced by the Administration for Native Americans

11TrainingTopicsParticipants1. Panel Review Training Part 1Overview of the FOA, how to analyze applications, and the panel review processPAMs, SAMs, Facilitators and Reviewers2. Panel Review Training Part 2Writing comments and scoring applicationsPAMs, SAMs, and Reviewers (Facilitators are optional)3. Facilitator TrainingFacilitating discussions, managing workloads, communication, developing panel summary reportsPAMs, SAMs, and Facilitators4. ARM Training (completed on your own via the Panel Review website)How to utilize the web-based ARM systemFacilitators and ReviewersANA staff will confirm all reviewers and facilitators participation in all required trainings.

All panel reviewers and facilitators are required to complete training prior to participating in the panel review. Reviewers and facilitators will be required to participate in Panel Review Training Part 1. The focus of the Panel Review Training Part 1 will be an overview of the Funding Opportunity Announcement, how to analyze applications, and an overview of the panel review process. Reviewers will also be required to participate in Panel Review Training Part 2. The focus of the Panel Review Training Part 2 will be on how to write comments and score applications.

Facilitators will be required to participate in the Panel Review Training Part 1 session and the Facilitator Training session. The focus of the Facilitator Training session will be on how to facilitate discussions, manage workloads, communicate with your panel and ANA staff, and how to develop the panel summary reports.

The format for these training sessions will be Webinars and they will be conducted the week before the panel review session begins. These trainings will take approximately 1-4 hours to complete.

As stated in the previous slide on technology requirements, ANA will utilize the web-based Application Review Module also known as the ARM system. This system is how reviewers will input their scores and comments that the facilitator will then summarize in to one panel summary report for each application. The ARM Training will be accessible online on ANAs website prior to the remote review of the applications. There is ARM training specifically for reviewers as well as for the facilitators. Please note the ARM system uses the term chairperson for the position ANA calls the facilitator.

Step 4 Panel AssignmentFacilitators and Reviewers will be assigned to panels. You will be notified of your panel number and the applications assigned to you. Facilitators and Reviewers will be required to complete Conflict of Interest Forms for applications assigned to you.Each person will be asked by ANAs contractor, Tribal Tech to complete a W-9 form for reimbursement and payment.

Produced by the Administration for Native Americans

12

Once you have completed all trainings, you will be ready to participate in ANAs panel review. Right before the panel review session begins, you will be assigned to a panel. You will be notified of your panel number and the applications assigned to you at this point.

You will then be asked to review the applications assigned to you to confirm you have no conflicts of interest. You will also be asked to complete a W-9 form in order to be paid for your work during the panel review session. At this point, you will now be prepared and ready to participate in the panel review session.

ANA greatly appreciates the time, commitment, and effort you give in order for panel review to be a success. We look forward to working with you throughout this process. If you have any questions or concerns, please contact the ANA Help Desk at 877-922-9262.

Please proceed to the next slide and click the link for the Survey Monkey to complete the Pre-requisite training commitment survey. It is important you complete the survey in order to be considered as a potential reviewer or facilitator.

You have now completed the Online pre-requisite trainingPlease follow this link to complete a short survey verifying your completion.13https://www.surveymonkey.com/s/ZLJPJXG13