in depth panel review training. activity: mock panel review to evaluate the need for assistance,...
Post on 31-Mar-2015
213 Views
Preview:
TRANSCRIPT
Activity: Mock Panel ReviewTo evaluate the Need for Assistance, reviewers will consider the extent to which the application includes: 1. A concise problem statement that identifies the current condition(s) to be addressed by the project2. Supporting information or data detailing the scope and nature of the problem3. Current challenges standing in the way of addressing the problem4. A clear description of the community to be served and who the intended beneficiaries are
What score, out of 10 possible points, did you give the application narrative?
Why?
Panel Review Roles
Reviewer: A member of the panel charged with evaluating
the application and documenting their evaluation
by scoring and commenting on the application strengths and
weaknesses
Facilitator: A member of the panel charged with managing
the panel discussions and writing the PSR
Subject Area Manager (SAM): An ANA representative charged with assisting the panel on the process and the editing of the
PSR
Program Area Manager (PAM): An ANA representative with the
responsibility to approve the PSR
Review Director: oversee the entire panel review process
Logistics Contractor: handles payment and ARM support
Panel Review Director
PAM
SAM
Facilitator 1 3 Reviewers
Facilitator 2 3 Reviewers
SAM
Facilitator 1 3 Reviewers
Facilitator 2 3 Reviewers
PAM
SAM
Facilitator 1 3 Reviewers
Facilitator 2 3 Reviewers
SAM
Facilitator 1 3 Reviewers
Facilitator 2 3 Reviewers
Panel Review Structure
Calendar of EventsPre-Panel
(April 1st – April 27th)
Reviewers and facilitators are identified and trained
Session 1: P&M, EMI, and ERE(April 25th – May 9th)
Reviewers and facilitators evaluate and panel applications
Session 2: SEDS and SEEDS(May 9th – May 23rd)
Reviewers and facilitators evaluate and panel applications
Post Panel
Payment is sent to reviewers and facilitators
The Application Review Module (ARM)
Do Enter the Score First Use the ARM text box
to write your comments
Spellcheck Save every 20-30
minutes
Do Not× Use Google Chrome× Click the ARM support
link on the ARM login page
Email ANAReviewer@acf.hhs.gov
with all ARM questions!
Reviewers read application & evaluate using
FOA criteria
Reviewers write comments &
determine preliminary score
Panel members discuss
applications
Reviewers revise comments & scores; PSR compiled by facilitator & submitted to
SAM/PAM
Three Principles for Reviewing Applications
Evaluate the extent to which the application responds to the FOA Evaluation Criteria
The FOA Evaluation Criteria is the only measure used to evaluate the application
Do not compare applications against each other
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary score
Panel members
discuss applicatio
ns
Reviewers revise
comments &
scores; PSR
compiled by
Facilitator &
submitted to
SAM/PAM
Narrative Descriptor Score
Excellent 93-100
Very Good 86-92
Good 78-85
Fair 70-77
Needs Significant Improvement 0-69
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary score
Panel members
discuss applicatio
ns
Reviewers revise
comments &
scores; PSR
compiled by
Facilitator &
submitted to
SAM/PAM
How Do You Determine the Significance of a Weakness and a Strength?
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary score
Panel members
discuss applicatio
ns
Reviewers revise
comments &
scores; PSR
compiled by
Facilitator &
submitted to
SAM/PAM
The applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)
The applicant did not describe anywhere in the application how community input was used in the development of the project, making it unclear if the project proposal is truly community-driven. (page 25, Approach)
The applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be evaluated, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected)
Examples of Good Weakness Comments
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary score
Panel members
discuss applicatio
ns
Reviewers revise
comments &
scores; PSR
compiled by
Facilitator &
submitted to
SAM/PAM
What is a panel
discussion like?
Not everyone agrees at the
beginning
Initial scores vary.
Sometimes by 20 points or
more
Structured and focused
We spent about an hour talking
about each application
Hearing the other reviewers
point of view made me
reconsider my score
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary score
Panel members
discuss applicatio
ns
Reviewers revise
comments &
scores; PSR
compiled by
Facilitator &
submitted to
SAM/PAM
The FACILITATOR sets group norms, focuses
the meeting, asks probing questions, shares information, and documents the
discussion
Panel Group Norms• Attend all
meetings• Keep Focused• Listen and
show respect
Focus the meeting by creating a meeting plan
Probing Questions• What criteria did you use? • How did you determine this
score? • What documentation did the
application provide that substantiates the approach?
Share initial reviewer
comments
Document the discussion to help write the Panel Summary Report
Requirements of a Panel Summary Report (PSR)
1. Factually Accurate2. Adheres to the FOA Evaluation
Criteria3. Consistent4. All comments justify the score5. Polished and Well-Written
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
Example of an Inconsistency in a PSR
Strength CommentThe applicant clearly showed how the current staff will manage and complete the project on time. (page 75, Objective Work Plan)
Weakness CommentThe applicant did not provide staff time commitments, making it unclear how proper direction, management, and timely completion of the project activities would be ensured. (page 13, Approach)
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
Writing a PSR that Represents the View of the Panel
Create an initial PSRTake notes during the panel discussionDraft the final PSRSubmit the final PSR to the PAM/SAM
for comments and approvalShare the final PSR with the reviewers
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
Tips on Writing a PSR that will be Approved
Reference Application Page NumbersUse the wording of the relevant FOA
evaluation criteriaMake it grammatically correct
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
Example of a Good PSRCommentThe applicant did not provide information indicating how the lives of community members would improve as a result of the project or how the impact of the project would be measured, raising significant questions about whether the conditions identified in the Need for Assistance will be addressed. (page 18, Outcomes Expected)
FOA Text in Outcomes Expected Evaluation CriteriaTo evaluate the projects intended impact, reviewers will consider the extent to which: 1. The condition(s) identified in the problem statement will be addressed2. The lives of community members and beneficiaries will improve
To evaluate the impact indicator(s), reviewers will consider the extent to which: How impact will be measured with at least one impact indicator using the same measure at three points in time, baseline (beginning of project), end of project, and three years post-project.
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
PSR Approval
Reviewers read
application &
evaluate using FOA
criteria
Reviewers write
comments &
determine
preliminary
score
Panel members discuss applicati
ons
Reviewers revise commen
ts & scores;
PSR compile
d by Facilitat
or & submitte
d to SAM/PA
M
Yes No
Is the PSR factually accurate?
Does the PSR adhere to the FOA Evaluation Criteria?
Is the PSR Consistent?
Do the comments justify the score?
Is the PSR polished and well-written?
Approved!
Roles and Responsibilities RecapReviewer
1. Evaluate the Application2. Document evaluation by scoring
and justifying score with comments
3. Submit score and comments to facilitator through ARM
Facilitator4. Create initial PSR by compiling all
reviewer comments5. Holds panel meeting with
reviewers6. Revises PSR based on reviewer
comments, discussion or SAM/PAM recommendations
7. Submit PSR to SAM through ARM
SAM/PAM Responsibility8. Determine whether the PSR can be approved. If not, back to step 6.
Facilitator Responsibility9. Share final PSR with reviewers to ensure it reflects panel opinion.
SAM Responsibility10. Submit PSR to PAM for approval through ARM
PAM Responsibility11. Approve the PSR
top related