Session Outline• Overview of session (5 minutes)

• Receive abbreviated version of the training given to typical grant reviewers (40 minutes)

• Break! (15 minutes)

• Read mock evaluation criteria and applications, score, and write comments (30 minutes)

• Hold mock panel reviews (30 minutes)

• Report out on panel scores and experiences (15 minutes)

• Discuss lessons learned and tactics for sustainability (45 minutes)

2

Overview• The goal of this session is for you to

decide what tactics would be the most efficient and effective to sustain the innovative and critical services that you’ve identified.

• As you listen to the training, read the mock application, write comments, and think about what evidence you would provide to justify sustaining each part of your program.

3

Mock Reviewer Training Overview• The following is a very abbreviated version of typical

training that reviewers receive prior to reading, scoring, or paneling an application.

• There are numerous protections in place for confidentiality, conflict of interest, and quality control, which we won’t be covering today for the sake of brevity.

• Application review processes can vary dramatically between funding organizations (government, business, foundations, etc.).

• The lessons learned from this process can help you articulate a clear plan to many potential stakeholders for how and why your program should be sustained.

4

Review Process1. Independently read and score the mock application

according to the evaluation criteria.

2. Write evaluative comments that support and justify your scores.

3. Do not discuss the application, scores, or your comments with anyone until steps 1 and 2 are complete.

4. Compare scores and comments in panels. Come to a consensus. Revise scores and comments as needed. Scores and comments do not need to be the same, but they cannot conflict with each other.

5. Comments reviewed by Panel Managers.

6. Prepare to report out on your experiences and comments.

5

Roles and Responsibilities• Reviewer: Reads, scores, writes comments, and

participates in panels.

• Chairperson: Reads, writes and compiles comments, facilitates panels, and creates summary reports. Chairperson does not score.

• Panel Manager: Checks summary reports of comments, scores, and answers questions related to process or evaluation criteria. Panel Manager does not read applications, does not score, and is not part of panels.

6

How to Read an Application1. Read the evaluation criteria and the mock

application.

2. Score the application according to the evaluation criteria and nothing else.

3. Never contact the applicant or any other outside source. Do not discuss the application with anyone. Do not use the internet, personal bias for or against the applicant, or any external source of information.

4. Every score must be fully supported and justified by detailed written evaluative comments.

7

What is an Evaluative Comment?

• Evaluative comments assess the value, worth, or quality of the information in the application.

• An evaluative comment describes the strengths and weaknesses of an application and justifies the scores given to an application.

• Think about what evaluative comments would be written about your program. What rationale and evidence do you have to support your program?

8

Why are Evaluative Comments Important?

• They determine scores. If a score isn’t supported by evaluative comments, it is sent back to the panel.

• They assist in determining funding and explain and validate scores to decision makers.

• They can be used for technical assistance to applicants, both funded and unfunded.

9

Evaluative Comments• A comment must be either a strength or a weakness.

• An individual comment can never be both.

• Avoid conjunctions such as but, although, however, and yet.

• Strength comments and weakness comments can never conflict with each other.

• If a particular element of an application has both strengths and weaknesses, you must write a separate comment for each, and they must be carefully written so as not to conflict with each other.

• When thinking of how you would write about your own program, if a reasonable person would have to parse what you’ve written into separate strength and weakness comments, you need to provide more evidence or change your approach. 10

Evaluative Comments (continued)

• A high scoring application should have many strength comments and few or no weakness comments.

• A perfect score should have no weakness comments. Anything less than a perfect score must have at least one weakness, and any application with a weakness comment must have points deducted.

• Don't simply restate what the applicant has written; evaluate what it says.

• Every comment should include a page and paragraph number citation.

11

Evaluative Comments (continued)

• Cite the rationale and evidence that the applicant provides to support your evaluative comments. If these don’t exist or are unconvincing, explain why.

• Assume that comments are being read by someone who has never read the application. A person reading your comments should understand the full story of what the applicant is proposing to do and why it should or should not be funded.

• Use complete sentences, proper grammar, and correct spelling.

12

Evaluative Comments (continued)

• Make comments tactful and constructive.

• Ask yourself, “Will my comments help the applicant to prepare a better application next time?”

• This is particularly important for weakness comments. Don’t just say that something isn’t there or isn’t good enough – explain why.

• Think about how you would explain it for your own program.

13

Evaluative Comments (continued)

• “The applicant uses appropriate outcome measures” is not an evaluative statement. You need to answer why the outcome measures are appropriate or inappropriate.

• Avoid personal pronouns and comparisons.

• Don’t write, “I think that this is bad. My program does it better.”

• Instead write, “The application states… This is ineffective because… A superior practice would be…”

14

Sample Strength Comments• Poorly written: The applicant demonstrates an

understanding of the needs of low-income adults. (Page 3, paragraph 1)

• Well written: The applicant clearly demonstrates evidence and understanding of the changing needs of low-income adults and their families by referencing the increasing numbers of families struggling with educational and emotional issues and developing relevant goals, objectives, and activities. For example… (Page 3, paragraph 1)

15

Sample Strength Comments (continued)

• Poorly written: The applicant’s needs assessments and focus groups demonstrate support for their proposal. (Pages 5, paragraph 2)

• Well written: The applicant carried out a series of assessments and focus groups with community partners. The results demonstrate strong community support for their proposal for a career pathway program to train nurses and the need to provide those students with support services, such as child care and transportation assistance. (Pages 5, paragraph 2)

16

Sample Weakness Comments• Poorly written: The objectives and the work are

too ambitious to accomplish. (Page 1, paragraphs 2-3)

• Well written: The applicant proposes to train 1,000 nurses in the first twelve months of the project. Their institution currently trains 100 nurses per year, and the labor market information provided shows that there are only 500 new job openings for nurses each year. Therefore, the objectives for the project are too ambitious given the program’s capacity to serve students and the community’s capacity to employ them. (Page 1, paragraph 2-3)

17

Sample Weakness Comments (continued)

• Poorly written: The applicant provides students with too many supportive services. (Page 2, paragraphs 1-2)

• Well written: The applicant does not provide a rationale or evidence to justify the number or type of supportive services they propose to provide, or how they will improve the program’s completion and employment outcomes. (Page 2, paragraphs 1-2)

18

Avoid Subjective and Personally Biased Comments

• “My agency would…”

• “They didn’t answer the criterion completely, but I know how this program works…”

• “I gave them a score of 15 out of 20. I thought it was a good application but not great.”

• “I don’t like what they are proposing…”

• “This is the worst application I’ve ever seen. What were they thinking?”

19

Next Steps• Break! (15 minutes)

• Read mock evaluation criteria and applications, score, and write comments (30 minutes)

• Hold mock panel reviews (30 minutes)

• Report out on panel scores and experiences (15 minutes)

• Discuss lessons learned and tactics for sustainability (45 minutes)

20

Break (15 minutes)

Read Mock Applications, Score, and Write Comments

(30 minutes)

Mock Panel Reviews (30 minutes)

Report Out on Panel Experiences

• What was your panel’s average score? Did you have difficulty coming to consensus, and why?

• What have you learned?

• What implications does this have for your program and your future funding strategy and marketing?

24

Key Resources

• University of Kansas Toolkits:http://ctb.ku.edu/en/toolkits

• A Financial Sustainability Resource Guide

25

Tactics for Sustainability

• Develop a funding strategy.

• Set realistic funding goals.

• Create an approach.

• Leverage your network.

26

Develop a Funding Strategy• Applying for publically available funding is

extremely time-consuming and highly competitive.

• Typically only very high scoring applications win an award.

• It’s better to write a smaller number of very strong applications instead of a large number of poorly written ones.

27

Develop a Funding Strategy (continued)

• As part of your preparation, we asked you to create a list of your program’s partners, employers, foundations, and institutional funders.

• Create a financial sustainability committee composed of key partners and stakeholders.

• Who would be on your sustainability committee and why?

• Have your committee hold mock panel reviews for your applications before submitting them, and involve them in all aspects of sustainability planning. 28

Set Realistic Funding Goals• Come up with a realistic business plan to

accomplish attainable goals.

• Know the cost to providing each of your services per person so it can easily be adjusted to meet different funding opportunities.

• Have evidence to support why funding each service accomplishes your goals so you can clearly explain the cost/benefit calculation for investing in your program.

29

Create an Approach• Have a short, easy-to-understand

document that outlines your program to include in outreach to funders and to build upon for potential applications.

• Think about your purpose, audience, message, delivery, and relevance.

• Make sure your program has the capacity to deliver the services you are promising before you apply for funding.

30

Leverage Your Network• Different organizations can apply for

different types of funding and support a variety of different missions, while still sharing common goals.

• Reach out to your community, build relationships with individuals, and build partnerships with organizations.

• Build a mutually beneficial relationship.

31

Identify Potential Funders• Government Funding• Federal Opportunities• State Opportunities• Local Opportunities• Private Businesses and Corporations• Volunteer Programs• Matching Initiatives• Impact Investments• Foundations• Fees for Goods or Services

32

Other Tactics• Avoid charts, graphs, and graphics

whenever reasonably possible, unless the evaluation criteria specifically calls for it. They can be difficult for a reviewer to translate into a written comment, illegible when photocopied, and difficult to read for reviewers with poor eyesight.

• Assume that the applicant cannot or will not click through hyperlinks to websites or other online materials.

33

Other Tactics (continued)• Use internal citations or footnotes. Avoid

endnotes or a bibliography unless it is specifically required.

• Internal citations and footnotes occur on the same page as the citation and allow the reader to easily see and substantiate the applicant’s claim.

• Endnotes and bibliographies obscure citations and are often skimmed or skipped by reviewers.

• Make sure there is real substance to any claim you make so that an expert reviewer can’t take issue with it.

34

Other Tactics (continued)• Applications should lay things out in the same

order as the evaluation criteria. Make it as easy as possible for reviewers to follow so they can easily map different parts of your application to comments.

• Use plain language so novice reviewers understand everything you’ve written.

• Don’t assume your reader understands the subject matter. Explain it in clear terms.

• Avoid jargon, acronyms, abbreviations, etc. (unless they are required).

35

Feedback• Tomorrow we’ll be addressing Business

and Marketing Plans.

• Any feedback on the Resource Guide?

• Do you have any insights from your mock panel experiences that can be applied to your sustainability tactics?

36

Discussion and Lessons Learned