![Page 1: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/1.jpg)
Using sound evaluation practices to support your project’s success
Evaluation for VISTAs
By Sara McGarraughImprove Group Research Analyst
![Page 2: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/2.jpg)
Let’s get to know each other
![Page 3: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/3.jpg)
Overview
Purpose and benefits of evaluationEvaluation tools that you can useDesigning an evaluation
![Page 4: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/4.jpg)
“In reality, (nonprofit) performance is all about
translating caring, believing, and compassion into results.”
Letts, Ryan & Grossman High Performance Nonprofit Organizations
1999
What is evaluation?
![Page 5: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/5.jpg)
What is evaluation?
Assessing strengths and weaknesses …to improve
effectiveness“
”
![Page 6: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/6.jpg)
Why Evaluation?
![Page 7: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/7.jpg)
Demonstrate results
*Fake data is used to create this dashboard
![Page 8: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/8.jpg)
Reflect
![Page 9: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/9.jpg)
Create a learning community
![Page 10: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/10.jpg)
Influence others
![Page 11: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/11.jpg)
Engage stakeholders
![Page 12: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/12.jpg)
Plan for sustainability
![Page 13: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/13.jpg)
What are the benefits of evaluation?
Knowledge and reflectionInformation to share with stakeholdersGood evaluation practices are also good program management practices
![Page 14: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/14.jpg)
Barriers to Evaluation
Humans resist changeWe want to get alongEvaluation might suggest we are doing something wrong or would need to changeTime and cost
![Page 15: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/15.jpg)
The Role of Evaluation Formative evaluation: How is the program delivered and is it meeting expectations?
![Page 16: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/16.jpg)
The Role of Evaluation
Summative evaluation: Did the program meet its goals?
![Page 17: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/17.jpg)
The evaluation process
![Page 18: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/18.jpg)
How are project and evaluation phases similar?
![Page 19: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/19.jpg)
PUTTING EVALUATION TO PRACTICE
![Page 20: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/20.jpg)
Start up: Define PurposeWho are the key stakeholders of the evaluation? What you hope to learn as part of the evaluationPrioritize competing interests
![Page 21: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/21.jpg)
Design & Planning: Refine goals
What are your program’s aspirations (your goals)?Goals should be SMART
Specific
Measureable
Attainable
Realistic
Time-bound
SMA
RT
![Page 22: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/22.jpg)
Design & Planning: Define project outcomes
What is the
CHANGE you wish
to see?
![Page 23: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/23.jpg)
PRACTICE: REFINING GOALS AND OUTCOMES
![Page 24: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/24.jpg)
Logic models: a basic evaluation tool
A Logic Model can describe how your program’s goals and
activities lead to results and how to measure them
![Page 25: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/25.jpg)
How are they useful?
Help get everyone on the same pageEncourage investment and buy-inFacilitate organizationProvide clear and concrete guidelinesServe as a roadmap throughout evaluation processUseful for grant proposals
![Page 26: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/26.jpg)
What does it show?
GOALS ACTIVITIES
OUTPUTS
INTENDED OUTCOME
S
MEASURES
![Page 27: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/27.jpg)
DefinitionsActivities:
What your program doesOutputs:
Countable products showing an activity occurred (evidence)
Outcomes: Benefit received from your programKnowledge, behavior, condition changes
![Page 28: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/28.jpg)
Example
GOAL:For all students
to be at or above their
grade reading level
ACTIVITIES:-Tutoring-Theater-Book club
OUTPUT: # of tutoring sessions# of books/articles read
INTENDED OUTCOMES:-Students are more confident in reading-Students think reading is fun
MEASURES:-Survey
-Check-in form
![Page 29: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/29.jpg)
An alternative to a logic model
![Page 30: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/30.jpg)
PRACTICE: LOGIC MODEL DEVELOPMENT
![Page 31: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/31.jpg)
BREAK
![Page 32: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/32.jpg)
DESIGNING DATA COLLECTION TOOLS
Is it a tool, instrument, or protocol?
![Page 33: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/33.jpg)
Design measures and toolsWhat type of evidence will demonstrate outcomes?Design evaluation tools
![Page 34: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/34.jpg)
Traditional data-gathering strategies
SurveysTests
Focus GroupsInterviewsAdministrative data
![Page 35: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/35.jpg)
Consider creative ways to gather data
![Page 36: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/36.jpg)
Matching evaluation tools to your work
Use available resourcesUse knowledge of staffHonor the wisdom of all of your stakeholders Supported by infrastructure
![Page 37: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/37.jpg)
PRACTICE: DATA COLLECTION TOOL DESIGN
![Page 38: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/38.jpg)
Say what? If the question is a scale…
“How valuable was your involvement with Sample Program?”
a) Very valuableb) Somewhat valuablec) Not valuable
![Page 39: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/39.jpg)
65% of all respondents found their involvement with Sample Program to be very valuable – OR – Over half of respondents found their involvement with Sample Org to be very valuable.
80% of all respondents found their involvement with Sample Program to be somewhat or very valuable.
Then, you can report outcome statements like:
![Page 40: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/40.jpg)
Say what? If the question is open-ended…
“What was the most valuable part of volunteering for Sample Program?”
![Page 41: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/41.jpg)
Respondents frequently reported community engagement, connecting with participants, and learning about issues in the community as the most valuable part of volunteering with Sample Program.
Then, you can report outcome statements like:
![Page 42: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/42.jpg)
Say what? If the question is a retrospective pre-test…
“Please rate your opinion of the importance of volunteerism before AND after participating with Sample Program.”
Before participation After participationVery
importantSomewhat important
Not important
Very important
Somewhat important
Not important
![Page 43: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/43.jpg)
“85% of respondents rated volunteerism as very important after participating with Sample Organization compared to only 42% before participating.”
Then, you can report outcome statements like:
![Page 44: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/44.jpg)
Types of questions
What are the pros and cons of these question types?
![Page 45: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/45.jpg)
Photo by Heather McQuaid
Focus on simple and effective
![Page 46: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/46.jpg)
Use parameters to make responding easier
Photo by Oli Shaw
![Page 47: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/47.jpg)
Avoid double-barreled questions, using jargon, or vague questions
Photo by Nate Bolt
![Page 48: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/48.jpg)
PRACTICE: SURVEY OR INTERVIEW QUESTION DESIGN
![Page 49: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/49.jpg)
Implementation: Data gathering
Smile!Practice in advanceHave a contact person for questionsRespect time & privacyCreate instructions
![Page 50: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/50.jpg)
Survey administration
TimingResponse rates
![Page 51: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/51.jpg)
Reflection: Share lessons and refine practices
Meet with the program teamPrepare report(s)Distill key findingsProvide recommendations/ suggestions/areas for consideration
![Page 52: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/52.jpg)
Evaluator’s ToolkitA logic model or a questions and sources gridA working knowledge of different methods
A network of people who can help you
![Page 53: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/53.jpg)
Create an evaluation plan that…
Addresses stakeholders’ needsIs validContributes to your organizationCan be implemented and takes into account available resources
![Page 54: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/54.jpg)
![Page 55: Using sound evaluation practices to support your project’s success](https://reader036.vdocuments.us/reader036/viewer/2022062501/5681639b550346895dd4938c/html5/thumbnails/55.jpg)
Questions or Comments?Sara McGarraugh
Continue the conversation
[email protected] www.theimprovegroup.com
Or find us on Facebook and LinkedIn Thank you!