evaluation – the why’s, the what’s and the how’s 2014 dr basma ellahi (pi) cat crum (phd...
TRANSCRIPT
Evaluation – the why’s, the what’s and the how’s
2014
Dr Basma Ellahi (PI)Cat Crum (PhD Student)
What is evaluation?
• Evaluation is a process that critically examines a program. It involves collecting and analysing information about a program’s activities, characteristics, and outcomes. Its purpose is to make judgments about a program, to improve its effectiveness, and/or to inform programming decisions (Patton, 1987).
Why evaluate?
• Answer the objectives of the scheme/project
• Demonstrate change due to the scheme/project
• Determine effectiveness i.e. causality or association
• Inform improvements
‘dull but important
When compared with the creative and exciting process of conceiving and initiating a project, evaluation can often be forgotten and be perceived as dull! However, some form of evaluation or formative feedback is the only thing that will show the effectiveness of the project
• Evaluations fall into one of two broad categories: formative and summative.
• Formative evaluations are conducted during program development and implementation and are useful if you want direction on how to best achieve your goals or improve your program.
• Summative evaluations should be completed once your programs are well established and will tell you to what extent the program is achieving its goals.
Type of Evaluation Purpose
Formative
1. Needs Assessment
Determines who needs the program, how great the need is, and what can be done to best meet the need. A needs assessment can help determine what audiences are not currently served by programs and provide insight into what characteristics new programs should have to meet these audiences’ needs.
2. Process or Implementation Evaluation
Examines the process of implementing the program and determines whether the program is operating as planned. Can be done continuously or as a one-time assessment. Results are used to improve the program. A process evaluation of a program may focus on the number and type of participants reached and/or determining how satisfied these individuals are with the program.
Summative
1. Outcome Evaluation
Investigates to what extent the program is achieving its outcomes. These outcomes are the short-term and medium-term changes in program participants that result directly from the program. For example, outcome evaluations may examine improvements in participants’ knowledge, skills, attitudes, intentions, or behaviours.
2. Impact Evaluation
Determines any broader, longer-term changes that have occurred as a result of the program. These impacts are the net effects, typically on the entire school, community, organisation, society, or environment.
Adapted from Norland (2004), Pancer and Westhues (1989) and Rossi et al. (2004).
Aims of evaluation
• How has the pilot been received by the different participants in each of the health communities/
• What have been the successes and the issues – lessons learned?
• How successful has the pilot been in moving the health communities towards working on the basis of the 5 key principles?
• How successful has the pilot been in helping the health communities meet their objectives?
Evaluation Objectives of the Malnutrition Prevention Pilot Programme
• Confirming outcomes and impact indicators
• Inputs and activities
• Appraising progress
• Identify limiters
• Identify unintended consequences of implementation
• Generate evidence base
• Use self generated data
Methodology
Impact Evaluation Framework
Theory of change
Logic Model
Linkages between inputs, activities, outputs and outcomes
Evaluation Plan
• Why?
• What?
• Who?
• How?
• When?
• Where?
Complex Evaluation
Range of activities-One-off information and/or taster session -Promotional stand-Promotional Campaigns-Health days-Demonstrations-Cooks and eat-Community Enterprise-Training courses
• Combination of evaluation methods
• Qualitative
• Quantitative
The tools of the evaluator
• Quantitative• Monitoring Information• Questionnaire/survey• Experimental evaluation – RCT, case control, cohort etc
• Qualitative• Observation• Interviews• Focus Groups• Case study• Documentation
Methods• Semi-structured interviews
(telephone)
• Action Learning sessions
• Audit - baselines and MUST
• Sampling strategy – purposive sampling
• Pluralistic model
• Ethics
• Frame of reference
What does good look like?
• Good evaluation is tailored to your program and builds on existing evaluation knowledge and resources.
• Good evaluation is inclusive• Good evaluation is honest. • Good evaluation is replicable and its
methods are as rigorous as circumstances allow.
Common dilemmas
• Intellectual property rights / Data protection and data sharing
• Be conscious of multiple roles• Follow informed-consent rules• Respecting confidentiality and privacy• Ethics• Complexity of data collection sites
How do I make evaluation an integral part of my program?
• Making evaluation an integral part of your program means evaluation is a part of everything you do. You design your program with evaluation in mind, collect data on an on-going basis, and use these data to continuously improve your program.
To build and support an evaluation system:
Couple evaluation with strategic planning.
Revisit and update your evaluation plan and logic model to make sure you are on track.
Build an evaluation culture
What are the benefits?
better understand your target audiences' needs and how to meet these needs
design objectives that are more achievable and measurable
monitor progress toward objectives more effectively and efficiently
learn more from evaluation
increase your program's productivity and effectiveness
10 reasons to evaluate your project
1. So you know whether it’s working
2. So you can be adaptable
3. To know how things are working
4. So you’re aware of unintended outcomes
5. To be able to better communicate the value of your work
10 reasons to evaluate your project cont.
6. To focus your work
7. To help look after the people you work
with
8. Build organisational resilience
9. Know why things are working
10. Life is complicated
Make evaluation part of your program; don’t tack it on at the end!
Resources
• Magenta Handbook - https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/220542/magenta_book_combined.pdf
• The Magenta Book: guidance notes for policy evaluation and analysis - http://www.civilservice.gov.uk/wp-content/uploads/2011/09/the_complete_magenta_book_2007_edition2.pdf
Contact details
Centre for Ageing Studies
Faculty of Health and Social Care
University of Chester
Riverside Campus
Chester
Dr Basma Ellahi – Reader in Food and Nutrition
Email: [email protected]
Professor Paul Kingston – Director of The Centre for Ageing Studies, Professor of Mental Health and Ageing.
Email: [email protected]
Cat Crum (PhD Student – sponsored by Age UK South Staffordshire)
Email: [email protected]