collaborative evaluation rita o'sullivanjohn o ’ sullivan evaluation, assessment, & north...
TRANSCRIPT
Collaborative Evaluation
Rita O'Sullivan John O’SullivanEvaluation, Assessment, & North Carolina A & T State Univ
Policy Connections (EvAP) NC Cooperative Extension School of Education Greensboro, NC Univ. of NC - Chapel Hill
[email protected] (919/843-7878) [email protected](336/285-4683)
Evaluation Steps Overview
• Introduction to Collaborative Evaluation• Collaborative Evaluation Techniques:
Collaborative Proposal DevelopmentClarifying Evaluation RequestsDesigning Collaborative EvaluationsEvaluation PlanningCollaborative Data CollectionCollaborative Data AnalysisConvincing Levels of Evidence Collaborative Reporting: Evaluation FairsCollaborative Evaluation Cycle
• Final Questions/Discussion
Starting Definition: COLLABORATIVE EVALUATION
• is an approach that engages program stakeholders actively in the evaluation process. When stakeholders collaborate with evaluators, their understanding increases and the utility of the evaluation is often enhanced.
Collaborative Evaluation as an Evaluation Approach
• Participatory Evaluation(e.g., Cousins, 1996; Patton, 1997)
• Empowerment Evaluation• (e.g., Fetterman, 2001; 2005)
• Collaborative Evaluation(e.g., O’Sullivan, 2004; O’Sullivan & O’Sullivan, 1998; Liliana Rodriguez-Campos; 2005)
Characteristics of a Collaborative Evaluation
• Stakeholders share responsibility for the evaluation
• Evaluation questions are developed together
• Evaluation expertise grows among board members, parents, teachers, service providers and other key stakeholders
• Evaluation evidence is used for program improvement
Multi-Site Annual Sequence
• Individual Site Evaluation Planning• Feedback on Evaluation Plans• Eval Plan Implementation with
Technical Assistance• Interim Sharing of progress• Technical Assistance as Needed• Evaluation Fair Sharing• Cross-Site Summaries
Advantages to Collaborative Evaluation
• Enhances the dynamics of eval, changing paradigm shift (interactions of ownership)
• External evaluators learn program dynamics
• Stakeholders understand the purposes of the evaluation
• Data collection and reporting are of better quality
• Access to additional evaluation resources
Characteristics of a Collaborative Evaluator
•Good listener•Flexibility•Understanding of evaluation
•Knowledge of the program
Collaborative Evaluation Techniques:
Developing an Evaluation Proposal
• Program Background Statement
• Evaluation Purpose• Evaluation Questions with
Proposed Information Gathering Strategies
• Timeline of Evaluation Activities
• Capability Statement• Budget
Collaborative Evaluation Techniques:
Clarifying an Evaluation Plan• Gathering information about the
programs nature and scope
• Determining the purpose of the evaluation
• Probing the resources available to conduct the evaluation
Mentoring Project
• Your are about to meet with the Director of the Mentoring Project
• You are an external evaluation consulting group, and she has asked for your help.
• Get together in small groups to decide on three questions to ask her that will clarify her evaluation needs.
Collaborative Evaluation Techniques: Evaluation Planning - Logic Models
GOALS & OBJECTIVES
ACTIVITIES RESULTS(Outcomes)
SMART:Specific
Measurable AttainableRealistic
Time-bound
MONITORING EVIDENCE(Outputs)
(Service Stats)+ Quality of
Activities
OUTCOME EVIDENCE:Immediate
IntermediateLong-Term
1. To educate teen parents about good parenting.
150 teen parents in parenting classes
Provide health clinics
Improve parenting practices (90% of participants)
Attendance lists
Class Evals/ Observations
RETRO Pre/Post Parenting Practices Surveys
Retrospective Pre/Post
• Ask respondents to tell you at the end of the program, where they were when they started and where they are now.
Program Theorizing/Theory of Change (Identify Indicators)
• Long Term: Healthy children ready for school
• Intermediate Outcome 1: Parents able to assist children with pre-school skills
• Intermediate Outcome 2: Program participants interact appropriately with children
• Immediate Outcome 2: Participants enhance understanding of good parenting
• Immediate Outcome 1: Develop and recruit program participants
Web Sites/References• www.eval.org (AEA website - Guiding Principles +…• www.wisc.ex (Ellen Taylor Powell)• www.wkkf.org (WK Kellogg Foundation - logic models + eval manual)• gi.unc.edu/research/k-12-toolkit (EvAP Evaluation Tool Kit for
International Presenter Programs)• www.gseis.ucla.edu/~srmevaluationgroup/reports/evap.PDF (Meta-
evalaution of Collaborative Evaluation of First 5 LA Childcare Initiative)• O ’Sullivan, R. G., & D’Agostino, A. (2002). Promoting evaluation
through collaboration with community-based programs for young children and their families. Evaluation, 3, 372-387.
• Campos-Rodriguez, L. Collaborative Evalaution. Tamarac, FL: Lumina Press.
• O ’Sullivan, R. G., Jay, M., & Powers Costello, B. (2006). Transcribing Focus Group Interviews: Is There a Viable Alternative for Analysis? Paper presented at the annual meeting of the American Evalaution Association, Portland, OR.
Collaborative Evaluation Techniques: Designing Collaborative Evaluations
EVALUATION QUESTIONS 1 2 3 4 5 6I. 1. To what extent have parenting classes promoted better parenting? X X X X
II. 2. How have project strengthened the capacity of rural communities? X X
IB. How can the program be improved?
Applicatio
n
Grantee Records
Evaluation Crosswalk:Foundation R
ecords
Collaborative Data Collection
• Explore information gathering alternatives with program staff
• Validate instruments by consulting with program experts, including staff
• Share summaries of information gathered with data sources and staff
Convincing Levels of Evidence
___ Newspaper story ___ Double-blind controlled ___ Comparison with group ___ A friend of a friend___ A friend ___ Description of events___ Pre-test/Post-test ___ Salesperson___ Testimonial ___ Treatment/control
group___ Celebrity endorsement ___ Expert critique___ ___
1 - NOT Convincing 2 = SOMEWHAT Convincing 3 = VERY Convincing
Collaborative Data Analysis
• Engage program staff and/or participants in data analysis
• Share data analyses with program staff
• Ask program staff for their interpretation of data findings
Collaborative Reporting: Evaluation Fairs &
Conferences
• promote evaluation
• enhance networking• suggest alternative program
ideas • allow programs to celebrate
achievements
Collaborative Evaluation and Evaluation Ethics
AEA Guiding Principles:
• Systematic Inquiry• Competence• Integrity/Honesty• Respect for People• Responsibilities for General & Public
Welfare
Collaborative Evaluation Cycle
Clarify Evaluation Request
Conduct Evaluation Fair
Design Evaluatio
n
Implement the Evaluation
COMMUNICATION
Discussion Items• Evaluator as technical assistance consultant• Staff changes create the need for training• Reluctance of program to take ownership• Reporting responsibility can become muddled.• Differences between internal and external evaluators• Objectivity of collaborative evaluators can become
suspect.• Some may expect distanced evaluation
Thank you!
• Please complete evaluation forms
and
• Contact us if we can be of assistance.