1 evaluation as continuous improvement the health disparities service- learning collaborative...
Post on 20-Dec-2015
212 views
TRANSCRIPT
1
Evaluation as Continuous Improvement
The Health Disparities Service-Learning Collaborative
Suzanne B CashmanFebruary 27, 2007
2
“Good evaluation” is nothing more than “good thinking”
It is the systematic collection of information about activities, characteristics and outcomes of programs, personnel, and products to use to reduce uncertainties, improve effectiveness and make decisions.
Patton, 1997
3
Evaluation as Assessment/Improvement
Mechanism to tell the story Becomes less of a burdensome add-on
Useful learnings For yourself For others
4
Why Evaluate?
Reduce uncertainties Measure program achievement Improve effectiveness Demonstrate accountability Make programmatic decisions Build constituency Influence policy
5
Why are you engaged in evaluation?
6
Comparison of Academic Research and Practical Evaluation
Academic Research Practical Evaluation
Purpose Test hypotheses Improve program/ practice
Method Controlled environment
Context sensitive
Statistics Sophisticated Simpler
7
Program Evaluation
Commitment to following the “rule” of social research
But more than application of methods… also a political and managerial activity; input
into process of policy making and allocation for planning, design, implementing, continuing programs
8
Program Evaluation
Rooted in scientific methodology, but responsive to resource constraints, needs/purposes of stakeholders, and nature of evaluation setting
9
Key Questions
What is the aim of the assessment? Who/What wants/needs the information? What resources are available? Who will conduct the assessment? How can you ensure results are used?
10
Evaluation should:
Strengthen projects Use multiple approaches Address real issues Create a participatory process Allow for flexibility Build capacity
WKKellogg Foundation, 1998
11
Evaluation(should tell us……………)
What has been done How well it has been done How much has been done How effective the work/program has been
12
Reasons to Evaluate
Measure program achievement Demonstrate accountability Examine resource needs and effectiveness Improve operations, obtain and give
feedback Influence policy Expand voices
13
Evaluation Framework (CDC)
I. Engage Stakeholders
II. Describe Program
III. Focus the Design
IV. Gather Credible Evidence
V. Justify Conclusions
VI. Ensure Use and Share Lessons Learned
14
I. Stakeholders
People who have a “stake” in what will be learned from an evaluation and what will be done with the knowledge
They include: People who manage or work in the program/organization People who are served or affected by the program, or
who work in partnership with the program People who are in a position to do or to decide something
about the program CDC, 1998
15
Stakeholders
Stakeholders’ information needs and intended uses serve to focus the evaluation
Variety of stakeholders may mean: more than one focus (policy implications vs
documentation of local activities) varied levels of involvement
16
Stakeholders
Who are your stakeholders? How do their needs and desires differ from
one another?
17
II. Describe Program
Need Expectations Activities Context
18
Expectations
Outcome Objectivesstatement of the amount of change expected for a
given problem/condition for a specified population within a given timeframe
Process Objectivesstatement regarding the amount of change expected in
the performance or utilization of interventions that are related to the outcome
19
III. Focus the Design
Questions to answer Process to follow Methods to use Activities to develop and implement Results to disseminate
20
Clarify
Individual, Systems, or Community Level Individual: individually targeted services or
programs, often for people at high-risk Systems: change organizations, policies, laws, or
structures Community: focus is on community norms,
attitudes, beliefs, practices
21
IV. Gather Credible Evidence
Types of data Demographic, health status, expenditures,
quality of life, eligibility, utilization, capacity
Sources of data Statistical reports, published studies, voluntary
organizations, program reports, media articles, government reports, state surveys
22
Thinking about data
Match the data to the questions – what kinds of information would be worthwhile?
As much as possible, use data that are being created as a regular part of the program
Collect and analyze data from multiple perspectives
Keep available resources in mind
23
Thinking about data(continued)
Where might we find them? How might we obtain them? What types should we consider? What do we do now that we have them?
24
Who can help us collect and make sense of data?
Community partners Student participants College administrative offices Faculty colleagues (and their students) Students who participated in previous
programs Campus service-learning centers
25
Indicators of Well-being: Dimensions to Consider
(Cohen, MDPH)
Traditional Less Traditional
Assets Social indications, Resiliency, Quality of life,Satisfaction, Self-reports Resources & Investmentof health
Deficits Disease, Utilization of Gaps among groups, medical services Education, Economics,
Cultural, Safety deficits
26
(Cont)
Traditional Less Traditional
Assets Use of pre-natal care Quality adjusted life years Self-reported health Social networks
Screening rates Rescue response time
% insured Support for needle exchange
Graduation rate Volunteerism
Deficits Age - adjusted death rate Lack of support for arts/culture
Hospitalizations Crimes per capita
Smoking prevalence
27
Specific Data Collection Methods
Surveys Interviews Focus groups Literature search Structured observations Critical events log Institutional documentation
28
Now that we have the data…...
Analyze Quantitative (statistical software) Qualitative (systematic review and assessment)
Synthesize information Follow framework of concepts
Write reports Disseminate
29
V. Justify Conclusions
Review findings What do they mean? How significant are
they? How do the findings compare to the
objectives of the program? What claims or recommendations are
indicated?
30
VI. Ensure Use and Share Lessons
Through deliberate planning, preparation, and follow-up
Collaborate with stakeholders for meaningful: Communication of results (process and outcome) Decisions based on results New assessment plans emerging from results Reflection on the assessment process
31
Challenges
Important things difficult to measure Complexity Measurement validity Time Proof of causation Need to be sensitive to context Resources
32
Challenges
What are the challenges you face?
33
Summary: Characteristics of Evaluation
Evolving process Variety of approaches More than collecting and analyzing data Critical design issues Reconciles competing expectations Recognizes and engages stakeholders
34
References
Bell R, Furco A, Ammon M, Muller P, Sorgen V. Institutionalizing Service-Learning in Higher Education. Berkeley: University of California. 2000.
Centers for Disease Control and Prevention. Practical Evaluation of Public Health Programs. 1998.
Kramer M. Make It Last: The Institutionalization of Service-Learning in America. Washington, DC: Corporation for National Service. 2000.
Patton M. Utilization-Focused Evaluation. Sage Publications. 1997.
WKKellogg Foundation. Evaluation Handbook. 1998.