if we build a partnership will they come? integrating needs assessment, process evaluation, and...
TRANSCRIPT
If we build a partnership will they come?
Integrating Needs Assessment, Process Evaluation, and Impact Evaluation
Ronald Jester and Robert Wilson, University of Delaware
A Program Evaluation may include
• Process Indicators
or• Impact Indicators
or• A combination of Process and Impact
Indicators
A Process Evaluation Analyzes Program Inputs, such as
•Outreach Activity•Services Delivered•Adherence to Grant
Proposal•Day-by-day Staff Activity•Costs •Materials
An Impact Evaluation Analyzes Program Outputs, such as
Productivity•Farm output•Individual farmer or waterman output•Efficiency•Benefits/Costs
Client Satisfaction with services
Improvements in Quality of Life
AgrAbility evaluations may include both process and impact indicators
The Delaware Maryland Program (DMAP) employs a systems model for program evaluation
This model starts with the target population - -2400 farmers with disabilities
The model tracks the target population through a service delivery system
The system includes a number of filters that regulate the flow of AgrAbility clients
The model includes a number of efficiency measures that identify impediments to efficient client flow
The AgrAbility Process includes 5 steps:
1. Identify farmers with disabilities.
2. Diagnose their problems.
3. Identify the services that will alleviate the problems.
4. Find resources to support for the services.
5. Deliver the services.
The Delaware-Maryland project relies on “partnerships” to deliver services.
The efficiency of the project depends on
1. Locating the target population 2. Engaging the target population 3. Linking the target population with the
partners 4. Routing the clients to the partners in the
network 5. Delivering appropriate services6. Producing specific outcomes (impacts)
as a result of the services
Overall efficiency is gauged by theproportion of the target population
• Located• Diagnosed • Referred to services • Provided services • Helped by services
Areas of Evaluation Process Evaluation Impact Evaluation
Locating Clients Diagnosing Clients Referring Clients to
services Providing Services
Measuring the Impact of Services
The Model (Figure 1) portrays the flow of a target population through a series of filters.
As the target population of 2400 farmers flowsthrough the service delivery system an increasing numberdrop out at each stage.
How to interpret Figure 1 The target group is the estimated 2400 disabled farmers who reside in
Delaware and Maryland. This estimate was generated from a random survey.
A farmer may seek help the AgrAbility program (columns B and C) or a program can reach out to a farmer (columns D,E, and F).
Line 1 shows that 280 of the estimated 2400 disabled farmers were contacted either by AgrAbility or another program.
Line 2 shows that 168 farmers out of the 280 farmers who were contacted were subsequently evaluated.
Line 3 shows that 120 of the farmers who were evaluated were subsequently referred for service.
Line 4 shows that 105 of those who were referred for service subsequently received a service.
Line 5 shows that 90 out of the 105 farmers who received a service subsequently experienced an improvement in their quality of life.
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
Six Process Measures are employed
Each measure indicates the efficiency of a different part of the
system
1. Market Penetration
The percentage of the target population that contacts social service agencies for help.
Include all agencies- AgrAbility and other agencies that provide services to farmers with
disabilities
Market Penetration Measurementis based on Target Population: the estimated 2400 disabled farmers residing in MD and DE
The farmers who were contacted are indicated in line 1 of Figure 1
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
1. Market Penetration(220)/(2400)
The percentage of the target population that contacts social
service agencies for help
1. Market Penetration(B1+ C1)/(2400)=.09
The percentage of the target population that contacts social
service agencies for help
Market Penetration is a crude measure of public service advertising reach.
2. Outreach
(B1 + C1+DI)/TP
% of the target population (TP)
identified and contacted by
a specific program
The target population that was reached is shown in cells B1, C1, and D1 of Figure 1.
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
3. Assessment Efficiency.
(B2+D2)/(B1+D1)
Assessment Efficiency is the percentage of potential clients eligible for assessment that was assessed
The clients who were assessed are show in cells B2 and D2 of Figure 1.
Clients eligible for assessment are shown in cells B1 and D1 of Figure 1
3. Assessment Efficiency. (100 +58)/
(110 +60)=.93
The percentage of potential clients eligible for for
assessment that was assessed
4. Referral Efficiency. (B3+D3)/(B2 with positive screen + D2 with positive
screen)
Referral efficiency is the percentage of clients with a positive screen that was referred for
service.
Clients who were referred for services are shown in cells B3 and D3.
Those with a positive screen are in cells B2 and D2.
Figure 1
Systems Model of Delaware-Maryland AgrAbility Program (DMAP), Target Population = 2400 Disabled Farmers
Sought Help From Did Not Seek Help Contacted by
DMAP
110Other
110DMAP
60Other
?None
?
Assessed
110Assessed
?
Referred
80Referred
?
Service Provided
60
Service Provided
?
Quality of Life Improved
50
Quality of LifeImproved?
Assessed
58Assessed
?Not Assessed
Referred
40Referred
?
Not Referred
Service Provided?
No Service Provided
Quality of Life Improved ?
Quality of Life Improved
40
Quality of Life Improved
?
B C D E F
1
2
3
4
5
Service Provided
45
5. Service Delivery Efficiency. (B4 + D4)/B3 +D3)=
(60+ 45)/(80 + 40)=.88
Service delivery efficiency is the percentage of clients referred for service
that received service.Clients who received a service are show
in cells B4 and D4.Clients who were referred for service are
shown in cells B3 and D3.
The Delaware-Maryland Project employs the SF-36 Quality of Life Instrument as an Impact Measure
Additional impact measures that are specific to farmers with disabilities
and assistive technology will be incorporated in the evaluation
These measures are being developed by a committee drawn
from AgrAbility programs and national program staff
The SF-36 is the most widely used quality of life instrument in health and social services
The SF-36 is calculated for 8 health scales (0-100)1. Physical limitations
2. Limitations in social activities
3. Limitations in work and home roles
4. Bodily pain
5. Psychological distress and wellbeing
6. Limitations because of emotional problems
7. Energy and fatigue
8. General health perceptions
The SF-36 has been validated through a
numerous studies
Several thousand scientific articles employ the SF-36 as a
quality of life measure for measuring program impact
All active Delaware and Maryland clients are currently completing SF-36 interviews
The SF-36 serves two purposes
The SF-36 is used in the initial assessment to develop a service plan.
The SF-36 is also used in a 1-year follow-up evaluation to assess changes in the quality of life subsequent to the initial assessment.
6. Improved quality of Life: (# of Clients with improved quality of life) /
(# of Clients that Received AgrAbility Service)
Clients with improved quality of life are shown in cells B4 and D4. Those who received a
service from AgrAbility are shown in cells B3 and D3.
(B4 + D4)/(B3 +D3)=
(50 + 40)/(60 + 45)=
90/105=
.86
Finally, 2 questions for the audience:
Which of the measures focus on the operation of partnerships in AgrAbility programs?
What other techniques should be used to evaluate partnerships?