a conceptual framework the experience lessons
DESCRIPTION
A syndrome of Irregular Enthusiasm: Increasing the Utilisation of Evaluation findings in the UPHOLD project BY Apollo Nkwake Visit http://www.uphold.jsi.com. Presentation Structure. A conceptual framework The experience Lessons. The dilemma of utilization. The Gap. Doing evaluations. - PowerPoint PPT PresentationTRANSCRIPT
A syndrome of Irregular Enthusiasm:
Increasing the Utilisation of
Evaluation findings in the UPHOLD project
BY Apollo NkwakeVisit http://www.uphold.jsi.com
A conceptual framework
The experience
Lessons
Presentation Structure
The dilemma of utilization
Doing evaluations
Using evaluations for evidence based policy
How Evals are done•Time•Quality•Relevance•Credibility•Participation•Evidence validity/rigor•Dissemination/access
Behavioral andinstitutional factors•A culture of evaluation•Capacity to use findings
The Gap
About UPHOLD (1)
Six-year integrated social services program designed by the Government of Uganda and USAID
Operated in 34 districts covering 42% Uganda’s population (~ 11.8m people)
Overall project aim was to increase access and utilization of sustainable and quality social services in Education, Health and HIV/AIDS in support of USAID’s Strategic Objective 8 (SO8) which aims to improve human capacity
Partnered with >120 CSOs and 34 local government
To strengthen existing data collection and information systems for Grantees ( both LGs and CSOs)
To build district capacities in planning and evidence-based decision-making by making accurate annual data available
To utilize performance results to target interventions
To document and share key lessons learned
The UPHOLD ‘Information’ Mandate
Context for doing and using evaluations in UPHOLD
Uganda is decentralized Local governments are CSO actors are closer to the
ground and need more localized information for programming
DHS’ and other higher level sources do not provide information disaggregate to local levels
Capacity issues-low capacity to generate reliable information
Resource constraints Varied capacity levels-planning, collection, processing
and use of information Availability of information is not linked to service delivery
planning
Using LQAS as a cost effective and simple tool
Originally developed in the 1920s to control the quality of output in industrial production processes
Involves taking a small random sample of a manufactured batch (lot) and test the sampled items for quality
If the number of defective items in the sample exceeds a pre-determined criteria (decision rule), then the lot is rejected
The decision rule is based on the desired production standards and a statistically determined sample size ‘n’ is chosen so that the manager has a high probability of accepting lots that meet the quality standards and rejecting lots that fail to meet those standards
Adapting LQAS for public health programs
Can be used locally, at the level of a “County or sub-county,” to identify priority areas or indicators that are not reaching average coverage or an established benchmark
Can provide an accurate measure of coverage or Service system quality at a more aggregate level (e.g. District coverage)
A
BC
DE
F G
Assume a district has seven Counties
Each County is ‘supervised’ by one person
The level of ‘success’ in each county is to be measured
The Basic LQAS Principles - I
A
B
C
DE
F G
Good
Below Averageor EstablishedBenchmark
Good
Below Average or Established Benchmark
Identify the reasons for program problems
Develop targeted solutions
Maintain the program at the current level
Identify Supervisors that can help other workers improve their performance
What can be done in the circumstances?
Each District is a ‘Supervision Unit’ and each County or Health sub-District or sub-County is a ‘Supervision Area’
Sample units are the Households, schools and health facilities
What is evaluated is the overall success in delivering specific services
Application of LQAS to monitor UPHOLD and Partners’ programs
Using the LQAS method (1)
~ 200 partner staff trained for 2 weeks in the LQAS methodology in 2004, 2005, 2006 & 2007
LQAS was done on an annual basis in with in the 34 project districts, involving CSO and LG partners in 2004, 2005, 2006 & 2007
Districts are divided into supervision areas District officers are involved in planning, data
collection, analysis and dissemination Fitted with in the planning cycle
Using the LQAS method (2)
19 villages are sampled from each of the five ‘Supervision Areas’ in each district
Data is Collected From, Households, Schools, Health Facilities
5 households are sampled from each village and a different questionnaire administered to each of the sampled households (~12,300 households covered)
Schools and Health Facilities also surveyed (423 health Units and 1,449 Schools)
What LQAS can do
A sampling method that:– Can be used locally, at the level of a ‘supervision
area’ (e.g., district) to identify priority areas (e.g., county, sub-county) or indicators that are not reaching average coverage or an established benchmark
– Can provide an accurate measure of coverage or health system quality at a more aggregate level (e.g., district, program catchment area etc.)
– Not diagnostic
Why we supported CSO and LG partners to adopt LQAS
Low sample size needs (n=19 in most cases) Simple to apply yet has very specific conclusions District level people could be trained to entirely ‘own’ this
methodology Provides high quality information at low & affordable cost Fast – ‘supervision areas’ are able to conduct self-
evaluation and obtain results immediately after the survey
Results are locally relevant and can be utilized in district level annual planning and decision-making
Example: monitoring malaria prevention in Bushenyi district-W. Uganda
010.5
5.30
5.3 4.211.7
5.3 7.913.2
5.315.4
9.417.2
36.8
21.128.9
13.2
28.9 25.8 26.8
79
36.8
55.3 52.660.5 56.8
43.9
0
10
20
30
4050
60
70
80
90
100
Per
cent
age
2004 2005 2006 2007
% Under-5 sleeping under ITN the night before survey
Example: monitoring uptake of reproductive health services in Wakiso district-C Uganda
47.4
79
63.2
94.7 94.7
41
63.2
89.5
73.779
89.5
79
45.952.6
7984.2
73.7
84.274.7
50
68.4
94.7
84.2
10089.5 87.4
57.8
75.8
0102030405060708090
100
Busiro A Busiro B Busiro C Entebbe Kyadondo Dist. Total All UPHOLD
2004 2005 2006 2007
% of mothers who delivered at health centers
What we learned
Sequence: there should be strong link between evidence generation, planning, resource allocation and service delivery/policy
In a decentralized setting with conflicting political interests, a piece of evidence does much
Results based reviews are very empowering, and enthusing
Emphasizing use is strengthening quality• It is possible to make comparisons within and across
districts and measure success in comparison to national targets
• LQAS has fostered more equitable allocation of resources at district level due to ‘evidence-based’ planning
What we learned from supporting CSOs and LGs to use LQAS (Contd)
The methodology is simple to use and CSO & LG personnel can be trained to carry out and analyze the annual surveys
‘Start-up’ training costs may be high, but this is often a once-off as persons already employed at district level are utilized for the surveys
At a cost of ~2,700 USD per district, LQAS is suitable for annual routine data collection even in resource limited settings
Visit http://www.uphold.jsi.com