idev 624 – monitoring and evaluation introduction to process monitoring payson center for...
TRANSCRIPT
IDEV 624 – Monitoring and Evaluation
Introduction to Process Monitoring
Payson Center for International Development and Technology Transfer
Tulane University
Process vs. Outcome/Impact Monitoring
Process Monitoring Outcome Impact
Monitoring Evaluation
LFM
USAID Results Framework
Program vs. Outcome Monitoring
• Program process monitoring: The systematic and continual documentation of key aspects of program performance that assess whether the program is operating as intended or according to some appropriate standard
• Outcome monitoring: The continual measurement of intended outcomes of the program, usually of the social conditions it is intended to improve
Process Monitoring
A Form of OutcomeEvaluation
04/20/23
What is the problem? Situation Analysis & Surveillance
What are the contributing factors?
Determinants Research
What interventions and resources are needed? Needs, Resource, Response Analysis & Input Monitoring
What interventions can work (efficacy & effectiveness)? Efficacy & Effectiveness Studies, Formative & Summative Evaluation, Research Synthesis
Are we implementing the program as planned? Outputs Monitoring
What are we doing? Are we doing it right?Process Monitoring & Evaluation, Quality Assessments
Are interventions working/making a difference? Outcome Evaluation Studies
Are collective efforts being implemented on a large enough scale to impact the epidemic? (coverage; impact)? Surveys & Surveillance
Understanding Potential Responses
Monitoring & Evaluating National Programs
Determining Collective Effectiveness
ACTIVITIES
OUTPUTS
INPUTS
OUTCOMES
OUTCOMES & IMPACTS
A Public Health Questions Approach to HIV/AIDS M&E
Are we doing the right things?
Are we doing them right?
Are we doing them on a large enough scale?
Problem Identification
(UNAIDS 2008)
(World Bank 2009)
04/20/23
Most Some Few*All
Input/ Output Monitoring
Input/ Output Monitoring
Process EvaluationProcess
EvaluationOutcome
Monitoring / Evaluation
Outcome Monitoring / Evaluation
Levels of Monitoring & Evaluation EffortLevels of Monitoring & Evaluation Effort
Number of
Projects
Number of
Projects
*Disease impact monitoring is synonymous with disease surveillance and should be part of all national-level efforts, but cannot be easily linked to specific projects
Strategic Planning for M&E: Setting Realistic Expectations
6
Impact Monitoring / Evaluation
Impact Monitoring / Evaluation
Project Monitoring PlanObjective 1:
Monitoring Strategy:
What (Indicators) How (Methods and Tasks)
When Who Where
What is Process Monitoring?
Process Monitoring• Process Monitoring: The systematic attempt
by evaluation researchers to examine program coverage and delivery
• Program monitoring provides an estimate of – the extent to which a program is reaching its
intended target population – the degree of congruence between the plan
for providing services and treatments (program elements) and the ways they actually are provided
• Does NOT attempt to assess the effects of the program on the program participants
(Rossi/Freeman 1989)
Process Monitoring (cont.)• Not a single distinct evaluation procedure but
a family of approaches, concepts and methods
• Focus on the enacted program itself: operations, activities, functions, performance, component parts, resources, etc.
• Often collects information about resource expenditures in the conduct of the program (cost-benefit analysis)
(Rossi/Lipsey/Freeman: 2004)
Process Monitoring Strategies• Process implementation evaluation:
– Conducted by evaluation specialists as a separate project, either stand alone or as a complement of impact evaluation
• Continuous program monitoring:– Continuous monitoring of key indicators,
routine data collection by management information system (MIS) or similar mechanism
(Rossi/Lipsey/Freeman: 2004)
Monitoring Service Utilization: Coverage
• Coverage: The extent to which participation by the target population achieves the levels specified in the program design– Both over-coverage and under-coverage are
problems, however, the most common problem is the failure to achieve a high target participation
(Rossi/Lipsey/Freeman: 2004)
Under- vs. Over-Coverage• Under-coverage: The proportion of the targets
in need of a program that actually participates in it
• Over-coverage: The number of program participants who are not in need, compared with the total number of participants in the program
Common problem: the inability to specify the number in need, the magnitude of the target population
(Rossi/Lipsey/Freeman: 2004)
Monitoring Service Utilization: Bias
• Bias: The degree to which some subgroups participate in greater proportions than others– Often caused by self-selection and/or program
actions (for example by focusing on most “success prone” targets)
(Rossi/Lipsey/Freeman: 2004)
Assessing Bias• Assessing Bias: Assessing the differences
between program participants and those that
1. Drop-out of the program (drop-out rate, attrition)
2. Are eligible but do not participate at all
• Accessibility: The extent to which structural and organizational arrangements facilitate participation in the program
• Data sources: Program records, specifically designed surveys incl. community surveys, census data and similar secondary data sources
(Rossi/Lipsey/Freeman: 2004)
Monitoring Organizational Functions
• Three kinds of implementation failure:– “Nonprograms” and incomplete interventions (nothing
or not enough delivered)– Wrong intervention (mode of delivery may negate
intervention, for example, may be too sophisticated)– Unstandardized intervention (implementation may
vary excessively across the target population)
• Focus on service activities AND monitoring of vital program support functions (fund-raising, training, advocacy, etc.)
(Rossi/Lipsey/Freeman: 2004)
Monitoring Data Collection
Planning for Data Collection
1. Choosing data collection method
2. Selecting indicators and developing questionnaires
3. Determining the sampling strategy
4. Assessing validity, reliability and sensitivity
5. Developing data analysis plan(King/Morris/Fitz-Gibbon: 1987)
STEP 1: Choosing Data Collection Method
• Examine the records kept over the course of a program
• Collect data to fill information gaps
(King/Morris/Fitz-Gibbon: 1987)
Data Collection Methods
1. Analysis of program records2. Key informant interviews3. Focus group interviews4. Observations5. Physical measurements6. Standardized tests7. Surveys (program participants,
community, etc.)
(Measure Evaluation Online Course)
Selecting Methods• Standardized approaches with ready-
made measurement instruments sometimes available
• However, often controversial, and ready-made measurement instruments may not be appropriate under all (most?) circumstances
- Example: Measuring well-being/ happiness across cultures?
Selecting Methods (cont.)
• Often evaluators have to develop their own tools and instruments
• Researchers have rarely sufficient time and resources to do this properly
• Requires significant amount of pilot testing, analysis, revision, and validation
STEP 2: Selecting indicators and developing questionnaires
• Use existing questionnaires and instruments, if feasible
• Use standardized indicators, if available
• Follow indicators standards if developing or modifying questionnaires and instruments
Project Indicators• An indicator is a measure of a concept or behavior• Principal types of indicators:
– Process indicators• Provide evidence of whether the project is moving in the right
direction to achieve an objective• Provide information about implementation of activities (quantitative
and/or qualitative)• They should be collected throughout the life of the project
– Outcome/Impact indicators• Provide information about whether an expected change occurred,
either at the program level or population level• Measure changes that program activities are seeking to produce in
the target population• Often stated in percentage, ration or proportion to show what was
achieved in relation to the total population• Should be a direct reflection of the objectives
Indicator Criteria1. Measurable (able to be recorded and
analyzed in quantitative or qualitative terms)
2. Precise (defined the same way by all people)
3. Consistent (not changing over time so that it always measures the same thing)
4. Sensitive (changing proportionally in response to actual changes in the condition or item being measured)
Indicator Selection
Matching Indicators and Methods of Data Collection
• Select more than one method to measure an indicator (if possible)
• Criteria for selecting methods: – Reliability, validity, and sensitivity– Cost-effectiveness– Feasibility– Appropriateness
Only measures that are valid, reliable, and sensitive will produce estimates that can be regarded as credible
STEP 3: Determining the Sampling Strategy
• Census vs. Sampling– Census measures all units in a population– Sampling identifies and measures a subset
of individuals within the population
• Probability vs. Non-Probability Sampling– Probability sampling results in a sample
that is representative of the population– A non-probability sample is not
representative of the population
Probability SamplingSample representative of the population, large sample size
• Simple random/systematic sampling • Stratified random/systematic sampling• Cluster sampling
.
Advantages– Research findings
representative of the population
– Advanced statistical analysis
Disadvantages- Costly and time
consuming (depending on target population)
- Significant training needs
Non-Probability SamplingSample not representative of the population, often small sample size
• Convenience/purposeful sampling• Quota sampling
• Relatively inexpensive• Can be implemented
quickly• Limited training needs
.
Advantages Disadvantages• Results not representative of
population• Limited options for statistical
analysis of the data• Results biased
STEP 4: Assessing Validity, Reliability and Sensitivity
• Validity– Is the instrument appropriate for what needs to
be measured?
• Reliability– Does the instrument yield consistent results?
• Sensitivity– Indicators changing proportionally in response
to actual changes in the condition or item being measured?
Reliability
• Reliability: The extent to which the measure produces the same results when used repeatedly to measure the same thing
• Variation in results = measurement error
• Unreliability in measures obscures real differences
(Rossi/Lipsey/Freeman 2004)
Reliability (cont.)• How to verify?
– Test-retest reliability: Most straightforward but often problematic, esp. if measurement cannot be repeated before outcome might have changed
– Internal consistency reliability: Examining consistency between similar items on a multi-item measure
– Ready-made measures: Reliability information available from previous research
Validity• Validity: The extent to which a measure
measures what it is intended to measure
• Usually difficult to test whether a particular measure is valid
• However, it is important that an outcome measure is accepted as valid by stakeholders
(Rossi/Lipsey/Freeman 2004)
Validity (cont.)• How to verify?
– Empirical demonstrations: • Comparison, often with another measure, that shows
that the measure produces the results expected • Demonstration that results of the measure “predict”
other characteristics expected to be related to the outcome
– Other approaches: Using data from more than one source, careful theoretical justification based on program impact theory, etc.
Sensitivity• Sensitivity: The extent to which the values of
the measure change when there is a change or difference in the thing being measured
• Outcome measures in program evaluation are sometimes insensitive because:– They include elements that the program could not
reasonably be expected to change– They have been developed for a different (often
diagnostic) purpose
(Rossi/Lipsey/Freeman 2004)
Sensitivity
Margoluis & Salafsky p94
Sensitivity (cont.)
• How to verify?– Previous research: Identify research in
which the measure was used successfully (need to be very similar programs, sample size needs to be sufficiently large )
– Known differences: Apply the outcome measure to groups of known difference or situations of known change, and determine how responsive it is
Measurement Errors
• Systematic error (we do not measure what we think we measure)
• Random error (inconsistencies from one measurement to the next)
STEP 5: Developing Data Analysis Plan
• Type of data– Qualitative and/or quanititative? Representative of
larger population? Standardized vs. open-ended responses?
• Type of comparison– Comparison between groups and/or over time?
• Type of variable– Categorical and/or continuous?
• Type of data analysis– Descriptive analysis and/or hypothesis testing?
Project Monitoring PlanObjective 1:
Monitoring Strategy:
What (Indicators) How (Methods and Tasks)
When Who Where