evaluation and monitoring methodologies
DESCRIPTION
Evaluation and Monitoring Methodologies. Strengthening the Legislature – Challenges and Techniques K. Scott Hubli, NDI. Overview. General Comments on Monitoring and Evaluation Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs - PowerPoint PPT PresentationTRANSCRIPT
Evaluation and Monitoring Methodologies
Strengthening the Legislature – Challenges and Techniques
K. Scott Hubli, NDI
Overview
General Comments on Monitoring andEvaluation
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Practical Tips and Considerations
General Comments onMonitoring and Evaluation
Evaluation (and Baseline Assessments) --Use to develop program design; use for major
course corrections --More costly and less frequent than monitoring
(every two to three years) --Typically done at the beginning and the end of a
program, but often also done after a major change in the political landscape (e.g., regime change, ethnic conflict settlement, etc.)
--Used for accountability to partners, donors, stakeholders, not for ongoing project management
General Comments onMonitoring and Evaluation
Performance Monitoring--Ongoing monitoring; used to manage performance
of implementation--Track changes (but less analysis)-- Informed by baseline assessment and, if well
designed, it can reduce future evaluation costs -- May indicate a need for a evaluation or updated
baseline-- Focus on low cost, regular data collection
(workshop evaluations, information available from parliament, regular focus groups, etc.)
General Comments onMonitoring and Evaluation
Always distinguish among: --Inputs (e.g., consultants, computers, etc.)
--Outputs (e.g., 40 people trained in a workshop onoversight techniques)
--Outcomes (e.g., increased knowledge of oversight investigation techniques)
--Objectives (e.g., increased oversight hearings)
--Goals (e.g., increased government accountability)
How are legislative strengthening programs different from other programs with respect to monitoring and evaluation?
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Legislatures are highly complex institutions --They involve multiple actors seeking to achieve multiple goals simultaneously
--Where possible, disaggregate data (by gender, party, region, etc.)
--Identify clear goals and targeted groups; watch for unintended consequences
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Long-term goals, short-term programs -- Resist the tendency to monitor outputs rather than progress in achieving desired outcomes, objectives and goals
-- Find ways to measure small changes in large goals; or outcomes that can be affected with the
project time frame
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Programs focus on process, not outputs--Example: number of laws passed--Emphasize qualitative over quantitative information--Use detailed process descriptions in establishing
baselines--Use monitoring and evaluation to help strengthen this process and to teach results-based management, where possible (“Monitoring and evaluation should be managed as joint exercises with development partners.”)
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Monitoring and evaluation is often highly political
--Involving partners can sometimes further politicize evaluation and monitoring; use caution and judgment--Can be hard to get necessary information--Politics may cause people to be less than fully honest--Results can be used as a political weapon
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Legislatures have natural cycles --Elections, post-election learning curves, legislative floor periods, recesses, budget processes, etc.--Example: constituency relations --Expect uneven development in performance monitoring, but try to attribute fluctuations in data--Time evaluations carefully – look for “normal” periods
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Many intervening variables --Economic conditions, geopolitical developments, ethnic conflict, death of a key politician, etc.
--No substitute for nuanced political analysis
--Measure outcomes, objectives, goals – not just outcomes; this can help identify these intervening variables
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Perceptions matter--Importance of qualitative over quantitative indicators
--Use of focus groups, opinion polls, etc.
--Even anecdotal evidence is useful if it captures a political mood or issue
Special Considerations in Monitoring and Evaluating Legislative Strengthening Programs
Difficulty of comparative benchmarking--Only one national legislature; cross-country
comparisons are of limited utility
--Comparisons across time more important; use ofthorough baselines
--Implications on setting goals and targets – use ofreasonable/consensus expectations
What are some practical strategies for dealing withthese unique aspects of monitoring and evaluating legislative strengtheningprograms?
Practical Tips & Considerations
General Issues--Be pragmatic in designing an evaluation or monitoring plan; tie
evaluation and monitoring to the purpose or objectives. Avoid evaluation for evaluation’s sake. Consider:
--resource availability for evaluation--novelty of the program--confidence in program design or
implementation--needs of funder
--Budget sufficient resources – (Costs for legislative strengthening evaluation may exceed those for other program types – soft assistance, new field, etc.)
Practical Tips & Considerations
Issues in Doing a Baseline--Limit scope to allow for detailed coverage of program areas
--Protect against biases of person(s) doing the baseline by: --Using teams
--Using clear, detailed terms of reference
--Incorporating documentary evidence
--Seeking consistency in future assessments
Practical Tips & Considerations
Issues in Doing a Baseline (cont.) --Pick timing carefully; describe any special circumstances--Prepare carefully for baseline assessment team
--Cover the range of stakeholders
--Get out of the capital
--Consider focus groups or creative methods for documenting perceptions and processes (e.g., a sample of 10 legislators to track periodically every 3 years)--Pay attention to protocol; build good will.
Practical Tips & Considerations
Using outside evaluators --Outside evaluators can not only provide objectivity but also insulation from the political consequences
of an evaluation--Combine multiple backgrounds (academic or legislative strengthening specialists and MPs or staff from similar systems)--Recognize value of “time in the trenches”--Designate a lead person with responsibility for producing the document--Get a sufficient time commitment
Practical Tips & Considerations
Issues in Performance Monitoring--Draw on baseline and prior evaluations
--Design performance monitoring plan up front; adjust it as project evolves:
--Imposes discipline; keeps program on track
--Provides clarity of expectations to partners
--Keep it current, modify as needed
--Make these changes explicit
Practical Tips & Considerations
Issues in Performance Monitoring (cont.)--Tie to likely performance issues--Draw on low-cost existing information sources; may be more quantitative, with less analysis--May focus on outcome level, rather than objective
or goal level--Consider quarterly or semi-annual monitoring--Expect, but explain, fluctuations--When you can’t explain repeated fluctuations, consider updating a baseline to try to identify issues--Often done, in part, by those implementing program
Final Thoughts
Be creative: legislative strengthening is anart, not a science
Be willing to accept criticism; fight structural bias for “spinning” results
Share lessons learned, both internally and externally