day 1 lecture 1 why evaluate 06jan13 gemma tweaks pptx final

Upload: mirza-arshad-baig

Post on 07-Jul-2018

218 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    1/26

    Why Evaluate? Gemma Stevenson

    Associate Director, CERP

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    2/26

    Quiz: Why Evaluate?a) To work out the best design of a programme

    b) To identify which programme delivers a desired outcome

    c) To determine how to reach a desired outcome in the most cost-effective way

    d) To be accountable to those paying for the programme (donors / tax

    payers / etc.)

    e) To settle an argument; show the impact of a program to skeptics or showthe inefficacy of a program to believers

    f) All of the above

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    3/26

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    4/26

    Aid optimists“I have identified the specificinvestments that are needed [toend poverty]; found ways to planand implement them; [and] shownthat they can be affordable.”

    Jeffrey Sachs

    End of Poverty

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    5/26

    Aid pessimists

    5

    “After $2.3 trillion over 5 decades, why are the desperate needs of the world's poor still so tragically unmet?

    Isn't it finally time for an end to theimpunity of foreign aid?”

    Bill Easterly The White Man’s Burden

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    6/26

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    7/26

    Debates surrounding development projects! Impact:

    • Do we agree on what the impact should be?• Can this impact be measured?• How do we know if a programme is effective?

    ! Prioritize:• How should we allocate/prioritise a limited budget?

    ! Benefit-Cost Calculations:• What is the cost-effectiveness of a program relative to other programmes?

    ! What role can evidence play in policymaking?• What is the most effective way to design a programme?• How to generate short and long run success?

    Can these questions be resolved?

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    8/26

    Needsassessment

    Programdesign - withM&Esystem

    embedded

    Pre-programlogistics/Program

    roll-out

    Processevaluation/

    Monitorprogresstowardstarget

    Learning&

    ongoingimprove-

    ments

    Reportingfindings -impact,progress

    Usingfindings

    toimproveprogram

    design &delivery

    Yes they can!By embedding M&E in policy-making

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    9/26

    Needsassessment

    Programdesign - withM&Esystem

    embedded

    Pre-programlogistics/Programroll-out

    Processevaluation/

    Monitorprogresstowardstarget

    Learning&

    ongoingimprove-

    ments

    Reportingfindings -impact,progress

    Usingfindings

    toimproveprogramdesign &

    delivery

    Highlight no.1 Do we agree on what the impactshould be?

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    10/26

    Needsassessment

    Programdesign forimplemen

    tation

    Pre-programlogistics/Programroll-out

    Processevaluation/

    Monitorprogress

    towardstarget

    Learning&

    ongoingimprove-

    ments

    Reportingfindings -impact,

    progress

    Usingfindings

    toimproveprogramdesign &delivery

    Highlight no.2

    Use Monitoring and evaluation to continually improve program design

    What is the most effective way todesign a program?

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    11/26

    Example: Property Tax Project! Tax systems throughout the developing world have poor performance

    ! Problems with tax inspectors (assess and collect taxes)• Corruption: tax inspector and taxpayer collude to reduce taxpayer liability• Effort: weak incentives for tax collectors lead to low effort and tax evasion

    ! Historically, problem solved by high-powered incentives (tax farming) for theseofficials

    ! But concerns about over-taxation and extortion led to a move away from high-powered incentives and towards balanced compensation for tax staff

    ! Project: Design optimal incentive schemes to improve tax collection and customerservice by CERP researchers engaging in a rigorous collaboration with the Punjab,Pakistan property tax department

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    12/26

    Process Evaluation! Is the program implemented as planned?

    • E.g.; tax collector incentive schemes in place & understood

    ! Are basic tasks being completed?• E.g.; tax collection data being regularly reported

    ! Is the intervention reaching the target population?• E.g.; tax inspectors, constables, and clerks informed and engaged

    ! Are the services being delivered?• E.g.; Challans issued, payments collected etc.

    ! Is the intervention being completed well or efficiently and to the beneficiaries’satisfaction?

    • E.g.; Payments made on time and on the agreed on criteria

    12

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    13/26

    Evaluation design! The program happened, how did it change lives?

    o Tax staff rewarded for efforto Taxpayer paid due and fairly treated

    ! Did we achieve our goals?o Tax collections improved

    o Taxpayer satisfaction/service quality raised

    ! Also distributional questions such as, did the program benefit somemore than others?

    o Some tax staff did bettero Some taxpayers benefitted moreo Collections and service improvements varied across tax circles/regions

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    14/26

    Policy design! Through (real-time) testing and evaluating different

    iterations of a program, you can compare theimpact levels to find the optimal design of the

    program

    ! The evaluation design should be developed inconjunction with the design of the intervention

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    15/26

    Needsassessment

    Programdesign - withM&Esystem

    embedded

    Pre-programlogistics/Programroll-out

    Processevaluation/

    Monitorprogresstowardstarget

    Learning&

    ongoingimprove-

    ments

    Reportingfindings -impact,progress

    Usingfindings

    toimproveprogramdesign &

    delivery

    Pre-emptive M&E

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    16/26

    What is Impact?! IMPACT

    • What happened because of the program?

    • What would have happened in the absence of theprogram? ( counterfactual )

    ! Take the difference between what happened (with the program) …and

    - what would have happened (without the program)= IMPACT of the program

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    17/26

    Intervention

    Time

    P r i m a r y o u t c o m e

    Counterfactual

    Impact

    What is Impact?

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    18/26

    Intervention

    Time

    P r i m a r y

    O u t c o m e

    Counterfactual

    Impact

    Beware!

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    19/26

    Intervention

    Time

    P r i m a r y

    O u t c o m e

    Without the counterfactual

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    20/26

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    21/26

    Intervention

    Time

    P r i m a r y o u t c o m e

    Without the counterfactual

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    22/26

    How to evaluate?! For any policy question of interest, need to be able

    to construct a plausible counterfactual

    ! Different ways to construct the counterfactual:• Pre-post• Simple Difference• Difference-in-Difference• Randomization

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    23/26

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    24/26

    Evaluate for cost-effectiveness

    ! Only once you accurately measure the impact of aprogram can you accurately calculate cost-

    effectiveness

    ! Though a comparative cost-effectiveness analysis,

    you can find the least expensive way of achieving adesired outcome

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    25/26

    Comparative cost effectiveness

  • 8/19/2019 Day 1 Lecture 1 Why Evaluate 06Jan13 Gemma Tweaks Pptx Final

    26/26

    So why evaluate?! For more focused and effective programme design! For understanding what works and what doesn’t

    work in a programme! For determining the impact of a programme! For understanding how far a programme is cost-

    effective