programpubdocs.worldbank.org/en/325301495057225074/12-ideasfor...01ways to tell if you are pforr...

DISCLAIMER

These tips would need to be adapted to local context. They have not been derived through rigorous

counterfactual analysis. They only provide some ideas emerging from available data (from Tanzania

Education PforR Process Evaluation (P147486)), operational experience (multiple PforR Operation

Documents including PADs, ISRs and Restructuring Papers), and operational guidelines. Also, this

general list of tips is not comprehensive in terms of areas covered.

PROGRAM

01 Ways to tell if you are PforR ready

02 Where to place your Program Boundary

RESULTS

03 Let your results chain guide you

04 Show the work (underpinning DLIs)

05 Simplicity trumps efficiency

06 DLIs are incentives — actors will respond only if they feel they can win

07 Refine DLIs again and again and…

08 The more important the DLI, the more money (especially process DLIs)

09 Year Zero DLIs can be powerful — use them

10 Certain DLI types are harder to meet (but very worthwhile)

11 Non-DLI results may be neglected — factor this in

IMPLEMENTATION

12 How to course-correct

12TIPS

A PforR leverages World Bank (WB) development

assistance to support a government program. At

the start of design, two things are usually known:

(i) size of the total Government program and (ii)

size of WB financing. The first thing to be decided

is: what part of the total Government program will

the WB financing support? This is called the PforR

Program and the dividing line between the

Government program and PforR Program is called

the program boundary.

Everything inside the program boundary is a part

of the PforR Program financed by the government

and the Bank; everything outside the program

boundary is a part of the Government program but

not PforR program. Designing the boundary

between the Government program and PforR

program is an art. Below are some tips on defining

program boundary.

THE “PROGRAM” IN A PforR

Tips for PforR Design: PROGRAM

1What are the enabling conditions for PforR? There are no straightforward answers to this – but the following

list can be useful. The key idea is that we often think of Government’s capacity and Bank team’s willingness.

However, it might be more useful to think of Government’s willingness and Bank team’s capacity, as follows:

TIP 1

WAYS TO TELL IF YOU ARE PforR READY

For Government

Willingness for open data: Does the government exhibit an appetite for data sharing and openness? This is not about the existing capacity or infrastructure around data. These can be – and should be – strengthened within the PforR. Rather, it’s about whether government is willing to consider open data platforms. This is a good litmus test for whether PforR can actually usher in more results-based policy-making.

Willingness to incentivize different levels of service delivery: Ultimate results occur at the point where frontline service providers (teachers, doctors) interact with the beneficiaries. If the government is not willing to pass-on some of the performance-based incentives inherent in DLIs down to these actors, then these DLIs are less likely to impact ultimate results. This can be done in easy ways – e.g. performance based grants or awards at the level of service delivery

1

2

For Team

Capacity to explain the implications of a PforR program to the government. PforR shifts the risk burden from the donors to the government. Its execution requires trust between government and TTLs. Therefore, it is crucial to signal good faith by being well prepared and honest about the pros and cons of a PforR with the government.

Capacity to discern key political economy constraints. Does the team have access to analytical and operational work to illuminate underlying technical and political constraints? The latter, in particular, is crucial for the design of effective DLIs.

1

2


2While defining the Program boundary, imagine placing the PforR Program on a sliding scale between two

ends, with WB financing on one end and the full Government Program on the other . Program boundary

should be large enough to maximize flexibility and influence, but small enough to manage risks and M&E

burdens.

While there is no exact science behind the size of the PforR Program, below are some aspects to consider

when deciding where to place the PforR Program:

Definitions: Government program: This is the overall government program which is usually based on an existing strategy and may involve several ministries/implementing agencies. PforR Program: This may either be equal to the entire government program or a slice of it. The boundaries of the PforR Program can be based on slicing based on time, geographical focus or results areas. World Bank Financing: This is the commitment made by WB to support the PforR Program. Note that in a PforR, the task team is accountable for all activities supported by the PforR Program and WB financing is disbursed in the form of payments against Program results (DLIs).

TIP 2

WHERE TO PLACE YOUR PROGRAM BOUNDARY

Flexibility vs FocusIf the team wants greater flexibility in the areas in which WB financing is leveraged, then Scenario B is more suitable. However, if the team wants a tighter focus and stronger, more targeted incentives then Scenario A is more suitable.

M&E BurdenM&E burden is more manageable under Scenario A

Influencing PolicyIf your PforR Program is a small slice of the overall government’s program, you may find it difficult to influence policy across the government program. Scenario B is more suitable if you want a greater influence on all results areas of the Government program.

Environmental & Social RisksEnvironmental and social risks are more manageable under Scenario A.

FOR EXAMPLEGovernment program (US$1 billion): Secondary Education

PforR Program under Scenario A: Improving Teacher Effectiveness (US$300 million)

PforR Program under Scenario B: Improving Quality of Secondary Education (US$700 million)


2

THE “RESULTS” IN A PforR

Tips for PforR Design: RESULTS

“

Data from the independent verification in Tanzania gave insights into areas where the results chain could be breaking down. DEOs reported high implementation rates but these were not being reflected in progress in intermediate/PDO indicators. Data was collected on head-teachers view of implementation and it reveals significant discrepancies from DEO reports. This suggests that using independent verification as a chance to gain insights into ‘why’ things might be going wrong can help.

3For successful PforR design, make the results chain the beginning, the middle, and the end of PforR design and implementation.

TIP 3

LET YOURRESULTS CHAIN GUIDE YOU

Make it the Key Focus of Technical Assessment Spend Technical Assessment resources and time mostly on the development of a robust and detailed results chain. If you know you will be developing a PforR, ensure that ESWs and AAAs are designed in service of the results-chain.

Update the Results Chain During Supervision Missions, ISRs, & MTRFeed in new information and insights into the results-chain, check to see which original assumptions are proving erroneous.

Insert DLIs & Results in the Results Chain Often DLIs are decided beforehand and then retroactively fitted within a results chain. Avoid this. Discussing DLIs – always in the context of the results chain – will help TTLs design more effective DLIs.

Underpin Results-Chain with as Much Information as You CanUnderpin your results chain with data, context-relevant evidence, and extensive discussions at all levels of service delivery. Focus group discussions can be your best friend in this exercise – they are cheap, quick, and full of crucial insights from the field.

Make All Underlying Assumptions ExplicitDo not assume that your assumptions are obvious — spell them out.

Received Toolkit Received Toolkit Training Received STEP Training

N-500

There is a Theory of Change risk that the interventions being supported may not be optimal or even sufficient to significantly improve teacher quality, student performance and attendance, and aggregate education outcomes”

TANZANIA BIG RESULTS NOW IN EDUCATION (P147486) Aide Memoire, Sept 2015

BRN Interventions; View on ImplementationPercentage of Schools Where DEO & Head Teacher Agree


4The process of DLI identification is as important as the DLIs themselves. And this process – including data,

discussions, tacit knowledge from past implementation experience etc. should be recorded in the Technical

Assessment. It is crucial to preserve this information to guide implementation, ISRs, and ICRs.

“

TIP 4

SHOW THE WORK (UNDERPINNING DLIs)

Describe the Thinking Behind DLIsFrom the beginning, describe the reasons why certain indicators are being chosen as DLIs. Maintain and update this write-up in the Technical Assessment, preserving notes on those indicators that were considered but rejected and why. As program is finalized, there should be a clear narrative about what makes a DLI a DLI? Also detail (i) pros and cons of DLI choices; (ii) data used; and (iii) findings from in-country workshop/focus-group discussions.

Brainstorm with the Government & StakeholdersIdentify DLIs through a collaborative, transparent, and data-driven process, preferably through multi-stakeholder workshop(s). This will do much to maximize ownership and minimize the confusion/lack of transparency that comes from bi-lateral close door discussions.

There are … cases where the DLIs … seem less well linked to program

objectives. The role of DLIs to incentivize performance is also not always

clear, and it sometimes seems that the DLIs have been designed to

ensure disbursements and are not very demanding… Political economy

constraints are behind many of the issues addressed by DLIs and PAPs,

but those constraints are rarely discussed directly.

IEG, PforR Evaluation, March 2017


5When weighing simplicity versus theoretical efficiency in terms of DLI design, err on the side of simplicity.

While an efficient design is ideal, when DLIs are not well understood — which evidence suggests is common-

they have little value. Other tips include:

Implementation evidence suggests that system-actors react to the simplest types of information. The more complicated the information contained within a DLI, the less powerful it would be as an incentive. For instance, to improve accountability in Tanzania, schools were given two types of inform

information: their ‘National Ranking’ and their ‘color band’ (reflecting whether school was doing well or poorly). Head-teachers are much more likely to remember the simplest piece of information – national ranking.

TIP 5

SIMPLICITY TRUMPS EFFICIENCY

Keep It ShortAvoid the temptation of sub-DLIs or turning each annual target into a mini-DLI. This will make DLIs easy to remember and therefore more conducive to action.

Keep It SimpleKeep the formulae simple and intuitive. Limit Number of DLIsNumber of DLIs needs to be reasonable — no more than 10. Also, minimize the number of discrete annual targets. This can be done by choosing targets selectively. Try prioritizing scalable input or output indicators, instead of too many discrete actions.


6Positive reinforcement from meeting DLIs can be a powerful force for success. On the other hand, if DLIs are

not seen as achievable, they will be completely ineffective.

In Tanzania education PforR, DLIs linked to ‘School Incentive Grants’ were designed to incentivize better school performance. However, 72% of DEOs and 51% of Head Teachers did not

believe that “schools receive more money if they show better results”. This would make this DLI relatively ineffective.

TIP 6

DLIs ARE INCENTIVES – ACTORS WILL RESPOND TO THEM ONLY IF THEY FEEL THEY CAN WIN

Threshold (All-or-Nothing) Disbursement is Best for Foundational Activities/Process Indicators While Pro-Rated Disbursements are Best for Output and Outcome IndicatorsTheory suggests scalable DLIs provide the most consistent incentivizing effect. However, when a specific task must be completed before other work can be done (e.g., creating an information management system or establishing baselines), a threshold disbursement works best.

Communicate DLIs at District (& School) Level as Early as PossibleThis can make a big difference in generating momentum, action, and trust around DLIs at the service delivery level. Communication should be designed to be user-friendly and context-friendly.

Try to Make Incentives Flow to the Responsible ActorsWhen possible, make payments at the same administrative level at which the actions being incentivized are conducted. This can be done by including ‘performance-based school grants’ or ‘performance-based teacher awards’ as DLIs.

Often What Appear to be Communication Problems are Actually Problems of ‘Trust’Teachers and schools may have low trust in the education system. In such cases, mere communication will not suffice. Early disbursement of Year 0/1 DLIs linked to schools can help build this trust and enhance future DLI achievement.

Information/Trust Around Incentives Flowing to Schools (Sig) was Sub-OptimalTRUE or FALSE?: Under BRNEd, schools get more money if they show better results.


76

When money is attached to results, all of a sudden a government’s focus on results changes from theoretical

to practical. In an IPF Results Framework, everyone maybe comfortable with aspirational or stretch targets

with a discussion at the end of why the results were or were not achieved. In PforR, disbursements are only

made when results are achieved and verified.

Experience has shown that several iterations of the DLIs may need to be made to make sure that they are

appropriate, clear, well understood and agree with all elements of the government, not just the central

Ministry of Education. It is important to discuss DLIs and the amounts allocated with the Ministry of Finance

as well as regions or schools that may be responsible for achieving the results.

1 This tip courtesy of David Webber (OPSPQ)

TIP 7

REFINE DLIs AGAIN andAGAIN and AGAIN…1

Tanzania Big Results Now in Education (P146486)


8Price your DLIs based on needed ‘incentives’ for action – not ‘costs’ of action.

“

TIP 8

THE MORE IMPORTANT THE DLI, THE MORE MONEY (ESPECIALLY PROCESS DLIs)

In the Tanzania Education PforR, substantial DLI funding was linked to Government fully executing its education budget. This had been a big bottleneck to education programs in the past. In Year 1, the GoT missed out on over $7 million worth of DLI payments because it failed to disburse a large portion of its education budget.

In response, the GoT made systematic changes to its funding mechanisms after its Year 1 review, leading to increased spending in Year 2. In its Year 1 review (MoEST & PO-RALG, 2015a), the GoT introduced two policy reforms intended to improve funding disbursements: (1) “ring-fencing” education program financing to ensure funding is consistently available and (2) sending capitation grants directly from the Ministry of Finance to schools, instead of through LGA offices. The GoT justified these policy shifts as “the only way the benefits can be realized and the learning outcomes for P4R achieved,” (MoEST & PO-RALG, 2015a, p 12). The GoT’s Mid-Term Review notes the financing indicator “provides a clear incentive for the Government of Tanzania to provide adequate resources to finance education” (MoEST & PO-RALG, 2016, p. 27). After these policy changes were instituted, disbursements improved significantly.

The mid-term review recognized that

timeliness of payments is one of the

most difficult DLIs to achieve, but also

one of the most important ones. Paying

beneficiaries on time is a key

fundamental aspect of an efficient

safety net program.

ISR of Kenya National Safety Net Program (P131305)


9Use Year 0 DLIs to boost morale and momentum: It is a good idea to have one or two easy to achieve Year 0 DLIs to jump start implementation and create positive momentum around the project.

“The delays in accessing funds for the pre-

financing of Program activities in the

participating Provinces has resulted in

considerable delays to Program

implementation. MARD may request an

advance of up to $ 50 million (25% of the

total Program funds) under the Program

however it has not yet done so, despite

reminders from the Bank team. In addition,

many of the Provinces have yet to sign on-

lending agreements and set up bank

accounts to receive the advance funds. The

delays relating to lack of access to advance

funds may adversely affect the

achievement of the Disbursement Linked

Results for the first Program period.

TIP 9

YEAR ZERO DLIs CAN BEPOWERFUL TOOLS —USE THEM

ISR of Vietnam Results-based Scaling Up RSWS Program National Safety Net Program (P152693)


Some types of DLIs are harder to meet and may require additional effort.

In Tanzania Education PforR, DLIs that were not directly in the control of central ministries proved harder to achieve.

Source: World Bank Disbursement Notifications

10TIP 10

CERTAIN DLI TYPES ARE HARDER TO MEET

THESE INCLUDE

DLIs That Require Coordination

Too much coordination among agencies to fulfill the DLI will weaken incentives and decrease its impact.

1 2 DLIs That Don’t Directly Incentivize Responsible Actors

When DLIs simply disburse to central ministries for results at the school level, impact may be limited. These DLIs need to be matched with direct incentives to the schools.

Share of Available Process Indicator Funding Released in Years 1 & 2, by Disbursement Level

Funding Disbursed at National Level Funding Disbursed at LGA Level


11Be prepared that non-DLI results may be overlooked during implementation – and design/plan

accordingly. One way is to leverage DLIs for non-DLI results.

In Tanzania education PforR, reading performance were a DLI but math performance was not. Improvements were seen in both reading and mathematics between 2013 and 2016, but reading improvements were much larger. The share of students considered to be progressing or proficient readers increased by 42% between 2013 and 2016. Students considered approaching benchmark or benchmark mathematics performers also increased, but by just 14%. This limited evidence shows that, in this case, the targeted outcome did improve more than another similarly important outcome not included as a DLI, but both the targeted and untargeted outcome did improve.

benchmark or benchmark mathematics performers also increased, but by just 14%. This limited evidence shows that, in this case, the targeted outcome did improve more than another similarly important outcome not included as a DLI, but both the targeted and untargeted outcome did improve.

Source: RTI International (2016)

“The GoT raised some concern that including reading speed as the only outcome indicator does not paint a complete picture of student learning. The GoT’s Mid-Term Report states that the one current outcome indicator “only capture[s] reading speed, which provides a partial assessment of educational outcomes. It would also be useful to capture comprehension ability. An Early Grade Mathematics Assessment (EGMA) was recently conducted, but this has not been included in the DLR indicator…[T]he Government needs to be able to assess whether the education system is succeeding in imparting mathematical skills

TIP 11

NON-DLI RESULTS MAY BE NEGLECTED — FACTOR THIS IN

Early Grade Reading & Math Scores in 2013 & 2016

READING MATHEMATICS

Non-Readers

Progressive Readers

Tanzania Ministry of Education, 2016, p.55). – Tanzania BRNEd PforR Process Evaluation

Beginning Readers

Proficient Readers

Non-Performers

Emergent Performers

Approaching Benchmarks

Benchmark Performance

IMPLEMENTATION

Tips for PforR Design: IMPLEMENTATION

12The Bank only disburses when results are achieved, so governments shoulder a higher financial risk under

PforR than they do under IPF. TTLs need to acknowledge this reality and address government concerns

by making provisions for course-correction around under-performing DLIs. A good results chain will make

it easy to do this.

Other Ideas: Use Independent Verification Data for

more than Independent Verification. Useit to figure out ‘why’ DLIs are not beingmet.

Do early & extensive planning to ensure that data on the validity of the results chain is available during MTR. Make this apart of your TA.

Don’t Be Afraid to Revisit Your DLIs During Implementation

Program M&E should report not just on whether DLIs are being met or not – but why. Use your results-chain to guide you on what additional data is required to answer this question

Get Data Directly From the Front Lines Use information from beneficiary feedback data and leverage phone surveys

In Tanzania, Head-Teachers were given training – not just on reporting data – but also on using it. This seems to be bearing fruit.

Head-Teachers’ Use of Census DataGive High Priority to Use Collected Data for:

TIP 12

HOW TOCOURSE-CORRECT