Knowledge-BasedCountry ProgramsAn Evaluation of the World Bank Group Experience

October 24, 2013

2

Knowledge activities of type evaluated here have more than doubled in past 10 years

► Increasing client demand for TA and ESW (blue line). Country programs moving to more knowledge services content.

► Account for about three quarters of all knowledge services.

► For the Bank: core knowledge services about one-third of administrative budget in recent years compared to a quarter ten years ago.

3

Knowledge-based country programs, knowledge activities, and method

► Rank based on four indicators of knowledge services use; selected country programs from high end of the ranking (KBCP’s).• Bulgaria, Chile, China, Kazakhstan, Kuwait, Malaysia,

Russian Federation, South Africa, Thailand.

► Knowledge activities selected linked to strategic priorities in CPS

► Method builds on former Quality Assurance Group’s knowledge services assessment:• Relevance, technical quality, results, sustainability of results.• Desk reviews CPSs and projects; visits to countries

4

The Nine Countries Selected are Representative Intensive Bank Knowledge Users

► High-income (Kuwait) and upper-middle-income, including EU and OECD members (Bulgaria, Chile)

► Knowledge services program more than 50 percent of country services budget on average

► Diversified economic structure

► No or moderate Bank lending

► Fee-based knowledge services

5

► Raise stakeholder awareness 75 percent

► Design reform, help consensus, build capacity, support dialogue, 50 percent of activities

► Supporting Bank strategy formulation or lending operations about a quarter of the activities

Most Bank knowledge activities raised stakeholder awareness, some supported lending

6

Good News on Outcome Achievement ofKnowledge Services► About 80 percent of the knowledge activities

reviewed were expected to have outcomes achieved or partly achieved

► No significant difference between TA and ESW► Agriculture, education, and financial sector

above average► Private sector/economic policy & governance

below average

7

Knowledge in support of lending had better results than self-standing knowledge services

► Knowledge services in support of lending more likely to succeed • Bank presence in the country• More established relationships with counterparts• More likely to have better results framework, including

M&E

8

More specifically, the ingredients of success

► Activities conveying best practice, using local expertise, and providing actionable recommendations were more likely to succeed• 90 percent of successful conveyed international best practice and

relevant examples (50 percent for unsuccessful)• 55 percent used local expertise (28 percent for unsuccessful)• 72 percent actionable recommendations (50 percent for unsuccessful)

► Activities addressing recipient capacity had better outcomes• 52 percent developed/strengthened institutions (16 % for unsuccessful)• 71 percent strengthened analytical or policy capacity (20 % for

unsuccessful)

► 60 percent of Bank knowledge services contributed to developing or strengthening institutions

9

Areas of weakness

► Less success when:• Unable to address issues relevant to the client (40

percent for unsuccessful activities)• No new evidence or data for decision making (30

percent for unsuccessful activities)• Activity not delivered on time for decisions (30 percent

for unsuccessful activities)• Activities requiring coordination of reforms across

sectors or new legislation (examples, private sector/economic policy & governance)

10

Monitoring and Evaluation a Significant Weakness of Knowledge Services

► Monitoring of capacity development outcomes and lesson learning were, on average, weak for Bank knowledge services evaluated.

• In only 17 percent of activities was there at least a partial results framework in the CPS.

• And only 23 percent of knowledge services included, at least partly, result indicators to track the achievement of the activities’ outcomes.

The Evaluation Links to Change Agenda on

Knowledge

12

A client focused engagement is essential

→ respond quickly to client requests, use programmatic approaches, tap local expertise.

►Activities achieved outcomes more often if: • stayed engaged and responsive in implementation• translated recommendations into actions fitting local

conditions

→ involve client early on in design and ensure that timing of delivery and follow up adequate to affect important decisions.

►Challenges for the Bank:• More client participation in design • Better timing of delivery for decisions • More use of local expertise

13

Strategic focus and agile delivery of global expertise required to become the best development solutions provider→ keep strategic focus, and ensure that delivery of knowledge services through a “consultant firm model” does not result in fragmentation of RAS programs

►With shift from finance to knowledge:• stay engaged in areas relevant to clients’ development agenda

(use programmatic knowledge services)• maintain capacity to see the big picture

→ deploy highly experienced staff with global perspective, able to formulate policy options that fit local conditions and generate data to support policy making.

►Challenges for the Bank to enhance global practices:• Relevance and technical quality • Generate timely evidence or data to support policy making • Knowledge flow across organization/mediate knowledge from

KBCP’s to LICs

14

Invest in monitoring and evaluation to learn and innovate

→ a significant investment in monitoring and evaluation will pay significant dividends for learning and for the effectiveness of knowledge services

►Three ingredients of science of delivery—evidence-based design, agile implementation, structured learning—require strong M&E

►In particular “rapid cycle learning” from project design to implementation requires a strengthened M&E framework for both projects and country programs

15

To Sum Up:

► Country clients acknowledged Bank:• independent and credible knowledge broker—familiarity w/local

context • capacity to customize international best practice solutions • ability to see the big picture and analyze cross-sectoral issues

► Results could be improved by:• more client participation in the design of knowledge services • timing better the delivery of services to inform important policy

decisions • making more use of local expertise• following up the knowledge services with policy implementation

support where needed

► In some instances Bank staff have insufficient experience/global expertise to deal with difficult policy problems:• deploying highly experienced staff with global perspective essential.

► Improving monitoring and evaluation systems is an essential investment

16

Thank you.

Additional Slides

These additional slides provide numbers supporting

statements made in the presentation.

18

Successful Knowledge Activities

► Activities conveying best practice, using local expertise, and providing actionable recommendations were more likely to succeed

Activity characteristicExpected Outcome of Knowledge Service Likely to be Achieved (in percent)

Yes Partly No

Activity Requested or Commissioned by Client 87.1 80.6 72

Activity Conveyed International Best Practice and Relevant Examples

87.1 73.8 52

Local Expertise Used by Activity 54.8 50 28

Actionable Recommendations Consistent with Findings Provided

72 73.6 52

Source: IEG knowledge activity reviews.

19

Less Successful Knowledge Activities

► Activities that were unable to address issues relevant to the client, with little new data to inform decisions, or not delivered in time for decisions were associated with less success in the achievement of outcomes

Activity characteristic Expected Outcome of Knowledge Service Likely to be Achieved (in percent)

Yes Partly No

Activity was Unable or Partly Unable to Address Some Issues Relevant to the Client

17.3 32 40

No New Evidence or Data to Inform Decision Making Generated

5.4 10 28

Activity Outputs not Delivered in Time to Inform Important Decisions

1.1 6.9 28

Source: IEG knowledge activity reviews.

20

Recipient Capacity Associated with Better Outcomes

► About 60 percent of Bank knowledge services reviewed contributed at least in part to developing or strengthening institutions.

► A large majority of Bank knowledge service and a significant part of IFC Advisory Services contributed to strengthening analytical or policy formulation capacity of recipients.

Activity characteristic Expected Outcome of Knowledge Service Likely to be Achieved (in percent)

Yes Partly No

Activity Contributed to Developing or Strengthening Institutions

51.6 31.9 16

Activity Contributed to Strengthen Analytical or Policy Formulation Capacity of Recipient

71 45.8 20

Source: IEG knowledge activity reviews.

Addressing issues of recipient capacity was associated with better outcomes

21

Improving Results of Knowledge Interventions Centerpiece of WBG Current Change Proposals

► Main proposals on knowledge and solutions:• Client Engagement: Strengthen client-focused

engagement model using science of delivery principles.• Global Practices: Become the best development

solutions provider by growing, supporting, and deploying global expertise for agile client delivery.

• Invest and fund transformational platforms for knowledge, research, learning, and innovation.

• Reinforce knowledge governance, strategy, and behaviors

22

Leadership needed to enhance the Bank global services

► Governance will benefit from management leadership to encourage knowledge services and develop stronger M&E for knowledge services.

► An additional challenge will be to bring in the knowledge acquired from knowledge-based partnerships to lower-income countries.

► Another challenge will require that staff and management have incentives to work on knowledge that are as strong as for other areas.

23

Focus on specific sectors more likely to achieve outcomes than multisector or thematic efforts

► Agriculture, education, and financial sector above average

► Average in infrastructure

► Poverty reduction average, with frequent partial achievement

► Private sector/economic policy & governance below average → difficulty coordinating reforms across sectors → if new legislation needed + complicated

24

Financing arrangements not a driver for results, but knowledge in support of lending better results

► Reimbursable activities significant in sample

► Their results similar to Bank-financed operations

► Knowledge services for lending more likely to succeed