Nonformal Staff Learning in the World Bank Heidi S. Zia Antonieta Romero-Follette

43168

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Nonformal Staff Learning in the World Bank

Heidi S. Zia Antonieta Romero-Follette WBI Evaluation Studies EG07-130 The World Bank Institute The World Bank Washington, D.C. June 2007

Acknowledgments

The World Bank Institute Evaluation Group (WBIEG) prepared this report under the direction of Richard Tobin. Heidi Zia is the principal author.

The authors thank Donald MacDonald who reviewed the study at several stages and offered suggestions for its improvement. Most notably, Don graciously agreed to formulate a definition of nonformal learning on behalf of the Office of the Chief Learning Officer and applied the definition to the Learning and Management System for proper recording of all nonformal learning in the World Bank Group.

The authors also thank Humberto S. Díaz for his assistance in formatting and graphics.

WBIEG evaluates learning by staff of the World Bank and activities of the World Bank Institute (WBI). The findings, interpretations, and conclusions expressed in WBI Evaluation Studies are entirely those of the authors and do not necessarily represent the views of the World Bank Group, including WBI.

WBI Evaluation Studies are available at http://www.worldbank.org/wbi/evaluation

Suggested citation: Zia, Heidi S., and Antonieta Romero-Follette. 2007. Nonformal Staff Learning in the World Bank. Report No. EG07-130. Washington, DC: World Bank Institute. Vice President, World Bank Institute Frannie Léautier Acting Chief Learning Officer Ana-Maria Arriagada Manager, Institute Evaluation Group Richard Tobin Task Team Leader Heidi S. Zia

ii

Contents

ACRONYMS AND ABBREVIATIONS .................................................................................... iv

1. OBJECTIVE ....................................................................................................................1

2. DATA COLLECTION.......................................................................................................2

3. DEFINITION AND SCOPE OF NONFORMAL LEARNING ..................................................2 Share of nonformal activities in total learning budget.......................................6 Share of nonformal learning in total number of events .....................................8

4. EVALUATION OF NONFORMAL LEARNING .................................................................11 References.................................................................................................................14

Appendix 1 Scoping nonformal learning in the Bank, FY04-06.............................15

Appendix 2 Mapping of formal and nonformal learning.........................................17

Appendix 3 Number of formal and nonformal learning events, FY04-06 ..............18

iii

Acronyms and Abbreviations

ACS Administrative and Client Support Group AFR Sub-Saharan Africa DEC Development Economics EAP/SAR East Asia and Pacific/South Asia ECA Europe and Central Asia ESSD Environmentally and Socially Sustainable Development Network EXT External Affairs, Communications & United Nations Affairs FSE/PSD Financial Sector and Private Sector Development Network GSD General Services Department HD Human Development Network HR Human Resources IEG Independent Evaluation Group IFC International Finance Corporation INF Infrastructure Network ISG Information Support Group LCR Latin America and Caribbean LEG Legal MIGA Multilateral Investment Guarantee Agency MNA Middle East and North Africa OED Operations Evaluation Department OPC Operations Policy and Country Services PREM Poverty Reduction and Economic Management Network SEC Corporate Secretariat SFR Strategy, Finance and Risk Management SLC Strategic Learning Centers WBI World Bank Institute

iv

1. OBJECTIVE

This study addresses the concern that the World Bank Institute Evaluation Group’s (WBIEG) approach to evaluation of staff learning does not fairly or well characterize the Bank’s training nor its overall quality because assessments of staff learning typically focus on formal learning activities included in the Bank’s Learning Management System (LMS). This concern is especially acute among parts of the Bank in which nonformal learning constitutes a large and major portion of the training provided.

Addressing this concern requires agreement on a definition of nonformal learning, an estimate of its size relative to formal learning, and reliable information about nonformal learning within the Bank. To address these issues, this report:

1. Provides a platform for dialogue among KLB members on the definition, scope, and role of nonformal learning in the Bank;

2. Provides a working definition of nonformal learning, supplemented by a literature review and a proposed definition from the office of the Chief Learning Officer (CLO);

3. Characterizes FY04-06 learning events as either formal or nonformal;

4. Identifies the relative amount and cost of formal and nonformal staff learning for three fiscal years (FY04-06) for the Bank’s major providers of staff learning; and

5. Identifies the extent to which nonformal learning events are or are not included in the LMS.

1

2. DATA COLLECTION

This study is based on data on learning activities from the Bank’s business warehouse (BW), which feeds into the LMS (i.e., the BW/LMS), and interviews with members of the KLB (see appendix 1 for the interview protocol).1

The LMS does not currently categorize staff learning as either formal or nonformal. To address this situation, WBIEG downloaded activity-level data by vice-presidential units (VPUs) and strategic learning centers (SLCs) and categorized staff learning activities as either formal or nonformal. Since the BW data provide only the count of activities in each of the VPUs and SLCs, and not their costs, representatives of all VPUs and SLC were interviewed to verify data in the BW and to obtain information on each unit’s expenditures for formal and nonformal learning.

3. DEFINITION AND SCOPE OF NONFORMAL LEARNING

WBIEG started with a definition of nonformal learning included in one of its prior reports. According to Prom-Jackson and her coauthors (2002), “Nonformal learning events, as distinguished from formal training for knowledge acquisition, are organized in the Bank as brown bag lunch (BBL) sessions, clinics, workshops, and work sessions for ‘learning groups’ or for a ‘community of learning practice’ engaged in learning, sharing and using learning for further reflections, for solutions or new perspectives.”

WBIEG modified this definition by categorizing workshops as formal instead of nonformal learning. WBIEG also asked the Office of the Chief Learning Officer (CLO) to provide definitions. This office proposed the following:

Nonformal learning represents structured and/or unstructured learning activities where participants have varying levels of inputs and controls over the objectives and agenda as the activity is being delivered.

1 When learning events are created, they are recorded both in the Business Warehouse and the LMS. WBIEG used data from the former, whose software has data quality-control functions not found in the LMS. For example, BW identifies and eliminates duplicate records.

2

Formal learning represents structured learning activities where objectives and agenda are for the most part, predetermined by the deliverer/supplier.

Based on a series of consultations with the CLO’s office, WBIEG developed a definition of formal and nonformal learning that categorizes World Bank Group staff learning into formal and nonformal (see table 1). The CLO’s office uses this categorization to map staff learning activities to the LMS.2

Table 1. Categorization of nonformal and formal learning events

Nonformal Formal

ART/workout Course Brown bag lunch Sector week Coaching/mentoring Workshop Team-based action learning Distance learning course Retreat Language learning Study tour Online course or module

Performance support tool Conference

Among those interviewed, nearly 60 percent agreed with this categorization, yet pointed to a number of limitations:

• Conferences may contain nonformal components. One interviewee said that conferences “are not fully formal learning” and another commented that they could also be categorized as nonformal learning because “they are not designed to impart specific skills.”

• Sector weeks, like conferences, also contain nonformal components. Two interviewees commented that sector weeks are mainly used as networking versus learning venues. One interviewee said that sector weeks “center on relationship building with some formal components.”

• Categorizing retreats is problematic. Some KLB members believe that retreats are not learning activities but rather “resemble business meetings or organizational support activities because they are not always designed with a learning objective in mind and are indeed funded by management budget and not the learning budget.”

In contrast, two people indicated that retreats are learning activities and “should get the same formal treatment as workshops and conferences, particularly in regions with highly decentralized staff resources where managers use retreats as joint learning venues for Washington-based and

2 See appendix 2 for KLB’s mapping of “on-the-job” and external training to the LMS.

3

field staff. One member also added that retreats are a result of a formal learning plan.

• Study tours may be formal learning events. Three KLB representatives, while agreeing “in principle” with the proposed categorization, indicated a need to revisit the grouping of study tours. They said that study tours could be categorized as formal learning because they have a precise learning objective, are structured, are carefully planned, and include all aspects of a typical course.

• BBLs may have a formal component. Several KLB members noted that BBLs could be characterized as formal learning as they are the means for addressing learning demands, and “despite their nonformal setting, the learning that takes place is in BBLs is formal.” An example of BBLs with formal aspects is the annual “Ramadan BBL Series” organized by MNA. These series require the same level of planning and preparation as formal courses.

• Performance support tools should be considered formal if they are part of a course. Examples of performance supports that are “not stand-alone products” are EXT support tools for courses on behavioral change.3

The KLB members who did not agree with the proposed categorization of formal and nonformal learning raised several concerns:

1. The proposed list of nonformal activities is not exhaustive. The views of several KLB members are largely captured by one member who said that “nonformal learning includes peer-to-peer learning, working by doing, learning from mistakes, learning over a conversation during lunch, and hallway conversations.” Other interviewees added that “there is a lot of team learning and mentoring on substantive operational issues that are not captured [in the proposed categorization]”; that “extensive on-the-job training and experiential learning should be counted as nonformal learning” and that “nonformal mentoring, advisory groups (hotlines) and nonformal networks are also a part of nonformal learning.”

2. Nonformal learning is part of program delivery. One interviewee claimed that “nonformal learning takes place when you have a project or a strategy that you are sharing with others ─ this is part of program delivery and should not come from the learning budget.”

3. The separation of nonformal and formal learning is artificial. Two KLB members noted that integrating formal and nonformal learning is part of a broader (and necessary) integration of learning and knowledge sharing. Another member noted that “by categorizing learning into formal and

3 EXT also considers developmental assignments and short-term staff exchanges as nonformal learning.

4

nonformal we are trying to draw a line that is arbitrary, as it can go either way.”

In addition to KLB contributions to characterizing and defining nonformal learning, the authors conducted a literature search on formal definitions of nonformal learning (box 1).

Box 1. Definitions of nonformal learning

Throughout the literature on adult learning, nonformal learning is often defined in the context of — and in contrast to — formal learning. It is defined by the lack of “formal” attributes in the learning process. In contrast to formal learning, which is universally known to involve such attributes as a dedicated time and space, a set curriculum, transfer of knowledge from instructor to student and sometimes a certificate of knowledge acquired, nonformal learning is broadly defined as a cumulative learning process, unrelated to any particular setting whereby the individual gains knowledge (Tissot 2000), either intentionally or unintentionally (Eraut 2000) and it is not necessarily certified. Examples of definitions that define nonformal learning in the context of formal learning include:

“Education for which none of the learners is enrolled or registered” (OECD 1977).

“Any organized, systematic educational activity carried on outside the framework of the formal system, to provide selected types of learning to particular subgroups in the population” (Coombs and Ahmed 1974).

“Learning which takes place in the work context, relates to an individual’s performance of their job and/or their employability, and which is not formally organized into a program or curriculum by the employer. It may be recognized by the different parties involved, and may or may not be specifically encouraged” (Dale and Bell 1999).4

A problem in defining nonformal learning as learning that lacks formal attributes is that it leaves a broad and vague residual from all that is counted as formal learning – that is also too broad and vague for analytical purposes. In addition, the content of “formal” is likely defined differently in different settings (in some settings, formal learning is by definition one that leads to a certification). Definitions of nonformal learning that are independent of definitions for formal learning include:

“Nonformal learning consists of learning embedded in planned activities that are not explicitly designated as learning, but which contain an important learning element” (Tissot 2000).

“Nonformal learning is not provided by an education or training institution, and as a rule it does not entail certification upon completion of the study. Nevertheless, nonformal learning is structured, in terms of learning objectives, learning time or learning support. Nonformal learning is intentional from the learner’s point of view” (The European Commission 2001). “Boundaries between formal, informal and nonformal learning can only be meaningful drawn in relation to particular contexts, and for particular purposes” (Colley et al. 2002).

4 In this view, the nonconventional learning setting and the absence of curriculum are central.

5

Share of nonformal activities in total learning budget

The BW’s cost data cover only part of the learning events and the coverage of nonformal events is particularly problematic, so WBIEG relied on learning coordinators’ estimates of the nonformal share of their learning budgets. Interviews revealed that nonformal learning represented over one-quarter (27 percent) of budgets for staff learning. Nonetheless, the share of the budget allocated to nonformal activities varied significantly across units ─ from 60 percent in ISN to 5 percent in ACS, OPC, PREM, and WBI respectively (see figure 1 and table 2).

Figure 1. Share of nonformal activities in learning budgets by unit, FY04-06*

0%

10%

20%

30%

40%

50%

60%

70%

NS

I IEG

H

DN

LE

G

LCR

E

SD

A

FR

EA

P/S

AR

S

FR

MN

A

IFC

E

CA

IN

F H

RS1

S

EC

IS

G

GS

D

WB

IP

RE

M

OP

C

AC

S

27 percent =nonformallearningshare ofbudget

* Weighted by units. Excludes PSD, DEC and EXT for which data was not reported.

6

Table 2. Estimated share of nonformal learning in learning budgets, FY04-06

SLC/VPU Share of nonformal

budget, FY04-06

Average annual nonformal learning

budget, USD '000

Total average learning budget, USD '000

A. Regions AFR 35% 3,313 9,467

EAP/SAR 33% 2,819 8,543 ECA 25% 1,667 6,667 LCR 40% 2,187 5,467 MNA 25% 670 2,680

B. SLC ACSa 5% 18 367 ESSD 40% 683 1,708 HDN 50% 1,150 2,300 INF 20% 373 1,867 ISG 10% 40 400

HRSb 20% 2,405 12,027 ISN 60% 360 600 OPC 5% 150 2,990

PREM 5% 132 2,633

PSDc … … 395 C. Other KLB members

DECc … … 250

EXTc … … 513 IEG 50% 325 650 WBI 5% 17 340 SFR 30% 149 497

D. Observers GSD 10% 120 1,200 IFC 25% 2,617 10,467

SECa 10% 33 330 LEG 40% 287 717

Overall shared (unweighted) 26%

Overall Shared (weightede) 27%

Total budget USD '000 19,515 73,073

Source: Bank’s Business Warehouse. a The estimated share of nonformal learning for ACS and SEC0 is based on data from FY06. b The estimated share represents the sum of budgets for management and CET. c Did not provide an estimated share of budget for nonformal learning. d Excludes units for which data were not available. e Weighted by units’ learning budgets.

7

All learning coordinators reported that nonformal learning activities are financed from the learning budget, but the estimates of expenditures on nonformal learning do not reflect staff time expended on attending, organizing, or leading the activities. Proper costing of nonformal training should include such costs. Consequently, if costs for staff time were included, the percentages of budget resources devoted to nonformal activities would increase.

Share of nonformal learning in total number of events

The BW/LMS constitutes the only comprehensive, Bank-wide source of data on staff learning activities. To gauge the accuracy of its data and to determine the extent to which the BW/LMS includes (or excludes) nonformal learning events, WBIEG compared the shares of nonformal learning indicated in the BW/LMS to estimates received from those interviewed (figure 2).

Figure 2. Shares in number of nonformal learning events, FY04-06

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

AC

S-1

O

PC

-2

PR

M-3

ISG

-1

LCR

-2

IFC

-3

HR

S-4

E

AP

/SA

R-5

E

XT-

6 A

FR-7

E

SS

D-8

WB

I-1

INF-

2 E

CA

-3

LEG

-4

HD

N-5

O

ED

-6

FSE

/PS

D -7

D

EC

-8

MN

A-9

According to KLB Members According to WB/LMS data

Group 1 Group 3Group 2

Source: Authors’ calculations based on interviews with KLB members.

At the level of individual VPU/SLCs, WBIEG found many differences between data in the BW/LMS and learning coordinators’ estimates. Indeed, data from the two sources matched infrequently (group 1 in figure 2). In most of the remaining cases, there was a difference of at least 10 percentage points between the estimates from those interviewed and the data in the BW/LMS. In four cases (LCR, MNA, ISG, and DEC), the differences exceeded 30 percent (table 3). There is no benchmark for what constitutes an acceptable difference, but the differences are nonetheless noteworthy and emphasize the desirability of agreement among VPUs and SLCs about which learning events are formal and which are nonformal.

8

Table 3. Average number and shares of learning events in FY04-06

Annual average number of nonformal and formal events in FY04-06 (BW/LMS data)

Percentage share of nonformal and formal events in FY04-06 (BW/LMS data)

Annual average share in number of events in FY04-06 (Interview data)

SLC/VPU NF F Total NF F NF F a. Regions AFR 19.3 26.3 45.6 42% 58% 50% 50% EAP/SAR 13.3 62.3 75.7 18% 82% 33% 67% ECA 38.7 49.0 87.7 44% 56% 25% 75% LCR 29.3 57.7 87.0 34% 66% 70% 30% MNA 10.3 2.7 13.0 79% 21% 30% 60% Unweighted Total 43% 57% 42% 56% Weighted Total 111.0 198.0 309.0 36% 64% 61% 59% b. SLCs ACS 2.0 49.0 51.0 4% 96% 5% 95% ESSD 113.7 88.0 201.7 56% 44% 60% 40% HDN 141.7 61.0 202.7 70% 30% 50% 50% HRS 23.7 297.0 320.7 7% 93% 25% 75% INF 65.0 49.0 114.0 57% 43% 42% 58% ISG 250.7 805.3 1056.0 24% 76% 66% 34% OPC 5.0 85.7 90.7 6% 94% 5% 95% PREM 86.3 65.7 152.0 57% 43% 55% 45% FSE/PSD 12.0 12.7 24.7 49% 51% 10% 90% Unweighted Total 37% 63% 35% 65% Weighted Total 700.0 1,513.3 2,213.3 32% 68% 52% 48% c. Other KLB members DEC 9.5 14.0 23.5 40% 60% 0% 100% EXT 2.5 38.0 40.5 6% 94% 20% 80% IEG 7.0 5.7 12.7 55% 45% 30% 70% SFRV a … … … … … 30% 70% WBI 9.0 41.0 50.0 18% 82% 5% 95% Unweighted Total 30% 70% 17% 83% Weighted Total 28.0 98.7 126.7 22% 78% 11% 89% d. Observers GSDa … … … … … 70% 30% IFC 2.0 5.0 7.0 29% 71% 50% 50% LEG 27.0 18.7 45.7 59% 41% 40% 60% SECa … … … … … 10% 90% Unweighted Total 44% 56% 43% 58% Weighted Total 29.0 23.7 52.7 55% 45% 41% 59% e. Other 41.0 196.3 237.3 17% 83% … … Overall Unweighted 34% 66% 34% 65% Overall Weighted b 909.0 2,030.0 2,939.0 31% 69% 51% 49% Overall Unweighted w/out ISG

35%

65% 33%

66%

Overall Weighted w/out ISG

658.0

1,224.7

1,883.0

35%

65% 43%

57%

Source: BW/LMS data and interview data. Notes: … Data not reported. a SFR, GSD, and SEC learning activity counts were not included in the BW/LMS at the time of the study. b Weighted by the number of events in the BW/LMS.

9

Contrary to expectations for a pattern of underreporting of nonformal learning activities by the BW/LMS, WBIEG found that in 9 of the 20 units data in the BW/LMS were higher than those reported by the KLB representatives (group 3 in figure 2).

Such differences suggest the need for:

(a) Both the BW/LMS and learning coordinators’ estimates to be taken with a degree of caution and at best as estimates for the actual extent of nonformal learning; and

(b) The Office of CLO and KLB members to become advocates for a systematic reporting of learning deliverables in the BW/LMS database. To do so, they may consider establishing monitoring systems coupled with incentive schemes within their respective VPU/SLCs to ensure accurate, comprehensive, and timely entry of training activities in the LMS.

Discrepancies between BW/LMS database and KLB reporting of nonformal learning shares within individual VPU/SLCs are reflected on the overall average shares of nonformal events in the Bank. On average, the estimated weighted share of nonformal learning reported by KLB members was 20 percentage points higher than indicated in the BW/LMS database (table 3).5 Adjusting for the size of learning offerings by VPU/SLCs, nonformal learning represents about 31 percent of all staff learning according to BW/LMS but as much as 51 percent according to the KLB members.6

In addition to differences between data sources, WBIEG also found a variation in nonformal learning shares across individual VPU/SLCs. Eight KLB representatives reported that one-half or more of their learning events were nonformal: LCR and GSD (70 percent), ISG (66 percent), ESSD (60 percent), PREM (55 percent), AFR, HDN, and IFC (50 percent respectively). In contrast, six reported that only 10 percent or fewer of their activities were nonformal: FSE/PSD and SEC (10 percent), WBI, OPC, and ACS (5 percent each), and DEC (0 percent) (figure 2). Additional data on the distribution of formal and nonformal learning events by provider are in appendix 3.

5 The unweighted averages of shares of nonformal learning from both sources are similar. 6 The 20 percent gap decreases to 8 percent when ISG, the largest provider of learning activities, is excluded.

10

4. EVALUATION OF NONFORMAL LEARNING

A decision to evaluate nonformal learning will depend on at least four factors: (i) the definition of nonformal learning; (ii) the size and scope of nonformal learning; (iii) the KLB’s opinion of the usefulness of an assessment; and, (iv) the likely costs, which depend largely on the methodology used.7

Considering that nonformal training, as defined in this report, captures over one-quarter of the learning budget, an assessment of effectiveness and impact may be warranted. A majority of respondents agreed.

Proponents of evaluating nonformal learning indicated that an evaluation of such learning could be useful because it would:

1. Promote an institutionally accepted distinction between formal and nonformal learning;

2. Encourage the development of adequate measures of and benchmarks for effectiveness for nonformal events;

3. Create a much needed instrument for measuring the impact of nonformal learning events; and,

4. Assess the usefulness of the largest component of nonformal learning events in the Bank ─ i.e., brown bag lunches.

Proponents of an evaluation indicated that a decision to conduct such an evaluation should depend on: (i) the degree to which the evaluation would impose on the participants; (ii) whether the KLB would provide resources to learning providers for evaluation of their nonformal offerings; and, (iii) a clearly defined and well-articulated objective and purpose for the evaluation to “avoid evaluating for the sake of evaluating” and imposing another evaluation.

The KLB members who did not support an investment in evaluating nonformal learning provided the following reasons:

1. “An evaluation of nonformal learning will not add any value because nonformal learning is a demand-driven type of activity which will happen

7 The definition of nonformal learning will affect the feasibility of an evaluation. For example, if nonformal learning is defined largely as peer-to-peer learning and hallway conversations, then it is infeasible to evaluate it in any meaningful way. The amount of nonformal learning is also a concern. If there is little such learning, an evaluation might not be desirable.

11

regardless of any evaluation…and there is no value added associated with such an evaluation.”

2. “Nonformal learning is too vague a notion and marginal to our unit…There is little value in quantifying nonformal learning, and if a lot of nonformal learning takes place, then why don’t we formalize it?”

3. One interviewee said that “anything one puts resources into is worth evaluating, but a Bank-wide evaluation of nonformal learning is useless. WBIEG must distinguish between different types of nonformal learning and develop specific instruments to be used by the SLCs and VPUs themselves, for measuring the effectiveness of their various types of nonformal events.”

4. WBIEG should evaluate nonformal training “only if we could evaluate the gains of on-the-job training experiences ─ particularly with respect to impact of decentralization [field assignments]. The compelling contribution to learning is on-the-job learning versus BBLs, which are awareness raising, networking, etc.”

5. Because of the many ways that staff members learn, assessing the impact of nonformal learning would depend on anecdotal evidence.

6. “It will be very hard to evaluate [nonformal training] because the learning goals of [nonformal training activities] are not articulated.”

Should the KLB decide to proceed with an evaluation of nonformal learning, at least three approaches deserve consideration. First, WBIEG can develop instruments for measuring various types of nonformal learning and distribute them to VPUs and SLCs to conduct self evaluations. With this option, WBIEG would design a customized instrument for various kinds of nonformal learning. The instrument would be standardized to accommodate all professional and technical nonformal learning events throughout the World Bank Group. Each unit would be responsible for conducting self-evaluations using these instruments and reporting results to the KLB.

Second, WBIEG could add nonformal learning events to its population of current activity-level evaluations. These assessments seek participants’ opinions about such topics as quality, usefulness, and relevance. To date, one criterion for eligibility for such assessments is that activities are registered as formal events in the learning catalogue at least two weeks before the activity begins. This criterion could be amended or discarded, and WBIEG’s sample of events to assess could be drawn from the entire population of learning events, both formal and nonformal.8 WBIEG does not recommend this option because of the diverse nature of nonformal learning events and the unsuitability of the evaluation instrument for certain kinds of nonformal events, such as coaching and BBLs.

8 This option requires that WBIEG specify the duration of nonformal learning activities that would be eligible for evaluation.

12

Third, WBIEG could conduct a Bank-wide survey to assess staff views on the usefulness, effectiveness, and impact of nonformal learning relative to formal learning. Such a survey would have a “public benefit component,” in that it would be useful not only to KLB members but also provide recommendations for task managers to improve the effectiveness and usefulness of various types of nonformal training offerings.

A decision to proceed with an evaluation of nonformal learning must consider the two challenges inherent in such an endeavor, namely data constraints on nonformal learning activities and on costs.

Data constraints on nonformal learning activities are a reflection of the nature of nonformal learning versus possible inadequacies in the LMS. Certain types of nonformal learning, such as use of performance support tools, learning that occurs during videoconferences with colleagues around the world, and other on-the-job trainings (such as peer learning, coaching/mentoring, and developmental assignments) cannot ─ on a practical level ─ be captured in any systematic and useful way.9

Obtaining cost data on nonformal learning is as equally challenging as is quantifying on-the-job learning sessions. Task managers typically do not report the cost of their nonformal learning activities in the LMS, and cost data do not include staff time for preparation of and attendance at events such as BBLs.

WBIEG also recommends that the reliability of the BW/LMS be increased. Any evaluation of learning programs, including nonformal learning, would benefit from a robust, all-inclusive, and reliable dataset of learning events in the Bank. To improve the reliability of BW/LMS, WBIEG recommends that KLB provide and promote the use of a common, Bank-wide definition of nonformal learning and then advocate for a systematic reporting of learning deliverables in the BW/LMS database. To do so, members of the Board may wish to consider establishing monitoring systems coupled with incentive schemes within their respective VPU/SLCs.

9 Some nonformal learning events (such as individualized training in software use and short-term staff exchanges for special projects) will never be entered into the BW/LMS ─ not because they cannot be captured and accounted for, but because it is too time consuming and costly to do so.

13

REFERENCES

Colley, Helen, Phil Hodkinson, and Janice Malcolm. 2002. Nonformal Learning: Mapping the Conceptual Terrain: A Consultation Report. Leeds: University of Leeds Lifelong Learning Institute. Available at http://www.infed.org/archives/e-texts/colley_nonformal_learning.htm.

Coombs, P. H., and M. Ahmed. 1974. Attacking Rural Poverty. How Nonformal

Education Can Help. Baltimore: John Hopkins University Press. Dale, M., and J. Bell. 1999. Informal Learning in the Workplace. DfEE Research

Report 134. London: Department for Education and Employment. Eraut, M. 2000. “Informal learning, implicit learning and tacit knowledge in

professional work” in F. Coffield, The Necessity of Nonformal Learning. Bristol: The Policy Press.

European Commission. 2001. Making a European Area of Lifelong Learning a Reality.

Brussels: European Commission. Available at http://europa.eu.int/comm/education/policies/lll/life/communication/com_en.pdf

Jeffs, T., and M. Smith. 1990. Using Informal Education. Buckingham: Open

University Press. Livingstone, D.W. 2001. Adults’ Informal Learning: Definitions, Findings, Gaps and

Future Research. NALL Working Paper No. 21. Toronto: Ontario Institute for Studies in Education. Available at http://www.oise.utoronto.ca/depts/sese/csew/nall/res/21adultsifnormallearning.htm

Organization for Economic Co-operation and Development. 1977. Learning

Opportunities for Adults Vol IV: Participation in Adult Education. Paris: OECD. Prom-Jackson, Sukai, M. Palmisano, W. Jackson, R. Novojilov, and T. Melon. 2002.

An Evaluation of Nonformal Learning in Professional Technical Networks, 2000-2001. Report No. EG-03-61. Washington, DC” World Bank Institute.

Tissot, P. 2000. Glossary on identification, assessment and recognition of qualifications

and competences and transparency and transferability of qualifications. In Bjornavold, J. Making Learning Visible: Identification, Assessment and Recognition of Non-formal Learning in Europe. Luxembourg: Office for Official Publications of the European Communities.

14

APPENDIX 1

SCOPING NONFORMAL LEARNING IN THE BANK, FY04-06

Interviewee:

VPU/SLC:

Date:

1. Do you agree with the following categorization of nonformal (vs.

formal learning)?

Formal and nonformal types of events (LMS)

Nonformal Learning Formal Learning

ART/workout Course

Brown bag lunch Sector week

Coaching/mentoring Workshop

Team-based action learning Distance learning course

Retreat Language learning

Study tour Online course or module

Performance support tool Conference

YES

NO

Comments:

2. Based on the definition above, what percentage of your activities

(number of activities) were:

Nonformal _____________

Formal _________________

15

3. What was your learning budget in:

FY 2004 $

FY 2005 $

FY 2006 $

4. What is your best estimate for the SHARE of your budget spent on

nonformal learning in:

FY 2004 %

FY 2005 %

FY 2006 %

5. What is the primary source of funding of:

Nonformal learning activities?_________________________________ Formal learning activities? ____________________________________

Comments:

6. Do you see a need for an evaluation of nonformal learning in the Bank?

YES NO If yes, what are the key evaluation questions that the study should pose? _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

16

APPENDIX 2

MAPPING OF FORMAL AND NONFORMAL LEARNING

Category Type Formal/Nonformal

Face-to-face ART/workout Nonformal

Brown bag lunch Nonformal

Coaching/mentoring Nonformal

Conference Formal

Course Formal

Language learning Formal

Retreat Nonformal

Sector week Formal

Study tour Nonformal

Team-based action learning Nonformal

Workshop Formal

Online Distance learning course Formal

Online course or module Formal

Performance support tool Nonformal

On-the-job Peer learning Nonformal

Coaching/mentoring Nonformal

Developmental assignment Nonformal

Other on-the-job learning Nonformal

External External Formal

Videoconference Videoconference Nonformal

17

APPENDIX 3

NUMBER OF FORMAL AND NONFORMAL LEARNING EVENTS, FY04-06

SLC/VPU NF F Total NF F Total NF F Total

a. RegionsAFR 27 30 57 22 29 51 9 20 29EAP/SAR 21 36 57 13 108 121 6 43 49ECA 72 43 115 34 37 71 10 67 77LCR 57 64 121 29 65 94 2 44 46MNA 1 3 4 5 2 7 25 3 28

Total 178 176 354 103 241 344 52 177 229

b. SLCs ACS a 3 56 59 1 47 48 44 44ESSD 151 105 256 124 68 192 66 91 157HDN 177 65 242 143 46 189 105 72 177HRS b 20 371 391 28 315 343 23 205 228INF 88 38 126 73 52 125 34 57 91ISG 175 937 1,112 295 661 956 282 818 1100OPC 5 90 95 1 55 56 9 112 121PRM 115 73 188 104 68 172 40 56 96PSD 26 10 36 7 10 17 3 18 21

without ISG 585 808 1,393 481 661 1,142 280 655 935Total 760 2,553 2505 776 1,322 2,098 842 2128 2970

DEC 18 18 36 1 7 8 17 17EXT 2 35 37 3 44 47 … 35 35OED 9 4 13 5 12 17 .. 1 1SFR 2 … 2 … … … … … …WBI 12 39 51 13 57 70 2 27 29

Total 43 96 139 22 120 142 2 80 82

d. Observers GSDDR … 3 3 1 1 … 14 14IFC 2 1 3 … … … … 9 9LEG 22 17 39 31 6 37 28 33 61SEC … … … … … … … … …

Total 24 21 45 31 7 38 28 56 84

e. Other 59 182 241 48 208 256 16 199 215

Overall 1,064 3,028 3,284 980 1,898 2,878 940 2,640 3,580W/out ISG 889 1,283 2,172 685 1,237 1,922 378 1,167 1,545

c. Other KLB members

2004 2005 2006

Source: Bank’s Learning Management System and Business Warehouse. a ACS is HRAC in LMS. b HRS is the aggregate of HRSLO & HRSDL. … = Not reported.

18