©FASID

The Users Guide to L-E-A-D

Log-frame Evaluation Application Design Year 2004 version

- A New Approach for Policy and Program Evaluation -

FASID Foundation for Advanced Studies on International Development

March 2005

Log-frame Evaluation Application Design

L E A D

©FASID

Preface

A decade ago, the Foundation for Advanced Studies on International Development (FASID) developed PCM (Project Cycle Management) that has become the de-facto standard management tool of international development projects.

Nowadays, evaluation is needed not only for a project, but also for a program or a

policy under which multiple projects are conducted. In 2002, FASID established an inter-institutional study group, named “Study Group for Evaluation on Public Policy and Programs”, to develop a new evaluation method for the next generation to be applicable for policy and program levels. With intensive efforts of experts participating in the study group, a new method “LEAD” (Log-frame Evaluation Application Design) was developed. In April 2003, “Production Unit for LEAD Users Guide” (PULUG) was established by young members of the study group. Since then, the unit pursued the refinement of the new method to be more practical and useful, in order to enable the method to be more easy to use and thereby leading to significant improvements for the users. This Users Guide was written to extend the practical knowledge of “LEAD” to people who are interested in using this new evaluation method. The readers are assumed to be planners, practitioners and managing officers in international development. In addition, this booklet will provide a new insight for others working for various social programs, such as public officers in national or local government, and it may be useful even for officers in private companies who wish to comprehensively improve the evaluation of their own projects.

This Users Guide is intended to use plain and clear expressions in explanation for every reader to understand, whilst it may still be difficult or ambiguous for some readers. Therefore, any comments, complaints and suggestions are welcomed as “the evaluation for improvement”.

We hope that LEAD leads you to the improvement of your projects and programs.

International Development Research Institute, FASID

©FASID

Contents １．INTRODUCTION ..............................................................................................................................1

0-0. FROM “EVALUATION” TO “IMPROVEMENT” ......................................................................................2 0-1. ABOUT LEAD ...................................................................................................................................3 0-2. CONTENTS OF LEAD .......................................................................................................................4 0-3. STRUCTURE OF THE USERS GUIDE .................................................................................................6 0-4. CASE.................................................................................................................................................6

２．LEAD IN PRACTICE........................................................................................................................8

1. INITIATION PHASE.................................................................................................................................9 1-1. Evaluation Scope Planning ........................................................................................................9

a) Outline ......................................................................................................................................................... 9 b) Output Sample: Scope of Evaluation ......................................................................................................... 10 c) Explanatory Notes:..................................................................................................................................... 11

1-1-1.Evaluation for Whom? ............................................................................................................. 11 1-1-2. Preparing “Scope of Evaluation” ........................................................................................... 11 1-1-3. Setting Evaluation Criteria ..................................................................................................... 11 1-1-4. Cases Which can not be Applied to Theory-Based Evaluation like LEAD...................... 12 1-1-5. Good Relationship with Stakeholders .................................................................................. 12 1-1-6. LEAD as a Tool of Capacity Building.................................................................................... 13

2-1. Hierarchy Tree Development .................................................................................................14 a) Outline ....................................................................................................................................................... 14 b) Output Sample: Hierarchy Tree ................................................................................................................. 15 c) Explanatory Notes:..................................................................................................................................... 16

2-1-1. Hierarchy Tree ......................................................................................................................... 16 2-1-2. Building a Hierarchy Tree ...................................................................................................... 16 2-1-3. Tips for Building a Hierarchy Tree ........................................................................................ 16 2-1-4. Coverage of a Hierarchy Tree ............................................................................................... 17 2-1-5. The Project without Activities and the Activity without the Project Name ....................... 17

2. PREPARATION PHASE ........................................................................................................................18 2-2. Logic Tree Development ........................................................................................................18

a) Outline ....................................................................................................................................................... 18 b) Output Sample: Logic Tree ........................................................................................................................ 19 c) Explanatory Notes...................................................................................................................................... 20

2-2-1. Logic Tree ................................................................................................................................ 20

©FASID

2-2-2. Setting Overall Goal ............................................................................................................... 20 2-2-3. “Means-Ends” Relationship ................................................................................................... 21 2-2-4. Tips for Setting Intermediate Objectives .............................................................................. 22 2-2-5. Theory-based Evaluation by Building a Logic Tree............................................................ 22

２-3. Projects Mapping ..…………………………………..…………………………………………...........23

a) Outline ....................................................................................................................................................... 24 2-4. Log-frame Formation ..............................................................................................................24

a) Outline ....................................................................................................................................................... 24 b) Output Sample : Log-Frame ...................................................................................................................... 25 c) Explanatory Notes:..................................................................................................................................... 26

2-4-1. Log-frame................................................................................................................................. 26 2-4-2.Special features of Log-frame for LEAD ............................................................................... 27 2-4-3. Log-frame as a Tool for Management .................................................................................. 27 2-4-4. Converting a Logic Tree into a Log-frame ........................................................................... 28 2-4-5. Setting Indicators and Means of Verification ....................................................................... 29 2-4-6. Setting Important Assumptions ............................................................................................. 31 2-4-7. Setting Inputs........................................................................................................................... 32

3. EVALUATION PHASE ...........................................................................................................................34 3-1. Evaluation Study Design .........................................................................................................34

a) Outline ....................................................................................................................................................... 34 b)Output Sample: Evaluation Grid................................................................................................................. 35 c) Explanatory Notes...................................................................................................................................... 37

3-1-1. Evaluation Purpose and Evaluation Criteria ....................................................................... 37 3-1-2. Program Level Evaluation...................................................................................................... 37 3-1-3. Designing Evaluation Study................................................................................................... 40 3-1-4. Setting the Methods of Data Collection ............................................................................... 41 3-1-5. Data Collection for Factor Analysis ...................................................................................... 42

3-2. Evaluation Study Implementation ...........................................................................................44 a) Outline ....................................................................................................................................................... 44 b) Output Sample: Evaluation Grid with Results ........................................................................................... 45 c) Explanatory Notes...................................................................................................................................... 47

3-2-1. Implementation of the Evaluation Study .............................................................................. 47 3-2-2. Question of Attribution ............................................................................................................ 47 3-2-3. Sorting the Results of the Evaluation Study ........................................................................ 49

3-3. Conclusions, Recommendations and Lessons Learned .........................................................50 a) Outline ....................................................................................................................................................... 50 b) Output Sample : Conclusions, Recommendations and Lessons Learned ................................................... 51 c) Explanatory Notes...................................................................................................................................... 52

©FASID

3-3-1. Drawing Conclusions.............................................................................................................. 52 3-3-2. Drawing Recommendations .................................................................................................. 52 3-3-3. Drawing Lessons Learned ..................................................................................................... 53 3-3-4. Preparing Presentation Materials ......................................................................................... 53

4. IMPROVEMENT PHASE .......................................................................................................................54 4-1. Feedback .................................................................................................................................54

a) Outline ....................................................................................................................................................... 54 b) Output Sample: Commitment to Improvement .......................................................................................... 55 c) Explanatory Notes...................................................................................................................................... 56

4-1-1. Commitment of Improvement ................................................................................................ 56 4-1-2. A Participatory Approach in Making the Commitment of Improvement ........................... 57

4-2. Improvement Planning .............................................................................................................58 a) Outline ....................................................................................................................................................... 58 b) Output Sample: Improved Log-frame ........................................................................................................ 59 c) Explanatory Notes...................................................................................................................................... 61

4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary ................................ 61 4-2-2. Workshop for Improving the Log-frame ............................................................................... 61 4-2-3. Ex-ante Evaluation of the Improved Log-frame .................................................................. 62

4-3. Improvement Implementation .................................................................................................63 a) Outline ....................................................................................................................................................... 63 b) Output Sample: Improvement Cycle.......................................................................................................... 64 c) Explanatory Notes...................................................................................................................................... 65

4-3-1. Progress Monitoring ............................................................................................................... 65 4-3-2.Monitoring System ................................................................................................................... 66 4-3-3. A Periodic Evaluation for Further Improvement .................................................................. 67

5. EPILOGUE: EVERLASTING CYCLE OF IMPROVEMENT ........................................................................68

©FASID 1

１．Introduction

©FASID 2

0-0. From “Evaluation” to “Improvement”

By definition, evaluation is to ascertain the achievement of a project or a program (i.e. a group of projects or activities) from a particular valuating point of view. Project cycle has long been considered in the general concepts of project management as follows.

The project cycle begins with making a plan, and is followed by the implementation process to carry out the contents of the plan. In following, evaluation is conducted by comparing the results of the project with the original plan. The evaluation is connected to the improvement of plan making in the next cycle. The project cycle works best when a well-documented original plan is prepared at the planning stage. This condition is, however, difficult to be realized in reality.

Moreover, “policy evaluation” or “program evaluation”, which evaluates a group of projects with similar overall goals, is one of the most popular but challenging issues in evaluation studies. However, there is not any standardized method to undertake it.

In both cases, the critical point is how to make the proxy plan to be compared with the results. Thus, it calls for a new approach that combines the various projects (or activities) and integrates them into a body of a program that can be evaluated in a logical manner. In this case, the cycle is not commenced from the planning stage, but rather it starts from the evaluation process to improve the plan. This is called “the cycle of improvement.”

Planning

(Plan)

Implementation

(Do)

Evaluation

(See)

①

②③

Planning

(Plan)

Implementation

(Do)

Evaluation

(See)

④ ②

③

Integration (Combine)

①

Cycle of Improvement

Program BProgram CProgram A

©FASID 3

0-1. About LEAD

LEAD stands for Log-frame Evaluation Application Design. This method is based on the PCM (Project Cycle Management) evaluation method that is the de-facto standardized method for evaluating international development projects of Japan and other donors. LEAD employs its functions to be applicable for the program-level evaluation. This method is applicable for the following needs.

- To evaluate various types of projects implemented independently, but

having similar overall goals as a comprehensive program. - To evaluate multiple projects with similar overall goals conducted by

various agencies as a comprehensive program. - To link the results of evaluation with improvement of management.

©FASID 4

0-2. Contents of LEAD

LEAD consists of several steps in four phases as follows.

Phase Step Issues to be addressed

1. Initiation Phase

1-1. Evaluation Scope Planning

For what, and to what extent will it be evaluated?

2. Preparation Phase

2-1. Hierarchy Tree Development 2-2. Logic Tree Development 2-3. Projects Mapping 2-4. Log-frame Formulation

What is the structure and content of the target program?

3. Evaluation Phase

3-1. Evaluation Study Design 3-2. Evaluation Study Implementation 3-3. Conclusions,

Recommendations and Lessons Learned

What are the results of the evaluation?

4. Improvement Phase

4-1. Feedback 4-2. Improvement Planning 4-3. Improvement Implementation

What should be improved? How is it to be improved?

Each step produces output as follows.

©FASID 5

Road Map：LEAD Method, Process and Outputs

2-1 Hierarchy Tree

Scope of Evaluation

Evaluation Result

LEAD Hierarchy Tree

LEAD Logic Tree

Projects Map

LEAD Log-frame

Evaluation Grid

Conclusion, Recommendation and Lessons

Learned

Action Plan for

Implementation

Continued Improvement

Based on the Cycle of

Improvement

1-1 Scope of Evaluation

4-3 Implementation of

Improvement Plan with Log-frame

4-2 Planning of Improvement

with Log-frame

3-3 Conclusion, Recommendation

and Lessons Learned

3-2 Implementation of Evaluation

Research

3-1 Evaluation Design

2-4 Log-frame

2-3 Projects Mapping

2-2 Logic Tree

Outputs LEAD Steps

Improved Log-frame

4-1 Feedback Commitment of Improvement

©FASID 6

0-3. Structure of the Users Guide

The LEAD Users Guide explains each step in three parts, those of “Outline”, “Output Sample” and “Explanatory Notes”.

a) “Outline” clarifies the main parts: objectives, procedures and output, on one page. b) “Output Sample” shows an example of output from an imaginary case. c) “Explanatory Notes” covers important points in a more detailed explanation.

0-4. Case

This guide employs an imaginary case as follows, and to be used for “Output Samples”.

Case: Evaluation of Assistance of Country J for Improvement of

Second-Grade Education in the Republic of Amelia.

The Republic of Amelia is located on the west coast of the African continent. Country J has

been assisting the development of Amelia since its independence in the 1960s, when Amelia

had been considered as one of the least developed countries. Recently, Amelia has been witnessing the significant

growth of agriculture as the core sector of the national economy, with the exports of its agricultural products having

been increasing., This has now drawn Amelia away from the group of poorest countries.

Along with national economic growth, however, the economic disparity between the rich and the poor has been

aggravated, and the trend seems set to continue without any intervention. As one of the main factors for the

disparity, the educational disparity is considered the most serious and fundamental factor, according to the results

of some social studies that have been conducted. Although more than 90% of Amelians have graduated from the

elementary schools on average, the secondary school education is not well extended yet in Amelia since the

graduation rate against the total population is still less than 40% on average.

In order to improve this situation, Country J has been implementing three projects and a scholarship program in

the capital city of Amelia for a decade.

List of Project Name

(Department of MOFA of Country J)

・Rehabilitation Project of Secondary School Facilities.

（Dept. of Grant Aid）

・Training Project of Secondary School Teachers

（Dept of Technical Assistance）

・Development Project of Secondary Education Curriculum

©FASID 7

（Dept of Technical Assistance）

・Scholarship Program for Secondary Education

（Dept of International Communication）

These projects are conducted independently by various departments of the Ministry of Foreign Affairs of Country

J, with no comprehensive evaluation yet being carried out. In order to improve effectiveness and efficiency of

country J’s assistance for Amelia, MOFA calls for a comprehensive evaluation from a broader point of view.

In addition, Country M, one of the other main donors in this field, has been conducting a “Project for Capacity

Building of Education Administrators”. In the comprehensive evaluation, this project needs to be considered as one

of the influential factors.

©FASID 8

２．LEAD in Practice

©FASID 9

1. Initiation Phase

1-1. Evaluation Scope Planning

a) Outline

Objective: To comprehend the needs of users (or clients) of evaluation in order to clarify the purpose and the scope of evaluation

Procedures:

1. Clarify the clients for whom evaluation is conducted, to whom the results will be fed back. (Who will be the user of evaluation?)

2. Set the purpose of the evaluation based on the needs of the evaluation.

3. Determine the scope (area) of the evaluation in order to deal with the needs.

4. Consider the main stakeholders, such as evaluators, clients, supporters, etc.

5. Consider the documents needed for the evaluation research.

6. Put the above into a “Scope of Evaluation” that will be shared with stakeholders.

Output:

Scope of Evaluation

©FASID 10

b) Output Sample: Scope of Evaluation Scope of Evaluation

Evaluation Purpose (What is the purpose of

evaluation?)

To evaluate assistance of Country J to Amelia’s secondary

education from a broader point of view, in order to redesign the

coordination of projects into an effective and efficient program.

Projects and Programs Concerned (What will be evaluated?)

・ Rehabilitation Project of Secondary School Facilities.

・ Training Project of Secondary School Teachers.

・ Development Project of Secondary Education Curriculum

・ Scholarship Program for Secondary Education

Evaluation Criteria (What viewpoints or criteria will be

focused in evaluation?)

・ Effectiveness (including impact)

・ Efficiency Etc.

Needs of Clients and Users (Who will be the direct

beneficiaries of evaluation?)

・ Management officers of Country J’s aid

（Needs: to improve the assistance program of secondary

education in Amelia.

Stakeholders/ Supporters (Information sources?)

・ Management officers of Country J’s aid ・ Project staffs of concerned projects ・ Relevant agencies of secondary education in Amelia ・ Ministry of Foreign Affairs of Amelia, etc.

Evaluation Team (Who conducts the evaluation?)

・ Ms. A (Leader) ・ Mr. B (Education expert) ・ Mr. C (Coordinator)

Schedule of Evaluation (How long is evaluation conducted

for?)

Period : June to December, year 20xx.

Progress Report: Monthly (to clients)

Final Report : December, year 20xx

Reporting and Feedback (How will the results be fed back?)

In addition to monthly reporting, the final report will be prepared

and submitted to stakeholders, including the clients.

Post-evaluation commitment (How will the evaluation be

utilized?)

By reviewing the evaluation results, Country J will draft a

redesigned assistance program for secondary education in

Amelia, which will be implemented after reaching a consensus

with relevant organizations in Amelia.

Budget No more than 30 thousand USD.

©FASID 11

c) Explanatory Notes: 1-1-1.Evaluation for Whom? The implementation of evaluation also requires the concept of result-based management.

Evaluation needs to achieve expected results, which should be clarified prior to the commencement of

the evaluation, by asking “what is the purpose of the evaluation?”, or “whose needs does this evaluation

expect to satisfy?”. Without clarification of such basic issues, the evaluation tends to be “evaluation

just for evaluation sake”, and which is pointless. Evaluation is conducted because of a context in which

there are stakeholders who need the evaluation to be undertaken for specific reasons. The evaluator

should comprehend the needs of the clients or the users of the evaluation results, in order to conduct

effective and efficient evaluation.

1-1-2. Preparing “Scope of Evaluation” Before beginning an evaluation, a Scope of Evaluation needs to be prepared. This is not

only for the evaluator who confirms the scope of works, but also for stakeholders (including

clients or users) in order to understand the evaluation’s expected outcomes and limitations. By

clarifying this in the printed document, the understanding of the evaluation can be accurate. When it

is necessary to change the contents of the evaluation because of, for instance, a lack of information

sources, the change can be properly done through the Scope of Evaluation, which works as a compass.

Sharing the Scope of Evaluation among stakeholders enables more support and collaboration for the

evaluation to be obtained.

It is important to note that the shown Scope of Evaluation is just one example template. The

evaluators should configure their own Scope of Evaluation by adding or deleting components according

to the needs of their specific evaluation. Any other topics that need to be evaluated should be clarified

in detail and confirmed with the stakeholders.

1-1-3. Setting Evaluation Criteria The evaluation criteria are particular points of view with which an evaluation is conducted. DAC

(Development Assistance Committee) under the OECD recommends five criteria for the evaluation of

international cooperation projects/ programs. Many international cooperation agencies employ all or

some of the five criteria for their projects and programs.

Efficiency.................................. Project has produced outputs with least inputs?

Effectiveness ............................ Project effectively achieved the project purpose?

Impact ...................................... Project accrued positive or negative impacts?

Relevance ................................. Project is still relevant to the needs of stakeholders?

Sustainability........................... Project’s outcomes can be sustained?

©FASID 12

There may be other criteria such as “urgency”, “equity as a public project”, “coherence with other

projects”, “validity and coverage of beneficiaries”, etc. The evaluation criteria can be different to meet

the purpose or the needs of evaluation.

Evaluation criteria is selected (even just tentatively) and clarified in the Scope of Evaluation, which

should be presented to the stakeholders, including the clients, before the start of the evaluation.

In reality, the greater the number of evaluation criteria, the more time and cost is expended.

Including all of the evaluation criteria is wasteful, not only in the matter of efficiency, but also in the

quality, since analysis with too many tasks tends to be flawed. On the other hand, selecting evaluation

criteria only by the consideration of time and cost is not appropriate in order to satisfy the needs of

evaluation. In practice, it is a matter of balancing the evaluation needs with the available resources.

1-1-4. Cases which can not be Applied to Theory-based Evaluation like LEAD There are some needs that can not be satisfied by LEAD. For example, the need to evaluate a model

project for poverty alleviation implemented in several places all over the world, or the need to evaluate

the several impacts which road construction would accrue, are not very applicable to LEAD. LEAD is

relying on the program theory (i.e. Logic Tree) that utilizes the logical relationship between “Means”

and “Ends”. If the “Means-Ends” relationship can not be developed with activities of target projects (or

programs), then evaluation by LEAD can not be proceeded. A model project of poverty alleviation that

is implemented at dispersed places can not be “combined” into a single logical diagram (a so called

“Logic Tree”), so it is impossible to apply a theory-based evaluation method like LEAD. With a

multiple impacts evaluation on road construction, applying LEAD is not recommended since it is not

necessary to form a complex logical model for evaluation.

Although it is true that advantages of LEAD are not fully mobilized in these cases, this Users

Guide covers various topics that may be very useful to all types of evaluation.

1-1-5. Good Relationship with Stakeholders In evaluation, establishment of a good relationship with stakeholders is a very important issue. It is

the project (or program) implementer who will receive and turn recommendations from evaluation into

actions. The recommendations will not be implemented when the project implementers have the

feeling of resistance towards the evaluator, much like the relationship between the criminal and the

judge or between the artist and the critique. Even in collecting the information, if the evaluator’s

attitude is over critical or judgmental, the evaluator will not be able to obtain the real information

from stakeholders, especially the implementers.

The collaboration of stakeholders, including the implementer, is indispensable for conducting LEAD,

which needs the formation of a logical diagram of target projects (or programs) at the preparation

©FASID 13

phase with these stakeholders. Thus, LEAD is a cooperative evaluation

method rather than a judgmental or criticizing one. In other words, LEAD is

“formative rather than summative.” The evaluator must fully understand

this characteristic of LEAD, and they must make the best effort possible for

forming a good relationship with all the stakeholders.

It is, however, a matter of balance. An evaluator that accepts all of the

demands and information from the stakeholders will lose the credibility in the end. The evaluator

must be independent and preserve the mind of objectivity and fairness.

1-1-6. LEAD as a Tool of Capacity Building In case that the evaluation is a program level evaluation concerned with several projects by multiple

agencies, it is necessary to clarify the coordinating agency who realises the recommendations from the

evaluation. If the coordinating agency is the local government, the evaluation team should contain

officers from the local government from the commencement of the evaluation. The participation of the

officers should be integrated into the evaluation, i.e. it is not just by the name of the team or the

attendance of periodical meetings. The evaluation should be designed in accordance with the

understanding of the participating officers. It is no one but the participating officers who actually

improve the projects with their coordinating capabilities. Coordination of multiple projects needs the

competent capacity of understanding and analysing of the projects. Thus, the participation in LEAD is

very effective capacity building for the coordinating officers.

Planning and implementing of LEAD are led actually by the evaluator (or the evaluation team), but

involvement of officers working for the implementing agency or a coordinating agency will enable the

recommendations of the LEAD evaluation to be more feasible and effective. By participating in LEAD

from the preparation phase, participating officers will learn the strengths and weaknesses of the

target projects (programs) at full length. Thus, the recommendations and lessons drawn from such a

participatory evaluation will be more concrete for realisation.

©FASID 14

2. Preparation Phase 2-1. Hierarchy Tree Development

a) Outline

Objective: To describe the whole picture of target projects (including activities) by stratifying them into a hierarchical tree.

Procedures:

1. List the name of all projects (or programs) to be evaluated.

2. List all of the main activities of the projects.

3. Arrange (or stratify) the projects into a hierarchical tree according to the way of the institutional arrangement or as expressed in reports (e.g. by implementing agencies or administrative departments).

4. Confirm the Hierarchy Tree with all stakeholders to form a consensus of understanding.

Output:

Hierarchy Tree

©FASID 15

b) Output Sample: Hierarchy Tree

J's assistance

Dept. of Grand Aid

Rehabilitation Project of Secondary School

Facilities

Dept. of Teachnical Assistance

Dept. of International Communication

Training Project of Secondary

School Teachers

Scholarship Program for Secondary Education

Development Project of Secondary

Education Curriculum

Construction of a building

Development of

Curriculum & Syllabus

Improvement of textbook program

Provision of scholarship

Development &

maintenance of database

Provision of Toilet

Procurement of teaching materials

Trainig on maintenance

and management

of facilities

・Omitted・

Organization of a

committee of textbook

development

Development of Teaching Materials

Provision of a water tank

Female education support

Public relations for female educaiton

Provision of free

textbooks

Training for Teachers

Implementation of a training

program

Training for teachers

Education for female education

Training female teachers

Developmeof manualon financimanagemen

・Omitted・

Development of a

guidebook on teaching materials

Development of a

guidebook on teaching methods

Planning of training

program for teachers

Curriculum Development

・Omitted・

Hierarchy Tree

Issue a news letter on female

education

Research on the

promotion of femnale

educaiton ・Omitted・

・Omitted・

Responsible Department

Projects

Activities

Agency

©FASID 16

c) Explanatory Notes: 2-1-1. Hierarchy Tree It is difficult to understand the whole picture of the projects to be evaluated when the numbers and

areas of the projects are many. In some cases, the project name exists but the activities are not clear

or not actually conducted. In other cases, the evaluation must cover some schemes like a scholarship

program that does not include proactive activities or regulations like legal punishment to prevent

crimes. The LEAD method needs to construct the logical “means-ends” relationship from these sorts

of things before proceeding to the phase of evaluation study. It is, however, very difficult to form the

logical relationship directly from these “raw” materials, and it is somewhat dangerous in the sense

that some important components may be lost when forming the rigorous logic at the beginning. Thus,

the preliminary stage is just to construct a diagram by following the existing institutional approach,

or the existing hierarchical order. The diagram is called “Hierarchy Tree ”. Building a Hierarchy Tree

is considered the preparation for the next stage of building a Logic Tree. For stakeholders, a

Hierarchy Tree would be more familiar than a Logic Tree and a Log-frame, therefore the Hierarchy

Tree would help their future understanding of the Logic Tree and Log-frame that are developed later.

2-1-2. Building a Hierarchy Tree It is recommended to build a Hierarchy Tree cooperatively with the main stakeholders based on

relevant documents and opinions of other stakeholders. It is begun with establishment of a database

into a table with a spread sheet software such as “Excel”. Database contains “Name of agency”,

“Name of department”, “Name of project”, “Main contents”, “Budget”, and so on. The data is put in

order of each column in a table, because it is easy to change it into a hierarchy tree. While a

table-style is handful as a database, the tree-style like “a hierarchy tree” is more powerful in

presentation. To draw the tree, it is recommended to use the drawing functions in a spreadsheet

software program, or in a drawing software program such as “Inspiration”. A more effective method

of building a Hierarchy Tree is undertaking a participatory hands-on workshop using paper cards

on which the name of the projects and activities are described. The Hierarchy Tree is then

constructed by arranging these cards on a large sheet of paper on a wall. Such a participatory

workshop is helpful in building a consensus among the stakeholders.

2-1-3. Tips for Building a Hierarchy Tree As indicated in the name, a Hierarchy Tree stratifies the projects to be

evaluated according to the way of the institutional arrangement. One of the

most common forms of arrangement is “by department” or “by type of

scheme” (e.g. technical cooperation or grant aid). The hierarchy (or

stratification) can be made into multiple layers. For example, the first layer

is “by country”, the second layer is “by organization”, the third layer is “by

An increase in

agricultural production in the Country

of C

Department A Department B

Program A Program B Program C Program D

©FASID 17

scheme”, the fourth layer is “by project name”, and the lowest layer is “by activities”. It is, however,

not recommended to build a complex diagram that has too many layers. The purpose is to build the

most recognizable diagram that can be understood easily by the main stakeholders. Thus,

participation of the main stakeholders in building a Hierarchy Tree is important.

2-1-4. Coverage of a Hierarchy Tree When there is any project or activity other than the target projects that influence the common

objectives, it is recommended to include them into a Hierarchy Tree. This is because such information

will be a useful reference when building a Logic Tree in the next stage, and when considering the

important assumptions in the Log-frame. In our case, the project conducted by Country M is

considered as one of the important external components, influencing the achievement of the outcome

of the target projects. Therefore, the Hierarchy Tree includes the Country M’s project within it.

2-1-5. The project without Activities and the Activity without the Project Name In developing a Hierarchy Tree, you may find a project that exists on paper but no actions are

taken at all, or you may find some activities that are not formed as a project. Moreover, there is an

institutional regulation, such as law with punishment, or a scheme like a scholarship program that

does not need activities, but that does influence the objectives. In making a Hierarchy Tree, such

things must be put in just as they are. For example, activities without any project name are put in

without a name. In this stage of building a Hierarchy Tree, it is not necessary to make a logical

arrangement. Instead, one should try to express the structure of subjects in a naturally and

institutionally accepted way. It is the preliminary stage, and so the Hierarchy Tree can be an

unorganized one as long as it reflects the reality.

Describe it as it is!

©FASID 18

2. Preparation Phase

2-2. Logic Tree Development

a) Outline

Objective: To clarify the logical structure of the target projects as a whole program by arranging their activities along the relationship of “Means-Ends”.

Procedures: 1. List all activities from the Hierarchy Tree and place them in

the bottom line.

2. Set an Overall Goal (a policy or program level goal) by referring to documents and opinions of stakeholders.

3. Consider Intermediate Objectives which bridge between activities and the Overall Goal in a logical sequence.

4. Put activities with the same (or very similar) Intermediate Objectives into groups.

5. Iterate the above processes to complete the Logic Tree in which “Activities – Intermediate Objectives” and “Intermediate Objectives – Overall Goal” can be expressed by the logical relationship of “Means-Ends”.

6. Confirm the Logic Tree with the consensus of stakeholders.

Output:

Logic Tree

©FASID 19

b) Output Sample: “Logic Tree”

Enrollment of middle school

education is improved in

Amelia

Running regular classes

Construction of building

Training on maintenance

J's Assistance

Improve a middle school

curriculum

Teacher's teaching methods improve

Students' morale toward

schooling increases

Equal educational opportunity for both boys and girls

Provision of a water tank

Provision of bathroom

Modification of curriculum and syllabus

Making a guidebook of teaching material

development

Organizaiton of Committee

for the improvement of textbooks

Training for teachers

Acceptance of trainees

Training for top managers

Provision of scholarship

Support for female

education

Procurement of teaching material

OverallGoal

Intermediate Objectives

Activities

Provision of free

textbooks

Training for female

education

Provision of free

textbooks・Omitted・・Omitted・

Logic Tree

Planning of a training program for teachers

Making a guidebook on teaching methods

Development of teaching materials

・Omitted・Omitted ・Omitted・

©FASID 20

c) Explanatory Notes 2-2-1. Logic Tree The Logic Tree expresses the logical structure of projects and activities to be incorporated into a

proxy program by reconstructing the logics of the components. Establishing a Logic Tree is necessary

to conduct a comprehensive evaluation. With the Logic Tree the following aspects can be evaluated.

- Holistic effects of various projects as a comprehensive program

- Relationships among projects and activities

- Single effects of projects, including the indirect effects

It is the evaluator who is responsible for developing a Logic Tree but the building process must be

participatory with stakeholders, as with the development of a Hierarchy Tree.

2-2-2. Setting Overall Goal Overall Goal is the common goal for which all related projects and activities are conducted. It is set

at the top of a Logic Tree, so that there is only one Overall Goal. If there is no Overall Goal identified in

advance, the evaluator sets a new Overall Goal. In setting this Goal , care should be taken when

deciding how to express it. It is NOT the summary of what the projects (or program) will do, but it

expresses a positive change or impact on the beneficiaries that the projects (or program) are intended

to accrue.

Bad and good examples of an Overall Goal in evaluating traffic-related projects are:

(Bad Example) × Traffic infrastructure is upgraded.

Overall Goal

(Good Example) ○ Traffic congestion is reduced.

In this case, the ’bad’ example is just describing what the projects do, and does not say what the

purpose of the traffic infrastructure development is for. Thus, it should be written in such a way as

that stated in the ‘good’ example, as this describes the outcome (or impact) of the development projects.

It can be said that the traffic infrastructure is not only for vehicle traffic, but also for pedestrians as

well. In this case, the expression of the Overall Goal could be “The city becomes a place where the

citizens feel able to be transported more easily”. This expression is a wider concept compared with the

original one. If an expression is too vague, some definitions may be added to the Overall Goal.

In practice, some clues for setting the Overall Goal can be found within the main goals of each

project, which is set above the direct project purpose. Referring to the needs and the purpose of

©FASID 21

evaluation, or meeting with stakeholders such as the program officers and representative beneficiaries,

will also give good clues. It is important to determine the Overall Goal by reaching a consensus among

stakeholders as explained in the previous steps.

2-2-3. “Means-Ends” Relationship The Logic Tree consists of components such as the Overall Goal, Intermediate Objectives and

Activities, among which the relationships are constructed in the logics of “Means-Ends”.

As stated below, an Activity “to teach mathematics to a student” is one of the Means for the End

(objective) that “the student can calculate”. For achieving this objective, there may be other means,

such as “to provide a textbook of mathematics”. Moreover, the statement that “the student can

calculate” can be one of the means for a higher objective such as “the student can promote to the next

grade”. Like this, a Logic Tree can be expanded flexibly.

The above Logic Tree may not be accepted by some readers. For example, it can be criticized that “to

provide a textbook” is not a direct means and that, instead, this should be “the student to study by

himself”. In practice, all Logic Trees can be true for the developers who create the tree with their own

logic. In other words, there is no “perfect” Logic Tree that everyone in the world would agree to as

being the ultimate model. A Logic Tree should of course be developed in an objective and rigorous way.

It is, however, impossible to form such a perfect model, because human beings cannot escape from

subjectivity. The state of objectivity results from a reorientation of the subjective opinions of human

–beings, in reality. In LEAD, the Logic Tree is a basis of following the evaluation, so it is necessary to

make a consensus on the Logic Tree with the stakeholders, including the groups for whom the

recommendations will be made. Thus, the evaluator should be good at not only creating logic, but also

dealing with the likely diverse opinions of stakeholders in order to reach a consensus.

The student can promote to the next

The student can calculate

To teach mathematics

End

↑

Means

End

↑

Means

To provide a textbook

©FASID 22

2-2-4. Tips for Setting Intermediate Objectives The Logic Tree consists of three layers such as the Overall Goal, Intermediate Objectives and

Activities. In practice, however, it is difficult to set Intermediate Objectives to be filled between the

Overall Goal and Activities at first. The following procedures are recommended.

For projects that have their own project purposes, just assume the project purposes as candidates for

the Intermediate Objectives, rather than decomposing them into activities. After setting the tentative

Intermediate Objectives directly from the project purpose, extract unnecessary activities which can not

be connected to Intermediate Objectives, and consider whether their purposes can become

Intermediate Objectives or not. If there are just activities or schemes without clear-cut purposes,

consider what they are intended to achieve, then set the Intermediate Objectives. When all activities

are set under Intermediate Objectives, check if there are any Intermediate Objectives that are the

same, or very similar. If you find any, then group them under a common Intermediate Objective.

Where there are a lot of projects and activities, input the information into a spreadsheet program

such as Excel. Put in the project names and main activities, and make a rough categorization of the

activities into several groups. By using the sorting function of the software, put the activities into

groups. Then consider the common objective of each group as Intermediate Objectives.

2-2-5. Theory-based Evaluation by Building a Logic Tree The establishing process of a Logic Tree is considered as evaluation, since this process can

distinguish unnecessary activities and illogical projects with unclear purposes. In evaluation, it is very

important to know which activities (or projects) are not necessary to achieve the Overall Goal. You

may be faced with an extreme situation in which you can not construct a Logic Tree since the purposes

of the activities are too diverse to be considered as a program. Continuing to the next step by setting an

irrational Logic Tree is a waste of time and resources. In this case, the evaluator can recommend fast

tracking the reforming of the program, jumping to the improvement phase to start reforming the

program as a new plan, rather than doing a wasteful evaluation study with such an illogical Logic

Tree.

Improvement

Phase

Preparation Phase

Research and Analysis Phase

Scope of Evaluation

Improvement Planning

Feedback

Implementation of Improvement Plan

Logic Tree

Hierarchy Tree

Log-frame

Evaluation Study Implementation

Evaluation Study Design

Conclusion, Recommendations and Lessons

No logic at all….

Need for re-planning.

Initiation Phase

©FASID 23

2. Preparation Phase 2-3. Projects Mapping

a) Outline

Output: Projects Map

Objective: To make geographical map that shows areas where are covered by each project.

Procedures: 1. Collect information in terms of areas where activities are

implemented or positive impact is expected to be reached by each project.

2. Draw above areas on the map by different colour. 3. Confirm the projects Map with the consensus of stakeholders.

©FASID 24

2-4. Log-frame Formation

a) Outline

Objective:

To form the Log-frame which describes the main components of the program based on the Logic Tree.

Procedures: 1. Transfer the components in the Logic Tree into corresponding

cells of “Narrative Summary” of the Log-frame.

2. Set external factors influencing the program into “Important Assumptions” of the Log-frame.

3. Set quantifiable indicators for Overall Goal and Intermediate Objectives.

4. Set “the Means of Indicators”

5. Set “Inputs” for enabling Activities

6. Confirm the Log-frame with stakeholders

Output:

Log-frame (Logical Framework)

©FASID 25

b) Output Sample: Log-frame

Log-frame of Country J’s assistance for secondary education for Amelia

Target Area: The Capital of Amelia Narrative Summary (of

Program/Policy) Objectively Verifiable

Indicators Means of Verification Major Important

Assumptions Super Goal： ・ Poverty in the capital of

Amelia is reduced.

・ The population below the

poverty line is less than 10% in the capital of Amelia in the year XXXX.

・ National statistics from the

Ministry of Social and Health.

・ UNDP Statistics Year Book.

Overall Goal： ・ The secondary school

enrollment in the capital of Amelia is improved.

・ The rate of students receiving

secondary education becomes more than 80% in the city in the year XXXX.

・ The ratio of graduation/entrance of secondary students becomes more than 1 to 10 in the year XXXX.

・ National statistics on

education derived from the Ministry of Education.

・ Government policies

on secondary education are kept unchanged.

・ Other planned poverty reduction programs are implemented.

Intermediate Objectives： 1. The number of school

facilities with sufficient educational function is increased.

2. School curriculum is improved.

3. Teachers’ skills and

knowledge are improved. 4. Students’ morale toward

schooling increases. 5. Accessibility of girls’

schooling is improved.

1-1. More than 90% of schools in the

city meet the national school facility standard level A.

2-1 An improved school curriculum

is endorsed by the educational committee and the third party in the year XXXX.

2-2 More than 80% of schools in the city introduce an improved curriculum in the year XXXX.

3-1 More than 70% of all teachers

have a license with which they can teach the curriculum in the year XXXX.

4-1 More than 90% of all students in

the district enjoy schooling in the year XXXX.

5-1 Less than 10% of all women feel

that a woman has a limited opportunity in the year XXXX.

1-1 Results of audit of the

national standards of school facility.

2-1 Minutes of the Boards of Education.

2-2 Statistics on education (the

Ministry of Education). 3-1 Statistics on education (the

Ministry of Education). 4-1 Results of Questionnaire

and Survey. 5-1 Results of Questionnaire

and Survey.

・ Trained teachers

continue working for the school.

・ School buildings constructed by the project are adequately maintained.

・ Social condition

surrounding students do not worsen.

・ M’s projects to improve secondary school education are implemented.

Activities： 1. Provide school facility 1-1. Set up national standards

for school buildings. Construct school facility

1-2. Equip the facility with bathroom

1-3. Equip the facility with a water tank

1-4. Training on how to manage and maintain the facility

1-5. Procure teaching materials

1-6. Audit the national facility standards

(The rest is omitted)

Inputs： Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・ Secondary school curriculum development project ・ Secondary school scholarship program Total Amounts: ooooooooo yen Major Inputs: ○○、��、×× Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance, Ministry

ofJustice Local Public Agency: Department of Secondary Education

Pre-conditions：

©FASID 26

c) Explanatory Notes: 2-4-1. Log-frame Log-frame (or Logical framework) was originated in a management tool of USAID in order to express

the summary of a development project. Log-frame consists of the main components of a project such as

project purpose, outputs, activities inputs, quantifiable indicators, external risks, etc. In PCM,

Log-frame is introduced as PDM (Project Design Matrix), which is created through a participatory

workshop with stakeholders. PDM is utilized in monitoring and evaluation in a project cycle.

Log-frame in LEAD is also a matrix table containing an overall goal, expected objectives, activities,

inputs, indicators, risks, etc., of the program to be evaluated. It is formed to confirm the structure of

the program prior to designing an evaluation study.

The form of a Log-frame is depicted as follows.

Program Title：

Target Area： Date：

Narrative Summary of the Program

Objectively Verifiable Indicators

Means of Verification

Important Assumptions

Overall Goal： Objective that the program should achieve within the program period

Indicators for measuring the level of achievement of Overall Goal

Data sources from which indicators are derived

Conditions for sustainability of the program effects.

Intermediate Objectives: Objectives which should be achieved in order to achieve the Overall Goal

Indicators for measuring the level of achievement of Intermediate Objectives.

Data sources from which indicators are derived

Conditions for Overall Goal besides that of Intermediate Objectives

Conditions for Intermediate Objectives besides that of Activities.

Activities： Summary of Actions to achieve the Intermediate Objectives

Inputs： Personnel, materials, equipment, facilities and funds required by the project

Pre-Condition Conditions that must be fulfilled before the program starts

©FASID 27

2-4-2.Special features of Log-frame for LEAD The Log-frame of LEAD omits the top row of the matrix, compared with

the general Log-frame used in project management. LEAD is developed for

a program or policy level evaluation method, so that the core objective of the

program is the Overall Goal level which is higher than the project purpose

level in the general Log-frame. In cases where there is a higher objective

than the Overall Goal, “Super Goal” should be inserted on the top row of the

matrix. For an example from the Amelia case, the Overall Goal of the

program (a group of projects) is “to increase the number of secondary

education graduates”. In the Amelia case, this Overall Goal is one of several

program goals of the poverty alleviation policy, and the Super Goal is “to

reduce the poverty population in Amelia”, which is located at the top of the

matrix.

On the other hand, there can be a case where the Logic Tree has several layers from the Overall goal

to Activities. In this case, the new row (e.g. “Outputs”) can be added between Intermediate Objectives

and Activities. The Log-frame can be modified flexibly for the needs of the evaluation. It is, however,

recommended to avoid making too complex a Log-frame, that would reduce the understanding of the

stakeholders and that would be difficult to be used in practice.

2-4-3. Log-frame as a Tool for Management The Log-frame contains important information of the program about its objectives and indicators,

and which becomes essential for evaluation on the achievement of the program. The distinct parts in

the Narrative Summary of the Log-frame, which segregate the program into “Overall Goal”,

“Intermediate Objectives”, “Activities”, “Inputs” and “Important Assumptions”, enable the evaluator to

pursue detailed factor analysis on the results of the achievement.

While the Log-frame is used for evaluation, it is the Log-frame’s strength that means it can be used

for program management in planning and implementation, too. PCM (Project Cycle Management)

applies PDM (Project Design Matrix = Log-frame) in all the phases of project management, such as

planning, implementation, and evaluation. By forming a Log-frame in evaluation, it can be used as a

basis for the following phases of planning and implementation. Forming the Log-frame may be

cumbersome if it is used only for evaluation. However, when considering that the evaluation within

LEAD is a step for the next phase of improvement, the Log-frame formation is effective and efficient in

the long-run. If the evaluation needs to be conducted just only for evaluation purposes, it is not

necessary to use LEAD. LEAD is an evaluation method that is integrated intrinsically with the

subsequent improvement of the program by using the Log-frame. Implementation with the Log-frame

is discussed in section 4-3.

Narrative Summary of Program/Policy

Objectively Verifiable Indicators

Super Goal: Poverty reduction in Amelia is promoted

Overall Goals:

Intermediate

Objectives:

Activities: Inputs：

©FASID 28

2-4-4. Converting a Logic Tree into a Log-frame If a Logic Tree has been formed logically, it is technically easy to convert the Logic Tree into a

Log-frame, since the Log-frame’s left column “Narrative Summary” consists of “Overall Goal”,

“Intermediate Objectives” and "Activities” that are connected in a logical relationship of “Means-Ends”,

just like the Logic Tree.

・ Overall Goal (i.e. Program Goal)

This is the goal (purpose) of the program (or a group of projects) to be evaluated, and it is

recommended to set only one Overall Goal. It should describe, in a sentence, the concrete

benefits or impacts given by the program to the beneficiaries (i.e. the target group). It is not a

description of what the program will be doing, instead it should clarify what the program

intends to achieve.

（Bad Example）×

（Good Example）○

Overall Goal：Rice production in village “A“ is increased.

What to achieve

What to do

Overall Goal：Technical assistance for Rice production in village “A“ is conducted

Overall Goal

Objective A Objective B

Activities

Logic Tree

Activities Activities Activities

Narrative

Summary (of Program/Poli

cy)

Objectively Verifiable Indicators

Means of Verificatio

n

Major Important

Assumptions

Overall Goals (Program Goals)

Intermediate Objectives

（Sub-program Goals)）

Activities：

Inputs：

Pre-conditions

©FASID 29

・ Intermediate Objectives（i.e. Sub-program Goal）

Intermediate Objectives are intermediate goals to achieve the Overall Goal. For one Overall

Goal, several Intermediate Objectives are set. They are described in a single sentence, as with

the Overall Goal. They are results from Activities. In other words, Activities should not be

confused with Intermediate Objectives.

（Bad Example）×

（Good Example） ○

・ Activities

Activities are descriptions of actions to achieve Intermediate Objectives. For one Intermediate

Objective several Activities are set, which are described with a “to do” expression. It is

recommended to put identical outline numbers (or reference codes) for activities.

2-4-5. Setting Indicators and Means of Verification Objectively Verifiable Indicator (OVI) is a quantifiable degree to be achieved for each objective

expressed in the Log-frame, such as the Overall Goal and Intermediate Objectives. For each objective,

one or more OVI is set. The OVI is described in a sentence, which includes the following information.

Information Example

・ When............................ “By September of year 2005”

・ Where........................... “In area A of country A”

・ Who/What.................... “Average income of farmers”

・ How much.................... “By 30% from the last year”

・ What to do ................... “To be increased.”

Intermediate Objective：Farmers apply the new techniques.

Result

2005 年９月までに、A国 A

地区の農民の平均所得が前

年比で 30%増加する。

Tips： In practice, it is recommended to set one Overall Goal, four-to-seven Intermediate Objectives, and

around four Activities for each objective, to enable a better understanding. Putting codes to

Intermediate Objectives and Activities will help further.

Intermediate Objective：Training for farmers are conducted.

Action

By September 2005,

average income of farmers

is increased by 30% from

the last year.

©FASID 30

By setting the OVI, vague expressions in the Narrative Summary in the Log-frame become clearer to

enable an objective evaluation to be conducted. In the course of setting the OVI, contents of the

Narrative Summary can be elaborated in its definition and modified if necessary.

In practice, the following information will be useful in setting the OVI.

・ Planned indicators in existing policy papers or program documents.

・ Standards regulated in public (National or international standards set by the governments, UN,

ISO, etc.).

・ Indicators used in similar projects.

・ Opinions of program officers, policy makers and other stakeholders.

・ Estimation from logical relationships expressed in the Log-frame.

Choosing the OVI requires utilising the following selection criteria.

・ “Objectivity” .......... It should be based on fact, and not influenced by personal beliefs or feelings.

・ “Direct”.................. The type of data should be exactly what should be measured. For example,

the type of data for an objective ”Income from agriculture is increased” is “income from

agriculture”. It is neither “agricultural production” nor “general income of households”.

・ “Adequate Level” .. The level of data should be exactly what should be measured. For example,

the OVI of an objective “Agricultural technique is upgraded” is not “the number of training

courses conducted”, since it is one of the activities for the objective. On the other hand, “the

amount of agricultural product” is not the proper OVI, since it is the result of upgrading the

agricultural techniques. Technically, there can never be any case where the same OVI is used in

different levels of the Narrative Summary in the Log-frame.

・ “Coverage”............. The OVI should cover all of what should be measured. Sampling should be

conducted without bias.

The cells in the right hand side of the OVI are for "Means of Verification", which indicate the source

of information. Means of Verification states relevant documents, methods, sourcing organizations,

locations and so on, in order to make the setting of indicators and their monitoring more efficient and

effective. When considering the Means of Verification, "availability", "reliability" and

"cost-performance" of information should be considered.

When there is no available source of information, it is necessary to collect information by conducting

researches by the evaluation team. It is, however, recommended to use the existing data as much as

possible in practice, since information collecting itself consumes a lot of time and monetary resource.

The selection of Means of Verification should proceed carefully, along with the selection of OVIs.

©FASID 31

Improved farmer’s skill

Important

Assumption

2-4-6. Setting Important Assumptions "Important Assumptions" are the conditions which are necessary for the program achievement, but

are not controllable by the program, and it is uncertain whether they will be realized or not.

Important Assumptions are found from logics within the Narrative Summary of the Log-frame. As

described below, Important Assumptions for "Activities" to "Intermediate Objectives" can be found by

asking whether there is any condition to be realized besides that of the Activities for accomplishing the

Intermediate Objectives.

For instance, the increase of agricultural production (as the Overall Goal) cannot be achieved only

by the improvement of farmer’s skills (as an Intermediate Objective), as there also needs to be rain. In

this case, "Amount of rainfall is enough" can be considered as one of the Important Assumptions. For

another example, the upgrading of the teaching quality in a school cannot be achieved if the trained

teachers leave the school, and therefore despite the fact that the training has been well conducted for

the teachers. In this case, "The trained teachers continue to work in the school" can be thought of as

one of the Important Assumptions. There are other issues such as economy, politics, environment,

society, culture and so on, for considering Important Assumptions.

Outcomes

Activities Important Assumptions

©FASID 32

When considering Important Assumptions, cross-cutting issues, which the OECD recommends, are

of some help.

Cross-cutting Issues Examples

・ Political ................................................. "The government keeps the policy of supporting

secondary education."

・ Technical ............................................... "The electricity necessary for agriculture is provided."

・ Environment ......................................... "More than 1000mm of annual rainfall is available."

・ Social/ Cultural ..................................... "Not many villagers migrate away for seeking jobs."

・ Institutional .......................................... "Farmers' Cooperative Organizations continue to exist."

・ Economic ............................................... "Market price of rice is maintained."

Projects or activities that are included in the Logic Tree, but are not included in the Narrative

Summary of the Log-frame, are also candidates for Important Assumptions. When there are other

organization(s) to do projects influencing the program, the projects should be considered as one of the

Important Assumptions. This will help in the later stages for analysing the attribution of the program.

In the Amelia case, the project conducted by Country M can be considered as an Important

Assumption.

"Pre-conditions" are the conditions that should be realised before the commencement of the program.

In practice, however, it is a rare case to need the pre-conditions because LEAD evaluation is conducted

mostly for on-going programs.

2-4-7. Setting Inputs The "Inputs" cell consists mainly of human resources, facilities and materials, and other operating

costs. If the inputs are diverse in various operating organizations, the inputs to be divided throughout

the organizations should be so described. This will help the later analysis of the contribution level of

each organization. In practice, the evaluator determines the method of setting the Inputs, but it is

recommended to clarify at least the main components of inputs (e.g. types and amounts of human

resources, facilities, materials and other costs) by each organization. When analysing the efficiency of

the program, it is recommended to put these inputs into each group of Activities or to enumerate the

inputs into monetary values.

©FASID 33

Well Done!

You have successfully achieved the halfway point to the final goal!

Relax for a while......

©FASID 34

3. Evaluation Phase

3-1. Evaluation Study Design

a) Outline

Objective: To clarify what is necessary to be studied and how it is conducted, in order to achieve the evaluation purpose.

Procedures: 1. Reconfirm the evaluation purpose from Scope of Evaluation

and Log-frame.

2. By defining the evaluation purpose, set evaluation criteria and evaluation questions.

3. Set study methods to answer the evaluation questions.

4. Compose the above information into a matrix named "Evaluation Grid", which should be shared with stakeholders.

Output:

Evaluation Grid

©FASID 35

b) Output Sample: Evaluation Grid Example of Evaluation Grid

Evaluation Purpose

Evaluation Questions (Criteria)

Evaluation Questions (Level 1)

Evaluation Questions (Level 2)

Research Methods Information Source

1-1-1. Whether the rate of secondary education enrolment in the capital city is increased up to 90%.

Analyse the past trends by using statistics on secondary education in order to predict future trends.

Statistical Document published by government.

1-1. Is the Overall Program Objective achieved?

1-1-2. Does ratio of graduation/ entrance increase up to 0.9 as planned?

Analyse the past trends by using statistics on secondary education in order to predict future trends.

Statistical Document published by government. Results from

sample survey.

1-2-1. To what extent J’s development assistance contributed to the increase of the education continuance rate?

Examine the association between the enrolment rate and J’s assistance by using regression model.

Document on J’s assistance in each school.

1-2-2. Which programs are significantly effective or ineffective?

Conduct questionnaire among stakeholders particularly on effective programs, including questions on the reasons why they think that is so.

Contract out the survey to a local consultant. Conduct interviews by the evaluation team.

Conduct qualitative evaluation of the impacts of the Country M’s education project based on its evaluation report.

Program evaluation report done by Country M.

1. Is the program effective?

1-2. Is the achievement of overall objectives attributed to Country J’s assistance? How much is its contribution?

1-2-3. To what degree was the contribution of assistance from other donors or agencies?

Conduct questionnaire and interviews among stakeholders on other factors influencing the school enrolment.

Contract out the

survey to a local

consultant.

Conduct

interviews by the

evaluation team.

2-1-1. Is there any redundant program or project?

Check with the Logic Tree Logic Tree.

To conduct comprehensive evaluation of middle school education support programs, which are currently implemented in the capital of Amelia, to modify these in order to make them more effective and efficient.

2. Is the program efficient?

2-1. Is there any redundant or overlapping program?

2-1-2. Is there any program that has identical inputs?

Lay out an actual use of inputs to identify redundancies.

Overall Inputs Matrix.

©FASID 36

2-2-1. Are there any inputs that have significantly low or high level utilization?

Ask stakeholders to rank the order of utilization of inputs and also ask for the reasons.

Contract out the survey to a local consultant.

2-2. Are inputs sufficiently utilized and effectively transformed to outcomes?

2-2-2. How is the transformation from inputs to outcomes?

Ask stakeholders to raise indicators to assess the contribution of inputs to outcomes, and analyse them by running a regression analysis.

Contract out the survey to a local consultant. Analyse the results of the survey by using in-house staff.

©FASID 37

c) Explanatory Notes 3-1-1. Evaluation Purpose and Evaluation Criteria In order to make the evaluation more effective (i.e. to achieve the purpose) and more efficient (i.e. to

consume the least cost), it is crucial to focus the evaluation purpose before designing the evaluation

study. LEAD requires the clarification of the evaluation purpose at the initiation phase (See 1-1

"Scope of Evaluation"), so the evaluation purpose is reconfirmed at this step. If necessary, the

evaluation purpose can then be adjusted and defined more clearly.

For evaluation of international cooperation projects, it has been recommended to consider the five

evaluation criteria of efficiency, effectiveness, impact, relevance and sustainability. In practice,

however, applying all five criteria simultaneously is not recommended when there is limited time and

resources. There is also a risk that all of the five may be outside of the clients' interests, so that the

evaluation would be useless in the end.

With limited time and resources, the evaluator should consult with the clients or the users of the

evaluation, and clarify their interests in the evaluation. For customisation, any basic improvements

needed should be clarified at this stage, such as whether the client wants to improve the effectiveness

or the efficiency of the program, to thereby alleviate the negative impacts, etc. Of course, there can be a

desire to pursue the improvement of every aspect of the program, however such an ambitious (but

ambiguous) comprehensive evaluation tends to be obligated with enormous amount of tasks, and with

a tendency to end with unsatisfactory results. If there is a request for several

evaluation criteria to be evaluated, then these should be prioritised by

ranking them with the stakeholders. The designing of the Evaluation Grid

can be commenced from the high ranked evaluation criteria.

3-1-2. Program Level Evaluation A project evaluation in general considers the whole effect of the project, while a program level

evaluation, in which a program includes several projects, needs to have some evaluation added on the

structure of the program and on the relationships among projects as sub-programs.

Especially for LEAD, which composes the proxy-program containing projects as sub-programs to be

evaluated, one of the most probable recommendations resulting from the LEAD evaluation is to

re-construct the program by adjusting and coordinating existing projects. Thus, the question about

which component (project) should be improved by adding, deleting and modifying is very crucial, so the

evaluation on each component within the program structure is indispensable in any evaluation criteria

selected.

©FASID 38

The Concept Map of Program Evaluation

Overall Goal (Program Goal)

Sub-program

Goal 1

Sub-program

Goal 2

Sub-program

Goal 3

Sub-program

Goal ….n

Activities Activities Activities Activities Activities Activities Activities

Inputs Inputs Inputs Inputs Inputs Inputs Inputs

Evaluation of Program Structure

Evaluation of Program

©FASID 39

Evaluation topics for each of the five evaluation criteria can be described as follows.

Five Evaluation Criteria in Program Level Evaluation

Evaluation Criteria Basic Approach (As a whole)

Advanced Approach (About the Structure)

Efficiency

Consider the Productivity in

implementation. How efficient is

the manner in which the outputs

are produced by the inputs? Are

any inputs wasted? What about

the adequacy of the inputs,

considering the outputs and

outcomes achieved?

Consider adequacy in the use of the inputs

provided by different agencies. Are there

any redundant inputs? What about

adequacy in the sequence of executed

projects? Is there any efficiency-making

effort, e.g. to share the same input by

different agencies?

Effectiveness

Consider the achievement level of

the Overall Goal. Consider as to

whether the Intermediate

Objectives within the program

contribute the achievement of the

Overall Goal.

Consider which component projects are

effective or ineffective to the achievement

of the Overall Goal. Consider the

synergistic effects among the component

projects.

Impact

Consider positive and negative

impacts (changes) accrued directly

or indirectly by the program. The

impacts that are not expected from

the planning are also included.

Consider which component projects are

the most or least influencing in causing the

impacts, in a positive or negative manner.

Relevance

Consider the relevance of the

Overall Goal to the needs of the

beneficiaries, the national and

international goals, socio-

economic trends, etc.

Consider the relevance of component

projects and activities to achieve the

Overall Goal. Consider the coherence

among the component projects. Consider

the maturity of the proxy-program as a

genuine program.

Sustainability

Consider the sustainability of the

impact and benefits accrued by the

program.

Consider which component projects

provide the most or least influence to the

sustainability of the program.

©FASID 40

The above definitions are rough examples to provide some hints for creating evaluation questions.

The evaluator should communicate with the clients and other stakeholders in order to clarify the

evaluation criteria and items to be studied in accordance with the Scope of Evaluation.

3-1-3. Designing Evaluation Study The purpose of the evaluation study is to satisfy the needs of evaluation. Thus, the evaluation study

should be designed by considering how to collect the necessary information in order to address the

needs. The process of designing can be defined as follows.

It is recommended to undertake the design by using the following matrix, known as the “Evaluation

Grid”, which contains the evaluation questions, methods and other necessary information.

Evaluation Grid: Example

Evaluation Purpose

Evaluation Questions

(Criteria Level)

Evaluation Questions

(Level 1…n)

Research Methods

Information Source

If the issues and questions are not clearly defined, specify these by breaking them down into smaller questions

Research methods to answer the evaluation questions in the cell to the left cell

Resources (cost, information) to meet the research methods

Issues and questions to be answered in order to meet the evaluation needs. As the starting point, selected evaluation criteria is clarified at this level

Needs for the present evaluation derived from the scope of evaluation

* Our Output Sample has Level 2 Questions.

For each evaluation criteria, several evaluation questions should be created from which the criteria

can be defined in a concrete manner. In cases where the question is broad and ambiguous, it is

What are the needs for evaluation (from the scope of evaluation)

What you should know to meet the needs？

What and how you should examine to answer the questions

The research method is both effective and efficient?

① ② ③ ④

Broad question

Middle question

Middle question

Narrow question

Narrow question

Narrow question

Narrow question

©FASID 41

recommended to decompose the question into a handful of sub-questions that are more easily

answerable in practice, such as questions at level 1, level 2, level 3 and so on. The method to collect

the information for answering each evaluation question is stated in the column to the right of each

question. The number of these research methods for each question can be more than one. The other

necessary information, such as the names of required documents can be added in the next column. As a

result, the Evaluation Grid gains more rows (or layers) as you move towards the right. The Evaluation

Grid can be used not only for designing the evaluation study, but also for placing the results of the

evaluation study within it.

3-1-4. Setting the Methods of Data Collection Setting the methods of collecting data should need consideration not only of the accuracy and

reliability of the data, but also of the time and resources that will be available in reality.

There are several options for data collection methods as follows.

Method Description Advantages Remarks 1. Literature

Survey

Collection and analysis of existing statistics, survey reports, work records, research data, etc.

Economical and efficient Difficult to verify the reliability of materials

2. Direct Measurement

Collection of data that can be measured and classified based on actual surveys

Provides accurate and reliable data; relatively low cost.

Facts only, no explanations

3. Sample Survey

An interview or questionnaire-based survey, targeting a sample population of a sufficient size with regard to the parent population.

Provides quantifiable and reliable data

High cost

4. Case Studies A detailed study of a small number of cases.

Provides self-contained detailed information

Difficult to generalize

5. Observation

Undertaken using a checklist or survey sheet. May also involve simple interviews.

Little need for preparation. Particularly suitable for a preliminary study.

Results can vary widely depending on the observer.

6. Key Informant Interviews

A technique often used in an evaluation. After selecting persons well-versed in the matters concerned, detailed interviews are conducted.

Low cost and easy to implement. Provides useful information and sometimes draws attention to unexpected problems.

Risk of bias due to the personal views of interviewer and interviewee. Data cannot be quantified. Time consuming

７Group Interviews Interviews with representatives of the community or groups (5-6 people each) in the form of an open meeting.

Background information can be obtained from interviewees’ attitudes and reactions.

There is a risk of creating biased conclusions through manipulation in the interviewee selection process.

©FASID 42

Although the selection of data collection methods is limited to available time and resources, it is

recommended to use the following selection criteria in order to select the best methods.

・Validity ·························Do the evaluation questions cover what should be studied?

・Importance ····················How important are the answers of the evaluation questions to the

evaluation ?

・Reliability······················How much are they reliable in their sources of information and methods?

・Accessibility··················· Is the necessary information easily accessed?

・Cost ······························ Is the cost of conducting the method appropriate?

In the process of designing the Evaluation Grid, participation of stakeholders including the clients is

important. The evaluator should share the process of designing the Evaluation Grid with stakeholders

from the draft version to the final version. This makes the evaluation more realistic and acceptable in

its conclusions and recommendations at the end.

3-1-5. Data Collection for Factor Analysis The designing process of the Evaluation Grid should consider the issue of a factor analysis on the

results of the evaluation study. Evaluation about effectiveness is concerned with the achievement of

the Overall Goal, and a factor analysis needs information about all factors (positive and negative)

influencing the Overall Goal’s achievement, such as the achievement of Intermediate Objectives,

Activities, Inputs, and relative Important Assumptions.

Such sub-topic information should be collected together with the main topic information for greater

efficiency. Therefore, the evaluator should include the data collection for factor analysis at the

Selection of Methods

Cost Accessibility

Validity Importance

Reliability

Selection Criteria of Research Methods

©FASID 43

beginning by examining the Log-frame or Logic Tree, having preliminary interviews with stakeholders,

and predicting the results with experiences of similar programs.

In practice, the following viewpoints are generally needed in collecting data for factor analysis. They

are (1) the achievement level of each components of the Log-frame, (2) the conduct of management in

Activities, and (3) cross-cutting issues.

“Degree of Achievement of the Log-frame” is checked at each level of the Narrative Summary,

such as Intermediate Objectives, Activities, Inputs and relative Important Assumptions.

“The conduct of management in activities” concerns the way of conducting the planned activities,

daily management style, and relationship among stakeholders in reality.

“Cross-cutting issues” consists of aspects of policy, technology, environment, society, institution,

and finance (or economy) that influence the program performance.

For example, the sample Evaluation Grid contains the evaluation question to check ,by conducting a

questionnaire-based survey, as to whether there are other social factors or situations influencing the

secondary school enrollment. This is one of the examples of collecting factor analysis data from the

view of cross-cutting issues in “society” and “environment”.

Factor Analysis

Degree of Achievement of the Log-frame

What are reasons for failure?

Process of Activities (i.e. Stakeholder Relationship)

Why are they achieved? Why are they not achieved?

YES NO

Are overall goals achieved?

Cross-Cutting

Issues

What are reasons for success?

©FASID 44

3. Evaluation Study Phase

3-2. Evaluation Study Implementation

a) Outline

Objective:

To collect the necessary information by conducting the planned evaluation study.

Procedures:

1. To reconfirm the evaluation questions in the Evaluation Grid.

2. To conduct the planned evaluation study using the designed methods.

3. To describe and sort the results of the evaluation questions into the Evaluation Grid.

4. To inform the progress of the evaluation study.

Output:

Evaluation Grid with Results

©FASID 45

b) Output Sample: Evaluation Grid with Results Evaluation Grid with Results: Example

Evaluation Purpose

Evaluation Questions (Criteria)

Evaluation Questions (Level 1)

Evaluation Questions (Level 2)

Research Methods

Results

1-1-1. Whether the rate of secondary education enrolment in the capital city is increased up to 90%.

Analyse the past trends by using statistics on secondary education in order to predict future trends.

The continuance rate has been increasing year by year, but the rate stays between 60% & 70% in the year XXXX. The rate of students from a poor background remains in the low level.

1-1. Is the Overall Program Objective achieved?

1-1-2. Does ratio of

graduation/ entrance increase up to 0.9 as planned?

Analyse the past trends by using statistics on secondary education in order to predict future trends.

The ratio of graduation/ entrance is diverse depending on the school, but the average ratio is approximately 8 to 10.

1-2-1. To what extent J’s development assistance contributed to the increase of the education continuance rate?

Examine the association between the enrolment rate and J’s assistance by using regression model.

Based on the results of regression analysis, J’s overall contribution is R^2=70%, which indicates a positive result, but the scale of the contribution is greatly different among programs.

1-2-2. Which programs are

significantly effective or ineffective?

Conduct questionnaire among stakeholders particularly on effective programs, including questions on the reasons why they think this is so.

Most stakeholders point out that program ○○ is particularly effective but, on the other hand, program xx is not quite effective.

To conduct comprehensive evaluation of middle school education support programs, which are currently implemented in the capital of Amelia, to modify these to make them more effective and efficient.

1 Is the program effective?

1-2. Is the achievement of overall objectives attributed to Country J’s assistance? How much is its contribution?

1-2-3. To what degree is the contribution of assistance from other donors or agencies?

Conduct qualitative evaluation of the impacts of the Country M’s education project based on its evaluation report.

A primary goal of M’s project is to give training on public administration, and therefore there is no direct impact.

©FASID 46

Conduct questionnaire and interviews among stakeholders on other factors influencing the school enrolment.

Educational support activities for women run by a local NGO have as big an impact on broadening educational opportunities for women as does J’s assistance.

2-1-1. Is there any redundant program or project?

Check with the Logic Tree.

Activities of A and B are partially redundant.

2-1. Is there any redundant or overlapping program?

2-1-2. Is there any program that has identical inputs?

Lay out an actual use of inputs to identify redundancies.

Although Programs A and B have almost the same items of inputs, they have been arranged independently and neither have been shared nor coordinated.

2-2-1. Are there any inputs that have a significantly low or high level of utilization?

Ask stakeholders to rank the order of utilization of inputs, and also ask for the reasons.

Opinions from Stakeholders reveal that inputs XX are the best to use, and for the reason _____

2. Is the program efficient?

2-2. Are inputs sufficiently utilized and effectively transformed to outcomes?

2-2-2. How is the transformation from inputs to outcomes?

Ask stakeholders to raise indicators to assess the contribution of inputs to outcomes and analyse them by running a regression analysis.

Input-Outcome Analysis says inputs ○○& �� has the highest transformation rate of all inputs. The overall transformation rate is XX %

©FASID 47

c) Explanatory Notes 3-2-1. Implementation of the Evaluation Study The evaluation study is conducted in accordance with the Evaluation Grid. Sub-contractors can be

procured to undertake the simpler tasks such as a questionnaire survey, so that the evaluator can

concentrate on the whole picture of the evaluation. In using the sub-contractors (e.g. surveyors or

private consultants), the scope of work of the contract should be linked coherently with the Evaluation

Grid, which is shared with the sub-contractors, so that the sub-contractors should conduct only but the

necessary tasks.

When progressing the evaluation study, there may be a need to alter some components of the

Evaluation Grid . In this case, the Scope of Evaluation should be reviewed before the identified

adjustment is accepted. In other words, “the Scope of Evaluation”, “the Evaluation Grid” and “the

Scope of Work with sub-contractors” should be integrated and managed consistently.

3-2-2. Question of Attribution One of the most challenging tasks of the evaluation process is the factor analysis to clarify the level

of attribution of each factor to the results. The following diagram depicts the hypothetical model of

attribution.

In the above diagram, there are three factors at the Intermediate Objectives level to attribute the

achievement of the Overall Goal. It is important to know which of these factors are the greater or least

contributors to the Overall Goal. It is, however, a daunting task to pursue the answer in an objective

manner. Currently, there are several techniques to do this type of analysis as follows.

The literacy of students from

District A is improved

School facility is renovated.

Teacher’s capability is

improved

Intermediate

Objectives

Overall Goal

1 2 3

Teaching materials are improved

©FASID 48

Techniques Description Remarks

Comparative Analysis

with a similar program

Compare with the similar program which particular component

projects or activities are not included. From the comparison,

estimate the negative effects of the lost part of the program. In

another word, benefits accrued from the program of our interest is

equivalent to the negative effects derived from the estimation.

It is rare to find a similar program.

Comparative Analysis

in Time Series

Measure the degree of the change of the achievement level of the

Overall Goal between before and after the target project or activity

started.

It is difficult to distinguish the sole effects from the others that have already existed.

Regression Analysis Enumerate the achievement of each item in the program and apply

the variables into the regression model to calculate the coefficients

(a1-aN).

(Overall Goal) = a1 x (IO1) + a2 x(IO2)+…+ aN x(ION)+ e

where “IO” means “Intermediate Objectives”

This can be applied into panel data or time series data.

Enumeration and modelling are difficult.

Qualitative Analysis Qualitative judgment made by stakeholders whom are well-versed.

May be quantified by making questionnaire surveys and analysed

with statistical techniques (Average and Variance).

Proper sampling is difficult. Result is still subjective.

Experimental

Techniques

From the beginning, apply the various sets of components in

various areas and monitor the difference of the results.

Not possible to apply in ex-post evaluation. Basically denied from the ethical point of view.

*Details of these techniques can be referred to in books listed in the annex.

There are other techniques applicable for attribution analysis, but there are no standardized

techniques in reality. From the other point of view, it is not recommended to use a highly complex

technique that is understandable only to a limited number of specialists, because such a complex

technique obstructs the understanding of the stakeholders, most of which are not experts on the

technical issues. The reduction of understanding among the stakeholders will be more harmful than

losing the technical robustness of evaluation, in the sense that such common stakeholders are the

users of the evaluation in practice. If the user cannot understand the results, the recommendations

won’t be accepted. Using highly complex techniques and modelling is also harmful in the sense that it

©FASID 49

would make the stakeholders, as well as the evaluator, lose the whole picture by focusing excessively

on detail.

Thus, when conducting the evaluation study, the evaluator should remember the needs and purpose

of the evaluation. The evaluation study should be implemented in order to make good

recommendations for improvement, and to give useful information for the users of the evaluation at

the end.

3-2-3. Sorting the Results of the Evaluation Study

Results of the evaluation study are put into the Evaluation Grid.

The style chosen for the description of the results is part of the evaluation process to be decided. The

results are placed to the right of the description of the research methods in the Evaluation Grid. If the

information in the results column is too extensive, the results should be summarised with the detailed

information being attached in an annex.

Evaluation Grid with Results

Evaluation

Purpose

Evaluation Questions (Criteria)

Evaluation Questions

(Level 1...n)

Research

Methods

Results

Notes

If the issues and questions are not clearly defined, specify these by breaking them down into small questions.

Research methods to give answers to the evaluation questions in the cell to the left.

Summary of Evaluation Results

Documents related to the results and their serial numbers.

Issues and questions to be answered in order to meet the evaluation needs. (Evaluation Criteria selected at this level)

Needs for the evaluation derived from the Scope of Evaluation

The attached documents should be written in a detailed but understandable form by using charts

and graphs. These documents do not just form a record. The documents in the evaluation should be

always be written by bearing in mind that they will be read by important stakeholders.

©FASID 50

3. Evaluation Study Phase 3-3. Conclusions, Recommendations and Lessons Learned

a) Outline

Objective::

To draw recommendations and lessons learned with overall

conclusions.

Procedures:

1. Draw overall conclusions by reviewing the results of the

evaluation study.

2. Draw recommendations.

3. Draw lessons learned.

4. Make presentation materials by summarizing the progress and results of the evaluation.

Output:

Conclusions, Recommendations and Lessons Learned

©FASID 51

b) Output Sample :”Conclusions, Recommendations and Lessons learned”

アメリア国中等教育支援プログラムの総合評価取りまとめ例

評価結果取りまとめ

総合評価

全体目標の達成度については堅実に中等教育の進学率は伸びてきて

おり、さらに卒業率の改善も目覚しい。一方で、その改善は貧困層

に属する学生とっては限られたものである傾向があり、また有効性

からみても、貧困層や女性をターゲットにした事業が十分にその効

果を発揮していないことが懸念される。特に貧困層をターゲットと

した一部の事業の有効性が低かったことは、貧困が一部の事業で対

応するには大きすぎる問題であることと、事業単位で実施されてお

り、J 国の他の事業や NGO など既存の活動との総合的なアプローチ

がとられていないことに起因するものと考えられる。

効率性の観点からも、同じような事業や活動をしている場合も多く、

また他の活動の経験や教訓などが十分に共有されていないなどの問

題が指摘されうる。投入財についてはおおむね効率的に成果に転換

されていると判断されるが、より事業を統合化し、共有するべきと

ころは共有するなどすれば、さらなる効率化が図られる可能性があ

る。

提 言：

(J 国に対して)

・ 貧困層支援の視点から、J国の各プロジェクトの組み合わせを見

直し、ロジック・モデル及びログフレームを改善することを提

言する。

・ 重複している活動については統合し、連携を強める。

・ 今後はそのログ・フレームを元に、各プロジェクトが進捗状況

や情報を共有し、連携を保ちながら実施する。

・ 上記プログラムを運営管理するための担当者を任命する。

(アメリア国に対して)

・ 以下の情報を、J 国のロジック・モデル作成の際には、それを提

供する。

- 中等教育に係る政府の中期戦略

- 中等教育にかかる現地 NGO の活動

- 中等教育にかかる他のドナ-の活動

教 訓：

中等教育に関する計画を検討する際には、貧困層への十分な配慮が

必要であり、NGO などのコミュニティーの協力は不可欠である。

Summary of Evaluation Results for Secondary Education Assistance of Country J Title Contents

Summary and

Conclusions

The degree of the achievement of the Overall Goal indicates that the enrolment of the secondary school has been improved, and especially that the graduation rate has been increased dramatically. There is, however, concern that the improvement seems to be enjoyed solely in the area populated mainly by middle income class households, and that the improvement is limited in the “poor” area, where the population of the residents have the lowest level of income. In the gender perspective, the improvement of the state of the female students is not significant compared with the male students.

The tendency for benefits not to be received by poor or female students should be of great concern, in the sense that Country J’s assistance is one of the programs for the poverty alleviation policy. There has been no significant improvement observed for the poorest group of students. There are some reasons for this. Firstly, poverty itself is the complex and difficult problem that the secondary school program can not solve on its own. Secondly, the projects have been conducted independently in practice and there is no effective coordination system among them. Thirdly, there has not been any cooperation with other active supporters, such as local NGOs and private schools.

The last two matters are concerned from the point of view of efficiency. Several inputs and activities are overlapped but neither shared nor coordinated. The lessons learned from a project did not get fed back to other projects, thus the same type of mistakes have been repeated. All inputs are efficiently utilized to the full, but a coordination of projects will increase the efficiency of the whole structure of Country J’s assistance.

Recommendations

�

(For Country J) ・ The coordination of projects should be fundamentally reviewed and

reconstructed from the view of assisting the poor, by improving the Logic Tree and the Log-frame.

・ The overlapped or very similar activities should be combined and coordinated consistently.

・ The multiple projects should be managed as a program by using the improved Log-frame.

・ A full-time program manager for managing the program should be assigned, while the project-level management should be conducted by the project leaders, and not by the headquarters office.

(For Amelia government) In the improvement process, the government of Amelia should provide the information as follows to Country J.

- The mid-term strategy for secondary education. - The detailed information of the local NGOs concerned.

- The detailed information of the main donors concerned.

Lessons

Learned

In planning the assistance of secondary education in Amelia, it is indispensable to consider the situation of poor people, and to pursue the support from the community such as local NGOs.

©FASID 52

c) Explanatory Notes

3-3-1. Drawing Conclusions Conclusions are not the summary of the results from the evaluation. They consist of the following

points.

・ How are the results relating to the needs and purpose of the evaluation expressed in the Scope

of Evaluation?

・ In the case of setting several evaluation criteria, which of the criteria are the results considered

to be very successful or unsuccessful?

・ Is there any underlying common factor which boosts or hampers the results?

・ In the cases of where problems have been found, is it necessary to implement any improvement

measures?

The conclusions are a basis from drawing the recommendations, addressing any problems pointed

out in the conclusions.

3-3-2. Drawing Recommendations Recommendations are drawn from the conclusions in order to improve the situation. In practice,

drawing recommendations can be promoted by using a participatory approach involving the relevant

stakeholders, including the program implementer who will be given the recommendations. Through

open discussions for giving ideas about possible recommendations, the final version of these

recommendations can be drafted and their feasibility improved. The following remarks should be

reiterated.

・ Recommendations should be drawn from the conclusions. Recommendations without any link

with conclusions are not accepted.

・ Clarify with whom the recommendations are made.

・ Avoid using ambiguous expressions, and make them as concrete as possible.

・ The number of the recommendations should be a handful, and less than seven.

・ In order to realize the recommendations, the recommendations can address the need for

structural reform of an organization or a detailed action plan, if necessary.

©FASID 53

3-3-3. Drawing Lessons Learned Lessons learned are lessons that should be considered when planning and implementing similar

programs. Like drawing the recommendations, a participatory approach is encouraged to be applied

in drafting the lessons, utilizing the various participatory methods such as brainstorming. The

finalizing of lessons learned should also be made with consideration to feasibility.

The following are critical points when drawing lessons learned.

・ Clarify “for whom” and “which kind of program” that the lessons learned contribute to.

・ Avoid using ambiguous expressions, and make them as concrete as possible.

・ The number should be limited to no more than three.

・ If the lessons learned can not be applied without a reform of the organisational structure, then

such a point should be included as a long-term recommendation.

3-3-4. Preparing Presentation Materials Preparing presentation materials is one of the most important tasks in evaluation, since an

evaluation that is not understood by the stakeholders is likely to be judged as a bad evaluation, even

though its methods and contents are correct. The Evaluation Study Phase is closed only after the

evaluation results are understood by the stakeholders, including the program implementers. Thus, the

documents of the evaluation should be carefully prepared for the purpose of presentation. The

presentation materials should consist, at least, of the following items.

・ Evaluation Report

Chapter 1 Program to be evaluated

Chapter 2 Evaluation Study Design

Chapter 3 Results of Evaluation

Chapter 4 Conclusions, recommendations and lessons learned.

・ Executive Summary of Report

Program to be evaluated

Conclusions, recommendations and lessons learned.

・ Slide show data with computer software (e.g. MS Power Point)

＊ In the cases where the client has a particular preferred format of reporting, then that should be

utilised.

©FASID 54

4. Improvement Phase

4-1. Feedback

a) Outline

Objective:: To acknowledge the results of the evaluation with the stakeholders Procedures:

1. Present the progress and results of the evaluation

2. Obtain comments from stakeholders

3. Discuss with stakeholders about improvement

4. Clarify the Commitment of Improvement with stakeholders

Output:

Commitment of Improvement

©FASID 55

b) Output Sample: Commitment of Improvement

Title: Commitment of Improvement for Country J’s assistance for the improvement of secondary education in Amelia.

Items Contents

Recommendation LEAD evaluation, conducted in the year XXXX, reported the following recommendations for Country J’s assistance for the improvement of secondary education in the Republic of Amelia:

1. The coordination of projects should be fundamentally reviewed and reconstructed from the view of assisting the poor, by improving the Logic Tree and the Log-frame.

2. The overlapped or very similar activities should be combined and coordinated consistently.

3. The multiple projects should be managed as a program by using the improved Log-frame.

4. A full-time program manager for managing the program should be assigned, while the project-level management should be conducted by the project leaders, not by the headquarters office.

Commitment of Improvement

In response to recommendations, the country of J will implement the following actions. 1. Hold a LEAD workshop to re-build a LEAD Logic Tree in reviewing

the combination of several projects, except for Project A. Also hold a discussion to construct the LEAD Log-frame.

2. Take necessary measures to cut off redundant activities based on the improved Logic Tree.

3. It is difficult to administer Log-frame management in J’s assistance where project-based management is done. Log-frame management should be implemented in a few years time, but is referred to in the comprehensive review.

4. Appoint a commissioner to promote project harmonisation between the country of J and the Republic of Amelia, and initiate research that helps to implement it in a few years time.

Responsible Personnel/

Agency

For actions 1 to 3, Department of XX, J’s Agency for Technical Assistance For 4, appoint a responsible person for coordination by XX, XXXX.

Others Next action: Run a preparatory committee for the workshop stated in the first action point in XX, XXXX.

Confirmed and signed by: Mr. X Ms. Y Mr. Z (J’s Agency for Technical Assistance)

©FASID 56

c) Explanatory Notes

4-1-1. Commitment of Improvement Unfortunately, there are a lot of cases that good recommendations from correct evaluations are not

implemented in reality. One of the main causes of this failure is the lack of follow-up action to ensure

the implementation of recommendations, while there are other causes such as the giving of useless

recommendations that are ambiguous, unfeasible and irrelevant to the needs of the program

implementers. Recommendations should be considered as a request for improvement that needs a

reply from the concerned party. The concerned party (the program implementer in most cases) that

receives the recommendations should clarify their commitment to the request for improvement by

issuing the Commitment of Improvement.

The Commitment of Improvement expresses the basic strategy to ensure the implementation of the

recommendations, and acknowledge the respective roles of the stakeholders. It is not necessary to

determine a detailed plan for implementation. Such a plan will be constructed in the next step.

The basic contents of the Commitment of Improvement are as follows (although they can be modified

if necessary).

Title Contents

Recommendations from the last

evaluation

State the recommendations and clarify them here if some ambiguity is

found

Basic Strategy of Improvement Clarify which of the recommendations are accepted and complied

with. Also clarify the measures taken concerning the

recommendations that are not accepted. Decide whether the

Log-frame should be changed or not.

Implementers

Clarify who is responsible for leading the improvement.

Schedule State when and how the first action is commenced, such as the initial

meeting.

Main Stakeholders Describe other stakeholders such as institutional and financial

supporters, and their roles.

Monitoring

How is the progress of the improvement monitored?

©FASID 57

Commitment of Improvement is the policy level document that describes a rough sketch of the basic

strategies and roles of the stakeholders. Individual stakeholders consider their own detailed

implementation plans.

4-1-2. A Participatory Approach in Making the Commitment of Improvement It is recommended to apply a participatory approach in making the Commitment of Improvement,

by holding a meeting of stakeholders after the presentation of evaluation results. In the meeting,

decisions on several topics such as whether all or part of the recommendations are accepted, and who

will take the leading role of implementation, are discussed and determined. Such a meeting will enable

the following actions to be implemented more realistically. The chairperson of the meeting should be

the representative of the concerned party for whom the recommendations are intended. In practice, the

meeting may need a facilitator or a consultant from the evaluation team, in order to support a smooth

procedure at the meeting.

The representatives of the concerned party produce the draft version of the Commitment of

Improvement. However, it should be shared and agreed by consensus of all the main stakeholders

before final authorization is given.

©FASID 58

4. Improvement Phase

4-2. Improvement Planning

a) Outline

Objective: To improve the Log-frame in line with the Commitment of Improvement

Procedures:

1. Reconfirm the Commitment of Improvement

2. Improve the Logic Tree through a participatory planning workshop

3. Form the Log-frame from the improved Logic Tree

4. Hold a meeting to discuss the feasibility of the improved Log-frame

5. Authorize the improved Log-frame to be shared with stakeholders

Output:

Improved Logic Tree Improved Log-frame

©FASID 59

b) Output Sample: Improved Log-frame Amelia: J’s assistance for the improvement of secondary education: Improved Log-frame

Target Area: Capital of Amelia Month/Date/Year Narrative Summary (of

Program/Policy) Objectively Verifiable

Indicators Means of

Verification Major Important

Assumptions Super Goal： ・ Poverty in the capital of

Amelia is reduced

・ The population below the poverty

line is less than 10% in the capital of Amelia by the year of XXXX

・ National statistics

derived from the Ministry of Social and Health

・ UNDP Statistics Year Book

Overall Goal： ・ The secondary education is

more accessible for the poor in the capital of Amelia.

・ The rate of poor population receiving secondary education is increased more than 80% in the capital by the year XXXX.

・ The ratio of graduation/entrance of secondary students in Special Assistance Districts is more than 1 to 10 by the year XXXX.

・ National statistics

on education derived from the Ministry of Education.

・ Social Statistics

・ Government policies

on secondary education are kept unchanged.

・ Other poverty reduction programs are implemented.

Intermediate Objectives： In Special Assistance Districts of the capital of Amelia- 1 The number of school

facilities with sufficient educational function is increased.

2 School curriculum is

improved. 3 Teachers’ skills and

knowledge is improved. 4 Students’ morale toward

schooling (learning?) is increased.

5 Accessibility of girls’

schooling is improved.

1-1.More than 90% of schools meet the

national school facility standards level B in the special district.

2-1 An improved school curriculum is

endorsed by the educational committee and by the third party by the year of XXXX.

2-2 More than 80% of schools in the city

introduce an improved curriculum by the year of XXXX.

3-1.More than 70% of all teachers in the

district have a license with which they can teach the program by the year of XXXX.

4-1.The rate of students from the special

school district who think they enjoy schooling increases to more than 90%.

5-1.The rate of women who think girls

have limited educational opportunities decreases to less than 10% by the year of XXXX.

1-1 Results of audit of the national standards of school facilities 2-1 Minutes of the Boards of Education 2-2 Statistics on education (the Ministry of Education) 3-1 Statistics on education (the Ministry of Education) 4-1 Results of Questionnaire and Survey 5-1 Results of Questionnaire and Survey

・ Trained teachers

continue working for the school

・ School buildings constructed by the project are adequately maintained

・ Social condition surrounding students do not worsen.

・ M’s projects to improve secondary school education are implemented.

Activities： 1. Provide school facility in a

special district 1-1. Select a school for the

program 1-2. Construction of a school

building 1-3. Equip the facility with a

bathroom 1-4. Equip with a water tank 1-5. Training on how to manage

and maintain the facility 1-6. Procure teaching materials 1-7. Audit the national facility

standards (The rest is omitted)

Inputs： Project Name ・ Secondary school reconstruction project ・ Secondary school teacher training project ・Secondary school curriculum development project ・Secondary school scholarship program Total Amounts: ooooooooo Yen Major Inputs: ○○、��、×× Stakeholders: Amelia: the Ministry of Education, Ministry of Social Insurance,

Ministry of Justice Local Public Agency: Department of Secondary Education

Pre-conditions：

Remarks: After conducting the participatory workshops and social surveys, the program is improved to thereby be focusing more on the poor.“ Special Assistance Districts” are areas where the average income of households is less than 30% of the average of the total population of the capital.

©FASID 60

Output Sample 2: Improved Logic Tree created for forming the Log-frame

Poverty

reduction in the capital

city of Amelia is promoted.

Enrollment rate for secondary education from

the poor families in

capital city of Amelia

increases.

The number of good school buildings increases

Teaching curriculum improves

Teaching methods improves

Student's morale for schooling improves

Female access to education increases

Activities Activities Activities Activities Activities Activities Activities Activities Activities・Omitted・ Activities ・Omitted・

©FASID 61

c) Explanatory Notes 4-2-1. Cases Where an Improvement of the Log-frame is Not Necessary In cases where the Commitment of Improvement does not include the restructuring of the Log-frame

in its basic strategy, those procedures in this stage can be skipped. In such a case, the existing

Log-frame is accepted as the base plan and the action plan is formed for the necessary improvements.

4-2-2. Workshop for Improving the Log-frame In cases where an improvement of the Log-frame is needed, it is recommended to hold a

participatory workshop with the stakeholders, rather than planning this at a desk or in a

meeting attended by a limited number of planners.

A participatory workshop has several advantages as follows.

・ It nurtures and encourages the ownership of the stakeholders for the Log-frame and the

action plan, which will increase their feasibility.

・ It improves the effectiveness of the plan by considering and adding various ideas from

multiple views of stakeholders.

・ It promotes team building among the stakeholders through such teamwork.

There are, however, some disadvantages with the participatory workshop, such as “the planning is

biased towards a few powerful participants”, “analysis tends to be short-sighted”, “the structure of the

plan is inconsistent”, etc. To avoid these weaknesses of the participatory workshop, it should be

carefully planned and managed by an experienced workshop facilitator.

In the participatory workshop for considering improvements, the participants are the stakeholders

indicated within the Commitment of Improvement. Discussions and analysis in the workshop can be

progressed smoothly by using paper cards (half the size of A4 paper) and larger pieces of paper in order

to visualize the discussions and analysis.

The goal of the workshop is to improve the Log-frame as the base plan.

Steps in the Participatory Workshop for Log-frame Improvement ① Review of the existing Logic Tree

- To review the Logic Tree critically from the Overall Goal, and identify the problems and

shortcomings

② Improvement of the Logic Tree

- To refine the structure of the Logic Tree to attain the Overall Goal effectively.

③ Formation of the Log-frame improved.

©FASID 62

The above sequence of procedures is a basic example, with the program of the workshop being

customized according to needs and constraints. When planning a participatory workshop of LEAD, it is

recommended to read the guidebook about the Participatory Planning method of PCM, since the PCM

is the foundation of the LEAD method. It is also a good idea to consult experienced moderators of the

PCM workshop first of all.

The evaluation study must have produced documents containing useful information. The results and

other information from the last evaluation should be fully utilized within the planning process,

especially when selecting the alternatives.

4-2-3. Ex-ante Evaluation of the Improved Log-frame After drafting the first version, the improved Log-frame is assessed and adjusted with the following

evaluation criteria.

・ Feasibility of the Plan

Is the program logic (or theory) well structured?

What is the reality of the setting of each component (especially the inputs)?

・ Five Evaluation Criteria

Is it “relevant” to the national policy and the needs of the beneficiaries?

Is it effective to achieve the Overall Goal?

Is it an efficient way for using the least inputs to produce the best outputs?

Are there any positive or negative impacts to be concerned?

Is it sustainable in its effects and good impacts?

Through the above ex-ante evaluation, the program expressed in the Log-frame is adjusted and

improved before commencement if necessary. The ex-ante evaluation is recommended to be a part of

the participatory planning workshop. For increasing the credibility of the plan, some of the ex-ante

evaluation should be conducted by the third party.

©FASID 63

4. Improvement Phase 4-3. Improvement Implementation

a) Outline

Objective: To implement the improved Log-frame with a monitoring system, which enables the continuous improvement of the program.

Procedures:

1. Implement the action plan that complies with the improved Log-frame.

2. Conduct monitoring on the progress of the plan.

3. Implement a periodical evaluation study.

4. Iterate the improvement phase with the periodical evaluation study.

Output:

Continuous Improvement within the Improvement Cycle

©FASID 64

b) Output Sample: Improvement Cycle

Commitment of Improvement

Recommendation

Implementation

（Do）

Evaluation

(See)

Planning

(Plan)

Regular Evaluation

Implementation of Activities

Planning of Activities

Planning of Improved Log-frame

Monitoring

Commitment of Improvement

Start from here!!

©FASID 65

c) Explanatory Notes 4-3-1. Progress Monitoring Progress monitoring is one of the most important activities after planning. Objectives of the progress

monitoring are to confirm the progress towards the Overall Goal, and to conduct a proper adjustment

if necessary. Thus, monitoring takes care not only of the Overall Goal, but also other components of the

Log-frame such as the Intermediate Objectives, Activities, Inputs and Important Assumptions. The

components to be monitored can be selected by referring to the Log-frame, in which the components

are structured in the “Means-Ends” relationship.

The most important item to be monitored is the achievement of the Overall Goal. It is, however, very

rare that effects on the Overall Goal can be measured during the early stages of the program.

Therefore, in the early period of the implementation, monitoring is concerned only with achievement of

Inputs and Activities. As the program is progressed, the effects on Intermediate Objectives can be

measured. Then finally the effects on the Overall Goal are reached, which is the final stage of the

program.

In analysing the causes of delay in progress, the “means-ends” relationship in the Log-frame can be

very useful. When finding that the effect on one of the Intermediate Objectives is not confirmed as

planned, the achievement of the activities is firstly examined as the means for the Intermediate

Objective. Next, the realisation of the relevant Important Assumptions to the Intermediate Objectives

can be referred to. If finding that some of the activities were not implemented, the cause can be

analysed by examining the achievement of the planned components in the Inputs section or the

realisation of the relevant Important Assumptions.

Narrative

Summary (of

Program/Policy)

Objectively

Verifiable

Indicators

Means of

Verificati

on

Major

Important

Assumptions

Overall Goals

(Program

Goals)：

Intermediate

Objectives

（ Sub-program

Goal）

Activities:

Inputs： Pre-condition

s

”Means-Ends”

relationship

©FASID 66

If these problems are found, it requires counter-measures such as adjustment, avoidance, mitigation,

and so on. The counter-measures are different in each situation, but it is important to plan for the

counter-measures within the basic organisation.

4-3-2. Monitoring System Monitoring should be conducted systematically within a monitoring system, which must consist of

“data collection”, “aggregation”, “decision” and “communication” as its components. The personnel and

organizations responsible for this should be clarified for each component. It helps the understanding if

the monitoring system is described in a diagram or a table as follows.

Example of a Monitoring System Narrative summary

Objectively Verifiable Indicators

Means of Verifi- cation

Data Collection ・ personnel ・ schedule ・ method

Aggregation ・ personnel ・ schedule ・ method

Decision ・ personnel ・ schedule ・ method

Method of counter- measures

(Intermediate Objective: 1) Literacy rate of women in village A is increased.

By the year XXXX, more than 80% of adult women in village A can read basic texts.

Monitoring report of the program

・ Chiefs of women’s groups

・ Every 3 months

・ Sample survey

・ Leader of literacy program

・ Every 3 months

・ Aggregate data

・ Steering Committee

・ Every 3 months

・ Discuss and vote

Necessary counter- measures can be conducted by the implementers, if they are at the activities level.

Intermediate Objectives are not achieved

Are activities successfully

implemented?

Are major important assumptions

realized?

Are inputs successfully

projected?

Do other activities achieve their

goals?

Are major important

assumptions realized?

©FASID 67

The above table is just an example. The monitoring system should be customized with the needs of

the stakeholders and based on utility.

4-3-3. A Periodic Evaluation for Further Improvement A periodic evaluation is necessary in order to improve fundamentally the structure of the program

as the program is being progressed. The periodic evaluation starts by creating a Scope of Evaluation,

and with the procedure then just following the steps from the evaluation phase of LEAD, as described

previously. Conducting the LEAD procedure the second time will be much easier than the first, since

the Log-frame has already been constructed and the stakeholders will be more experienced.

©FASID 68

5. Epilogue: Everlasting Cycle of Improvement

The concept of PCM (Project Cycle Management) is to manage the entire cycle of a project, consisting

of planning, implementation and evaluation, consistently with a prepared plan named PDM (Project

Design Matrix), which is the Log-frame.

In contrast, LEAD (Log-frame Evaluation Application Design) starts from the evaluation by

combining related projects and activities into a proxy-program into a Log-frame, which is then

evaluated as a program. LEAD continues to the following improvement phase, in which the

recommendations from the evaluation lead to the improvement of the existing program by a

re-planning of the Log-frame. The improved Log-frame will be used in the management of the program

cycle. LEAD results in management via a program cycle of continuous improvement, known as

“KAIZEN” in Japanese.

The basic concept of the cycle of continuous improvement is the same as that of the PCM, such as a

cycle of “plan”- “Do”- “See”. LEAD places greater emphasis on improvement, so that a PDSA

(Plan-Do-See-Action) cycle applied in Japanese TQC (Total Quality Control) is more suitable to LEAD.

The message of the PDSA cycle is that the evaluation should never be conducted only just for the

purpose of evaluation, and that the evaluation must be strongly linked with further actions to improve

the situation.

The PDSA cycle never stops, with improvement being continuous. The objective of the LEAD

evaluation is not to give recommendations that are only tentatively worthy. Rather, it is to install a

cycle of continuous improvement by the use of a Log-frame. LEAD leads you to establish such an

integrated management system with a Log-frame.

Program A

Integrating as Program

Evaluation

Improved Plan

Implementation of Plan

Regular Evaluation

Cycle of Improvement

Program B

Program C

Program D

©FASID 69

Reference Books (not completed, but useful) ・ FASID (2002), PCM Project Cycle Management Participatory Planning,

FASID ・ FASID (2000), PCM Project Cycle Management Monitoring and Evaluation,

FASID ・ Rossi, Peter H, Forward E. Freeman, and Lipsey, Mark W.(1999),

Evaluation: A Systematic Approach, 6th ed., SAGE Publications ・ Weiss, Carol H.(1998), Evaluation: methods for studying programs and

policies, 2nd ed.., Prentice Hall ・ Wholey Joseph S., Harry, and Kathryn E. Newcomer, eds(1994), Handbook

of Practical Program Evaluation, Jossey-Bass ・ Patton, Michael Quinn. (1997). Utilization-focused evaluation: The new

century text (3rd ed.). Thousand Oaks, CA: Sage

©FASID 70

Member List Members of Study Group for Evaluation on Public Policy and Programs - Ryo SASAKI - Keiko NISHINO - Kiyoshi YAMAYA - Masaki SAITO - Yujiro HAYAMI - Masafumi YAGI - Yoko ISHIDA - Naonobu MINATO Members of PULUG (Production Unit for LEAD Users Guide) - Takahiro MIYOSHI - Yasushi HAYASHI - Mari FURUYA - Mitsuru KAYAMA All comments and inquiries must be received by Naonobu Minato. The contact address is minato@fasid.or.jp Foundation for Advanced Studies on International Development (FASID) Chiyoda Kaikan Building (4th and 5th fl. ) 1-6-17, Kudan-Minami, Chiyoda-ku, Tokyo 102-0074, Japan