module 5 project evaluation jan 2004.ppt...

Global M&E InitiativeGlobal M&E Initiative5-1

Module 5

Project Evaluation at IUCN


Project Cycle

Project Identification &

Design

Implementation

Evaluation

Project Approval & Funding

Monitoring Reporting

Communication

Situation Analysis


Objectives of the Module

• At the end of Module 5 participants will:

- Be familiar with the concept of project evaluation as practiced in IUCN

- Understand the responsibilities of project managers for evaluation

- Understand the key elements of an evaluation process and the standards to which they should be carried out in IUCN

- Understand the different ways of effectively communicating evaluation results


Perspective Taken in this Module• Most IUCN evaluations are initiated by external donors• IUCN is seen as a full participant in managing all phases

of the evaluation.• IUCN managers are responsible to ensure that the IUCN

evaluation policy and standards are integrated into all evaluations.

• As such this module focuses on what needs to be done in order to manage an IUCN project evaluation.

Note: for those of you interested in improving your skills as an evaluator, please contact the Evaluation Coordinator in IUCN


Evaluation in IUCN• What is it?

- A systematic and objective assessment of an ongoing or completed project

• Design• Implementation• Results

- Involves gathering, analysing, interpreting and reporting information

- Should be based on credible data

• Purpose • Learning and improvement• Accountability


Types of Evaluation• Two ways of classifying:

- When it is done• Ex-ante evaluation• Formative evaluation• Summative – end of project • Ex-Post evaluation

- Who is doing it• External evaluation• Internal evaluation or self-assessment


Types of Evaluation • Ex–ante evaluation

- Conducted before the implementation of a project as part of the planning

- Also referred to as appraisal or quality at entry• Formative evaluation

- Conducted during the implementation of the project- Used to determine the efficiency and effectiveness of

the implementation process - Provides information to improve processes and learn

lessons


Types of Evaluation • Summative evaluation

- Conducted at the end of the project- To assess state of project implementation and

achievements at the end of the project- Collate lessons on content and implementation

process

• Ex-post evaluation- Conducted after the project is completed- Used to assess sustainability of project effects,

impacts - Identifies factors of success to inform other projects


Types of evaluation• External evaluation

- Initiated and controlled by the donor as part of contractual agreement

- Conducted by independent people – who are not involved in implementation

- Often guided by project staff


Types of Evaluation • Internal or self assessment

- Internally guided reflective processes- Initiated and controlled by the group for its own

learning and improvement.- Sometimes done by consultants who are outsiders to

the project- Need to clarify ownership of information before the

review starts


Evaluation Criteria for IUCN

1. Relevance

2. Effectiveness

3. Efficiency

4. Impact

5. Sustainability


Inputs Activities Actual Results

Reasonsfor Project

Project Definedand Described

ImplementationProcess

Evaluation Criteria, Project Makeup and Logic

Outcomes

The Situation to Improve Problems

and Visions

IMPACT

Project Plan(Planned Results

and Activities)


Evaluation Criteria, Project Makeup and Logic

1. Relevance -Design

2. Effectiveness(Performance)

5. Sustainability

Project Plan(Planned Results

and Activities)

Outcomes

The Situation to Improve Problems

and Visions

Inputs Activities Actual Results

4. IMPACT

3. Efficiency


Evaluation Criteria and Questions • Relevance - Was/is the project a good idea given the situation to

improve? Was the logic of the project correct? Why or Why Not?

• Effectiveness - Have the planned results been achieved? Why or Why Not?

• Efficiency - Have resources been used in the best possible way? Why or Why Not?

• Impact - To what extent has the project contributed towards its longer term goals? Why or Why Not? Have there been any unanticipated positive or negative consequences of the project? Why did they arise?

• Sustainability – Can the outcomes be sustained after the project funding to ensure continued impacts? Why or Why Not?


Evaluating IUCN Projects

• Results-oriented accountability

• Improving planning and delivery

• Quality control• Supporting an

evaluation culture

• Work in partnership• Transparency • Ethical • Credible • Accessibility• Utility

Will be done in accordance with the organisation's evaluation policy

Guiding Principles


Roles and responsibilities of Project managers

• Varies in different types of evaluations but includes:- Know and implement donor contractual requirements

for evaluation - Identify needs for internal evaluations and self-

assessment processes that foster learning, reflection, and improvement

- Ensure that resources for evaluation are built in at the target level of 3-5% of the total budget for your project

- Ensure staff have specialised training needed to keep up-to-date on the use of evaluation techniques and tools


Roles and responsibilities of Project managers cont’d

• Varies in different types of evaluations but includes:

- Link project evaluations to the programme results (outputs and outcomes) in both the component programme and the IUCN Intersessional Programme

- Build time into workplans for anticipated evaluations

- Liase with the M&E global and regional staff to obtain the necessary support to design, manage and deliver high quality evaluations

- Create a culture of learning and reflection by actively using the results of evaluations for project planning improvements


Steps in Managing a Project Evaluation

1. Establishing the need for an evaluation

2. Initial Planning and Resourcing

3. Developing Terms of Reference

4. Engaging the Evaluator or Evaluation Team

5. Approving the Workplan

6. Implementing and Monitoring the Evaluation

7. Assessing the Results of the Evaluation

8. Developing a Plan for Follow-up


Step 1: Establishing the need for an evaluation

• IUCN managers need to clarify the purpose of evaluations. Examples:

- Donor requirement- Accountability- Innovation- Learning and change - Responding to changed circumstance


Step 2: Assessing the ability and readiness to evaluate

•Evaluations take up significant time and resources•Need to ensure that the costs are appropriate for the anticipated benefits. Some considerations:

1. Importance of the evaluation to IUCN or the donor-does it need to take place?

2. Ability and readiness of IUCN and partners to engage in the evaluation-Is there a reason that the evaluation should be postponed? Cancelled?

3. Size of the evaluation. Setting the focus and scope for the evaluation

4. Resourcing the evaluation: Money? Technical expertise?


Defining scope and size• Clarify

- if external or internal- Level of effort and resources required /

available- Stakeholder groups to be involved and how.


Defining scope and size cont’d• Full stakeholder desirable, but could be limited to the

following:- Deciding whether or not to evaluate.

- Defining the type of evaluation, its scope, and criteria.

- Defining the evaluation questions, what are the key issues to explore in the evaluation?

- Defining evaluation workplan. Evaluation activities must be scheduled and fit into the stakeholders' agendas.

- Deciding which recommendations to adopt and which to reject.

- Disseminating and gathering feedback on the results.


Providing resources for the evaluation

• Evaluations require substantial investments of financial and human resources.

• IUCN’s Evaluation Policy recommends projects allocate 3-5% of the total budget to evaluation.

• These funds often used for external evaluations

• Managers may face challenge of funding internal evaluation

• Project funds allocated to training, communication & learning can often be used to finance monitoring and evaluation and to collate lessons learned.

• If funds are insufficient consult M&E for advice.


Developing Terms of Reference (TORs)• TOR are the key guide for an evaluation. • They should

- clarify reasons for the evaluation- highlight issues that have become apparent- indicate the general depth and scope required- spell out any imperatives for the evaluators- provide details about methodology, scheduling, cost and the

qualifications of the members on the evaluating teams

• The project manager is responsible for ensuring clear and focused TORs

• M&E staff will help where necessary.


Contents of Terms of ReferenceSuggested Outline For A TOR1. Context for the evaluation2. Rationale or purpose for the evaluation3. Evaluation issues and questions4. Evaluation stakeholders5. Methodology6. Qualifications of evaluators7. Schedule8. Outputs and Deliverables9. Cost10. Action Plan11. Appendices - Evaluation Matrix, Evaluation Policy, LFA


TORs – involving stakeholders – whose questions?

List made by committeeNot conceptual or strategicToo many issues

Ownership of processIncreased likelihood of implementation of results

Involving Stakeholders

No buy-in from stakeholders

AccountabilityEvaluation owner identifies the issues

Contracting Authority – IUCN

WeaknessesStrengths


Formulating evaluation questions

• Using the evaluation matrix

- Evaluations designed to answer a series of questions about the project.

- Evaluation questions help to focus the evaluation to respond to stakeholder needs

- Good questions are clear, focused, and relevant and make sense to all stakeholders.

- Five broad issues guide evaluation questions in IUCN relevance, effectiveness, efficiency, impact, sustainability.

• Questions can be raised on each of the issues.


Developing evaluation questions• Factors guiding choice of questions

- Objectives and results of the project- Stage of the project in the project cycle- Stakeholders likely to use the findings

- Other guides:• Relative importance of each question• Relative ease, resource requirements and data

availability for answering the possible questions.

• Selected questions should show clear link to issues of concern and purpose of the evaluation.


Developing evaluation questions

• Resources to consult:

- Project design document – Does it specify objectives, outputs, outcomes and indicators for the intervention?

- Funding agreements - were specific evaluation questions part of the funding agreements

- “Stepping into the new millennium: IUCN’s Intersessional Programme” - How do the questions link to the key result areas (KRAs) or the KEGO?

- Other evaluations or studies - What questions did they ask and what did they learn?

- Key project stakeholders - What do they think are important questions?

- Experts in the field - What insights can they bring to bear on formulating the evaluation questions?


Developing evaluation questions• Tips

- Remain open to a wide variety of possible questions- Resist locking yourself into a question before having

consulted adequately - Be realistic about the number of questions that can be

answered in a single evaluation- Focus on priority questions

• Evaluation matrix (next slide)- Used to organise evaluation questions around issues -

provides structure- Ensures that multiple data sources are used to answer

each question- Ensures a balance of stakeholder views is sought during

the process.


Developing an evaluation matrix

Other evaluation criteria

Sustainability

Impact

Efficiency

Effectiveness

Relevance

E.g. data sources

IndicatorsSub-questions

Key questions

Issues


Engaging the Evaluator or Evaluation Team

• Evaluators can be selected by IUCN, imposed by donors or jointly agreed to.

• In all instances IUCN standards provide guidance to managers on engaging evaluators.

• This guidance includes:- The appropriate level of technical expertise or

evaluation expertise - The previous experience or profile of the evaluator

(IUCN has suggested profiles)- Suggested profile of a good evaluation team - Using peers as evaluators - Roles and responsibilities


Engaging the Evaluator or Evaluation Team

• Important issue in IUCN - Balance of both technical expertise and evaluation expertise

• Past practice – reviews done by technical experts –interesting – but often don’t address evaluation questions

• Need a balance• M&E staff can assist• Using peers as evaluators – challenges, opportunities


Reviewing and Approving the Workplan

• The evaluation work plan is developed by the evaluator and the evaluation team

• It should:- provide roadmap for conducting the evaluation

(interprets TORs)- include proposed methodology and means of analysis

• A poor work plan leads to poor evaluation• IUCN has checklist criteria for reviewing a work plan• Important that project Manager and Regional Evaluation

Coordinator review and approve the evaluation work plan


Reviewing and Approving the Workplan cont’d

• Suggested outline of a Workplan

Introduction - purpose and stakeholdersEvaluation Questions (framework)Methodology (sources, methods)Schedule (Gantt chart)Resource Allocation and BudgetEvaluation TeamOutline of Evaluation Report


Implementing and Monitoring the Evaluation Work

• Managers required to facilitate evaluators work by:- Supporting field data collection- Making documents available

- Responding to regular evaluation reports and feedback

- Distributing draft reports for comments to appropriate IUCN staff and partners

- Participating in donor and evaluator meetings when requested

- Reviewing drafts of findings and reports and providing feedback


Communicating results to stakeholders

• It is critical to communicate findings to relevant stakeholders in a timely manner

• Use appropriate means- Phone calls and emails are informal - Complement by formal briefings and presentations- Always remember different audiences will have

different needs- Reporting approach should reflect those needs.


Different Audiences may have Different Needs

• Internal staff might need a verbal report and a memo with key points

• Donors and external stakeholders might need a full report

• Ministries might need an abstract

• Public at large might need an abstract of findings only

• Know your audience and match your reporting approach


Effective Communication of Evaluation Results

• Captures the data in its conclusions• Speaks in language of users• Detached, non-possessive stance• Objective - “truth” to power, but• Is pragmatic - goes only as far as the key stakeholders

will accept


Assessing the quality of an evaluation report and process

• Meeting needs – commissioning managers, stakeholders• Relevant scope• Suitable methods• Reliable data• Sound analysis• Credible findings• Impartial conclusions• Clear reporting


Developing a Follow-up Plan• Critical step – often missed• Follow-up plan supports use• Decide which recommendations to accept and which to

reject – together with relevant stakeholders

• Develop a follow-up plan showing: Conclusion / recommendationRecommendations accepted and how they will be implementedRecommendations not accepted and whyWhat needs to be doneWho will do itWhen


Using Evaluation Results• Use of results by managers is weak in many

organisations • Many evaluation reports sit on shelves• Ignored until a donor brings them to our attention - then it

is too late!• Need to ensure use at an early stage• Put evaluation results on staff agendas – debate,

discuss, decide how to implement, build into workplans• Review evaluation results of other agencies,

organizations, competitors


Review, approve and follow up results• Evaluations are costly exercises; results are expected to

be used by program and project managers.

• Typically IUCN evaluations include an action plan as part of the evaluation.

• IUCN evaluations should be made available to partners and a copy sent to HQ.

• When appropriate, evaluations should be used to help our partners learn about what has happened.

• The project manager is responsible for communicating the evaluation results to different audiences


Exercise 5.1 - Developing ToRs and workplan for an evaluation

• Time: 3 hours• Instructions: Working in a small group design TORs for the

evaluation including an evaluation matrix, as described in this module

• Five years later - Assume the project we have developed was funded and has been implemented over the last five years.

The project is about to undergo a formative evaluationYou are managing the evaluation process

1. Identify evaluation questions for the five evaluation criteria (40 minutes)2. Outline the methodology (data required, data sources & data collection

methods) to be used in answering the evaluation questions (40 minutes)3. Determine the qualifications, skills and experiences mix of an evaluation

team (15 minutes)4. Draft the evaluation workplan and a budget (30 minutes)5. Provide the evaluation report outline (15 minutes)6. We will discuss this in plenary (40 minutes)


Test your understanding1. What are the purposes of project evaluation in IUCN?2. List, with examples, the common types of evaluation a project

manager is likely to be involved with in IUCN3. What is evaluation criteria? What evaluation criteria are adopted in

IUCN?4. What are the roles and responsibility of the manager in a project

evaluation?5. What are the steps involved in managing a project evaluation?6. Give an example of an outline for TORs of an evaluation. 7. Why is it important to agree on an evaluation workplan?8. What are evaluation questions? Why are they important guides to

evaluation?9. In your experience, how well are results of evaluations used to

improve management in IUCN?

module 5 project evaluation jan 2004.ppt...

Documents