Monitoring and Evaluation of Monitoring and Evaluation of GeSCI’s ActivitiesGeSCI’s Activities

GeSCI

Team Meeting 5-6 Dec 2007

Why should GeSCI Monitor and Evaluate our Why should GeSCI Monitor and Evaluate our activities?activities?

• To Evaluate: The quality and effectiveness of our advice

• To Promote: to MoE, partners, stakeholders, donors

• To Acknowledge: Important Milestones

• To Learn: GeSCI as a “learning organisation”

• TO IMPROVE

The challenges to GeSCI Monitoring and The challenges to GeSCI Monitoring and Evaluating our activitiesEvaluating our activities

• Without critical analysis of the M&E information, it will not turn into knowledge

• Without “learning agility” M&E will just increase everyone’s workload

• M&E will not be dynamic if it doesn’t feed back into the learning cycle

• “Hidden assumptions” – do we all think it is important enough?

Questions to ask at the beginning of Questions to ask at the beginning of programmeprogramme

1. Are there clear programme objectives?

2. Are there aligned benefits or performance indicators defined for the project?

3. Are those benefits measurable?

4. Do the benefits have owners / beneficiaries?

5. Is there a baseline against which to measure each benefit?

6. Is there clear accountability for each objective?

7. Have the objectives, benefits and targets been communicated?

8. Is the process / responsibility for measuring the benefits established?

Questions to ask during the programmeQuestions to ask during the programme

1. Are the benefits being monitored?

2. Are remedial actions in place?

3. Is the progress being communicated?

4. Are all of the programme activities aligned to the delivery of the benefits?

5. How engaged are the benefits owners/recipients of change?

6. Is the senior behaviour supportive of the benefits delivery?

7. Are the benefits aligned to the objectives of individuals?

What is Outcome Mapping?What is Outcome Mapping?

• Research carried out by IDRC’s Evaluation Unit and Michael Quinn Patton

•Takes the form of periodic facilitated workshops and follow up interviews with beneficiaries, partners and stakeholders

• Outcome Mapping assesses contributions made by development projects and programmes, or organizations to the achievement of outcomes

•Outcomes are defined as changes in the behaviour, relationships, activities, or actions of the programme’s partners

•Outcome Mapping should not replace traditional forms of M&E that look at changes in state. - OM to support our workplans?

Why not focus on impact?Why not focus on impact?

• Demonstrating impact requires isolating key factors that cause results and attributing them to one program or organisation

• Linear “cause and effect” thinking contradicts the complex process of development

Feedback on performance should concentrate on improving rather than proving, understanding rather than reporting, creating

knowledge rather than taking credit

Outputs of WorkshopOutputs of WorkshopStage 1: Design Stage

•Planning: Why? How? Who? vision, mission, boundary partners

What? outcome challenges, progress markers

Stage 2: Performance Monitoring

• Strategy maps

• Organisational Practices Journal

• A monitoring framework outcome journal, strategy journal, performance journal

Stage 3: Evaluation

• Evaluation Plan

The Questions that Outcome Mapping The Questions that Outcome Mapping will help to answer :will help to answer :

• In designing and articulating the program’s logic

- What are our development goals?- How can our programme contribute to those development goals?- How can we help our partners contribute to the broader

development goals?

• Recording internal and external monitoring data

- How far have our beneficiaries progressed towards achieving outcomes?

- What are we doing to support the achievement of those outcomes?

- How well have we performed?

• Indicating cases of positive performance and areas of improvement

- What worked well? Why?- How can we maximise our contributions?

The Questions that Outcome Mapping The Questions that Outcome Mapping will help to answer will help to answer (continued)(continued)::

• Evaluating intended or unexpected results

- Who changed / what changed? How?- If they did not change as expected, do we need to do something

different or reorient our expectations?

• Gathering data on the contribution that a program made to bringing about changes in its partners

- What activities / strategies were used?- How did the activities influence the individuals, groups, or

institutions to change?

Examples of Outcome MappingExamples of Outcome Mapping

• SchoolNet Namibia

– Used in 2005 to explore details of the programme and lay down a road map for replication in other situations

• Uganda Health Information Network

- Used in 2004 to monitor their organisational practices, progress made towards change, strategies employed to achieve

outcomes

School Net Namibia ExampleSchool Net Namibia Example

SchoolNetSchoolNetNamibiaNamibia

EducationEducationDepartmentsDepartments

University andUniversity andPolytechnicPolytechnic

NamibiaNamibiaTelecomTelecom

TargetTargetschoolsschools

ICT suppliersICT suppliersand serviceand serviceprovidersproviders

National Planning National Planning CommissionCommission

Namibian Namibian Telecoms Telecoms RegulatorRegulator

Namibia PowerNamibia Power

Boundary partners

School Net Namibia ExampleSchool Net Namibia Example

Boundary partner Outcome challenge

Government officials and policymakers

Education Department through the Ministries of Higher Education and Basic Education.

Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation.

School Net Namibia ExampleSchool Net Namibia ExampleBoundary partner Outcome challenge

Ministries of Education(s)

Endorsement of the ICT initiatives, approval of the implementation details and establishing guidelines for future implementation.

Progress markers

Expect to see1. Negotiating with SchoolNet and other service providers on details

of implementation2. Setting a priority list for implementation

Like to see1. Efficient coordination of partnerships for delivery of ICT services

to schools.2. Establishing guidelines for integrating ICT effectively into learning

and teaching

Love to see1. Creating the critical mass of ICT skills at school level to be able to

realise the country’s vision for the future.2. Setting specific guidelines on cost-effectiveness for ICT solutions

for schools or putting implementation out to tender to see who offers the best option

School Net Namibia ExampleSchool Net Namibia Example

Organisational Practices

1. Seeking feedback from key informants

A feedback mechanism was put in place from the outset and maintained throughout the life cycle of the programme through ongoing discussions with the key stakeholders

Other methods of M&EOther methods of M&E• Logical Framework Analysis

– Work Plans and Monitoring and Evaluation method combined– A framework incorporating the concepts of continued rationale,

efficiency, effectiveness, effects and lessons learned, which is used during implementation/completion and after the completion stages for ex-post evaluations

• Results Based Management– Feedback on actual outcomes and goals of programme actions– Help answering to questions: what are the goals; are they being achieved;

and how can achievement be proven?• Balanced Scorecard

– A management system to enables organizations to clarify their vision and strategy and translate them into action

– A feedback around both the internal processes and external outcomes in order to continuously improve strategic performance and results

– Provides a framework for managing the implementation of strategy while also allowing the strategy itself to evolve in response to changes in the organization’s competitive market, and technological environments.

If you do not measure results, you cannot tell success from failure.

If you cannot see success, you cannot reward it.

If you cannot reward success, you are probably rewarding failure.

If you cannot see success, you cannot learn from it.

If you cannot recognize failure, you cannot correct it.

If you can demonstrate results, you can win public support.

Adapted from Osborne & Gaebler 1992

THANKS