m&e overview (ii)

7/27/2019 M&E Overview (II)

1/41

M&E Overview


2/41

At the end of this session, participants will

be able to:

Understand concepts of monitoring andevaluation

Understand how to set goals and objectives

Prepare an M&E plan

T. Bezabih


3/41

What Is Monitoring? and What IsEvaluation? How are they different?

How do they fit together?


4/41

Monitoring is the routine

process of data collection

and measurement of

progress toward programobjectives.

Monitoring involves

counting what we are

doing.

Monitoring involves

routinely looking at the

quality of our services.


5/41

The regular follow-up of the implementation of

planned activities. It also involves

documentation of project activities

The systematic and continuous process of

following and keeping track of indicators in

order to ensure that the project/program isproceeding according to plan and modifying the

plan as necessary.


6/41

Are projected outputs being met?

Are we heading in the right direction?

Are we in good time?

Are the indicators appropriate?

Did you identify the correct problem and has this problemchanged?

Are the intervention strategies appropriate to the targetpopulation?

What can be improved in our project?

Are we utilizing resources efficiently?

Gives us the strengths and weaknesses of our project

Provides updates for stakeholders


7/41

Assesses progress against setobjectives/outputs

Supervises implementation

Assesses effectiveness of implementationstrategies

Identifies new issues and/or unforeseencircumstances that may be obstacles

T. Bezabih


8/41

Identifies necessary corrective measures (strategymodification)

Verifies information first-hand for immediate

feedback Strengthens relationships between collaborators(donors, implementers and beneficiaries)

Serves as a motivation to implementers andbeneficiaries

Provides an opportunity to verify whether resourcesare being used effectively (cost-effectiveness)


9/41

Review existing information related to the project. Make a conceptual framework of the project for monitoring.

Identify monitoring goals and objectives.

Identify indicators. Determine which categories of workers, supervisors orothers will be responsible for the collection of each categoryof monitoring data.

Develop a timetable for frequency of monitoring.

Develop/strengthen a management information system. Develop monitoring instruments.

Conduct monitoring activities.


10/41

Analyze monitoring data.

Write a report.

Make recommendations.

Implement recommendations.

Identify new indicators based on the

recommendations.

Modify the monitoring system if necessary.

Continue to monitor.


11/41

Evaluation is the use of social

research methods to systematically

investigate a programs

effectiveness. Evaluation requires study design.

Evaluation sometimes requires a

control or comparison group.

Evaluation involvesmeasurements over time.

Evaluation involves special

studies.


12/41

It is a means of problem verification

It maximizes utilization of resources

It identifies the strengths/weaknesses of the project

It provides information for planning and re-planning

It provides learning opportunities

It provides an opportunity for problem solving (strategymodification)

It is a basis for maintaining and/or improving the existing

strategy It measures the effectiveness of the project/program

It is a check whether the project was implementedaccording to the detailed plan/design


13/41


14/41

DIFFERENCES BETWEEN MONITORING

AND EVALUATION

Monitoring Evaluation

Who M. Internal management responsibility all levels

E. Usually incorporates external inputs (objectivity)

When M. Ongoing

E. Periodic mid-term, completion, ex-post

Why M. Check progress, take remedial action, update

plans

E. Learn broad lessons applicable to other

programs/projects, policy review, etc

Focus on M. Inputs, activities, outputs, process

E. Outcome, impact


15/41

Lack of appreciation of the role of monitoring andevaluation.

Fear of finding mistakes/ Fear of failure

Lack of transparency and accountability by project

managers. Lack of knowledge and skills in monitoring and evaluation.

Cost of re-designing the overall project

Resistance to change by entire project staff

People are overwhelmed by more work, lack of time

Restrictive budgets (lack of funds to accommodatemonitoring and evaluation).

Fear of piracy by external evaluators.


16/41

Process Evaluation

Mid-term Evaluation

Impact Evaluation Summative Evaluation

What is it?

Why is it? When?

Who conducts it?

How should the findings be used?


17/41


18/41

Global Fund wants to know how many PLHIV have beenreached by your HBC program this year.

A local community-based organization wants to start addressingHIV in its community with a comprehensive BCC program and

starts by collecting key bits of information to find out who ismost in need of the services.

After a year of conducting your program, you want to know ifthe budget is being spent in the most efficient way.

NAP+ is interested in finding out if the HBC services providedare being carried out according to national standards of quality.

HAPCO wants to know if the programs being carried out inAmhara Region are changing the risk behaviors of havingmultiple sexual partners in the Region.


19/41

The core of any M&E system is the

goals and objectives of the program

to be monitored and evaluated.

If the program goals and objectives

are written in such a way that they

can be easily distinguished from

one another and measured, the jobof the M&E specialist will be much

easier.


20/41

General statement that describes the hoped-for

result of a program (e.g., reduction of HIV

incidence).

Goals are achieved over the long term and

through the combined efforts of multiple

programs.

T. Bezabih


21/41

Specific, operationalized statement detailing thedesired accomplishment of the program.

A properly stated objective is action-oriented, starts

with the word to, and is followed by an actionverb.

Objectives address questions ofwhat and when,but not why" orhow.

Objectives are stated in terms of results to beachieved, not processes or activities to be

performed.


22/41

(S) Specificity Is it specific? Does it covers onlyone rather than multiple activities?

(M) Measurability Can it be measured or countedin some way?

(A) Attainability Is the objective actually doable?Can we achieve this goal?

(R) Relevance How important is this objective tothe work that we are doing? How relevant is it toachieving our goal?

(T) Time Does the objective give a timeframe bywhen the objective will be achieved, or a timeframeduring which the activity will occur?


23/41

Start with the outcome of theintervention (purpose), thendefine the outputs and then

activities that would achievethe outputs. Then finally theimpact.


24/41

MONITORING &EVALUATION INDICATORS

An indicator is the quantitative or qualitative

evidence that will be used to assess progress

towards an objective.

Indicators provide the basis for monitoring

progress and evaluating the achievement of

outcomes.


25/41

CLASSIFICATIONS OF INDICATORS

Input indicators: measure all the investment

and recurrent cost resources (e.g. human,

financial, facility, equipment, supplies) needed

to enable the activities to be delivered

Process indicators: attempt to set standards

for the quality of activities to be carried out,

such as appropriate training methods or an

adequate supervision plan.


26/41


Output indicators: measure result of inputs &process:

Level at which program is functioning (operations):

e.g. number of services delivered, medicinesprovided, IEC messages delivered

How well program is functioning in terms of itsobjectives (performance): e.g., adequacy of output

(e.g. services delivered) in terms of quality, equity,utilization/access, cost-effectiveness, efficiency,sustainability


27/41


Outcome indicators: measure the main

changes in the program population that are

expected in the short-term, such as improved

knowledge, attitudes or behaviors

Impact indicators: measures a long-term

effect that the program aims to produce, such

as reduction of HIV prevalence


28/41

Practical Examples

Examples of Outcomes, Indicators,

Baselines and Targets (Prevention):

Table 2 Examples of Outcomes, Indicators,

Baselines and Targets (Treatment)- Table

3


29/41

FEATURES OF GOOD INDICATOR

Valid: actually measures the phenomenon (e.g.,self-reported vs. facility based data)

Reliable: produces same results when used morethan once to measure precisely the same

phenomenon

Specific: measures only the phenomenon it isintended to measure

Sensitive: reflects changes in the state of the

phenomenon Operational: measurable with developed and

tested definitions and reference standards


30/41

TYPES OF INDICATORS

PROCESS - show whether the activities that wereplanned are actually being carried out, and carried outeffectively

IMPACT - to assess what progress is being made towards

reaching the objectives, and what impact the work has hadon the different groups of people affected by the work

QUANTITATIVE indicators involve the definition ofnumerical measures eg number of meetings attended

QUALITATIVE indicators refer to definingcharacteristics that cannot be quantified. For example,changes in behaviour orpeoples perceptions

PROXY indicators measure things that represent (orapproximate) changes that cannot be measured directly.


31/41

CHOOSING INDICATORS

Select among possible choices based on:

What level of system youre measuring(inputs, process, outputs, outcomes)

Data availability & quality

Cost of collecting data

Comparability with other programs, projects


32/41

CRITERIA FOR DEVELOPING AND SELECTING

INDICATORS

Result oriented: should focus at measuring resultexpected from the program

Direct: should always be direct as much as possible.Proxy indicators could only preferred when direct

indicators are not possible or difficult to use for differentreasons

Objective: Could be understood without differentinterpretations

One-dimensional/Independent: output indicatorscannot be used to prove achievement of purpose and

purpose indicators achievement of goal

Quantitative: (Whenever possible). However, thereare instances where qualitative indicators could be

desired or even more useful.


33/41

CRITERIA FOR DEVELOPING AND

SELECTING INDICATORS

Disaggregated: (Whenever possible) usedisaggregated indicators. e.g. by sex, age,geographical location, education level, etc.

Simplicity/Unambiguous: Simple and clearly definedin the program's context

Validity/Consistency: The values of the indicatorsshould stay valid/constant as long as they are collectedin identical conditions, no matter who does the

collecting Specificity: Should measure specific conditions that

the program aims to do & change

Sensitivity: should be highly sensitive to changes in a

program situation.


34/41

CRITERIA FOR DEVELOPING AND

SELECTING INDICATORS

Cost-Effectiveness/Ease of Data Collection:

Identify and select indicators for which data, time

and resource (budget) is available

Relevance/Reliable: Should be relevant to program

objectives and measure what is expected to achieve

Timeliness: Should be sensitive to the time it is

done and possible to collect the data reasonably andquickly.


35/41

DATA GATHERING, ANALYSIS, AND

REPORTING STRAGEY

The HIV/AIDS Monitoring and EvaluationFramework, should clearly define:

Indicators for the specific HIV/AIDS

intervention areas Sources of data to generate the indicators

Measurement tool

Frequency of data collection

Responsible body for data collection,compilation, analysis and reporting bylevel


36/41

Suggested reporting schedules for

EMSAP II and GF projects

Level of Indicators Recommended frequency of reportingExample of data collection methods used

Input/process Continuously Health and non- health servicesstatistics,

Program monitoring

Output Quarterly, semi-annually, or annually Health and non-healthservices statistics,

Program monitoring

Outcome 1 to 3 years Population-based surveys,

Health facility surveys,

Special studiesImpact 2 to 5 years Surveillance,

Population-based surveys,

Special studies


37/41

Refer M&E OVER


38/41

The proposal submitted to the Global Fund by

NAP+ has five major core areas: ART

adherence literacy and counseling, home based

care, job creation, food and nutrition support

and capacity building

Break into five groups and write a goal and

three SMART Objectives for each of core areasfor the M&E system.


39/41

Careful selection of the questions you want answered

through monitoring and evaluation will greatly help you

develop your M&E processes and work plan.

At the outset of the planning process, program managersshould ask themselves where they want the program to take

them.

Many of these questions will be reflected in the goals and

objectives.


40/41

Was the activity carried out as planned?

Did it reach its target beneficiaries?

Did any changes in exposure to HIV infectionresult?

How will the risk behaviors of the targetpopulation be affected?

What sort of coverage do you expect to have? Did STI/HIV incidence change?

How much did it cost?

T. Bezabih


41/41

Refer back to your previous group activity

where you developed a goal and a set of

objectives. Now look at these goals and

objectives and come up with at least three

monitoring questions and at least two

evaluation questions.

m&e overview (ii)

Documents