1 |

Monitoring and Evaluation

Module C5

2 |

Module objective

By the end of this module, you will be able to

● Describe the importance of monitoring and evaluation in risk communication for a public health emergency

● List key components in the logical framework for evaluation

● Describe the difference between output evaluation and outcome evaluation

Photo:WHO/Yu Zhao

3 |

Why monitoring and evaluation (M&E for emergency risk communication (ERC)?

● Investment in terms of human, financial and other resources requires us to prove we have achieved results

● Being able to measure and communicate results of risk communication work has a huge impact on 'trust' and 'credibility' – itself a fundamental principle in risk communication

● By communicating effectiveness of activities undertaken, products produced etc., more resources can be mobilized to strengthen ERC work in the future

Image: http://wesleyboardresources.org/wordpress/wp-

content/uploads/2015/02/teacher-evaluation.jpg

4 |

What is monitoring?

● A continuous internal process for making sure that the activities under the programme/project (e.g., the implementation of ERC capacity building as stated in the national strategy) are on track

● Monitoring of the activities should be done on a regular basis

● The process should provide regular feedback

● Progress of project activities, use of resources, results achieved and institutional systems (staffing, policies, etc.) should be monitored

Image:

http://www.galilcol.ac.il/pictures/pictures/M&E_1.jpg

5 |

What is evaluation?

● Systematic way of collecting, analysing and using information to answer basic questions about a project/programme.

● Can be internal, external or involving many key stakeholders

● Can be done by

– Process evaluation: assess whether an intervention/model was implemented as planned, whether the target population was reached, and what were the major challenges and successful strategies used- Example: Have trainings in ERC been completed as planned?

– Outcome evaluation: determine whether and to what extent the expected changes occurred and whether these changes can be attributed to the programme activities. Example: Have trained personnel been deployed in an emergency response in a timely way?

Image:

http://ieg.worldbank.org/Data/influential_eva

luation_blog.gif

6 |

The logic model

● Helps to map out where the programme or intervention is headed. Planning and evaluation go hand-in-hand. There is an evaluation logic model that needs to be developed early in the planning phase.

Impact

Outcome

Output

Input

Components of the logic model

Source: Rice Project, Harvard School of Public Health

7 |

The logic model approaches

❖ Outcomes….the consequences of the

outputs…e.g. Information reaches the

public, new journalists report the facts

correctly, people adopt public health

advice such as hand washing,

vaccination, etc.

❖ Impact ….. Contributes to less people

dying, less sick, limits damage to

economies, etc.

Output evaluation: Activity driven

Outcome evaluation: Change driven

❖ Activities……e.g., message

development, stakeholder

engagement

❖ Outputs........e.g., products

developed, or number of community

outreach activities conducted,

trainings conducted, posters

distributed, TV or radio spots

produced and disseminated, number

of press briefings, etc.

8 |

The logic model Public health

event or

emergency

Activities

9 |

Output evaluation

● Outputs are products of a program’s activities or services, often expressed in terms of units, i.e., number of IEC materials produced and disseminated, number of people trained, etc.

● In your logic model, outputs refer to the activities we do and who we reach (target groups)

10 |

Outcome evaluation

● A systematic process of collecting, analysing and using information to assess the effectiveness, relevance and impact of achieving the programme/project goals within a specific period of time

● It involves measuring change in knowledge, attitude, behaviours, skills, norms and practices such as vaccine uptake

● More than one output (awareness raising campaigns, community engagement, IEC material, radio spots, etc.) is needed to produce an outcome

Oral Cholera Vaccination Campaign in

Southern Malawi, 2015

Photo: WHO/L. Pezzoli

11 |

Preparing for M&E

● Set out indicators based on the programme/project objective

● The indicators are agreed measures to see progress

● Indicators need to be comparable to the baseline data/information before and after the programme/project is implemented

Image:

https://www.google.ch/search?q=indicators&hl=en&source=lnms&tbm=isch&sa=X&ved

=0ahUKEwj_09PT3b3KAhUB1RQKHQ5XBz8Q_AUIBygB&biw=2048&bih=749#hl=en

&tbm=isch&q=SMART+indicators&imgrc=CR_UHMu30v7FSM%3A

12 |

What are the key questions?

● Did the programme/project achieve its objectives?

● Did the target group benefit from the programme/project?

● Was it cost effective?

● Can the improved outcomes be attributed to the programme/project?

● Which activities were more or were less effective?

13 |

Indicators

S •Specific

M •Measurable

A

A •Achievable

R •Realistic

T •Time-bound

14 |

Data collection methods

● Qualitative

– In depth interviews, observations, document review, participatory assessment, focused group discussions

● Quantitative – Structural methods for data

collection, including household surveys, data collection, statistical surveys, etc.

Image:

htItps://www.google.ch/search?q=indicators&hl=en&source=lnms&tbm=isch&sa=X

&ved=0ahUKEwj_09PT3b3KAhUB1RQKHQ5XBz8Q_AUIBygB&biw=2048&bih=749#

hl=en&tbm=isch&q=Data+collection+methods&imgrc=qq82HO9mDqZ8cM%3A

15 |

Example of a risk communication response logical framework

Situational

assessment

Community

engagement/Social

mobilization

Media monitoring Number of media

outreach

Inputs

Human resources

Communication

systems

Stakeholders trained

Activities Outputs Outcomes

Message development

Information sharing,

dissemination

Planning, strategic

direction,

implementation

Number of community

outreach activities

conducted

Number of releases of

public information

Number of products

(brochures, posters)

distributed

Numbers of

community

organization engaged Communication

coordination

Community outreach,

visits, meetings

Information reaches

audiences quickly

Messages are

consistent

Change in public

opinion, reaction

Transparency is

achieved

Change in

population KAP

Barriers to adoption

of messages are

removed

16 |

Example of outcome approach vaccine distribution model

Input

•Providers, authorities and pandemic planners participate in pandemic planning efforts

Activities

•Activities that encourage and enable providers, authorities and planners to achieve common pandemic goals

Output

•Providers, authorities and planners achieve mutual agreement for distribution of vaccines

Outcomes

•Increased availability of providers administering the vaccine

Impact

•Vaccination leads to health and social benefits

17 |

Evaluating risk communication capacity

● The latest model for monitoring and evaluating national risk communication capacity examines

– 5 risk communications domains : systems, partner and internal communication, public communication, community engagement, listening and rumour management

– 5 levels of capacity: from none to sustainable capacity

Image:

https://www.google.ch/search?q=indicators&hl=en&source=lnms&tbm=is

ch&sa=X&ved=0ahUKEwj_09PT3b3KAhUB1RQKHQ5XBz8Q_AUIBygB

&biw=2048&bih=749#hl=en&tbm=isch&q=capacity+assessment&imgrc=

huBpl0UV4kdJJM%3A

18 |

WHO risk communication capacity assessment

logical framework

Scoring

Indicators R.5.1 Risk

Communication

Systems

(Plans,

mechanisms, etc.)

R.5.2 Internal and Partner Communication and Coordination

R.5.3 Public Communication

R.5.4 Communication Engagement with Affected Communities

R.5.5 Dynamic Listening and Rumour Management

1. No Capacity

2. Limited Capacity

3. Developed Capacity

4. Demonstrated

Capacity

5. Sustainable Capacity