union-imdp.org

THE HEALTH MANAGERTHE INTERNATIONAL MANAGEMENT DEVELOPMENT PROGRAMME® (IMDP)

MONITORING OF HEALTH PROGRAMMES

EXECUTIVE PERSPECTIVE: DR. ADAM KARPATI

EVALUATION FOR FRONT-LINE PUBLIC HEALTH PROGRAMMES

IN THIS ISSUE:

ISSUE 2, 2014

Health solutions for the poor

WELCOME TO

The Health ManagerISSUE 2, 2014

MONITORING & EVALUATION OF PUBLIC HEALTH PROGRAMMES

The properly designed and executed Monitoring and Evaluation (M&E) system addresses the critical issues of performance assess-ment and programme impact. It measures and assesses both pro-gramme efficiency and effectiveness, and M&E is an important contributor of a country’s response to improve public health.

In this second 2014 issue of The Health Manager, we have invit-ed key global practitioners, including The Union’s own in-house M&E experts, to share insights on the key distinctions between monitoring, evaluation, and importance of investing in strength-ening M&E system to deliver results and save crucial resources.

José Luis Castro Interim Executive Director

The Union

3

INSIDE THIS ISSUE

Letter from The IMDP Director . . . . . . . . . . . . . . . . 4

Monitoring of Public Health Programmes . . . . . . . . . . 6

Evaluation for Front-Line Public Health Programmes . . . . . . . . . . . . . . . . . . . . . . 10

Executive Perspective: Dr. Adam Karpati, MD, MPH Senior Vice President For Public Health Impact, The Union North America. . . . . . . . . . . . . . . . . . . 14

The Health Manager is published by The Union’s International Manage-ment Development Programme (IMDP). Director of Publications: José Luis Castro, Director of the IMDP: Tom Stuebner, Marketing and Busi-ness Development Manager: Stephan Rabimov – © 2014 The Union.

4

A MESSAGE FROM THE IMDP DIRECTOR

Dear Colleagues:

Monitoring and Evaluation (M&E) are key components of any project or programme.

Evaluation keeps us focused on our goals and objectives – are we addressing the right issues with the appropriate strategies? Evalu-ation is about effectiveness.

Monitoring keeps us focused on our processes – are we perform-ing our strategies with the correct application of resources. Moni-toring is about efficiency.

As public health leaders, we strive to demonstrate programme im-pact (via evaluation) and good stewardship (via monitoring). M&E is one of the pillars of health systems strengthening (HSS), and HSS is a critically important outcome of the investments made by donors.

In this issue of The Health Manager and in the IMDP’s upcom-ing training on M&E, you are invited to explore the importance of M&E and its application to delivering high-quality, high-impact programmes.

Tom Stuebner Director of the IMDP

5Photo: The Union/IMDP exhibition booth at the 2nd Conference of The Union South-East Asia Region in Dhaka, Bangladesh (March 2014).

6

Monitoring and evaluation are two related yet distinct con-cepts and processes. Monitoring is about controlling the

execution of the project. It consists of reviewing processes so that potential problems can be identified and corrected in a timely manner.

MONITORING OF PUBLIC HEALTH PROJECTSRAJESH KAPOOR IMDP FACULTY

Photo: Courtesy of Rajesh Kapoor / The Union

7

Monitoring does not happen by chance – it has to be planned and budgeted. It is a skill and competence that needs to be developed in all project teams, ideally at the beginning of a project. It needs to be built into the project at the time of detailing the project prior to implementation. This “up front” investment in establishing a monitoring system significantly increases the probability of proj-ect success.

Monitoring helps us to control the process so that the project func-tions within tolerances, with immediate corrective and preventive actions. This requires that we need to have clearly defined norms and standards to be able to do process control. A simple example is of a heart monitoring process. It has a set of activities with de-fined parameters for heart rhythm, blood flow and pressure. Each of these parameters has very clearly determined tolerance limits. The moment any reading goes out of the tolerance limits we know that immediate action needs to be taken to first correct the situa-tion and then prevent the same problem for reoccurring.

The problem in many health projects is that the process control points (which most often constitute the activities) do not have clearly defined parameters and tolerances. In other words, we need to identify the right indicators for our activities with tolerances. In their absence we are unable to make out when the process has gone out of control, and therefore we are unable to take immediate

8

corrective and preventive action. This leads to project delays and cost overruns.

Too often monitoring is done haphazardly and in a too-general sense, without specific remedial correction. By contrast, a well-monitored project requires ongoing review, supervision and cor-rection so that the targeted performance is achieved without cost or time overruns.

Monitoring is a critical function for project managers. That func-tion includes clearly defining and identifying the right indicators and tolerance specifications. Once that is established, other vari-ous elements of data collection and analysis, supervision and re-porting complete the monitoring system. These elements each re-quire detailed deliberation and must interconnect logically with each other.

In conclusion, monitoring activities should be time-bound and de-fined in terms of both quantity and quality measures. Monitoring keeps the project on track according to plan and for the ultimate benefit of project recipients.

9

MONITORING AND EVALUATION 2014 COURSEMay 12 – 16, 2014 • Register at union-imdp.org

REGISTRATION DEADLINE: MARCH 28TH, 2014

PROGRAMME AGENDA

Monitoring and Evaluation (M&E) addresses the critical issues of performance assessment and pro-gramme impact. This five-day course prepares participants to measure and assess both programme efficiency and effectiveness.

BENEFITS OF TRAINING

1) Develop different levels of performance indicators

2) Strengthen the processes of supervision, audit and data collection

3) Demonstrate different means and methods of M&E for improved accountability

4) Analyse data to draw meaningful conclusions

5) Improve content and quality of reporting

6) Design elements of an M&E system

WHO SHOULD ATTEND

Middle- to senior- level managers working in public health, as well as managers who oversee personnel across multiple levels, divisions and locations.

DATE LOCATION FACULTY LEVEL FEE LANGUAGE

May 12 – 16, 2014

Kuala Lumpur, Malaysia

Rajesh Kapoor and Swati Devgon

Middle- to senior-level professionals

Euro 1,500 Course fee includes accommodation, break-fast, lunch, tea breaks, and course materials

English

“When we initiated our work two years ago we didn’t have any idea of the challenges we would face, and that we had to be managers,

administrators, leaders, and communicators. Attending The Union courses gave us the necessary tools to solve these situations.”

Dr. Natalia Celauro Organización Libre del Tabaco, Paraguay

Contact IMDP: imdp@theunion.org

Application deadline is 45 days prior to course. Upon acceptance into course, applicants will be invoiced. Fee payment is due within 30 days of invoicing.

10

How do you know if your programme has produced the desired results and impact? At the end (or some other designated

point such as when funding needs to be secured again) it is essen-tial to conduct an official evaluation of the programme addressing quality, cost-effectiveness and how significant the impact on the target population has been.

EVALUATION FOR FRONT-LINE PUBLIC HEALTH PROGRAMMESPROFESSOR IAN DARNTON-HILL AO, MBBS, PHD, MPH*

*MSC(Med), P/GDIPNUTRDIET, FACN, FAFPHM, Adjunct Professor at University of Sydney, Australia and Tufts University, USAPhoto: Courtesy of Dr. Ian Darton-Hill / The Union

11

A programme in Mexico that gave a monthly allowance of flour to poor mothers found that iron deficiency anemia was not being re-duced, even though the flour provided was iron-fortified. Why not? After a thorough evaluation was conducted to assure that recipients had complied with conditions of participation, it was discovered that the form of iron being used to fortify the flour was not being well absorbed. Based on this discovery, the flour was changed to a more bio-available formula and anemia levels began to improve.

The US Centers for Disease Control and Prevention (CDC) have suggested asking the following six questions when evaluating your programme:

• What specifically is being evaluated (the entire programme, parts of it, the local cultural context and so on)?

• What aspects of the programme (adoption of changed be-haviors, accessibility of supplements, etc.)?

• What are the standards to measure against (and are these donor requirements)? What will be the benchmark against which the programme can be considered successful (e.g., what proportion of the population would have to be reached with the intervention to make a significant difference)?

• What evidence will be collected, who will measure it and are both objective (often it is better to use an outside group that is less likely to be biased)?

12

• What conclusions can be drawn from the collected evidence?

• How will the results and any lessons learnt be used to im-prove the programme and any new one being planned to im-prove public health outcomes in general?

Despite the seemingly obvious benefits of evaluations, they are of-ten not done done well or the outcomes are not applied. This is often because adequate funding was not available in the original design of the programme. This is a misconception that “we know what to do” so let’s spend all the funds on implementation actions.

Evaluation requires training and hands-on experience. An effec-tive monitoring and evaluation (M&E) system incorporates relevant performance indicators, data collection, analysis and reporting. Re-porting is critical since it is your opportunity to properly demon-strate programme effectiveness to beneficiaries and funders alike.

Even the biggest organisations are increasingly emphasising mon-itoring and evaluation and how it can be used to gain the results people want. For small and middle-sized programmes, good eval-uations are even more critical because the margins are so much tighter and the link to target beneficiaries closer.

13Photos credit: Jens Jeske / The Union

COURSE DATE LOCATION FACULTY LEVEL FEE LANGUAGE

Monitoring and Evaluation

May 12 – 16, 2014 (registration deadline March 28)

Kuala Lumpur, Malaysia

Rajesh Kapoor and Swati Devgon

Middle- to senior-level professionals

Euro 1,500 English

Management and Leadership

May 19 – 23, 2014 (registration deadline April 5)

Bangalore, India

Viswanath Gopalakrishnan and Gayatri Sriram

Middle- to senior-level professionals

Euro 1,500 English

Project Management

June 16 – 20, 2014 (registration deadline May 2)

Entebbe, Uganda

Rajesh Kapoor and Swati Devgon

Junior- to middle-level professionals

Euro 2,000 English

Human Resources Management

June 23 – 27 2014 (registration deadline: May 9)

Cuernavaca, Morelos, Mexico

Elsie Zamora Middle- to Senior- level professionals

Euro 1,500 Spanish

Leading Management Teams

July 7 – 11, 2014 (registration deadline: May 23)

Kuala Lumpur, Malaysia

Rajesh Kapoor and Akhil Kapoor

Middle- to senior-level professionals

Euro 1,500 English

Strategic Planning and Innovation

July 14 – 18, 2014 (registration deadline: May 30)

Kuala Lumpur, Malaysia

Viswanath Golpalkrishnan and Gayatri Sriram

Middle- to senior-level professionals

Euro 1,500 English

APPLICATION DEADLINE IS 45 DAYS PRIOR TO COURSE / UPON ACCEPTANCE INTO COURSE, APPLICANTS WILL BE INVOICED / FEE PAYMENT IS DUE WITHIN 30 DAYS OF INVOICING

COURSE FEE INCLUDES ACCOMMODATION, BREAKFAST, LUNCH, TEA BREAKS AND COURSE MATERIALS.

UPCOMING 2014 IMDP COURSESREGISTER AT union-imdp.org

14

EXECUTIVE PERSPECTIVE: DR. ADAM KARPATI, MD, MPH SENIOR VICE PRESIDENT FOR PUBLIC HEALTH IMPACT, THE UNION NORTH AMERICA

Photo: Courtesy of Dr. Adam Karpati / The Union

15

At a big-picture level, monitoring and evaluation are nearly as important in public health work as the actual delivery

of treatment: without “M&E,” we wouldn’t know what we are ac-complishing, how well we are performing relative to our objectives, or whether our programmes are likely to impact public health as broadly as we might wish. In sum, we wouldn’t know much.

On the ground, it sometimes looks different: organisations and their people can struggle with the implementation of effective M&E, and may question whether it is “worth the effort.”

One person who would answer that question with an emphatic “Yes!” is Adam Karpati, MD, MPH – the newest addition to The Union North America’s leadership team. As Senior Vice President for Pub-lic Health Impact, Dr Karpati will help The Union assess the impact of its work through new epidemiological and evaluation strategies.

For the last 14 years, Dr Karpati has been highly steeped in moni-toring and evaluation: as an epidemiologist and senior executive in the New York City Department of Health and Mental Hygiene, he served during the tenure of one of the most data-driven and results-oriented mayors ever: Michael Bloomberg, whose philan-thropic activities have helped improve health outcomes globally. Metrics, data collection and evaluation were integral to the public health agency’s activities in this era, and many health outcomes improved measurably.

16

“The full cycle of public health work, in all aspects from planning a project through assessing it, needs to be informed by data,” Dr Karpati says. “Monitoring and evaluation measures your produc-tivity and effectiveness. Are you reaching the number of patients you need to? On a broader level, are you reaching your overall pro-grammatic goals?”

New York City is far more diverse than the concentration of gleam-ing towers shown in movies and travel posters: it contains many communities affected by high burdens of illness, disease and pre-mature mortality. The public health challenges include infectious diseases such as HIV; non-communicable diseases, such as asth-ma, maternal and child health; and behavioral health (mental ill-nesses and substance abuse).

“Though the range of programmes managed by the public health department was extremely broad, monitoring and evaluation was a common theme” he says. “Investing in strong M&E systems will pay off by improving the quality and efficiency of the services those programmes are delivering.”

Dr Karpati believes that beyond providing a critical understanding of programme performance, careful and thorough monitoring and data collection yield many important benefits. Among them:

17

• Assessing the broader impact of the programme on public health. For example, how does the number of individuals the programme is serving compare to the need in the community at large? What is the community rate of the condition you are targeting? How does that rate break down across subpopula-tions? Is the programme targeted accordingly? “Detailed data are necessary to determine resources needed, and provide a baseline from which to measure results,” he says.

• Understanding not only what has resulted from your effort, but why: “It is not always enough to know how many people were cured or reached. You need an analytic approach, with data that is specific, if you want to learn – as you should – why some were cured and others not; why you reached some people and not others.”

Dr Karpati highlights a key aspect of M&E: collecting detailed enough information to allow analysis broken down (stratified) by subgroups. “Averages,” he notes, “can be misleading.”

He offered a theoretical example of a province-wide child vaccina-tion programme that was in aggregate 85% successful in reach-ing its target population. Without stratifying its data further, the health programme would not have known if any narrow geograph-

18

ic or demographic areas had been less successfully reached. That programme might have been 95% successful for half of its popula-tion and 75% for the other half.

Dr Karpati recognises that, to the individual health worker, data collection can sometimes seem onerous. He suggests at least two ways to minimise this effect:

• Dedicate time and energy to involving the entire staff in the M&E process. This will make them more understanding of, and invested in, the effort’s desired outcomes.

• Integrate monitoring and data collection into the actual work flow of the intervention.

“When this happens,” Dr Karpati says, “monitoring and evaluation isn’t just filling out forms; it’s engaging the staff and helping them do their jobs better. That in itself is a positive outcome.”

19

“Dedicate time and energy to involving

the entire staff in the M&E process.

This will make them more understanding of,

and invested in, the effort’s desired outcomes.”

Dr. Adam Karpati, MD, MPH Senior Vice President for Public Health Impact, The Union North America

The Union, North America 61 Broadway, Suite 1720 · 10006 New York, NY, USA Tel: (+1) 212 500 5736 · Fax: (+1) 347 772 3033 email: imdp@theunion.org · www.union-imdp.org

Health solutions for the poor union-imdp.org

ISSUE 2, 2014

THE HEALTH MANAGERTHE INTERNATIONAL MANAGEMENT DEVELOPMENT PROGRAMME® (IMDP)

ISSUE 2, 2014