guidelines for writers (soph post …€¦ · web viewtitle guidelines for writers (soph...

The Context of Programme Evaluation

Introduction

Welcome to the first unit of Monitoring and Evaluation of Health and Development Programmes.

The module is based on the understanding that you have a fair amount of experience in some aspects of projects or programmes management and that we are building on this foundation.

Neither monitoring nor evaluation can be effectively undertaken unless the programme being evaluated has a sound conceptual framework. Accordingly, one of the key messages we put across is that monitoring and evaluation processes must be conceptualised early in the Programme Management Cycle. We therefore devote a session to the development of a conceptual framework.

At the end of the unit we invite you to add monitoring and evaluation objectives to the evaluation plan you will develop for your first assignment.

We will use material from a programme for primary prevention of Non-Communicable diseases implemented in Cape Town by staff of the School of Public Health in 2002, as an example.

There are five Study Sessions in this unit. Study Session 1: An Introduction to Monitoring.Study Session 2: An Introduction to Evaluation.Study Session 3: Approaches to Evaluation.Study Session 4: Locating Monitoring and Evaluation in the Programme Management Cycle.Study Session 5: Developing a Conceptual Framework for a Programme.Study Session 6: Programme Goals and Objectives.Study Session 7: Monitoring and evaluation objectives.

In Sessions 1 and 2, we define the concepts of monitoring and evaluation and debate their different purposes. Session 3 introduces you to the approaches to evaluation. In Session 4, we will take a look at the components of the SOPH, UWC, Master of Public Health: Monitoring and Evaluation of Health and Development Programmes - Unit 1

1

UNIT 1

Programme Management Cycle and the position of monitoring and evaluation within it.Session 5 focuses on identifying the four principal elements of a programme, and on the development of a conceptual framework for a programme.In Session 6, we explore the concepts of programme goals and objectives and discuss their significance in the monitoring and evaluation processes. Session 7 guides you through developing monitoring and evaluation goals.

Here is an overview of the intended outcomes of Unit 1.

Learning outcomes of Unit 1

By the end of this module, you are expected to be able to: Understand key concepts relevant to monitoring and evaluation. Make a reasoned case as to why projects should be monitored and

evaluated. Explain the importance of being critically reflective of your own evaluation

paradigm. Explain the difference between interpretivisim and positivism. Discuss the strength and weaknesses of your own evaluation approach. Locate monitoring and evaluation in the Programme Management Cycle. Develop understanding of the components of a project using a conceptual

framework. Develop programme goals and objectives. Develop monitoring and evaluation objectives. Design and motivate for a monitoring system as part of an evaluation

plan.

Please be aware that there is a glossary of concepts that are commonly used in monitoring and evaluation, in section 5 of the Module Introduction. Refer to these explanations when you come across an unfamiliar term.

Preparing for your assignmentBy the end of this unit, you should be in a position to begin your first assignment and prepare goals and objectives for a programme as well as a conceptual framework. This will provide the basis for a programme evaluation. Try to develop your assignment as you proceed through the module.

Before you start on Session 1, here is an opportunity to reflect on your existing knowledge of monitoring and evaluation.

Review your knowledge of monitoring and evaluationIt is always good academic practice to review what you already know about the topic and to identify where you need to deepen your understanding. Here is a short self-evaluation task, which will enable you to:

SOPH, UWC, Master of Public Health: Monitoring and Evaluation of Health and Development Programmes - Unit 1

2

Reflect upon your previous experience (if any) of monitoring and evaluation;

Review your current knowledge and skills relevant to this topic; Identify your learning and development needs.

Answer this questionnaire while previewing the Module Guide and the Reader.

a) My previous experience of evaluation consists of:

b) I have knowledge about the following aspects of monitoring and evaluation:

c) I have skills in the following aspects of evaluation:

d) What I would like to learn (or further develop) on this topic:

You might like to record your expectations for learning under the topics used above – knowledge, skills and experience.

When you have finished this task, briefly scan through this Module Guide and read the table of contents and reflect on these questions:

e) Does it appear that your expectations will be met?


3

f) Does the material appear relevant to your needs?

g) How can you use this module to meet your learning and development goals?

h) Make notes about any aspects which are missing and send these to your lecturer.

i) What else can you do about these missing aspects?

Hopefully this task has helped you to get to grips with what lies ahead, what you already know, and what is expected of you by the end of the module.

Resources available to you

There are many areas throughout this module where additional references, readings or websites are given. These are avenues where you can follow up material that is of particular interest to you.

Your lecturer can also help you find additional materials in monitoring and evaluation. Make contact early in your studies so that she may get to know you and your learning needs. A starting point will be to fax your lecturer a copy of your audit task with your name on it. Take a look at it again when you have completed the module to see whether you feel that gaps still exist. Then alert us to them when you evaluate the module.


4

Unit 1 - Session 1 An Introduction to Monitoring

Introduction

The fact that you have chosen this module suggests that you already recognise the importance of monitoring and evaluation within the life of a programme or project.We devote this session to clarifying key concepts. We will also raise some of the debates related to the processes of monitoring and evaluation. We emphasise the importance of monitoring and evaluation within the life of a programme, an issue that you, as Programme Manager, may have to take up with your colleagues.

Contents

1 Learning outcomes of this session2 Readings3 What is monitoring?4 Why we monitor programmes5 Identifying monitoring activities6 Using District Health Information Systems for monitoring7 Planning monitoring activities8 Session summary9 References

Timing of this session

In this session there are three readings and five tasks. It should take you about two and a half hours to complete.


5

1 LEARNING OUTCOMES OF THIS SESSION

By the end of this session you should be able to:

Explain the concept of monitoring. Discuss the role of monitoring. Plan monitoring activities.

2 READINGS

There are three readings for this session. You will be referred to them in the course of the session.

Author/s Publication detailsMwadime, R. et al. (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related

Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy, & Tufts University: 1.15-1.16.

Feuerstein, M-T. (1986). Ch 6 - Using Your Evaluation Results. In Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London: Macmillan: 160-165.

Rossi, P. H. & Freeman, H. E.

(1993). Ch 4 - Program Monitoring for Evaluation and Management. In Evaluation: A Systematic Approach. Newbury Park, Ca: Sage Publications: 163-213.

3 WHAT IS MONITORING?

Start this session by considering these examples of monitoring.

When we weigh children between 0-6 years at six-week intervals, we are monitoring children’s growth.

When you check the patient’s temperature twice a day, you are monitoring the patient’s condition.

When you check your bank balance every month, you are monitoring your finances.

Now go through Reading 1; use it to help you to define monitoring in Task 1


6

READING

Mwadime, R. et al. (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy, & Tufts University: 1.15-1.16.

TASK 1 – DEVELOP YOUR OWN DEFINITION OF MONITORING

What is monitoring? Write your own definition.

FEEDBACK

Now compare your notes with the following definition of monitoring:

Monitoring is a process of continuous and periodic surveillance of the physical implementation of a programme, through timely gathering of systematic information on work schedules, inputs, delivery, targeted outputs, and other variables for the programme, in order to have the desired effects and impact. Monitoring is an integral part of a management support function and of the Management Information System. Monitoring is thus a management tool. Monitoring reports can be used as a basis for internal review (evaluation) of programme operations at the management and technical levels.

Sometimes monitoring is referred to as process evaluation because it concentrates on what is done in the process of running a service or programme, i.e. “within”. This includes the programme activities, the personnel who perform activities and other matters of implementation.

4 WHY WE MONITOR PROGRAMMES

What is the role of monitoring within the life of a programme? Why do we monitor programmes?

In general, the purposes of monitoring are: To oversee the physical implementation of the programme by providing

information on its current status, thus ascertaining that implementation is proceeding as planned;


7

To provide the basis for corrective measures to be taken through identifying early indications of deviations, performance gaps and other problems requiring immediate attention for the programme to succeed;

To verify proper utilisation of programme resources, including that inputs are made available on time and that inputs are transformed through activities into outputs;

To verify that activities are undertaken and transform into outputs.

Note that monitoring activities are carried out around inputs, outputs and processes, that is, activities which enable inputs to be transformed into outputs. Monitoring data should be collected from the inception of the programme, and it should be utilised on an ongoing basis for programme improvement. If there is no monitoring system in place at the inception of the programme, it is not too late to start at a later stage in the life of the programme.

TASK 2 - IDENTIFY REASONS FOR MONITORING A PROGRAMME

Think about a Primary Health Care programme with which you are familiar. List the reasons for conducting monitoring activities in this programme. If there are no monitoring activities, are there any reasons why there should be?

Now compare the reasons you have listed with those in the feedback.

FEEDBACK

Monitoring of programmes is done in order to: Assess the quantity, quality and timing of programme inputs; Verify that inputs are transformed, through activities, into outputs that

generate results; Provide information to improve targeting; Identify operational constraints to programme effectiveness thus helping

managers to improve implementation; Determine whether a programme is servicing the target groups.


8

5 IDENTIFYING MONITORING ACTIVITIES

The programme narrative in the box below is based on a programme for primary prevention of non-communicable diseases, which was implemented in Cape Town by staff of the SOPH in 2002.

After reading the narrative, determine which activities of this programme lend themselves to monitoring.

AN INTERVENTION FOR PRIMARY PREVENTION OF NON-COMMUNICABLE DISEASES: EXPERIENCES FROM KHAYELITSHA, A BLACK TOWNSHIP IN CAPE TOWN, SOUTH AFRICA (2002)

Objectives: To increase the knowledge of Community Health Workers (CHWs) who participate in the programme about 5 risk factors for non-communicable diseases (NCDs) by the end of 2006.To reduce the prevalence of obesity among CHWs participating in the programme by 10% by the end of December 2006.To increase the number of CHWs attending all training sessions on primary prevention of NCDS by the end of December 2003.

Methods: Twenty-three CHWs employed by Zanempilo, an NGO, in Site C, Khayelitsha, participated in the programme. Firstly, focus group discussions were conducted with seventeen purposively selected CHWs to explore attitudes, beliefs and perceptions regarding hypertension and diabetes. Secondly, interviews assessed basic knowledge about causes, prevention and control of hypertension and diabetes. Thirdly, anthropometric measurements (weight, height, waist circumference) were taken and Body Mass Index (BMI) was kg/m2 was calculated and used as an indicator of obesity. Fourthly, level of physical activity and associated barriers were assessed. Lastly data was collected on eating patterns (type and frequency of consumption of certain foods). Focus group discussions revealed that CHWs were uncertain about the causes of hypertension and diabetes and found it difficult to grasp the fact that even people without risk factors of these diseases, such as overweight and family history, could be diagnosed as hypertensive/diabetic. Many CHWs believed in traditional medicines and home brewed beer as best treatments for hypertension, and that bitter substances could be used to dilute blood sugar levels in diabetic patients. They also believed that people who took medical treatment became sicker, and their health deteriorated rapidly. A large percentage of CHWs were overweight and some were extremely overweight. Barriers to physical activity included fear of losing weight and lack of environmental safety. Eating patterns revealed a lack of nutrition related knowledge.A training program for the primary prevention of non-communicable diseases was developed and implemented: The objective of the training was to build the capacity of the Community Health Workers (CHWs) in making healthy choices about the food they eat and in increasing physical activity. The training programme emphasised the role of nutrition, physical inactivity, body weight, smoking and alcohol in causation of hypertension and diabetes.

Implementation: Training sessions were held once a week for a period of six months. Pre- and post-tests were given to assess the effectiveness of the training. Lectures were given at a central location in the community. Facilitators of training workshops came from different tertiary education institutions and the training was participatory. Learning was reinforced by use of visual aids and real animal organs where applicable. Case studies, specifically developed for this population group, were used to reinforce learning and to encourage critical thinking. For each topic, its role in causation of NCDs was explained, preventive measures were highlighted and a key message was provided which could be used to inform the community.

Pens and writing pads were given to the participants during the training. Refreshments including tea, biscuits, and a light lunch were also served.

At the completion of the training programme, one year later, the CHWs completed evaluation forms to evaluate the training programme. Focus group discussions were also held to determine changes in


9

perceptions about risk factors for non-communicable diseases. Anthropometric, or body measurements, were also taken to evaluate the impact of the training on lifestyle modification.

TASK 3 - IDENTIFY POTENTIAL MONITORING ACTIVITIES

For the case study above, suggest what monitoring activities could have been carried out to ensure the success of the programme.

FEEDBACK

The following issues could have been monitored in this programmme:

The adherence to the training programme by the facilitators; The quality and content of lectures; Whether lectures were given as planned in terms of timing; The type and the quantity of stationery supplied and used; The type, amount, quality and quantity of refreshments given during

training; The number of CHWs who attended the training on a weekly basis; Whether CHWs understood the lectures given.

6 USING DISTRICT HEALTH INFORMATION SYSTEMS FOR MONITORING(MANAGEMENT INFORMATION SYSTEM)

A point that has been made in a number of other modules, such as Health Management I and II, Measuring Health and Disease I as well as in many of the modules from the Health Information Stream, is the importance of relevant record-keeping and information systems. Read the short extract from Reading 2a by Feuerstein for a brief overview of this issue and its link to monitoring. It is important to bear this issue in mind when conceptualising a monitoring system.

READING

Feuerstein, M-T. (1986). Ch 6 - Using Your Evaluation Results. In Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London: Macmillan: 160-165.

A District Health Information System (DHIS) is a tool, a strategy and a method which managers have at their disposal for collecting, storing, organising, and reviewing information resulting from monitoring activities, using it as feedback for the purpose of decision-making. As such, a DHIS is part of a monitoring system.


10

A DHIS can exist at many sites and levels within a programme. For example, a given programme can collect, manage, and act on information at the community, district or national levels. This information can also be analysed, reviewed and used for decision- making purposes at higher levels. An effective DHIS will also have mechanisms in place at all levels for feedback to be given and received on such decisions.

A DHIS can help improve the quality (adequacy and accuracy) of data collected. It is one way of developing a reporting system that facilitates information flows and reporting levels, by increasing ease of access to information.

Note that DHIS data can be used at different levels, by different decision-makers, for different types of decisions and that any given programme may have several Management Information Systems in place to monitor different programme activities.

READING

Rossi, P.H. & Freeman, H.E. (1993). Ch 4 - Program Monitoring for Evaluation and Management. In Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications: 163-213.

TASK 4 – REVIEW IMPORTANCE OF DISTRICT HEALTH INFORMATION SYSTEMS IN MANAGING PROGRAMMES

Think of an example of a DHIS or monitoring data system from your own programme. How are the data used and what is the importance of this system in monitoring?

FEEDBACK

When designing a monitoring system, it is important to determine:

What information will be most useful for programme management? What information will be most useful for programme implementers (at

different levels)? What information will be readily available through programme

implementation? What additional information needs to be collected as part of programme

monitoring? Which indicators can programme staff realistically monitor? How will the information be stored and retrieved? What will be the most appropriate and effective feedback channels?


11

7 PLANNING MONITORING ACTIVITIES

In this section you will consider the monitoring activities in a health promotion programme or any type of a programme.

TASK 5 – PLANNING MONITORING ACTIVITIES

Note down all the monitoring tasks that take place or are necessary in your programme; then, brainstorm the detailed steps towards accomplishing them.Don’t forget to consider data collection systems for monitoring.

Once you have completed your brainstorm, compare your answer with the information given below.

FEEDBACK

Steps in conducting monitoring activities Review existing information related to the project. Familiarise yourself with the conceptual framework of the project or

develop a conceptual framework for the project. Identify monitoring objectives. Identify indicators. Determine which categories of workers, supervisors or other will be

responsible for the collection of each category of monitoring data. Develop a timetable for frequency of monitoring. Develop/strengthen a management information system. Develop monitoring instruments. Train staff in monitoring activities. Conduct monitoring activities. Identify feedback channels. Analyse monitoring data. Write a report. Make recommendations. Implement recommendations. Identify new indicators based on the recommendations. Modify the monitoring system if necessary. Continue monitoring.

(Mwadime et al, 1999: 1.22)

To be able to carry out monitoring activities successfully, these steps need to be taken into consideration. For example, you need to review existing information about the programme so as to know what monitoring activities were conducted previously, and to familiarise yourself with the conceptual framework of the programme - this will guide you to the components of the SOPH, UWC, Master of Public Health: Monitoring and Evaluation of Health and Development Programmes - Unit 1

12

programme which need to be monitored. Some of the information may be readily available, for example, indicators to be monitored, as these may have been identified during programme planning.

8 SESSION SUMMARY

You will have noticed that we seem to have run ahead of ourselves by introducing monitoring objectives and indicators, which will also be dealt with in later sessions. This illustrates the central role objectives and indicators have in all aspects of monitoring.

Before we go any further with monitoring, however, work through Session 2, which introduces evaluation and its purposes. As you study, consider the differences between monitoring and evaluation.

9 REFERENCES

Mwadime, R. et al. (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy & Tufts University.


13


14

Unit 1 - Session 2 An Introduction to Evaluation

Introduction

There is much debate about the meaning and purpose of evaluation. This is due to the fact that there are different ways of conceptualising evaluation, and each way influences the choice of methodology and approach when evaluating. So regard this session as the introduction to a debate which is ongoing!

Generally speaking, the term evaluation means to make an assessment or judgment about the value of something. In this module, evaluation is presented in relation to making assessments or judgments about the value of programmes, projects or policies. Part of health programme evaluation involves making judgements about how successfully a programme or project has, for instance, translated the principles of Primary Health Care into practice - for example, the programme’s effectiveness in promoting community participation, its ability to reach minority or disadvantaged groups in the community, or its ability to work towards environmental sustainability. Evaluation in this context may involve making judgements about various aspects of a programme including effectiveness, quality or efficiency, or even value for money. Bear this in mind as you select the aspect of the programme that you plan to evaluate in your assignment.

Contents

1 Learning outcomes of this session2 Readings3 What is evaluation?4 Why, and what, we evaluate 5 Linking monitoring and evaluation6 Barriers to programme evaluation7 Session summary8 References

Timing of this session

In this session there are four readings and four tasks.


15



Explain the concept of evaluation. Discuss the purposes of evaluation. Demonstrate understanding of the relationship between monitoring and

evaluation.

2 READINGS

There are four readings for this session. You will be referred to them in the course of the session.

Author/s Publication detailsWeiss, C. H. (1998). Ch 1- Setting the Scene. In Evaluation. New Jersey: Prentice Hall:

1-19.

Feuerstein, M-T. (1986). Ch 1 - Understanding Evaluation. In Partners in Evaluation: Evaluating Development and Community Programmes with Participants. London: Macmillan: 1-7.

Oshaug, A. (undated). Evaluation of Nutrition Education Programmes: Implications for Programme Planners. Oslo: Nordic School of Nutrition, University of Oslo, Norway. [Online]. Available: www.fao.org/docrep/W3733E/w3733e06.html Downloaded: 13/12/02: 1-26.

Eusebio J. S., Dacanay, R. C., Ramos, M. A. C. P. & Lantican, L. L.

(1991). Field Guide on Comprehensive Planning, Monitoring and Evaluation of Nutrition-oriented Rural Development Programs at Local Levels. Barangay Integrated Development Approach for Nutrition Improvement (BIDANI) of Rural Poor. Institute of Human Nutrition and Food College of Human Ecology: 39-46.


16

http://www.fao.org/docrep/W3733E/w3733e06.htm

3 WHAT IS EVALUATION?

When you compare the quality and quantity of a meal with the price you paid for it, you are determining the value of the restaurant – or evaluating it. When you make a judgement about the reception you received at the hospital or a bank, you are evaluating the service offered by these institutions.

Michael Patton defines evaluation quite broadly:

[Evaluation] includes any effort to increase human effectiveness through systematic data-base inquiry. Human beings are engaged in all kinds of efforts to make the world a better place. These efforts include assessing needs, formulating policies, passing laws, delivering programs, managing people and resources, providing therapy, developing communities, changing organizational culture, educating students, intervening in conflicts, and solving problems. In these and other efforts to make the world a better place, the question of whether the people involved are accomplishing what they want to accomplish arises. When one examines and judges accomplishments and effectiveness, one is engaged in evaluation. (1990a: 11)

This definition emphasises the goal-directed nature of evaluation - that is, improving programmes or services through a systematic process of examining and judging. Patton suggests that the main purpose of evaluation is to inform action, enhance decision-making and apply knowledge to solving practical problems. This focus helps to differentiate evaluation from basic research. As Patton explains “… evaluation is applied research, or a type of action science”. (1990a) Another point that Patton makes is that such judgements are made on the basis of data or evidence.

Some definitions place an even stronger emphasis on the judging aspects of evaluation, like this one by Wright, Haycox & Leedham (1994):

Evaluation is about establishing the worth of a policy, service, activity or whatever is under consideration. It is, therefore, an examination of objectives of service provision, how well these objectives are achieved and at what cost.

Other definitions tend to view evaluation as primarily a learning process. Marie-Therese Feuerstein (1986), in her book Partners in Evaluation: Evaluating Development and Community Programmes with Participants, describes evaluation as looking for and learning from programme successes and failures in order to decide how and where changes are needed and to help plan for the future. You will examine this reading in the next section, for now take a look at the reading by Weiss (1998).


17

READING

Weiss, C. H. (1998). Ch 1 - Setting the Scene. In Evaluation. New Jersey: Prentice Hall: 1-19.

In this chapter, Weiss proposes another definition of evaluation, and briefly outlines the history of evaluation, discussing how evaluation differs from research. Weiss reminds us that the term evaluation encompasses many different kinds of judgement (for example, of effectiveness, efficiency and quality) to be applied to many phenomena (for example, programmes, policies and projects).

As with Patton, Weiss, in defining evaluation, compares evaluation with research. Weiss proposes that the main difference between research and evaluation is the intended use or purpose. While evaluation often shares the same methods or tools as research, its purpose, or intention, is different. Evaluation is action-oriented and has direct relevance to a programme, project or policy. It seeks to answer a range of questions including:

Is the programme addressing a real, or the right problem? Is the intervention correct or appropriate? Are additional interventions necessary to achieve the objectives? Is the intervention being implemented as planned? Is it an effective way of addressing the problem, for example in terms of

cost and inputs?

In addition, evaluation can serve as an accountability tool that enables programme management to show the achievements of the programme to the stakeholders as objectively as possible.

Like monitoring, it is therefore a programme management tool, a verification process for measuring achievement and programme results and for assessing its relevance, efficiency and effectiveness in relation to its objectives within a given budget or available resources. Use this as a working definition of evaluation, but be aware that you should be prepared to expand it in the course of your studies.

4 WHY, AND WHAT, WE EVALUATE

As you may be aware, evaluation is often challenging, can be complex and always consumes time and resources. It may deplete the resources which are required to implement programmes or deliver services. So the question, “Why evaluate?,” is a valid and important question.


18

One way to justify evaluation is in terms of measuring programme effectiveness. Evaluation processes may be used to demonstrate to planners, donors and other decision-makers that the programme activities have achieved measurable improvements. Monitoring and evaluation can also indicate whether and where resources are being used efficiently and where strategies for resource allocation may need to be considered, or reconsidered.

READINGS

Feuerstein, M-T. (1986). Ch 1 - Understanding Evaluation. In Partners in Evaluation. Evaluating Development and Community Programmes with Participants. London: Macmillan: 1-7.

Oshaug, A. (undated). Evaluation of Nutrition Education Programmes: Implications for Programme Planners. Oslo: Nordic School of Nutrition, University of Oslo, Norway: 1-26.

TASK 1 - IDENTIFY THE PURPOSES OF EVALUATING PROGRAMMES

a) Feuerstein’s consideration of why people evaluate in the reading (pages 2-3) generated ten key reasons or rationales for evaluation. Compare these with the reasons listed by Oshaug.

b) Think of an aspect of a programme that you would like to evaluate (as part of your assignment). What uses could this evaluation have in the life of this programme?

For feedback, revisit the two readings, and amend your notes as necessary. Do the uses you listed for the evaluation you intend to conduct accord with the reasons given in the two readings?

In the next task, you will return to the case study introduced in Unit 1 Session 1, focussing on Primary Prevention of non-communicable Diseases.

TASK 2 - IDENTIFY EVALUATION ACTIVITIES

Read through the programme description of Primary Prevention of non--Communicable Diseases in Unit 1 Session 1, section 5.

a) Identify the components of the programme that would, or could, be evaluated.

FEEDBACK


19

Compare your answers with those below. If you had problems identifying what would be evaluated, note down your questions so that you can discuss them with the course convener.

Aspects of the programme that were evaluated: Knowledge, beliefs and attitudes of CHWs about risk factors for Non-

Communicable Diseases before and after the intervention, i.e. training programme, to assess changes.

Anthropometric measurements (BMI, waist circumference): these were measured before and after the intervention to measure the impact of the programme in modifying behaviours towards adopting a healthy lifestyle, which in turn may result in weight loss.

Level of physical activity: this was also measured pre- and post- intervention to determine if there was a change in the level of physical activity (i.e. the outcome of the programme).

Eating patterns: data was collected before and after the intervention to assess the ability to make healthy choices about food, (i.e. this focuses on the intended outcome of the intervention).

5 LINKING MONITORING AND EVALUATION

Having explored various definitions of monitoring and evaluation, take a few minutes to answer the following question for yourself before studying the reading by Eusebio et al (1991), pages 39-42.

TASK 3 – IDENTIFY LINKS BETWEEN MONITORING AND EVALUATION

a) Where does monitoring end and where does evaluation begin in a programme? Is there any overlap?

FEEDBACK

READING

Eusebio J. S., Dacanay, R. C., Ramos, M. A. C. P. & Lantican, L.L. (1991). Field Guide on Comprehensive Planning Monitoring and Evaluation of Nutrition-oriented Rural Development Programs at Local Levels. Barangay Integrated Development Approach for Nutrition Improvement (BIDANI) of Rural Poor. Institute of Human Nutrition and Food College of Human Ecology: 39-46.

Note that Figure 5 in Eusebio et al illustrates an overlap in the monitoring (process) and evaluation (result) of the programme.

While monitoring activities focus on programme inputs, project activities or processes, evaluation focuses on outcomes and impact components of the SOPH, UWC, Master of Public Health: Monitoring and Evaluation of Health and Development Programmes - Unit 1

20

programme. Monitoring and evaluation overlap in the area of output. This means that outputs can be monitored on an ongoing basis and evaluated at the end of the programme.

For example, if one of the programme objectives is: To empower the 23 CHWs about 5 risk factors for non-communicable diseases by the end of the year, a programme manager may check the number of CHWs trained on a monthly basis (monitoring) and evaluate at the end of the training programme to see whether the intended number of trained CHWs has been achieved.

6 BARRIERS TO PROGRAMME EVALUATION

In this section we consider some of the barriers to programme evaluation. In the reading that follows, Feuerstein describes some of the difficult issues that evaluation sometimes raises, and proposes that evaluation involves looking for, and learning from, a programme’s successes and failures.

READING

Feuerstein, M-T. (1986). Ch 1 - Understanding Evaluation. In Partners in Evaluation. Evaluating Development and Community Programmes with Participants. London: Macmillan: 1-7.

TASK 4 – IDENTIFY BARRIERS TO EVALUATION

From the reading and from your own experience,

a) What are some of the barriers that prevent programme managers from evaluating programmes?

b) To what extent do you believe that Primary Health Care or Health Promotion workers are open to evaluating their failures – programmes or interventions which have failed, or to identifying and acknowledging weaknesses in existing or ongoing programmes? Cite some examples.

FEEDBACK Compare your examples and reasons with the following response.

Programme managers often resist evaluating their programmes for the following reasons: Lack of time, knowledge / skills and fear of change. Poor project design, e.g. if monitoring and evaluation was not built into the

project plans, it would mean additional workload, and a restrictive budget. Stakeholders may expect positive results and question the value of

monitoring and/or evaluation if it yields negative findings.


21

Stakeholders may be concerned that monitoring/evaluation results might be negative.

Monitoring and evaluation is sometimes seen as “police work” or a fault-finding exercise.

Stakeholders may argue that monitoring and evaluation resources, not just money, but staff time, would be better spent on expanding the programme.

7 SESSION SUMMARY

In this session, we have discussed the nature and purpose of evaluation, and the distinction between monitoring and evaluation.

As we have said, there is a vast range of viewpoints on what evaluation is, and the role it should play in a programme. In the next session, we will examine different approaches to evaluation in some detail, and try to show why the question of approach matters.

8 REFERENCES

Patton, M. Q. (1990a). Qualitative Evaluation and Research Methods. Newbury Park, Ca: Sage Publications.

Wright, K., Haycox, A. & Leedham, I. (1994). Evaluating Community Care. Buckingham: Open University Press.

Feuerstein, M-T. (1986). Partners in Evaluation: Evaluating Development and Community Programmes With Participants. Oxford: Macmillan.


22

Unit 1- Session 3Approaches to Evaluation

Introduction

When we speak of an approach to programme evaluation, we are acknowledging the range of conceptualisations and methods for the evaluation of programmes, and that programmes can be evaluated at different points in their lifespan – for example, at baseline (before programme implementation) or after an impact has been made. Furthermore, when evaluators undertake programme evaluation, they do so from different conceptual positions or worldviews. In this session, we will broaden the use of the term approach to encompass paradigms employed in research and therefore in evaluation. We will examine three different paradigms, from which we will demonstrate that, in carrying out evaluation activities, evaluators start with particular assumptions about the nature of the world, the nature of knowledge and how knowledge is produced through research.

We also raise the issues of choosing a research methodology. This can vary from a quantitative to a qualitative approach, or comprise a combination of the two, depending on what is appropriate to the evaluation purpose and context.

Contents

1 Learning outcomes of this session2 Readings3 Why do the evaluator’s assumptions matter?4 Critical views of Positivism5 Session summary 6 References and further readings

Timing

This session contains two readings and three tasks. It should take you about two and a half hours to complete.


23


By the end of this session, you should be able to:

Explain the importance of being critically reflective of the paradigm underpinning your own evaluation.

Discuss the strengths and weaknesses of your own evaluation approach. Explain the difference between Interpretivism and Positivism.

2 READINGS

The readings for this session are listed below.

Author/s Publication detailsGephart, R. (Summer 1999). Paradigms and Research Methods. In RMD Forum,

4. [Online] Available: //www.aom.pace.edu/rmd/ Downloaded 2003/09/09. 1-10

Marsden, D. & Oakley, P.

(1991). Future Issues and Perspectives in the Evaluation of Social Development. Community Development Journal, 26(4). 315-328

3 WHY DO THE EVALUATOR’S ASSUMPTIONS MATTER?

In this section we discuss how you need to be aware that your assumptions about how the world works will probably influence your programme design. Herrington proposes that we, as researchers (or evaluators), should engage in critical reflection of ourselves as researchers. She advocates:

“… examining [one’s own] research practices and assumptions in relation to alternatives and trying to understand the consequences of each for the findings that result from a study. … While I believe that we should engage in such critical reflection for our own personal growth, we also have a


24

professional obligation to do so. … For many of us, examining these assumptions means starting to ask new questions from the ones we have been accustomed to ask as researchers and readers of research. We have been used to asking questions like the following: What do I want to focus on … what guiding question? What and who will be my sources of information and the methods of

obtaining it? How will I analyze and interpret the information? …

[In order to understand the assumptions behind our actions, she suggests alternative questions which are:]

Whose views are included in a research study and by whom are they represented? More generally, what are participants’ roles throughout a study?

In what ways do researchers recognize their own roles in shaping research findings?

In what ways do researchers recognize the functions of ideology and, more generally, values in research studies?” (Herrington, 1993: 40-41)

TASK 1 – CONSIDER YOUR POSITION AS AN EVALUATOR

a) What is the difference between Herrington’s first and second set of questions?

b) To what extent do you see this sort of critical reflection as important for you as an evaluator? Why do you say so?

FEEDBACK

Herrington’s first set of questions focuses on the programme itself and the way you presently understand it. Her second set of questions asks you to reflect on the way you understand the programme before you decide what to evaluate. She suggests that you take a step back from the programme and analyse the power arrangements within the programme, your own power as a researcher, and the influence of the researcher’s position in even choosing the focus of the evaluation.

Herrington is in fact asking us to consider our broad research paradigm, or the position from which we view the world.

Here is a scenario to help you understand the point that is being made:You are an evaluator. You believe that the impact of your programme intervention can be fully known or understood. You assume that this knowledge exists separate from you, the evaluator, in other words that you have no influence over it. Your assumption is that all that is required in order


25

to understand the impact objectively is to collect data, analyse and interpret it, and you will gain a better understanding of the situation.

This is what is called a Positivist position on research and evaluation.

However, if you accept that knowing something is more than simply searching out and managing the objective facts of a situation, you are actually accepting a different worldview or paradigm from the Positivist paradigm. You are making the assumption that things are only knowable in context, and that different players within a context may interpret things differently. You are therefore recognising your own role in “shaping the findings” (Herrington, 1993: 41), and that your own values and ideology may be shaping the research.

In relation to research and evaluation, the nature of knowledge itself has been hotly debated. Instead of phenomena being regarded as objectively knowable, some sociologists and philosophers have argued that knowledge is related to the social conditions under which it is created, and that it is subjective and political in nature. The study of the “origins and nature of knowing and the construction of knowledge” is called epistemology. Thinking about epistemology requires us to ask the questions: “What is the relationship between the knower and the known?” and “What role do values play in understanding?” (Maykut & Morehouse, 1994: 4)

Broadly, it could be said that there are three positions or worldviews available to the researcher or evaluator. The first is a Positivist position and this remains the dominant approach to research, both in its usage within the field of health, and its power (or credibility) within the scientific community.

The Positivist Approach makes the following assumptions: We can understand reality by studying its parts. It is possible to be truly objective and stand outside of the object of your

study. You can suspend your own values as a researcher and evaluator. Explanations from one time and place can be generalised to other times

and places. One’s aim is to seek to verify a proposition.

(Adapted from Maykut & Morehouse, 1994: 12)

In this module, we have taken a primarily Positivist Approach; however, as you will see in some of the readings, and in the adaptations that we suggest in the approach, a purely Positivist Approach is not compatible with Primary Health Care or evaluation of Health Promotion.

A very helpful summary of the key paradigms is to be found in the reading by Robert Gephart. Take a look at Table 1 on page 2 of the reading in which the author proposes a typology of three paradigms, Positivism, Interpretivism and Critical Theory/Postmodernism.


26

He argues that these three paradigms are presently “…shaping social, organizational, and management research.” (Gephart, 1999: 7)

READING

Gephart, R. (Summer 1999). Paradigms and Research Methods. In RMD Forum, 4: 1-10. [Online] Available: //www.aom.pace.edu/rmd/ Downloaded 2003/09/09. Look carefully at the underlying assumptions of the three paradigms, their main ideas, their goals, units of analysis and research methods.

TASK 2 – DISTINGUISHING THE PARADIGMS

a) Try to locate the evaluation study you propose doing within the descriptions of the three paradigms. Does it fall within one of them exclusively, or is it a mixture? Are there any risks to the validity of your research findings because of your chosen approach?

b) In relation to your evaluation, is it important to be conscious of the research paradigm within which you are working?

c) Which of these three paradigms invites more participant control, and which is likely to be more researcher-controlled? Why do you say so?

FEEDBACK

a) The first question is for you to answer in relation to your programme evaluation.

b) The importance of being conscious of the research paradigm within which one is working is in part because one’s assumptions influence every aspect of the evaluation process. The choices of approach, of design and of data collection method, influence the possible results of the evaluation. For example, a Positivist evaluation paradigm may provide statistics on how many youth took up information on HIV/AIDS prevention measures. However, in order to understand why fewer than 40% of a sample of youth (Blecher et al, 1995) felt any risk to themselves from AIDS, one needs to undertake an Interpretivist evaluation and recognise that possible explanations and interventions lie within the youth themselves, and their own perceptions of unsafe sex within their social context.

c) There is a likelihood that a researcher who holds with the more Positivist view of the world will prefer to have more control over the process. This may be because the method requires more technical expertise, but also because the Positivist researcher is likely to feel that participatory research is more subjective.


27

On the other hand, we have directly contradicted this position by offering you training in a Positivist method of evaluation while at the same time advocating that Participatory Evaluation is in keeping with the principles of Primary Health Care. This hopefully makes you realise that there are no hard and fast rules in this regard. Most approaches are hybrid, or reflect mixed positions.

4 CRITICAL VIEWS OF POSITIVISM

Marsden and Oakley take a critical position on Positivism, regarding it as unlikely to address the “deeply political nature of evaluation.” (Marsden & Oakley, 1991: 316) They propose instead the Critical Theory position in relation to evaluation. They identify two broad paradigms from which research is conducted, and these merge the second and third strands discussed in Gephart’s paper.

Most of the evaluation approaches, methods and processes we have presented represent a Positivist approach to evaluation. This does not mean we value this one over all others, or that we negate any others. Both explanations are valid and reflect different theorists’ interpretations; we have merely focussed on one, to give you a thorough grounding in at least one. While absorbing this approach, it is also important to be sure to understand why Marsden and Oakley are so critical of the Positivist position or, as they call it, the Instrumental position.

They write:…[O]ne might identify two broad approaches to evaluation. Firstly there is the specific instrumental approach: the desire to know how effective interventions in the name of social development have actually been. This approach might be termed instrumental/technocratic. It usually attempts to extend current quantitatively-based practices into the new, uncharted seas of social development. Secondly, there is the approach which attempts to confront the dominant instrumental paradigm through ... [examining] the latent functions that evaluations serve, in reinforcing certain sorts of control. This emphasises the political processes involved in valuations and is not normally characterised by an over-concern with the elaboration of more appropriate practice. For want of a better word, this approach might be called interpretative. (1991: 315)

Read the full article by Marsden and Oakley which argues that the Interpretivist position is more legitimate for an evaluator because it critically confronts the very act of evaluation, and questions whether evaluation is not simply a way of maintaining conditions brought about by the dominant worldview and those in power at the time. The authors place subjectivity at the


28

centre of the experience of culture and history and call positivistic social science into question.

READING

Marsden, D. & Oakley, P. (1991). Future Issues and Perspectives in the Evaluation of Social Development. Community Development Journal, 26(4): 315-328.

TASK 3 – COMPARE A POSITIVIST WITH AN INTERPRETIVIST APPROACH a) What criticisms do these authors advance on the Positivist or

Instrumentalist approach?

b) What advantages do they put forward for the Interpretivist approach?

c) Would you say that your programme design is Positivist? Why or why not?

d) What value would an Interpretivist Approach have for your programme evaluation?

FEEDBACK

The Positivist approach is criticised because it fails to address the political nature of evaluation and because of its rationalist assumptions and the belief that an evaluation can be value-neutral. The authors characterise Positivist evaluation as primarily “a tool for management used to attempt to gain increased control … Evaluation is reduced to a static and mechanical operation designed to point out irrationalities, inconsistencies and bad practices, in pursuit of the ideal organisational form.” (Marsden & Oakley, 1991:316) Do you agree with the authors in their critical position on Instrumentalism?

The assertion that evaluations are political is further explained by Dennis Palumbo. He notes that a programme evaluation can be political in a number of ways: In that evaluations have bearing on the decisions made around the

programme, evaluation is political. Even by passing judgement on how successfully the programme is doing,

a programme evaluation is taking a political position. In addition, many evaluations may be undertaken simply to build the

position or reputation of a programme, and this in itself is a political and not entirely ethical act. (Palumbo, 1987: 12)

Many evaluators and evaluation theorists debate these issues in journals, websites and publications. Take a look at the further readings in Section 6 if you wish to explore these issues further.


29

5 SESSION SUMMARY

In this session, we have introduced a broad framework for understanding the positions taken by different evaluation paradigms. We have also taken a critical view at Positivism and invited you to consider your own evaluation plan in relation to the critical issues raised.

In the next session we move on to locating evaluation in the broader management context.

6 REFERENCES AND FURTHER READING

Blecher, M.S. et al. (1995). AIDS – Knowledge, Attitudes and Practices among STD Clinic Attenders in the Cape Peninsula. South African Medical Journal, 18: 1281-1286.

Caro, F. G. (1977). Readings in Evaluation Research. New York: Russell Sage Foundation.

Herrington, A. J. (1993). Ch 2 - Reflections on Empirical Research: Examining Some Ties Between Theory and Action. In L. Odell (Ed). Theory and Practice in the Teaching of Writing: Re-thinking the Discipline. USA: Southern Illinois University Press: 40-70.

Marsden, D. & Oakley, P. (1991). Future Issues and Perspectives in the Evaluation of Social Development. Community Development Journal, 26(4): 315-328.

Maykut, P. & Morehouse, R. (1994). Beginning Qualitative Research: A Philosophic and Practical Guide. London: The Falmer Press.

Palumbo, D. (1987). The Politics of Programme Evaluation. Newbury Park, Ca: Sage.

Schwandt, T. A. (Winter 2001). Ch 6 -Responsiveness and Everyday Life. New Directions in Evaluation, 92: 73ff.


30

Unit 1 - Session 4 Locating Monitoring and Evaluation in the Programme Management CycleIntroduction

Monitoring and evaluation are essential skills for programme managers. Here are two testimonies from programme managers who took this module as a SOPH short course recently.

“My job description includes evaluation of programmes. I thought I knew how to conduct an evaluation. I thought that I was doing the correct thing. It is after I attended the Winter School course on Monitoring and Evaluation that I realised that I have been doing it wrong.”

“I wish all programme managers could attend this course as most of the time people do not know what questions to ask about their programme effectiveness.”

We have used these comments to demonstrate the importance of this skill to programme managers. In essence, monitoring and evaluation are crucial processes because their ultimate aim is to use the information gathered from the operation of the programme to make subsequent decisions about the ways in which health-improvement activities can be better implemented.

The key responsibility of anyone who manages a programme is to ensure that effective planning and implementation of that programme takes place in order to reach intended goals or outcomes. The potential of monitoring and evaluation is to contribute to this management process and thereby to the effectiveness of the intervention.

In this session we are going to review the Programme Management Cycle in order to clarify where the monitoring and evaluation components fit into it.


31

Contents

1 Learning outcomes of this session2 Readings3 The Programme Management Cycle4 Planning a monitoring and evaluation strategy5 Session summary6 References and further reading

Timing

There are three tasks and no readings in this session. It should take you no more than an hour to complete.


By the end of this session, you are expected to be able to:

Describe where monitoring and evaluation fit into the Programme Management Cycle.

2 READINGS

There are no readings in this session.

3 THE PROGRAMME MANAGEMENT CYCLE

Many of you will have come across a model called the Programme Management Cycle: this is a planning framework or tool developed to assist managers and personnel in conceptualising, planning and managing projects. Its value lies in that it demonstrates that programme implementation is a continuous process. In the diagram illustrating this idea, the activities carried that constitute a programme are usually arranged in the form of a cycle.

Designing a programme implementation plan.These activities include: Developing goals and objectives.


32

Problem identification and cause analysis. Monitoring and evaluation. Implementation and service delivery. Now refresh your thinking about the Programme Management Cycle and

where the processes of monitoring and evaluation fit into the cycle.

TASK 1 - MAP OUT A PROGRAMME MANAGEMENT CYCLE

a) Arrange the five activities that constitute a programme to form a cycle. Draw connecting arrows so that all the activities are connected to each other.

b) Indicate clearly where monitoring and evaluation are situated within the Programme Management Cycle.

FEEDBACK

Take a look at Figure 1 and note where monitoring and evaluation have been located in the diagram. Does it differ from your cycle?

Figure 1 - A Programme Management Cycle

Source: Mwadime R et al. (1999).

The Programme Management Cycle is a model which depicts all the processes that go into planning and implementing interventions. For example, in programme planning, the programme manager and his or her team start by identifying and analysing the problem that requires intervention. They then go


33

on to develop programme aims and objectives followed by a process of designing and implementing intervention strategies.

The Programme Management Cycle suggests that monitoring and evaluation is regarded as a continuous process, which is applied throughout the programme cycle. Can you think why? It is because, for example, when the programme manager reviews the problem identified in relation to the programme objectives, s/he should also assess whether the objectives are realistic and achievable and whether the implementation process is related to the identified problem. In addition, the manager has to monitor the quality of activities involved during implementation and service delivery. The manager therefore has to continuously check if the steps taken are appropriate for the planned programme. In addition, the manager has to evaluate whether the services delivered are relevant to the target group, etc.

TASK 2 – IDENTIFY MONITORING AND EVALUATION ACTIVITIES SUGGESTED BY THE PROGRAMME CYCLE

Consider the Programme Cycle: jot down the key issues you think should be monitored and evaluated within this process and explain why you think so.

FEEDBACK

In the Programme Management Cycle, there are several key issues that must be monitored and evaluated at key moments during implementation of the programme. Note that the purposes are noted in italics. You have possibly thought of additional issues, but your list should include:

Keeping track of the programme implementation process to enable adjustments to be made where needed.

Keeping the programme on track in terms of timing and objectives. Accounting for resources used against activities to ensure efficiency and

best use of resources. Ensuring that expected results will be achieved in terms of programme

objectives. Determining the extent of results achieved through the attainment of

desired outputs for the purpose of checking the validity of the objectives. Determining the impact of the results on the lives of the target group/s to

demonstrate effectiveness of the programme.

As this list shows, for effective implementation of the programme, monitoring and evaluation components should be built to run throughout the programme cycle, and run from its beginning to its end. This implies that monitoring and evaluation activities require resources to be allocated specifically for them and


34

must be built into the programme budget of time, money and personnel throughout the life of the programme.

4 PLANNING A MONITORING AND EVALUATION STRATEGY

As we have noted, the process of planning your monitoring and evaluation strategy at an early stage is critical, but what does it involve? When putting together a proposal for an intervention, you should try to identify what information will tell you that your programme is effective. You should also think about who will collect the information, when and how often.

Planning for monitoring and evaluation in the early stages of the programme may also indicate a need for a baseline survey or needs assessment before the programme begins. Consider this kind of evaluation as an option for your second assignment. Building monitoring and evaluation components into the planning may also help in identifying a need for control groups, which are often important for credible evaluation.

The monitoring and evaluation planning process also gives programme managers an opportunity to experience the value of monitoring and evaluation, such as increasing the efficiency and effectiveness of resource usage throughout the life of the health programme or intervention.

TASK 3 - REFLECT ON PAST EVALUATION EXPERIENCES

Part 1

Think of a Health Care programme you have worked in or in which you are currently involved, and answer the following questions:

a) Was evaluation incorporated during the planning of the programme?

b) Was any baseline information collected?

c) If baseline information was collected, what was its purpose and value?

d) Now read about the Community Health Workers’ lifestyle intervention which is repeated in the box below. Try to decide whether evaluation was built into the project from the start. Cite examples from the case study to support your answer.


35

In this health intervention, a Participatory Evaluation Approach was used.

AN INTERVENTION FOR PRIMARY PREVENTION OF NON-COMMUNICABLE DISEASES: EXPERIENCES FROM KHAYELITSHA, A BLACK TOWNSHIP IN CAPE TOWN, SOUTH AFRICA (2002)

Objectives: To increase the knowledge of Community Health Workers (CHWs) who participate in the programme about 5 risk factors for non-communicable diseases (NCDs) by the end of 2006To reduce the prevalence of obesity among CHWs participating in the programme by 10% by the end of December 2006To increase the number of CHWs attending all training sessions on primary prevention of NCDS by the end of December 2003

Methods: Twenty-three CHWs employed by Zanempilo, an NGO, in Site C, Khayelitsha, participated in the programme. Firstly, focus group discussions were conducted with seventeen purposively selected CHWs to explore attitudes, beliefs and perceptions of hypertension and diabetes. Secondly, interviews assessed basic knowledge about causes, prevention and control of hypertension and diabetes. Thirdly, anthropometric measurements (weight, height, waist circumference) were taken and Body Mass Index (BMI) was kg/m2 was calculated and used as an indicator of obesity. Fourthly, level of physical activity and associated barriers were assessed. Lastly, data was collected on eating patterns (type and frequency of consumption of certain foods). Focus group discussions revealed that CHWs were uncertain about the causes of hypertension and diabetes and found it difficult to grasp the fact that even people without risk factors of these diseases, such as overweight and family history, could be diagnosed as hypertensive/diabetic. Many CHWs believed in traditional medicines and home brewed beer as best treatments for hypertension, and that bitter substances could be used to dilute blood sugar levels in diabetic patients They believed that people who took medical treatment became sicker and that their health deteriorated rapidly. A large percentage of CHWs were overweight and some were extremely overweight. Barriers to physical activity included fear of losing weight and lack of environmental safety. Eating patterns revealed lack of nutrition related knowledge.A training program for the primary prevention of non-communicable diseases was developed and implemented: The objective of the training was to build the capacity of the Community Health Workers (CHWs) in making healthy choices about the food they eat and in increasing physical activity. The training programme emphasised the role of nutrition, physical inactivity, body weight, smoking and alcohol in causation of hypertension and diabetes.

Implementation: Training sessions were held once a week for a period of six months. Pre- and post-tests were given to assess the effectiveness of the training. Lectures were given at a central location in the community. Facilitators of training workshops came from different tertiary education institutions and the training was participatory. Learning was reinforced by use of visual aids and real animal organs where applicable. Case studies, specifically developed for this population group, were used to reinforce learning and to encourage critical thinking. For each topic, its role in causation of NCDs was explained, preventive measures were highlighted and a key message was provided which could be used to inform the community.

Pens and writing pads were given to the participants during the training. Refreshments including tea, biscuits and a light lunch were also served.

At the completion of the training programme, one year later, the CHWs completed evaluation forms to evaluate the training programme. Focus group discussions were also held to determine changes in perceptions about risk factors for non-communicable diseases. Anthropometric or body measurements were also taken to evaluate the impact of the training on lifestyle modification.

FEEDBACK

In the above programme, monitoring and evaluation was built into the planning stage of the programme, although this is not explicitly stated.

For example, baseline information was collected on the following: Knowledge, beliefs, attitudes and perceptions about diabetes and

hypertension.SOPH, UWC, Master of Public Health: Monitoring and Evaluation of Health and Development Programmes - Unit 1

36

Weight, height, waist and hip circumference measurements to determine level of obesity as a risk factor for non-communicable diseases.

Eating patterns and levels of physical activity (risk factors for non-communicable diseases).

Collecting baseline information prior to introducing intervention strategies (in the form of training) helped the co-ordinator to understand the context, and what would be the most appropriate intervention/s. More importantly, it would later serve as the basis for analysing whether behaviour changes had resulted from the intervention.

Comparison of the outcomes with this baseline information yielded empirical proof that the activities of the project had demonstrable effects on the desired objectives (to develop awareness of risk factors and to change behaviour in relation to eating and exercise habits). In other words, it would give an indication of the effect of the intervention on the target population (CHWs).

This example demonstrates the importance of planning the evaluation strategy in advance: without doing so, there would have been no basis on which to judge the impact of the intervention.

How did your experience of a programme intervention compare with this one?


37

TASK 3 – Part 2

Now read another example of another intervention programme aimed at providing support for HIV infected mothers. Try to decide whether evaluation was built into the project from the start. Cite examples to support your answer.

Mothers-To-Mothers-To-Be:

Peer Counselling, Education and Support for Pregnant Women in Cape Town, South Africa

Mothers-to-Mothers-to-Be (M2M2B) is a mentorship programme for HIV infected pregnant women. Recently delivered HIV infected mothers who have passed through the antenatal programme are invited to return to the antenatal clinic to act as mentors, educators, counsellors, friends and confidants for recently diagnosed, HIV infected pregnant women. At the first antenatal visit, a pregnant woman is matched with a woman who has recently delivered – a mentor. This mentor sits with the pregnant woman – keeps her company, helps her with questions, the logistics of negotiating the hospital, reinforces messages about taking nevirapine and choosing a feeding method. At each appointment, the patient sees the same mentor.

The mentor-mothers also visit mothers in labour and the post-partum wards. Post-partum education is so important for mothers going home with new babies. HIV positive mothers are challenged by uncertainties about feeding regimes and they need to be certain about follow-up care for themselves and their babies.

At a clinic, each day, different mentor-mothers are scheduled to be part of the care team. Ten mothers provide care each week; each mother has her day so patients know when to come back to see a familiar face. Each mentor-mother holds her position for six months. After six months, the mothers graduate to be replaced by new mentor-mothers. The mentor-mothers are paid a stipend for their work.

The first M2M2B programme was started at Groote Schuur Hospital in Cape Town in October 2001. M2M2B has subsequently opened in Khayelitsha’s Site B MOU, a primary care maternity centre and at Mowbray Maternity Hospital, a district hospital. Additional sites in the Cape Town area, Paarl, Kimberley, East London and Johannesburg are awaiting M2M2B programmes. The programme has been a tremendous success, with rapid expansion only limited by funding constraints.

The programme has resulted in an increase in pre-natal visits, and an increase in knowledge on the issues surrounding HIV and pregnancy resulting in greater adherence to therapies that will reduce vertical transmission of HIV.

FEEDBACK

Although this is not explicitly stated, monitoring and evaluation was built into the planning stage of the programme.


38

For example, baseline data was collected on the number of pre-natal visits, knowledge about issues surrounding HIV and pregnancy, and the number or percentage of people who adhered to therapy.

5 SESSION SUMMARY

In this session, we have tried to contextualise monitoring and evaluation within the Programme Management Cycle. You have also been encouraged to reflect on your own experiences of these activities and to consider the importance of incorporating monitoring and evaluation during the planning stages of the programme. In the next session, we will focus on developing a conceptual framework for the programme as one of the key elements of monitoring and evaluation.

6 REFERENCES AND FURTHER READING

Coulson, N., Goldstein, S., & Ntuli, A. (1998). Chapter 13 - Project Planning and Evaluation. In Promoting Health in South Africa. An Action Manual. Sandton: Heinemann and Further Education: 158-171.

Mwadime, R. et al. (1999). Monitoring and Evaluation of Nutrition and Nutrition-Related Programmes. A Training Manual for Programme Managers and Implementers. The Applied Nutrition Programme, University of Nairobi School of Nutrition and Policy, & Tufts University.


39


40

Unit 1 - Session 5Developing a Conceptual Framework for a Primary Health Care ProgrammeIntroduction

If you have been involved in developing a programme proposal, it is more than likely that you had a template or set of questions which you were trying to answer at the planning stage. This is what we call a conceptual framework for a programme.

Its purpose is to guide the planning of a programme and to ensure that all the important issues have been considered in the process of conceptualising the programme. A conceptual framework is also useful in enabling others to understand and analyse a programme, and it plays a crucial role in developing sound monitoring and evaluation plans. The process of designing a programme according to a framework ensures that there is a clear understanding of the goals and objectives of the programme, and places emphasis on measurable short-term and long-term objectives. Developing a conceptual framework also helps programme teams to clearly define the relationships between factors that are key to the implementation and success of a project. Such factors may be internal, and also interact with the external environment or programme context.

Designing a conceptual framework also deepens the understanding of managers, implementers and other partners in many practical ways, including serving as the foundation for selecting appropriate and useful monitoring and evaluation indicators.

In this session, you will develop two types of conceptual frameworks for a Health Care programme of your choice – this task is part of your assignment.

Contents

1 Learning outcomes of this session2 Readings3 Three types of conceptual frameworks 4 Strategic Frameworks5 Research Frameworks6 Logical Frameworks7 Session summary


41

Timing

This session includes one reading and involves a substantial explanation of three types of conceptual framework followed by a task which will feed into your assignment. If thoroughly done, it should take up to three hours to complete.



Explain the value of a conceptual framework in programme planning and evaluation.

Use selected conceptual frameworks to describe a familiar Health Care programme.

2 READINGS

There is one reading in this session.

Author/s Publication details

Bertrand, J. T., Magnani, R. J. & Rutenberg, N.

(1996). Ch II – Defining the Scope of Evaluation. In Evaluating Family Planning Programs - with Adaptations for Reproductive Health. USA: The EVALUATION Project, University of North Carolina: 14-18.

3 THREE TYPES OF CONCEPTUAL FRAMEWORKS

You will recall from the previous session that it is necessary to develop the plans for monitoring and evaluation during the planning stage of the programme itself. This can be achieved by developing a conceptual framework for the programme, using a tool which is simple and readily applicable in any programme. The M&E plan is thus integrated into the overall conceptual framework of the programme. From here on, we are going to talk of the Programme Framework and the M&E framework interchangeably, concentrating on the framework as a tool for effective monitoring and evaluation.


42

Three basic types of conceptual frameworks are generally used; different organisations are likely to have a preference for a particular type. These are: a) Strategic or Results FrameworkA Strategic or Results framework is a diagram that identifies and illustrates the causal relationships linking all levels of a programme’s strategy including objectives and impacts.

b) Research Framework The Research framework (also known as a Conceptual Framework) is a diagram that identifies and illustrates the relationships among systemic, organisational or individual salient factors that may influence programme/ project operation and successful achievement of the programme or project goals. Note that when we use the words Conceptual Framework with capital letters, we are talking of this particular type of framework, i.e. research framework, rather than the generic idea of frameworks.

c) Logical FrameworkA Logical framework is a diagram that identifies and illustrates the linear relationships between programme inputs, processes, outputs and outcomes.

The following reading introduces these frameworks in more detail.

READING

Bertrand, J. T., Magnani, R. J. & Rutenberg, N. (1996). Ch II – Defining the Scope of Evaluation. In Evaluating Family Planning Programs - with Adaptations for Reproductive Health. USA: The EVALUATION Project, University of North Carolina: 14-18.

We are now going to take you through the development of each of these types of frameworks.

4 STRATEGIC FRAMEWORKS

A Strategic Framework is a diagram that identifies and links all levels of the programme with its potential results; it also illustrates the causal relationships between different levels of the programme’s objectives.

The purpose of this framework is: To provide a clear picture of the causal relationships that connect

incremental achievement of results with the comprehensive programme impact;

To clarify project or programme mechanics and the relationships between factors which suggest ways and means of objectively measuring the achievement of desired goals.

Here is an example of such a framework.


43

EXAMPLE 1: STRATEGIC OR RESULTS FRAMEWORK

In this example of a Strategic Framework, you will notice that each of the boxes contains a specific result or achievement. Beginning from the lowest levels, the diagram depicts the causal flow of outcomes which the managers of this programme expect to flow from their activities, to the overall objective at the top.

Developing a Strategic Framework clarifies the points at which the results can be monitored and evaluated. In addition, Strategic Frameworks clearly depict the causal relationships that the project design assumes will connect the different actions of the project with the outcomes. For example, training provision is linked causally with, or is expected to result in, the provision of a higher quality of care at facilities where more trained providers work; this ultimately leads to an improvement in health status or health outcomes for the targeted population. The effectiveness of the project’s related activities can be measured at each step in the causal framework.


44

INCREASED USE OF FAMILY PLANNING, MATERNAL AND CHILD HEALTH SERVICES & HIV/AIDS PREVENTIVE MEASURES

Availability of quality services

Demand for services

Client knowledge of reproductive and child health improved

Information and services increased

Health workers’ skills and knowledge increased

Sustainable effective management

5 RESEARCH FRAMEWORKS

Research Frameworks (also known as Conceptual Frameworks) identify and illustrate the relationships amongst relevant systemic, organisational, individual, or other salient factors that may influence programme or project operation and successful achievement of the programme or project goals.Research Frameworks take a broad view of the project itself in order to clarify the relationship of its activities and main goals to the context in which it operates.

The purpose of this framework is: To provide a perspective for understanding programme objectives within a

comprehensive (complete) context of relevant factors in a programme’s operating environment;

To clarify analytic assumptions and their implications for programme possibilities or limitations on success, as well as measuring and analysing that degree of success.

A Research Framework is a useful tool for identifying and illustrating a wide variety of factors and relationships that may affect the (successful) outcome of the programme. These factors include programme plans and operations, its operating environment in terms of target or non-target populations, government or institutional policies, infrastructure and other characteristics of the socio-economic or political system, and also include intermediate changes or ultimate goals. The design of the framework should show the interrelationships between all factors that are relevant to achieving the project’s goal or goals.

Constructing a Research Framework thus clarifies the complete context affecting the outcome of a programme or intervention.

It clarifies:

Your assumptions about the causal relationships connecting significant features of the programme context;

Aspects that your planned intervention may affect as well as other factors beyond your control.

Identifying the variables that factor into the programme performance, and organising the explicit ways in which they interact with each other, sets the stage for outlining the objective results which you can reasonably expect from your programme activities. Clarifying all of these issues is a critical step towards designing valid measures for analysing or evaluating the success of those interventions.


45

The following diagram illustrates the Research Framework in use:

EXAMPLE 2: A RESEARCH FRAMEWORK

This is a general example of a Research Framework in the health field. Factors shown include individual, systemic and programme variables.

Note that the programme in this diagram has activities that aim to influence both service utilisation and institutional capacity. A different project might aim at individual characteristics such as increasing knowledge of service providers, which would affect service utilisation only indirectly.

In this framework, there are two outcomes of interest, the sustainability of the health programme and the health status of the population.

6 LOGICAL FRAMEWORKS

In this section, we will use a case study to develop a Logical Framework.

Logical Frameworks identify and illustrate the linear relationships between programme inputs, processes, outputs and outcomes. Inputs or resources


46

Technical inputs

Institutional capacity

Individualcharacteristics

Service utilisation

Healthypractices

Health status

Programmesupply Programme

sustainability

affect processes or activities, which produce immediate results or outputs, ultimately leading to longer-term or broader results, or outcomes.

The purposes of this framework are: To provide a streamlined interpretation of planned use of resources and

desired ends; To clarify project or programme assumptions about linear relationships

between key factors relevant to (the achievement of) the desired ends.

This type of framework allows staff to articulate how they expect programme inputs and activities to achieve the desired effects. This is also a process of reaching consensus on the details of the programme and clarifying the terminology that will be used for different components of the programme.

The process of developing a Logical Framework starts with an understanding of the problem or need which the programme is addressing. In other words, one is answering questions like: What is the problem? How big is the problem? Who does it affect? What are the cause(s) of the problem?

If the programme defines the problem wrongly, everything thereafter will also be wrong. The framework indicates what elements need to be monitored and/or evaluated – to assess, and show, whether the intended objective is being achieved. These elements can later be translated into indicators. (You will learn more about indicators in Unit 4.) Dividing a programme into its components makes it easier to create the necessary indicators that will enable assessment of the programme; it also makes it easier to identify specific constraints to programme effectiveness while the programme is being implemented.

In this particular framework, there are four principal elements that can be translated into indicators and that are particularly useful for monitoring and evaluating health care programmes. These are inputs, outputs, outcomes, and impacts. We make assumptions about the relationships between these elements when creating this diagram. Let us explore the four principal elements in more detail.

6.1 The programme elements

Inputs are all those resources that go into developing the programme at the onset or start-up phase and during the implementation, to enable the programme to achieve its objectives. The inputs are things such as the number and qualifications of personnel, the financial resources, the institutional set-up, the timing, etc, and they must be designed to address the problem. The inputs should be distributed to meet all needy groups and be accessible financially, socially and technically. If this does not happen, the inputs are useless and the outputs may not be met.

Outputs are all the goods and services delivered to the target population by the programme. The inputs are to be transformed into outputs in the


47

implementation of the programme. The quantity and quality of the outputs is very important.


48

For instance, if the inputs are the work of trainers of CHWs, the outputs are the number of trained CHWs. The quality of the training should also be appropriate, because just any training would not help in effectively meeting the needs of the community. However, having well-trained staff does not necessarily ensure that the quality of the output will be good. Success and impact are created by making sure that trained personnel are enabled to do the work that they were trained for.

Outcomes are changes in behaviours or practices as a result of programme activities. The outputs, if they are of the right quantity and quality, should produce an outcome. For example, the skills of the CHWs should change, and if they do their tasks well, the detrimental behaviour or practices of the community should change for the better. The change in skills of the CHWs and/or the change in behaviour and practices of the community are the outcomes of the programme. The outcome is expected to influence the problem as defined at the beginning of the programme.

Impacts are the effects of the programme on the beneficiaries. The change in the problem is the impact of the programme on the beneficiaries or clients.

Assumptions are the external factors, influences, situations or conditions that are necessary for project success. Assumptions are important for the success of the programme, though they are largely or completely beyond the control of programme management. For example, in nutrition education, we may assume that community workers who are trained will understand the training and be motivated to do what they have been trained to do. However, we cannot be sure that this actually will happen. Accordingly, it is necessary to make assumptions explicit and list them in the framework as elements to be monitored or evaluated.

The elements of the programme can be presented in a Logical Framework or a diagram. The advantage of a logical framework is that it enables you to discuss the assumptions you have made about the implementation of the programme.

We will use the case study “Training Community Health Workers in the Primary Prevention of Cardiovascular Diseases” to illustrate a logical framework.

SOPH, UWC, Masters of Public Health, Health Promoting Schools: The Context of Programme Evaluation - Unit 1

65

PROGRAMME UPDATE: TRAINING OF CHWS ON PRIMARY PREVENTION OF CVD

The Provincial Administration of the Western Cape funded the training of CHWs. Lectures were given by lecturers from various institutions in collaboration with Zanempilo Trust.

The training was participatory and lectures were given once a week for a period of six months. Case studies were used to reinforce learning and to encourage critical thinking. Topics included basic anatomy of the heart, lungs, pancreas and relevant blood vessels, basic physiology, basic research methods and how to conduct a situational analysis. Handouts were given at the end of each lecture. Refreshments were served during teatime.

Based on the above programme, the following logical framework was developed.

EXAMPLE 3: A LOGICAL FRAMEWORKINPUTS PROCESS ASSUMPTION

SOUTPUTS ASSUMPT-

IONSOUTCOMES ASSUMPT-

IONSIMPACT

Training programme for CHWs.

Training materials developed.

Trained personnel.

Refresh-ments, transport,stationery.

Recruitment of CHWs.

Development of the curriculum.

Conduct training activities.

CHWs motivated to attend training.

CHWs understand the materials and are motivated to adopt healthy lifestyles.

23 CHWs trained.

Number of lectures given per month.

Number of education sessions.

The training will improve the knowledge of CHWs about risk factors for CVD.

Education will lead to healthy food choices and increased physical activity.

CHWs will be change models in their community.

CHWs modify their behaviour by adopting healthy lifestyles.

CHWs increase theirphysical activity.

Mean BMI isreduced.

CHWs will utilise this knowledge to motivate other community members to adopt healthy lifestyles.

Community members will also adopt healthy lifestyles.

Reduction in the prevalence of obesity.

Reduction in the prevalence ofhypertension and diabetes.

Reduction in prevalence of CVD.

This framework presents a straightforward view of the project, which was designed to reduce the prevalence of cardiovascular diseases by training CHWs in risk factors for cardiovascular diseases and in the benefits of adopting healthy lifestyles. As you can see, it does not account for all the factors that may influence operation and results, such as media influences, maturation of the target population, etc. Instead it focuses on the project’s activities, outcomes and impacts. This narrow focus is useful to programme managers and M&E planners, because it clarifies the direct relationships among elements of particular interest within a particular programme.

Logical frameworks can usefully show the flow of the programme’s operations while keeping the ultimate goals in view. Process indicators are sometimes useful for monitoring, and can also provide insight when evaluating to pinpoint breakdowns within the project, for example, if objectives are not fully met.


66

When you develop a Logical Framework, it is always advisable to work through several drafts before it is finalised. The first draft is usually very broad, but a review of the goals and objectives may help you to trim and focus the framework. We are now going to look at a Logical Framework developed by Elizabeth, a Health Information Officer who was one of the participants in the SOPH Winter School 2003. We are grateful to her for providing this example.

In this example, we will show how the framework was modified through discussion and debate, with the focus and programme objectives gradually being refined. Elizabeth started with a summary of her programme framework. The italicised text is our commentary.

PROGRAMME FOCUS: TB CONTROL

GoalTo reduce morbidity and mortality of Tuberculosis in Philadelphia sub-district.

Programme Objectives1. To reduce the risk of TB infection by 5% in the Philadelphia district by the end of

December 2004.2. To achieve a minimum of a 25% cure rate, an improvement from 2%, of all new

sputum positive TB cases, by the end of December 2004 in the Philadelphia district.

3. To achieve a turnaround time of at least 48 hours, from 72 hours, for sputum results, in more than 90% of cases by end of July 2004.

4. To build capacity for both the health personnel and community DOTS supporters on information collection.

With the help of the group, Programme Objectives 1-3 were deleted because the activities which were required to achieve those objectives were outside her scope of practice. Remember that she is working as a Health Information Officer. Having objectives 1-3 was too ambitious for her. She then focused on one objective (no 4) which she used for the development of her logical framework. The programme’s objective thus became:

To build capacity for both the health personnel and community DOTS supporters on information collection.

The goal remained unchanged as this objective is related to the goal. Building of capacity of personnel and DOTS supporters on information collection will lead to improved (increased accuracy in) data collection and reporting, which will lead to proper treatment and to a reduction in mortality and morbidity.


67

A LOGICAL FRAMEWORK FOR A TB CONTROL PROGRAMME


68

Develop a training programme for health personnel and Community DOTS supporters on data collection

Develop IEC materials for Community DOTSsupporters

Sustain the DOTS supporters

Develop a referral system

Number of health personnel trained on data collection

Number of DOTS supporters trained

Referral System

Health personnel gain knowledge and skills on information management

DOTS supporters gain skills on information management

DOTS supporters realise the importance of real and valid data

Reduction in morbidity and mortality due to tuberculosis

Improvement on sputum for testing at 2 months

Tracing of defaulters for testing at 2 months

INPUTS

OUTPUTS

PRIMARYOUTCOME

SECONDARYOUTCOME

IMPACT

Number ofIEC materials for DOTS supporters developed

Resources: transport stationery and food packages for DOTSsupporters

Number of km, and stationery used, no. of food packages distributed

That concludes the discussion of the three types of conceptual frameworks. We are now going to challenge you to spend some time developing examples of two of these conceptual frameworks for the health care programme which you intend to evaluate for your assignment. In the next session, we will focus on programme objectives as a key element of evaluation.

TASK 1 – DEVELOP TWO CONCEPTUAL FRAMEWORKS

Develop a Logical Framework and a Research Framework for the Primary Health Care programme that you intend to evaluate.

FEEDBACK

At this stage, there is no generic feedback except the preceding input on the frameworks. However, you are welcome to e-mail or fax your task to your lecturer for feedback. Mark it clearly with your lecturer’s name. Call it Logical and Research Framework Task for M&E in health care programmes.


69

Unit 1 - Session 6Programme Goals and Objectives IntroductionIn the previous session you developed a conceptual framework for your programme. For you to be able to evaluate the effectiveness, efficiency or cost-effectiveness of a programme, you need to be aware of what the programme is trying to achieve. This is expressed in the programme goals and objectives, also part of the Programme Management Cycle. Clearly written objectives will tell you exactly what the project will achieve. They will also inform the workers of what activities need to be completed to ensure that the objectives are met. Clearly stated objectives are also important in identifying appropriate indicators which can be used to measure the success of the programme.

In this session you will review the goals and objectives of that aspect of a programme you would like to evaluate.

Contents

1 Learning outcomes of this session2 Readings3 Programme goals and objectives4 Session summary5 References and further readings

Timing

This session includes one short reading and three tasks. It should take you about an hour to complete.


70



Differentiate between programme goals and objectives. Develop programme goals and SMART objectives for a limited scale Primary

Health Care programme of your choice. Explain the importance of clear objectives for monitoring and evaluation.

2 READINGS

You will be referred to the following reading in the course of this session:

Author/s Publication detailsBertrand, J. T., Magnani, R. J. & Rutenberg, N.

(1996). Ch II – Defining the Scope of Evaluation. Evaluating Family Planning Programs - with Adaptations for Reproductive Health. USA: The EVALUATION Project, University of North Carolina: 13-14.

3 PROGRAMME GOALS AND OBJECTIVES

TASK 1 – CLARIFY KEY CONCEPTS

Take a few minutes to jot down what you understand by the terms goals and objectives in the space provided below. Develop and list examples to clarify the terms.

Goals


71

Objectives

FEEDBACK

Now read pages 13-14 of the reading by Bertrand et al (1996).

READING

Bertrand, J. T., Magnani, R. J. & Rutenberg, N. (1996). Ch II – Defining the Scope of Evaluation. Evaluating Family Planning Programs - with Adaptations for Reproductive Health. USA: The EVALUATION Project, University of North Carolina: 13-14.

Here are some explanations and examples to compare with your own.

Goals

“A goal is a statement, usually general and abstract, of a desired state towards which a program is directed.” (Rossi and Freeman, 1993:104)

Goals refer to significant, long-term changes that planners expect to occur in people’s lives but which may not be totally achieved during the life of the programme, and may often refer to the vision of the programme.Programmes and projects often make only a small contribution towards achievement of a goal.

Here are some examples of goals: To improve the nutritional status of children. To reduce mortality due to cardiovascular diseases. To reduce infant mortality rate. Health for all by the year 2000.

Note that goals do not specify concrete expectations, nor do they specify how programme success will be measured, i.e. the criteria for success. It is also important to note that goals are not time-bound, i.e. do not specify what will be done, or achieved within a specific period.


72

Objectives

“Objectives, by contrast, are specific, operationalised statements detailing the desired accomplishments of a program.” (Rossi and Freeman, 1993: 104)

Objectives are derived from goals and, for both the programme and monitoring and evaluation, are usually stated in the form of a declarative statement, stating the purpose of the programme and the precise ground to be covered.

Objectives tell us exactly what the project intends to achieve. The achievement of the objectives contributes to the realisation of the goal.

It is very important that objectives be carefully thought out and stated clearly and precisely. To achieve this, we say that objectives must be SMART: Specific, Measurable, Achievable, Relevant and Time bound. This implies:

Specific: Must describe exactly what you want to achieve, with which target group, in which place.

Measurable: Must have a component that can be accurately measured, preferably without costing too much in time, money and effort.

Achievable: Must be realistically achievable within the circumstances, budget and time frame. It is better to be cautious rather than too ambitious.

Relevant: Must clearly contribute to achieving the goal. Time-bound: Must be achieved within a certain time period.

Examples of objectives: To increase the caloric intake among children under 24 months by 30% in

Mount Frere district by year three of the programme. To reduce the level of smoking by 15% among the youths of Khayelitsha

township by October 2005. To reduce the prevalence of obesity (BMI > 30) by 10% among the

residents of Khayelitsha township by the end of 2005. To reduce case fatality rate due to malnutrition by 10% in Limpopo district

by 2004.

The above objectives were set in relation to the goals of the programmes they refer to. For example, the goal for the programme on primary prevention of CVD, which was: “Reduction in morbidity and mortality due to CVD”, will be met by the objective: “To reduce the prevalence of obesity by 10% among residents of Khayelitsha township by year 2005.”

It is important to note that the programme is evaluated against its objectives – which speak to results. However, outputs are used to monitor the programme, as it is the outputs that contribute to the attainment of results and the achievement of objectives.


73

In the following task, you are invited to review the goals and objectives of the programme that you have chosen to evaluate. It is a very important exercise as you will later use these objectives to develop your monitoring and evaluation objectives for the first assignment.

TASK 2 – REVIEW THE GOALS AND OBJECTIVES OF A PROGRAMME

Take this opportunity to review the goals and objectives of the programme you have chosen. Do this by following these steps: Make sure that the objectives are related to the goals of the programme. Assess whether the objectives are SMART. Modify them if needed.

FEEDBACK

It is important that you do this exercise thoroughly as you will use the goals and objectives later when developing monitoring and evaluation objectives.We cannot really provide comment here but you would be welcome to e-mail or fax through your programme goals and objectives with a short programme description to your lecturer for feedback. Call it Programme Goals and Objectives Task for M&E in health care programmes. Put the lecturer’s name prominently on your fax.

The final task of this session explores the link between clearly stated programme goals and objectives, and programme monitoring and evaluation.

TASK 3 – CHECK YOUR UNDERSTANDING

a) Why is it important to develop clear and SMART programme goals and objectives?

b) What is their role in the monitoring and evaluation process?

FEEDBACK

The most obvious reason for having clear goals and objectives is to be able to implement a health care programme effectively. Without identifying your destination, you are unlikely to get there. Goals and objectives are just as important in developing a monitoring and evaluation strategy, the purpose of which is primarily to check progress and improve effectiveness of the programme as well as to enable assessment of programme impact.

As has been noted, the programme goals dictate the objectives. The programme is in turn evaluated against its objectives. Since objectives also enable us to develop proposed or likely outputs, the programme can then be SOPH, UWC, Masters of Public Health, Health Promoting Schools: The Context of Programme Evaluation - Unit 1

74

monitored against these anticipated outputs. All these processes contribute to the attainment of results and the achievement of objectives and ultimately, goals.

4 SESSION SUMMARY

Hopefully you have now developed a strong sense of the importance of clearly stated goals and measurable objectives for programme development and monitoring and evaluation. It is also important to ensure a logical relationship between the programme’s goals and its objectives. For a programme to proceed successfully, it is not only important that goals and objectives are clear, the programme plan should also be thorough and comprehensive. For monitoring and evaluation to proceed smoothly you need to have monitoring and evaluation objectives which are linked to the project objectives. For this reason, we will devote the final session of this unit to developing monitoring and evaluation objectives.

5 REFERENCES AND FURTHER READINGS

Rossi, P.H., & Freeman, H. (1993). Evaluation: A Systematic Approach. Newbury Park, Ca: Sage Publications.


75

Unit 1- Session 7Monitoring and Evaluation ObjectivesIntroduction

You are now familiar with the terms monitoring and evaluation and the uses of these processes. In addition, you have examined a range of issues which are important when embarking on an evaluation.

We now explore monitoring and evaluation objectives, which are derived from the programme goals and objectives. This session therefore builds on your own selected programme’s goals and objectives. At the same time, evaluation objectives are often linked to the intended use of the information collected in the process.

For the purpose of this session, we are going to use the goals and objectives that were used in Unit 1 Session 4.

Contents

1 Learning outcomes of this session2 Readings3 Developing monitoring and evaluation objectives4 Session summary

Timing

There are no readings and two tasks in this session. The final task feeds into your assignment. The session should not take you more than an hour to complete.


76



Develop monitoring and evaluation objectives for a health care programme evaluation.

2 READINGS

There are no readings in this session.

3 DEVELOPING MONITORING AND EVALUATION OBJECTIVES

We will start this session by comparing a set of programme goals and objectives with related monitoring and evaluation objectives.

PROGRAMME GOAL

PROGRAMME OBJECTIVES

MONITORING OBJECTIVES

EVALUATION OBJECTIVES

TO IMPROVE THE NUTRITIONAL STATUS OF CHILDREN.

To increase the calorie intake among children under 24 months by 30% in Mount Frere district by Year three of the programme.

To determine the trend in the frequency of feeding children; To identify the types of foods fed to children of less than 24 months in three villages of Mount Frere district.

To determine whether the calorie intake of children under 24 months has increased by 30% by Year three of the programme.

In this example, you will notice that the Programme Goal is broad while the Programme Objectives are relevant to the goal and are specific. They must be achieved within three years, so they are time-bound, and they are measurable, i.e. increased calorific intake for a specific group of children by a specific amount. We will leave you to assess whether you think this programme objective is attainable.

Now consider the Monitoring Objectives. Monitoring objectives are an expression of what you will check on a regular basis in order to see whether you are moving towards accomplishing the Programme Objectives. In this SOPH, UWC, Masters of Public Health, Health Promoting Schools: The Context of Programme Evaluation - Unit 1

77

instance, the planner has determined that in order to monitor whether the Programme Objectives will be reached, it will be helpful to take note of “the frequency of feeding children and the types of foods fed”. The types of food and the frequency of feeding will show whether there is an increase of calorific intake. A baseline study will however be essential in order to determine whether there is an increase. Do you think that the target percentage increase of 30% should be used from the outset? Is this realistic, or should it be increased gradually?

The Evaluation Objectives are an expression of what the programme should have accomplished, and are therefore almost exactly the same as the Programme Objectives.

Monitoring and evaluation objectives should shed light on monitoring and evaluation questions – what will be monitored, and what will be evaluated. These questions in turn should define the information which the monitoring and evaluation activities will collect. They should also guide the development of data collection instruments.

These questions should not be confused with those used in the actual data collection process when evaluating the programme, for instance, in interviews. These questions are only a guide to the development of the data collection tool. For example, an evaluation question for a programme involved in the training of community nutrition workers could be: For each of the training sessions conducted, was a training plan developed and was a report of the activity written?

TASK 1 – FORMULATE MONITORING AND EVALUATION OBJECTIVES

Now develop monitoring and evaluation objectives for the two examples below.

PROGRAMME GOALS PROGRAMME OBJECTIVES



TO REDUCE MORTALITY DUE TO NON-COMMUNICABLE DISEASES.

To reduce the prevalence of Type 2 diabetes by 15% among the residents of Khayelitsha township by 2005.

TO REDUCE THE INFANT MORTALITY RATE.

To reduce the case fatality rate due to malnutrition by 10% in Limpopo district by 2004.

FEEDBACK


78

Compare your suggested monitoring and evaluation objectives with these ones. Yours may be slightly differently expressed.

PROGRAMME GOALS

PROGRAMME OBJECTIVES



TO REDUCE MORTALITY DUE TO NON-COMMUNICABLE DISEASES.

To reduce the prevalence of Type 2 diabetes by 15% among the residents of Khayelitsha township by 2005.

To assess the number of people with Type 2 diabetes on a yearly basis.

To determine if the prevalence of Type 2 diabetes among Khayelitsha residents has been reduced by 15% in 2005.

TO REDUCE THE INFANT MORTALITY RATE.

To reduce the case fatality rate due to malnutrition by 10% in Limpopo district by 2004.

To obtain monthly case fatality rates due to malnutrition.

To determine if the case fatality rates due malnutrition in Limpopo district has been reduced by 10% in 2004.

TASK 2 – DEVELOP MONITORING AND EVALUATION OBJECTIVES FOR YOUR OWN PROGRAMME

Now refer back to the work you did on your the programme you plan to evaluate. Formulate your own monitoring and evaluation objectives following from the programme objectives you developed. Note the differences between monitoring and evaluation as illustrated in the examples above. Also note the differences in purpose between monitoring and evaluation, as discussed in the preceding sessions. Finally, revisit and make sure you understand how the monitoring and evaluation objectives relate to the programme objectives.

Please send them to your lecturer if you would like feedback. Call your work Monitoring and Evaluation Objectives Task for M&E in programmes and make your lecture’s name prominent on your fax.

7 SESSION SUMMARY

By now you will have seen that monitoring and evaluation form part of the programme implementation cycle, and that for monitoring and evaluation to be effective, it should be built into the programme planning. You have identified the most important elements of a programme and different types of conceptual frameworks for programmes. You have also identified programme goals and objectives, as well as monitoring and evaluation objectives.


79

The unit as a whole forms the essential background for monitoring and evaluating programmes. I hope you found it useful.

Hopefully you are moving steadily towards developing a monitoring and evaluation plan for the health care programme that you have chosen and also towards completing your first assignment. At this stage, it is essential to check the size and scope of your proposed evaluation project with your lecturer before you take it any further.

This is the end of Unit 1 – in Unit 2 we will introduce you to a range of issues which you should be familiar with when entering monitoring and evaluation.

Before you start Unit 2, be aware that there is a glossary of common terms used in evaluation in section 5 of the Module Introduction. Scan through the terms, noting which ones you are familiar with and marking those that are new to you. Refer to the glossary when you come across an unfamiliar concept.


80


81

guidelines for writers (soph post …€¦ · web viewtitle guidelines for writers (soph...

Documents