Monitoring and Evaluation

Module 12 – March 2010

Project Partners

Funded by the Health Resources and Services Administration (HRSA)

Module Overview

Monitoring & Evaluation (M&E) framework and components

Records, registers and reports

Recording and evaluating response to TB treatment regimens

Supervision

International Standards 13 and 21

Learning Objectives

At the end of this presentation, participantswill be able to: Describe what is meant by “Monitoring and

Evaluation”

Discuss the importance of collecting data and ensuring the accuracy of the data

Explain ways in which the data are used to evaluate treatment

Describe how M&E activities can benefit both TB and HIV/AIDS programs

Monitoring & Evaluation System

A key element of the Stop TB Strategy

Allows programs to:

• Monitor progress and treatment outcomes of individual patients

• Evaluate the overall performance of the TB program at various levels (local, district, national)

• Identify areas of program improvement and weakness

• Ensure accountability

Monitoring

What is it?

• Routine tracking of services and program performance

Monitoring (2)

How is it done?

• Through information collection, data input, analyzing the data, and reporting what is found in that analysis outcome report

Why should we do it?

• To better assess how well a policy or program is achieving its intended target

Evaluation

What is it?

• Episodic assessment of results that can be attributed to program activities

• Types of evaluation related to M & E:

➜ Process evaluation: assesses the progress in program implementation and coverage

➜ Outcome and impact evaluation: measures the effect of the program activity on the target population

What are the Targets?

Stop TB Partnership/WHO• 70% TB case detection and 85%

cure rate by 2005

Millennium Development Goals (MDG):• Halt, and begin to reverse, the incidence of major

diseases such as HIV/AIDS and TB by 2015 Decrease TB prevalence and death rates to 50% of

the 2000 estimates

United Nations General Assembly Special Session (UNGASS) – global targets

Where do Indicators Fit In?

Indicator: a specific, observable, and measurable characteristic or change that shows the progress a program is making toward achieving a specific outcome

Indicators may be expressed in terms of:• Number

• Rate

• Proportion

• Percentage

Limitations of Indicators

Indicators DO NOT:

Measure everything

Tell us why a problem may exist or how to fix it

Determine if problems identified are amenable to intervention

Tell us which interventions are most cost effective

What are some possible uses of data collected by the National HIV/AIDS and

TB Programs?

Using and Disseminating Data

M & E can improve and enhance NAP and NTPwork by: Identifying areas of strengths and weaknesses

Helping plot progress toward program goals

Allowing a program to see trends and to identify high risk groups in order to better target TB control efforts

Providing justification for needed resources

Identifying training and supervision needs

Increasing public awareness about TB

Advocating for policy changes and allocation of funds

Monitoring & Evaluation Framework

CONTEXTEnvironmental, cultural, political, and socio-economic factors external to the programme

Monitoring/Process Evaluation Outcome/Impact Evaluation

INPUT

Basicresourcesnecessary

• Policies• People• Money• Equipment

PROCESS

Programmeactivities

• Training• Logistics• Management• IEC/BCC

OUTPUT

Results at theprogrammelevel(measure ofprogramme activities)

• Services• Service use• Knowledge

OUTCOME

Results atlevel of targetpopulation

• Behaviour• Safer practices

IMPACT

Ultimate effectof project inlong term

• TB incidence• HIV prevalence• Morbidity• Mortality

Types of M & E Activities

Staff training

Supervision

Completion of reporting forms

Discussions during staff meetings

Ensuring medicine and laboratory stock supplies

Quality control activities

Analysing data and preparing reports

Standard 13: Recording & Reporting

A written record of all medications given, bacteriologic response, and adverse reactions should be maintained for all patients

Standard 21: Recording & Reporting

All providers must report both new and re-treatment tuberculosis cases and their treatment outcomes to local public health authorities, in conformance with applicable legal requirements and policies

Form 3: BASIC MANAGEMENT UNIT TB REGISTER – LEFT SIDE OF THE REGISTER BOOK

Type of patient 4 Date of

registration District TB No.

Name Sex

M

/F

Age

Address Health facility1 Treatment supporter2

Date treatment

started

Treatment category 3

Site P / EP N R F D T O

1 Facility where patient’s treatment card is kept. In case several copies are kept, the most peripheral facility should be entered. 2 including community worker/volunteer, family members or friends. 3 Enter the treatment category: CAT I: New case

CAT II: Re-treatment e.g. 2(HRZE)S/1(HRZE)/5(RHE) Chronic:patient sputum positive at the end of a re-treatment regimen. Chronic cases still alive and not started on Category IV treatment should be re-entered at the beginning of each year. Patients who are started on Category IV treatment should be entered in a separate Category IV register and separate Category IV treatment cards should be used for them. 4 Tick only one column : N=New – A patient who has never had treatment for TB or who has taken antituberculosis drugs for less than 1 month. R=Relapse – A patient previously treated for TB, declared cured or treatment completed, and who is diagnosed with bacteriological (+) TB (smear or culture).

F=Treatment after failure – A patient who is started on a re-treatment regimen after having failed previous treatment. D=Treatment after default – A patient who returns to treatment, positive bacteriologically, following interruption of treatment for 2 or more consecutive months. T=Transfer in – A patient who has been transferred from another TB register to continue treatment. This group is excluded from the quarterly report on registration. O=Other previously treated– All cases that do not fit the above definitions. This group includes smear-positive cases with unknown outcome of previous treatment, smear negative previously treated, EP previously treated and chronic case (i.e. a patient who is sputum positive at the end of a re-treatment regimen)

Form 3: BASIC MANAGEMENT UNIT TB REGISTER – RIGHT SIDE OF THE REGISTER BOOK

Results of sputum smear microscopy and other examination Treatment outcome

& date TB/HIV activities

Before treatment 2 or 3 months 1 5 months End of treatment

Smear result

Date/ Lab. No.

X-ray Date/ Result

4

Smear result

Date/ Lab. No.

Smear result

Date/ Lab. No.

Smear result

Date/ Lab. No.

Date

Outcome in text

2

HIV result3

/ Date/

No. HIV reg

ART Y/N

Start date/ No. ART reg

CPT Y/N Start date

Remarks

1CAT 1 patients have follow-up sputum examination at 2 months; CAT II patients have follow-up sputum examination at 3 months. CAT 1 patients with extended phase 1 to 3 months have follow-up sputum examination at 2 AND 3 months with results registered in the same box. 2Enter the code (1-6) as follows:

1-Cure: Sputum smear positive patient who was sputum negative in the last month of treatment and on at least one previous occasion. 2-Treatment completed: Patient who has completed treatment but who does not meet the criteria to be classified as a cure or a failure. 3-Treatment failure: New patient who is sputum smear (+) at 5 months or later during treatment, or who is switched to Category IV treatment because sputum turned out to be MDRTB. Previously-treated patient who is sputum smear positive at the end of his retreatment or who is switched to Category IV treatment because sputum turned out to be MDRTB. 4-Died: Patient who dies from any cause during the course of treatment. 5-Default: Patient whose treatment was interrupted for 2 consecutive months or more. 6-Transfer out: Patient who has been transferred to another recording and reporting unit and for whom treatment outcome is not known.

3 + positive, - negative, U unknown, ND Not Done. Documented evidence of HIV test performed during or before TB treatment is reported here. 4 + : suggestive of TB, -: not suggestive of TB, ND: not done.

Why is accurate reporting and record keeping important?

Data Quality Assurance

Ensures that the information collected adequately represents the program’s activities

Accurate data – measuring what it is intended to measure

Reliable data – collected and measured the same way by all program personnel over time

Reporting Forms and Registers

Request for Sputum Examination

Tuberculosis Treatment Card

Tuberculosis Identity Card

Basic Management Unit TB Register

TB Laboratory Register

Quarterly Report on TB Case Registration

Quarterly Report on Sputum Conversion

Register of TB Suspects

Quarterly Report on Treatment Outcomes

Yearly Report on Program Management in Basic Management Unit

Tuberculosis Treatment Card

Tuberculosis Treatment Card BMU TB Register No._____________

Name: ________________________________________________________ Disease site (check one) Pulmonary Extrapulmonary, specify ___________

Type of patient (check one) New Treatment after default Relapse Treatment after failure Transfer in Other, specify ___________________

Sex: M F Date of registration: ____________________________

Age: ________ Health facility: _________________________________

Address: ________________________________________________________ ________________________________________________________________

Name / address of community treatment supporter (if applicable) ________________________________________________________________

Sputum smear microscopy Weight (kg)

Month Date Lab No. Result 0

TB/HIV Date Result* HIV test

CPT start ART start

* (Pos) Positive; (Neg) Negative; (I) Indeterminate; (ND) Not Done/unknown

I. INITIAL PHASE - prescribed regimen and dosages

CAT (I, II , III):

Number of tablets per dose and dosage of S: (RHZE) S

Referral by :

Self-referral Community member Public facility Private facility/provider Other, specify --------------------------------

Cotrimoxazole ARV Other

Tick appropriate box after the drugs have been administered Daily supply: enter . Periodic supply: enter X on day when drugs are collected and draw a horizontal line ( ) through the number of days supplied. Ø = dr ugs not taken

Day Month

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Please turn over for continuation phase

Tuberculosis Treatment Card (2)

Comments: _________________________________________

___________________________________________________

___________________________________________________

___________________________________________________

___________________________________________________

II. CONTINUATION PHASE

Number of tablets per dose Daily supply: enter . Periodic supply, enter X on day when drugs are collected and draw a horizontal line ( ) through the number of days supplied. Ø = d rugs not taken

Day Month

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

X-ray (at start) Date: Results (-), (+), ND

Treatment outcome Date of decision ____ Cure Treatment completed Died Treatment failure Default Transfer out

HIV care Pre ART Register No. CD4 result ART eligibility (Y/N/Unknown) Date eligibility assessed ART Register No.

______________________________________________________________________________________

______________________________________________________________________________________

______________________________________________________________________________________

______________________________________________________________________________________

______________________________________________________________________________________

______________________________________________________________________________________

Name and address of contact person: ______________________________________________________________________________

(RH) (RHE) (Other)

Preparing a TB Treatment Card

Activity

Tuberculosis Identity Card

This card contains an extract of information on the treatment card

It is given to the patient at the start of treatment

It is used to record daily DOT and must be used during the intensive phase of treatment

It also serves as a reference document for TB status after treatment

It should be presented to the doctor whenever the patient falls ill in the future

Tuberculosis Identity Card (2)Tuberculosis Identity Card

Name __________________ BMU TB Register No. _____ Address __________________ Date of registration: _______ Sex: M F Age ______ Date treatment start _______ Health facility: ______________________________________ Supporter (name and address) __________________________

Sputum smear microscopy Weight (kg)

Month Date Lab No. Result

Disease site (check one) Pulmonary Extrapulmonary, specify _______

Type of patient (check one) New Treatment after default Relapse Treatment after failure Transfer in Other specify ______________

I. INITIAL PHASE CAT (I, II , III): (RHZE) S Other Drugs and dosage:

II. CONTINUATION PHASE (RH) (RHE) Other Drugs and dosage:

Appointment dates: ________________________________ __________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________ __________________________________________________

REMEMBER

Basic Management Unit TB Register

This revised register is the cornerstone of an NTPs monitoring & evaluation system

It records essential information for notification & treatment outcome by district

It should always be kept up to date with data on sputum smear examinations and treatment outcome

Where electronic data collection systems are available, the information from the register should be entered into the database at least once every month

Basic Management Unit TB Register – Left side of the register book

Date of registration

BMU TB No.

Name Sex

M

/F

Age

Address Health facility1

Date treatment

started

Treatment category2

Site P/EP

Type of patient3

N R F D T O

Footnotes appearing on first page of the register only. 1 Facility where patient’s treatment card is kept. In case several copies are kept, the most peripheral facility should be entered. Use standardized type of health

facilities according to block 2 of the Yearly Report on Programme Management in BMU. Health facility is defined as any health institution with health care providers formally engaged in any of the following TB control functions (DOTS): referring TB suspects/cases, laboratory diagnosis, TB treatment and patient support during treatment.

2 Enter the treatment category: CAT I: New case of sputum smear microscopy positive, severe

sputum smear microscopy negative PTB & EPTB e.g. 2(RHZE)/4(RH)

CAT II: Re-treatment e.g. 2(RHZE)S/1(RHZE)/5(RHE) CAT III: New sputum smear microscopy negative PTB and EPTB

e.g. 2(RHZE)/4(RH) 3 Tick only one column:

N=New – A patient who has never had treatment for TB or who has taken antituberculosis drugs for less than 1 month.

R=Relapse – A patient previously treated for TB, declared cured or treatment completed, and who is diagnosed with bacteriological (+) TB (sputum smear microscopy or culture).

F=Treatment after failure – A patient who is started on a re-treatment regimen after having failed previous treatment.

D=Treatment after default – A patient who returns to treatment,

positive bacteriologically, following interruption of treatment for 2 or more consecutive months.

T=Transfer in – A patient who has been transferred from another TB Register to continue treatment. This group is excluded from the Quarterly Reports on TB Case Registration and on Treatment Outcome.

O=Other previously treated– All cases that do not fit the above definitions. This group includes sputum smear microscopy positive cases with unknown history or unknown outcome of previous treatment, previously treated sputum smear microscopy negative, previously treated EP, and chronic case (i.e. a patient who is sputum smear microscopy positive at the end of re-treatment regimen)

TB Register in Basic Management Unit using Routine Culture and DST – Right side of the register book Results of sputum smear microscopy and other examinations Treatment outcome & date TB/HIV activities

Remarks

Before treatment 2 or 3 months 1 5 months End of treatment

Date

Outcome in text 7

ART Y/N Start date

CPT Y/N Start date

Sputum smear micros-

copy date/No./ Result

2

HIV result3/

Date

X-ray Result

4/ date

Culture

date/No./ Result

5

DST date/No./ Result

6

Sputum smear micros-

copy No./

Result2

Culture No./

Result5

Sputum smear micros-

copy No./

Result2

Culture No./

Result5

Sputum smear micros-

copy No./

Result2

Culture No./

Result5

Footnotes appearing on first page of the register only 1 CAT I patients have follow-up sputum smear microscopy examination at 2 months; CAT II patients have follow-up sputum smear microscopy examination at

3 months. CAT I patients with initial phase of treatment extended to 3 months have follow-up sputum smear microscopy examinations at 2 AND 3 months with results registered in the same box.

2 (ND): Not done; (NEG): 0 AFB/100 fields; (1-9): Exact number if 1 to 9 AFB/100 fields; (+): 10-99 AFB/100 fields; (++): 1-10 AFB/ field; (+++): > 10 AFB/ field 3 (Pos):Positive; (Neg):Negative; (I):Indeterminate; (ND):Not Done / unknown. Documented evidence of HIV test performed during or before TB treatment is

reported here. Measures to improve confidentiality should accompany recording of HIV status. 4 (Pos): Suggestive of TB; (Neg): Not suggestive of TB; (ND): Not Done. 5 (Pos): Positive; (Neg): Negative; (ND): Not Done. 6 (ResistR): Resistant to Rifampicin; (ResistH): Resistant to Isoniazid; (ResistE): Resistant to Ethambutol; (ResistStrept): Resistant to Streptomycin;

(ResistRH): Resistant to Rifampicin and Isoniazid; (Suscept): Susceptible; (ND): Not Done. 7 Write clearly ONE of the following outcomes per patient:

Cure: Patient with culture or sputum smear microscopy positive at the beginning of the treatment who was culture or sputum smear microscopy negative in the last month of treatment and on at least one previous occasion. Treatment completed: Patient who has completed treatment but who does not meet the criteria to be classified as a cure or a failure. Treatment failure: New patient who is culture or sputum smear microscopy positive at 5 months or later during treatment, or who is switched to Category IV treatment because sputum smear microscopy turned out to be MDRTB. Previously-treated patient who is culture or sputum smear microscopy positive at the end of his re-treatment or who is switched to Category IV treatment because sputum turned out to be MDRTB. Died: Patient who dies from any cause during the course of treatment. Default: Patient whose treatment was interrupted for 2 consecutive months or more. Transfer out: Patient who has been transferred to a health facility in another BMU and for whom treatment outcome is not known.

Treatment Outcomes

Cured

Treatment completed

Treatment failure

Died

Default

Transfer out

Treatment Outcomes

Cure Patient whose sputum smear or culture was positive at beginning of treatment but who was smear- or culture-negative in the last month of treatment and on at least one previous occasion

Treatment Completed

Patient who has completed treatment but who does not meet the criteria to be classified as a cure or a failure

Treatment Failure

Patient who is sputum smear-positive at five months or later during treatment– or –Patient found to harbor a MDR strain at any point of time during treatment, whether smear-negative or -positive

Treatment Outcomes (2)

Died Patient who dies for any reason during the course of treatment

Default Patient whose treatment was interrupted for 1 month or more

Transfer Out

Patient who has been transferred to another recording and reporting unit and whose treatment outcome is not known.

Treatment Success

A sum of cured and completed treatment (smear-positive or culture-positive patients only)

Supervision

How is supervision used in your TB and HIV/AIDS Prevention and Control

Programs?

Role of Supervision in M & E

Supervision is a process of guiding, helping, training, and enabling staff to improve their performance in order to provide high quality health care services

Purpose of Supervision

Provide leadership and direction to staff

Ensure effective program implementation

Monitor operations and evaluate achievement of goals

Ensure adherence to laws and policies

Avoid confusion or duplication of efforts

Purpose of Supervision (2)

Monitor that all necessary tasks are properly performed

Ensure that resources are properly used and are available to staff, including training and supplies to carry out their duties

Ensure accountability

Barriers to Effective Supervision

Lack of commitment

Lack of proper planning and time management

Lack of tools for Monitoring & Evaluation

Insufficient staff

Problems with transportation

Lack of confidence or preparation

Roles and Responsibilities

National Level – NTP and NAP

• Planning, implementing, monitoring, and evaluating Program at all levels

Regional/District/Parish Levels

• Coordinating, supervising, planning, implementing, monitoring and evaluating all aspects of TB and HIV/AIDS Programs in the region

How Can M&E Information be used in TB and HIV Programs?

Identify gaps in performance

Monitor treatment outcomes

Measure the impact of an intervention or policy change

Identify populations for enhanced control and prevention efforts

Identify local problems as they arise

Ensure high quality TB and HIV prevention and control strategies are consistently provided

Summary: ISTC Standards Covered

Standard 13: A written record of all medications given, bacteriologic response, and adverse reactions should be maintained for all patients

Standard 21: All providers must report both new and re-treatment tuberculosis cases and their treatment outcomes to local public health authorities, in conformance with applicable legal requirements and policies

Summary

Several approaches are used to monitor and evaluate TB and HIV/AIDS programs including supervision, training and the keeping of records and registers

Accurate and timely reporting and record keeping is important. It allows true assessment of Program achievements