module 9 part a: monitoring and evaluation *image courtesy of: world lung foundation
TRANSCRIPT
Module 9 Part A: Monitoring and Evaluation
*Image courtesy of: World Lung Foundation
• Describe the difference between monitoring and evaluation
• Explain why we monitor programmes
• Explain how to monitor a TB programme
– What to monitor– Principles and techniques
• Describe how to conduct a supervisory visit
Learning Objectives
What is M&E?
*Image courtesy of: World Lung Foundation
Differences between Monitoring and Evaluation
Why Monitor and Evaluate?
1. Collect accurate information about the TB Programme
2. Use that information to improve the TB Programme
Management Reporting Accountability Advocacy Evaluation
Monitoring serves several purposes
• Episodic assessment of specific indicators– determine effectiveness or impact of services or
activities – during a given interval
• Determine whether goals are being met• Assess impact of a specific service or
intervention- HIV testing among TB patients
• Advocacy
Why Evaluate?
• Determine if staff activities follow BNTP guidelines
• Measure key indicators related to case detection, quality of diagnosis, and TB treatment
• Identify problems in implementing TB control activities (e.g., laboratory reporting delays)
• Inform the annual evaluation of TB Programme
• Use the findings to modify programme goals and strategies (e.g., implementing RHT)
Benefit for BNTP
Surveillance Data and Programme Monitoring within BNTP
Clinic-level: TB Suspect and TB Case Registers
District-level: District TB Register and ETR
National-level: ETR
How should I monitor?
The first step is observation, but talking and analysis help you understand the cause
Observe Talk Analyse
Actions Attitudes Consistency
Processes Understanding Accuracy
Conditions Morale Effectiveness
Observe
Take a look around the clinic
Are there signs of disorganization? Are the patient treatment cards for all registered patients available? Are the cards in a binder in numerical order?Are the meds organized?
Are there IEC materials? Are there masks for patients and respirators for staff? Are patients being triaged appropriately?Is the condom dispenser full?
Other Visual Indicators
Communication with Clinic Staff
Find Out More from People
Through communicating we can:
Test level of knowledge
Gauge attitude and morale
Seek guidance on priority areas
DO’s: Listen and be prepared to learn Take notes – it’s your job. Don’t be embarassed! Ask follow-up questions and explore the issues Compare one story against another! “Triangulate”
DONT’s: Don’t worry if the person knows more than you Don’t take anything at face value – CHECK all verbal information against the data Don’t threaten or intimidate the people you speak to
Some Communication Tips
Level of knowledge: ‘Who should be entered into this register?’ ‘Explain the process of how the register is used’
Attitude / Perception ‘Whose job is it to enter the data?’ ‘How important is it that this register is
used properly?’ ‘Are you comfortable with using it?’
Seek guidance ‘What needs to be done to improve how it is used?’
Apply this Approach to Determine How Register is Used
Analysing the Suspect and Sputum Dispatch Register
• % of new pulmonary TB (NPTB) suspects who have 3 initial sputums collected consecutively
• Formula:
Number of who have 3 initial sputums collected consecutively
Number of NPTB suspects worked up for TB
• Tip: compare the result you get to last time. You can learn a lot by comparing indicators over time
Indicator: Quality of Programme Management
The TB Register
Use your experience and common sense Are the entries recorded correctly?
– Proper chronological sequence– Proper identification of class and type
Are there signs that the register is being completed in “batches”? – Multiple entries on the same date with different
treatment start dates (all in the same pen!)
How does the data ‘look’?
Analysing the DistrictTB Register
Recording and reporting Calculating Indicators
Use your experience and common sense Are the entries recorded correctly
– Proper chronological sequence– Proper identification of class and type
Are there signs that the register is being completed in “batches” – Multiple entries on the same date with different
treatment start dates (all in the same pen!)
How does the data ‘look’?
Example: Review of District TB Register
Is there a report for each facility in this quarter? For cases registered 3 months ago, are there follow-up
sputum examination results (that is, for those collected at month 2 / end of the intensive phase?)
For cases registered > 6 months ago, are there more follow-up sputum exam results, or blanks in these columns?
For cases registered 12 months ago, are there treatment outcomes, or gaps in that area of the register?
Are there cases who were registered on the basis of sputum exam results but never started treatment?
Analysing the DistrictTB Register via the ETR
Calculating Indicators Recording and reporting
Calculating TB Rates
Death Rate = deaths / proportion of incident cases that die
Case Detection Rate = annual new smear-positive notifications (country) / estimated annual new smear positive incidence (country)
Other Useful TB Rates Include:Notification Rate, Cure Rate, Treatment Success
Rate, Default Rate, Treatment Failure Rate, Transfer Out Rate
TB CasesPulmonary
EP Total %Smear + Smear - No Smear Total
New cases 91 15 105 211 45 256 97%
Relapses 3 2 3 8 0 8 3%
After default 0 0 0 0 0 0 0%
After failure 0 0 1 1 0 1 0%
Total 94 17 109* 220 45 265 100%
% 35% 6% 41% 83% 17% 100%
* of which children aged 0-7: 6
TB Cases 0-14 15-24 25-34 35-44 45-54 55-64 65-74 >75 Total %
AllTBCases
M 9 16 66 44 17 5 2 0 159 60%
F 5 17 39 33 9 3 0 0 106 40%
Total 14 33 105 77 26 8 2 0 265 100%
% 5% 12% 40% 29% 10% 3% 1% 0% 100%
AllSmear + Cases
M 1 7 23 15 6 2 0 0 54 59%
F 0 7 15 13 3 0 0 0 38 41%
Total 1 14 38 28 9 2 0 0 92 100%
% 1% 15% 41% 30% 10% 2% 0% 0% 100%
AllSmear +Re-treat
M 0 0 2 0 0 0 0 0 2 67%
F 0 1 0 0 0 0 0 0 1 33%
Total 0 1 2 0 0 0 0 0 3 100%
% 0% 33% 67% 0% 0% 0% 0% 0% 100%
0 record(s) with missing age
Botswana Tuberculosis Programme
Case Finding Report
Report on New and Retreatment Cases of Tuberculosis(WHO)
15 – GABORONE Quarter 1 of 2004
Formula
SS+ cases
Registered cases
Example
94= 35%
265
AFB Diagnosis RateThe proportion of notified cases diagnosed with sputum smear microscopy
43%
What can it mean if cases diagnosed through AFB is under 50%?
% Cases diagnosed with AFB Microscopy
0
10
20
30
40
50
60
70
80
90
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
District A District B District C
1. Too many cases being diagnosed through x-ray
2. Too many extra-pulmonary cases3. In comparison with previous
quarters an increase may imply increasing reliance by clinicians on smear microscopy…• Is that good or bad?• What could have caused the
change in District A?• Take a look at District C – if you
were monitoring this province what would you think if you saw this trend?
• What about District B?• Which is probably the best
result?
Category Outcome Status No %
AllTB
Cases
Treatment completedSmear negative at completion 28 8%Smear positive at completion 0 0%Smear results not available 67 19%Treatment not completedDied during treatment 12 3%Transferred to another unit 26 7%Defaulted from treatment 20 6%Treatment outcome not evaluated 195 56%All tuberculosis cases 347 100%
AllSmear +New
Treatment completed
Smear negative at completion 21 18%Smear positive at completion 0 0%Smear results not available 16 13%Treatment not completedDied during treatment 3 3%Transferred to another unit 11 9%Defaulted from treatment 6 5%Treatment outcome not evaluated 63 53%All tuberculosis cases 120 100%
AllSmear +Re-treatment
Treatment completedSmear negative at completion 0 0%Smear positive at completion 0 0%Smear results not available 1 25%Treatment not completedDied during treatment 0 0%Transferred to another unit 0 0%Defaulted from treatment 0 0%Treatment outcome not evaluated 3 75%All tuberculosis cases 4 100%
Botswana Tuberculosis Programme
Treatment Outcome ReportReport on the Outcome of Tuberculosis Treatment
15 – GABORONE Quarter 1 of 2004
Formula
No of registered cases that default
Total no of registered cases
Example
20= 6%
347
Program Default Rate:How many patients are defaulting from treatment. In other words, how many people stop taking their medications.
Less than
5%
Something is wrong in the program and needs to be fixed.The patient is quite likely sill sick
and contagious.
Less than
5%
It’s important to understand what‘default’ means:
“A TB Patient is classified as a ‘default’ when their treatment is interrupted for
2 consecutive months or more.”
This is an easy indicator to get from the data, but it can be deceptive!
Every defaulting patient = a risk of MDR-TB in the community.What do we know for sure if
we’re missing the target?
1. Is it to do with the patient?• Is it embarassing for the patient?• Are patients scared of the
side-effects?• Does the patient know that you have to KEEP taking
the tablets?2. Is it to do with the nurse?
• Are nurses doing the DOT?• Is it dangerous / difficult to get to the patient?
3. Is it a problem of drug supply?
The defaulter rate is an important indicator for measuring quality of treatment. But what are its limitations? Take a closer look at the definition of ‘default’.
What does it mean if the program default rate is not on target?
Category Sputum Conversion StatusAt 2 months At 3 months
No % No %
All Smear +Cases
Treatment still ongoing
Converted to smear negative 29 23% 49 40%
Remaining smear positive 9 7% 7 6%
Smear results not available 75 60% 54 44%
Treatment discontinued
Died during treatment 1 1% 1 1%
Transferred to another unit 9 7% 10 8%
Defaulted from treatment 1 1% 3 2%
All smear + cases 124 100% 124 100%
All Smear +New
Treatment still ongoing
Converted to smear negative 28 23% 46 38%
Remaining smear positive 8 7% 7 6%
Smear results not available 73 61% 53 44%
Treatment discontinued
Died during treatment 1 1% 1 1%
Transferred to another unit 9 8% 10 8%
Defaulted from treatment 1 1% 3 3%
All smear + new cases 120 100% 120 100%
AllSmear +Re-treatment
Treatment still ongoing
Converted to smear negative 1 25% 3 75%
Remaining smear positive 1 25% 0 0%
Smear results not available 2 50% 1 25%
Treatment discontinued
Died during treatment 0 0% 0 0%
Transferred to another unit 0 0% 0 0%
Defaulted from treatment 0 0% 0 0%
All smear + new cases 4 100% 4 100%
Botswana Tuberculosis Programme
Sputum Conversion ReportReport on Response to Initial Phase Tuberculosis
Treatment
15 – GABORONE Quarter 1 of 2004
Formula
SS+ converting to smear negative
SS+ registered cases
Example
29= 23%
124
The Sputum Conversion RateAre enough patients converting to smear negative at the end of 2 months of treatment?
To do this we use a conversion Rate
Equal to or greater than
85%
What can it mean if sputum conversion rate is under 85%?
Equal to or greater than
85%
0
10
20
30
40
50
60
70
80
90
100
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Sputum Conversion (%)
1. The drug treatment regimen may not be working• Are the correct regimens being
used?• Could MDR-TB be an issue?
2. Patients are not taking their TB medications• Are the nurses doing their job?• Is Directly Observed Treatment
actually happening?• Are patients scared of taking
the tablets?
What kind of detective work would you do to find the cause?
Formula
No of registered cases with test taken
Total no of registered cases
Example
6= 84%
7
Proportion of notified cases tested for HIV:
What % of TB cases were tested for HIV?
100%
HIV StatusResult
Date
Test 1 Test 2
N
P
N
N
N
N
6
RegNumber
Reg
isteredD
ate (m
md
dyy)
001/05 01/01/05
002/05 01/02/05
003/05 01/03/05
004/05 01/04/05
005/05 01/05/05
006/05 01/06/05
007/05 01/07/05
7
What can it mean if less than 100% of patients are tested for HIV?
1. Lack of training among clinical staff
2. Shortage of test kits3. High refusal rates among
patients4. Other factors requiring
consultation with the TBFP or relevant clinical staff
100%
Low testing rates may indicate that the policy of routine HIV testing has not been implemented properly in a site or district.
Analysing the treatment card
• Patient’s information is complete• Patient’s DOT and weights correctly recorded• All HIV Status results are correctly recorded• For HIV-infected patients
– Receipt of ART noted?– Receipt of IPT?
Reviewing the Treatment Card
We have: An understanding of the principles of M&E Some experience and guidelines you can take
away with you on good ways to: Observe Communicate Analyse
By now (hopefully)…
Next steps
In the next session we will apply these practices to the supervision checklist, which means we have to: Use the whole supervision checklist Practice communicating with actual field staff Interpret actual data to develop indicators Observe the environment Develop a list of action-items
Thank youand
Good Luck!