![Page 1: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/1.jpg)
Insert name of presentation on Master Slide
Trials and Tribulations of Monitoring and Evaluation
7th July 2011
Presenter: Dr Kerry A Bailey MBBS BSc MSc DLSHTM MRCGP FFPH
![Page 2: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/2.jpg)
OutlineIntroductionsDefinitionStepsExamples10.55am – small group discussion11.15am – feedback and
discussion11.25 am summary11.30 am coffee
Trials and tribulations of M and E
![Page 3: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/3.jpg)
Pennard Africa Link Swaziland PALS Introductions – experience, any hoped for learning points
![Page 4: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/4.jpg)
Insert name of presentation on Master Slide
definitionevaluation‘A process that attempts to
determine as systematically and objectively as possible the relevance, effectiveness, and impact of activities in the light of their objectives’
Last Dictionary of Epidemiology
![Page 5: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/5.jpg)
Are we doing what we planned?Are we doing more good than harm?Can we show to others that we are?
Who are ‘others’?- funders- staff on the ground- other link members- other links- publication
![Page 6: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/6.jpg)
Monitoring vs evaluationMonitoring: routine tracking of service and
programme performance using input, process and outcome information collected on a regular and ongoing basis …is used to assess the extent to which a policy or programme is achieving its intended activity targets on time.
VersusEvaluation: episodic assessment of results
that can be attributed to programme activities; it uses monitoring data and often indicators that are not collected through routine information systems. Allows exploration of causes…
WHO, 2004
![Page 7: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/7.jpg)
Steps in developing M&E plan
Trials and Tribulations of M& E
Engage appropriate stakeholders
Identify goals/objectives of programme
Develop M&E framework
Define & select indicators
Identify sources & collect data
Analyse & interpretDisseminate & use
Talk to right peopleWhat are you trying to
do?How can you show
you are doing that?Routine data,
numbers interviews?process, outputs or
outcomes Information for action
Adapted from WHO 2004/CDC
![Page 8: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/8.jpg)
Programme evaluationTypes
◦ Formative evaluation (development)◦ Process evaluation (implementation)◦ Impact evaluation (achieving goals)
Wynn BO, Dutta A, Nelson MI, Challenges in Programme Evaluation of Health Interventions in Developing Countries, RAND, 2005
A guide to monitoring & evaluating TB/HIV systems, WHO, 2004
![Page 9: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/9.jpg)
Health Service EvaluationDonebedian
Structure – often qualitativeInputsProcessOutputsOutcomes – 5 Ds – death, disability,
disease, dissatisfaction, discomfortMaxwell – quality
Effectiveness, equity, efficiency, accessibility, acceptability ( also retrospective vs prospective)
![Page 10: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/10.jpg)
Practical examples
![Page 11: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/11.jpg)
Swaziland Sub Saharan Africa Population 1 million Size of Wales Absolute monarchy, King appoints prime
minister, cabinet, judiciary, controls media
Lowest life expectancy in the world – 33.7 years
HIGHEST HIV rate in the world (26% DHS, 2007)HIGHEST TB rate in the world (1262/100,000 (WHO,2007)69% live below the poverty line78% < $2 a day, 48% < $1 a day (UNDP 2007)
![Page 12: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/12.jpg)
ExamplesHIV Testing and
counsellingINH prophylaxis
as an adjunct to TB screening programme
![Page 13: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/13.jpg)
Examples of Evaluation of Programmes 1. HIV testing and counselling HTC – Prospective, evaluation not specifically funded
Background - WHO guidance(2007): HTC rather than VCT
Vision – All rural clinics to HIV test (prior to this not testing), phlebotomy and TB screen, regular specimen transport with driver trained up to do all to support
- Formative – qualitative: interviews with partners, clinics, nurses, regional supervisors, matron – emphasized transport of samples not people
- funder identified, proposal, M and E plan- Process outcomes – (1st ¼) numbers of nurses
trained, number lay people appointed and trained in testing, counselling and phlebotomy, vehicle attained
qualitative – interviews with staff (BSc student)
![Page 14: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/14.jpg)
challengesWhat - Routine data?
monthly clinic reportsVs Test kit orders?How Separate booksAccess file at main
baseResults structure
( team going out to clinics to assist, counsellor in hosp), process (nurses trained) and outputs
- when
![Page 15: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/15.jpg)
Impact Numbers of people tested for HIV and breakdownnumbers of CD4s being reported to patient
(previously a problem) (rose from <40% to 93%) average CD4 count (now mean 437, median 371)
monitoring should show increaseNumbers initiating on ART
Numbers tested by HTC in rural clinics
0100200300400500600700
month 08-09N
umbe
r of
pat
ient
s te
sted
![Page 16: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/16.jpg)
INH prophylaxisEvidence,
discussions, politics
‘feasibility study’3 clinics, higher
intensity of support
Early clinics demonstrated the importance of this
![Page 17: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/17.jpg)
Q.1.Have you had a cough for more than two weeks?
YES
If NO to allIf YES to 2 or more
•Give sputum bottles•Stamp card and date•Refer for clinical assessment•Cough hygiene
If YES to one“Screen Positive”
“Screen Negative”
•Clinical r/v•Treat antibiotics and/or investigate (CXR)•Repeat 5 question screen after 2 weeks
Prescribe ISONIAZID(INH) ONLY when all 5 answers are ‘no’
Pregnant?Liver disease?Delivered within 3 months?
If NOIf yes
Sputum Negative
May still have TB - repeat screening questions
If screen negative r/v after 2 weeks and follow flow chart Still “Screen
Positive”
If YES to one
refer CXR & medical r/v
If NO ask:Q2 Have you had noticeable weight loss in the last month?Q3 Have you had night sweats > 3 weeks?Q4 Have you had persistent fever? (>3 wks)Q5 Do you have chest pain? (>3 weeks)
Sputum Positive
Refer to diag centre for TB TREATMENT(re emphasize cough hygiene)
Swaziland Cough Screening -Good Shepherd Hospital PilotAll attendees at ART are screened on first presentation, and 1 monthly intervals
![Page 18: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/18.jpg)
REALITY CHECKRolled out too quicklyMisunderstandings,
poor data – although seemed to be good
But don’t assume data is robust
Results seemed good – but could have been doing dreadful harm
HIV +ve Number screened
Screen negative
INH
100 90 50 40
![Page 19: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/19.jpg)
Key PointsQualitative alone –
rarely enoughQuantitative alone –
understand qualityProcess and output data
is only part of the storyOnly collect what is
absolutely necessary Information for Action -
don’t just disseminate link to teaching, peer review, expert discussion opportunities
![Page 20: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/20.jpg)
Your turnIn 2-3 peopleIs there a project
you are running/planning
Have you an evaluation plan
Discuss/reflect
![Page 21: Trials and Tribulations of Monitoring and Evaluation Insert name of presentation on Master Slide 7 th July 2011 Presenter: Dr Kerry A Bailey MBBS BSc MSc](https://reader038.vdocuments.us/reader038/viewer/2022110100/56649ddc5503460f94ad40a9/html5/thumbnails/21.jpg)
AcknowledgementsDr Ciaran HumphreysProf John WrightProf John WalleySabelo Nkwanazi Dr Canaan MamvuraGcinaSteven LukheleKwanele DlaminiAll the GSH TB team