Insert name of presentation on Master Slide

Trials and Tribulations of Monitoring and Evaluation

7th July 2011

Presenter: Dr Kerry A Bailey MBBS BSc MSc DLSHTM MRCGP FFPH

OutlineIntroductionsDefinitionStepsExamples10.55am – small group discussion11.15am – feedback and

discussion11.25 am summary11.30 am coffee

Trials and tribulations of M and E

Pennard Africa Link Swaziland PALS Introductions – experience, any hoped for learning points

Insert name of presentation on Master Slide

definitionevaluation‘A process that attempts to

determine as systematically and objectively as possible the relevance, effectiveness, and impact of activities in the light of their objectives’

Last Dictionary of Epidemiology

Are we doing what we planned?Are we doing more good than harm?Can we show to others that we are?

Who are ‘others’?- funders- staff on the ground- other link members- other links- publication

Monitoring vs evaluationMonitoring: routine tracking of service and

programme performance using input, process and outcome information collected on a regular and ongoing basis …is used to assess the extent to which a policy or programme is achieving its intended activity targets on time.

VersusEvaluation: episodic assessment of results

that can be attributed to programme activities; it uses monitoring data and often indicators that are not collected through routine information systems. Allows exploration of causes…

WHO, 2004

Steps in developing M&E plan

Trials and Tribulations of M& E

Engage appropriate stakeholders

Identify goals/objectives of programme

Develop M&E framework

Define & select indicators

Identify sources & collect data

Analyse & interpretDisseminate & use

Talk to right peopleWhat are you trying to

do?How can you show

you are doing that?Routine data,

numbers interviews?process, outputs or

outcomes Information for action

Adapted from WHO 2004/CDC

Programme evaluationTypes

◦ Formative evaluation (development)◦ Process evaluation (implementation)◦ Impact evaluation (achieving goals)

Wynn BO, Dutta A, Nelson MI, Challenges in Programme Evaluation of Health Interventions in Developing Countries, RAND, 2005

A guide to monitoring & evaluating TB/HIV systems, WHO, 2004

Health Service EvaluationDonebedian

Structure – often qualitativeInputsProcessOutputsOutcomes – 5 Ds – death, disability,

disease, dissatisfaction, discomfortMaxwell – quality

Effectiveness, equity, efficiency, accessibility, acceptability ( also retrospective vs prospective)

Practical examples

Swaziland Sub Saharan Africa Population 1 million Size of Wales Absolute monarchy, King appoints prime

minister, cabinet, judiciary, controls media

Lowest life expectancy in the world – 33.7 years

HIGHEST HIV rate in the world (26% DHS, 2007)HIGHEST TB rate in the world (1262/100,000 (WHO,2007)69% live below the poverty line78% < $2 a day, 48% < $1 a day (UNDP 2007)

ExamplesHIV Testing and

counsellingINH prophylaxis

as an adjunct to TB screening programme

Examples of Evaluation of Programmes 1. HIV testing and counselling HTC – Prospective, evaluation not specifically funded

Background - WHO guidance(2007): HTC rather than VCT

Vision – All rural clinics to HIV test (prior to this not testing), phlebotomy and TB screen, regular specimen transport with driver trained up to do all to support

- Formative – qualitative: interviews with partners, clinics, nurses, regional supervisors, matron – emphasized transport of samples not people

- funder identified, proposal, M and E plan- Process outcomes – (1st ¼) numbers of nurses

trained, number lay people appointed and trained in testing, counselling and phlebotomy, vehicle attained

qualitative – interviews with staff (BSc student)

challengesWhat - Routine data?

monthly clinic reportsVs Test kit orders?How Separate booksAccess file at main

baseResults structure

( team going out to clinics to assist, counsellor in hosp), process (nurses trained) and outputs

- when

Impact Numbers of people tested for HIV and breakdownnumbers of CD4s being reported to patient

(previously a problem) (rose from <40% to 93%) average CD4 count (now mean 437, median 371)

monitoring should show increaseNumbers initiating on ART

Numbers tested by HTC in rural clinics

0100200300400500600700

month 08-09N

umbe

r of

pat

ient

s te

sted

INH prophylaxisEvidence,

discussions, politics

‘feasibility study’3 clinics, higher

intensity of support

Early clinics demonstrated the importance of this

Q.1.Have you had a cough for more than two weeks?

YES

If NO to allIf YES to 2 or more

•Give sputum bottles•Stamp card and date•Refer for clinical assessment•Cough hygiene

If YES to one“Screen Positive”

“Screen Negative”

•Clinical r/v•Treat antibiotics and/or investigate (CXR)•Repeat 5 question screen after 2 weeks

Prescribe ISONIAZID(INH) ONLY when all 5 answers are ‘no’

Pregnant?Liver disease?Delivered within 3 months?

If NOIf yes

Sputum Negative

May still have TB - repeat screening questions

If screen negative r/v after 2 weeks and follow flow chart Still “Screen

Positive”

If YES to one

refer CXR & medical r/v

If NO ask:Q2 Have you had noticeable weight loss in the last month?Q3 Have you had night sweats > 3 weeks?Q4 Have you had persistent fever? (>3 wks)Q5 Do you have chest pain? (>3 weeks)

Sputum Positive

Refer to diag centre for TB TREATMENT(re emphasize cough hygiene)

Swaziland Cough Screening -Good Shepherd Hospital PilotAll attendees at ART are screened on first presentation, and 1 monthly intervals

REALITY CHECKRolled out too quicklyMisunderstandings,

poor data – although seemed to be good

But don’t assume data is robust

Results seemed good – but could have been doing dreadful harm

HIV +ve Number screened

Screen negative

INH

100 90 50 40

Key PointsQualitative alone –

rarely enoughQuantitative alone –

understand qualityProcess and output data

is only part of the storyOnly collect what is

absolutely necessary Information for Action -

don’t just disseminate link to teaching, peer review, expert discussion opportunities

Your turnIn 2-3 peopleIs there a project

you are running/planning

Have you an evaluation plan

Discuss/reflect

AcknowledgementsDr Ciaran HumphreysProf John WrightProf John WalleySabelo Nkwanazi Dr Canaan MamvuraGcinaSteven LukheleKwanele DlaminiAll the GSH TB team

Siyabonga Thank you

kerrybaileyjones@gmail.com