Participatory performance monitoring of WASH services at scale in BRAC WASH Programme

1

Monitoring Sustainable WASH Service DeliveryAddis Ababa, Ethiopia

9-11 April, 2013

Mahjabeen AhmedBRAC

An Introduction to BRAC

BRAC Coverage in Bangladesh:

Districts 64

Field Offices 2661

Pop. Covered 113 Million

2

BRAC’s presence in the world

BRAC WASH Programme

The Programme started in 2006 with 150 sub-districts, and has reached 248 sub-districts till date.

Components Target (in million people)

Achievements till Dec 2012 (in million people)

Water 2.5 1.97

Sanitation 35 27.81

Hygiene 51 42

3

Key Programme Strategies

• Creating demands

• Establishing Village WASH Committees

• Strong interpersonal communication component to change behaviours

• Tailored support to ensure that hardcore poor (grants) and poor (soft loan) are reached.

• To meet the demand supporting Rural Sanitation Centres (loans & orientation)

• Stimulate innovation through action research programme

4

Monitoring Methodology

• WASH I had MIS to monitor inputs and outputs

• Inputs: e.g., number of visits, trainings

• Outputs: e.g., no. of VWCs established, no. of toilets built, of different types

• WASH II also needed performance monitoring: that is, how well toilets are used; how well VWCs continue to perform; to what extent women are integrated in planning and management; etc.

Now: MIS + QIS (quantified Qualitative Information System)

5

History & Rationale

• Quantified qualitative assessment methodology

• First developed by IRC and WSP in 1998

• Aim: to replace surveys, because they are extractive and inform only central management and donors, not the users, the VWCs and field workers

• QIS:

1. visualises to all participants where they perform well and where they can improve (‘climb the ladder’)

2. produces statistics that inform management and donors on progress of the whole programme

3. allows to compare results over time and between locations

6

Scoring methodology

• QIS uses Likert scales: Participants score on a scale from 1-5, in which 1 is lowest and 5 is highest.

• Two differences: Each scale consist of “progressive mini-scenarios” Each scale starts at 0, not at 1

• Participants can see their level and can climb from 0 (“nothing to show”) to 4 (“the ideal of 4 key measurable criteria”). The scores can be analysed statistically.

• A scale consists of “no x, x+1, x+1+1, x+1+1+1 and x+1+1+1+1” in which each 1 is a criterion for the indicator.

7

Example of a QIS scale

8

Indicator 1: PERFORMANCE OF VWC SCORE

IDEAL: (1) Committee (male and female members) meets every 2 months + (2) maintains list of decisions and meeting minutes + (3) identifies gaps and takes action + (4) mobilizes ADP funds for hard core poor

4

(1) Committee (male and female members) meets every 2 months + (2) maintains list of decisions and meeting minutes + (3) identifies gaps and takes action

3

BENCHMARK: (1) Committee (male and female members) meets every 2 months + (2) maintains list of decisions and meeting minutes

2

(1) Committee (male and female members) meets every 2 months 1

No full VWC, OR, VWC exists but does not meet 0

Reason for high/low score:

QIS development process (1)

• Workshop 1 (Jan 2012, 1 week) with WASH staff from HQ, all regional program managers(20) & IRC advisors: Set indicators Formulate & review scales Define terms Design scoring process Finalize scales & work plan

• Workshop 2 (March 2012, 1 week) First pilot training & field testing of QIS

9

QIS development process (2)

• Adjustment of scales, process and manual (April 2012)

• Full-scale pilot (August-Sept 2012) with 432 households (144 UP/P/NP, 36 VWCs, 12 schools and 12 RSCs in 4 upazilas in 4 geographical zones

• Workshop 3 (Oct 2012, 1 week):

Analysis of data & experiences and report

Adjustment of scales and manual

Selection of independent monitoring teams

Sample design: sample frame, size, sampling methods

10

QIS development process (3)

• Training of 30 independent monitoring teams (Dec, 2012):

1 male and 1 female in each team, 3 batches for 6 days

• 3-stage sample: (1) Random choice of 50 Sub districts from WASH I and 50 new unions from WASH II with probability proportional to size

(2) Minimally three or 3 VWCs/ sub-district

(3) 27 HHs per VWC (3x9 HH for UP/P/NP)

•

11

8100 HHs 400 schools 300 VWCs 300 RSCs

The QIS indicators for BRAC WASH

Village WASH Committee 1. Safe and protected drinking water source2. Performance of VWC3. Women’s participation

Households 4. Safe and protected main drinking water source5. Drinking water management from source to cup6. Sanitary and hygienic household latrine7. Who uses the latrine8. When are latrines used9. Hand-washing provisions after defecation10. Sludge management when latrine pit is full

Schools 11. Sanitary and hygienic school toilets12. Student brigade13. Menstrual hygiene management14. Performance of school WASH committee

Rural sanitation centers 15. Performance of RSCs

12

Data collection and analysis

• 1 male & 1 female staff collected data

• Data directly entered into a smart phone

• Data sent by smart phones to the QIS data base

• No data entry persons needed in Dhaka

• ICT and WASH staff cleaned the database

• BRAC’s QIS manager and IRC’s QIS advisors did a first analysis of the data using Epi-info open source software

13

Sample locations of: 8100 HHs 400 schools 300 VWCs 300 RSCs

14

• Scientific sampling methods

• Unique bar code avoids duplication when data is submitted more than once, and allows re-visit if needed

• Greater reliability through: No manual data entry (high error rate) Double data entry in smart phone and

on paper Team compared scores on internal consistency

before sending/uploading

15

Quality Assurance

Implementation and support

• The teams started the data collection just after the training

• Each team could always call Managers or ICT to solve any problems with data collection or smart phones

• Real time data

• The hard copies were checked rigorously to detect errors & for improvements next time

• One person was called back for retraining when consistent data errors detected

• For gender equality, team members alternated roles of process facilitator and data entry

16

Summary of preliminary findings

17

• All but 1 VWC are functional and meet regularly

• 99% VWCs at or above bench mark

• Almost 1 in 3 VWCs meet all 4 criteria (meet 1x/2mnth, record decisions and minutes, identify gaps and take action, mobilise govt. funds for ultra poor)

BUT • Almost 70% can still improve

performance further by mobilizing government funds meant for the ultra poor

• 26% can also improve by noting gaps and taking actions

QIS Findings on VWC performance

PERFORMANCE OF VWC %

IDEAL: (1) M+F members meet every 2 months + (2) record decisions & minutes + (3) identify gaps & takes action + (4) mobilizes ADP funds for HCP

31

(1) M+F members meet every 2 months + (2) record decisions & minutes + (3) identify gaps & takes action

42

BENCHMARK: (1) M+F members meet every 2 months + (2) Record decisions and minutes

26

(1) M+F members meet every 2 months 0

No full VWC, OR, VWC exists but does not meet

1

TOTAL 100

18

Safe management of drinking water in homes

• Above benchmark, P and NP perform better than HCP (43%, 52%, 35%)

• 22% HCP, 27% P and 33% NP meet all 4 criteria on safe drinking water from source to cup

• Others (67% HCP, 67% P and 64% NP) still 1-4 risky DW management practices

HCP POOR NON-POOR0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

5% 4% 5%

19% 19% 19%

30%28%

21%

13%16%

19%

22%27% 33%

11%5%

4%

Nalevel 4level 3level 2 level 1level 0

BM

19

Hand Washing Provisions at toilet

IDEAL: (1) Enough water to wash hands carried or available in or near latrine + (2) soap/soap solution in plastic bottle at latrine + (3) water for hand-washing is from safe source + (4) there is a special hand-washing station

(1) Enough water to wash hands carried or available in or near latrine + (2) soap/soap solution in plastic bottle at latrine + (3) water for hand-washing is from safe source

BENCHMARK: (1) Enough water to wash hands carried or available in or near latrine + (2) soap/soap solution in plastic bottle at latrine

(1) Enough water to wash hands carried or available in or near latrine

(0) No provisions for hand-washing carried or available in or near latrine.

• Presence of hand wash facility as proxy for hand washing behaviour.

• Majority of scores at or above BM. HW provisions at toilets is still a focus area: 10% are still at the score 0.

20

QIS findings from the School

21

 

Toilets for girls Student brigade Menstrual

hygiene management

Above BM 80% 68% 68%

At BM 12% 20% 5%

Below BM 8% 12% 26%

Total 100% 100% 100%

Experiences

• Performance monitoring concept was new: VWCs, householders and field staff initially experienced it as evaluation/judgemental

• Once understood, the ‘climbing the ladder’ approach is much appreciated by all

• Small details could be easily captured

• Comparing written and phoned data showed need for some correction and is good learning for next round (Dec. 2013)

22

ধন্য�বা�দ Thank you

23