quality assurance & accountability project · routinely and systematically monitored and...

Date: 25th April, 0930-1600 | 26th April; 09:30-1430 CEST

Location: Solidarites International, 89 Rue de Paris, 92110 Clichy, France

Attendees: Oxfam: Eva Niederberger, James Brown

SI: Aude Lazzarini, Laurène Barlet

GWC: Franck Bouvet

UNICEF: Carla Daher, Masumi Yamashita

Groupe URD: Lisa Daoud

ACF: Tom Heath

Objectives:

• Review progress made to date, identify challenges and ways forward

• Discuss conceptual models and agree how they should be applied in practice

• Agree how the project can be communicated to external stakeholders

• Develop project roadmap based on new timescale

• Agree planning for South Sudan, Colombia and key outputs

Day 1:

For overview, see p.3

Day 2: Project planning

MA

Y

GBV Stocktaking Workshop

First ro

un

d o

f co

un

try

vis

its

Mo

nth

ly Q

AA

P T

WG

Ca

lls /

Up

date

s

South Sudan Visit 1

Guidance note table of contents

Literature review final draft

JU

N

Colombia visit

JU

L

Re

mo

te

su

pp

ort

AU

G

Guidance note draft 1

SE

PT

Phase II concept note

Se

con

d r

oun

d o

f

co

un

try v

isits

OC

T

Guidance note draft 2

Communications strategy

NO

V

DE

C

Guidance final draft

Quality Assurance & ACCOUNTABILITY PROJECT:

TEAM REVIEW MEETING NOTES

GBV Stocktaking workshop:

The QAAP will be presented at the Stocktaking workshop: Lessons on promoting the dignity, safety,

health and privacy in humanitarian action organised by the GBV AoR, the IASC GBV Guidelines

Implementation Team, IOM and UNICEF. This will be an opportunity to raise awareness of the QAAP

project amongst participants (including shelter, CCCM and WASH sector coordinators in Bangladesh,

Iraq, Nigeria, Somalia and South Sudan) and to learn from existing GBV risk mitigation projects and

good practice.

South Sudan Visit 1:

The visit to South Sudan will be conducted by James and Helen from 20-31st May. The team will visit

field locations in Malakal, Mingkaman and Mangateen and further develop the quality analytical

framework.

Guidance note:

The QAAP Guidance note is a key output of the project and will be drafted progressively between

May and December 2019. An initial table of contents (ToC)will be proposed at the end of May,

followed by a first draft in August, a second draft in October and a final draft in December. The QAAP

TWG will have the opportunity to review and comment on these drafts as they are developed.

Colombia visit:

The Colombia visit will take place between 2-9 June.

Literature review:

The literature review will be circulated to the QAAP Team and TWG in mid-June for comment.

Phase II concept note:

Achieving the long-term behaviour change amongst NHWCPs and partners to ensure that quality is

routinely and systematically monitored and improved will require ongoing efforts. In order to secure

funding to continue the QAAP beyond 2019, a concept note setting the direction and objectives for a

QAAP Phase 2 will be drafted in September.

Communications strategy:

Ensuring that the findings, guidance and recommendations of the QAAP are effectively disseminated

to different stakeholders will be an important driver of the overall project impact. The communications

strategy developed by October will ma key stakeholders and identify communications and

engagement channels for each. The communications strategy will be used to plan dissemination and

engagement opportunities over the final quarter of the project. The QAAP TWG will be engaged

through monthly calls and updates from June.

PARIS MEETING PRESENTATION

QUALITY ASSURANCE &

ACCOUNTABILITY PROJECTREVIEW MEETING: PARIS

25th April 2019

Page 2

SESSION PLAN: THURSDAY

MORNING

0930 Introduction:

• Objectives setting

• Progress overview

1000 Conceptual models of quality

• Presentation of literature review

• Discussion around different models

1100 Break

1130 Presentation and discussion of the two field visits

• Bangladesh

• Myanmar

1230 Lunch

AFTERNOON

1330 Presentation and review of quality frameworks

• Process centred (Bangladesh)

• Outcome monitoring (Myanmar)

1430 Break

1500 Communication, dissemination

• Engagement of TWG

• Working with other initiatives

• Wider communications

SESSION PLAN: FRIDAY

MORNING

0930 Day 1 review

1000 Project road map

• Guidance and advocacy

• Second round of visits

• Remote support to countries

1100 Break

1130 Planning for South Sudan, Colombia

1230 Lunch

AFTERNOON

1330 Wrap-up

1430 Break

1500 Flexible time

OBJECTIVES

THURSDAY

• Review progress made to date, identify challenges and ways forward

• Discuss conceptual models and agree how they can be applied in practice

• Review quality analytical frameworks and agree way forward

• Agree how the project can be communicated to external stakeholders

FRIDAY

• Project roadmap

• Planning next steps

Page 5

PROGRESS REVIEW

DESK REVIEW

• Draft to be shared after this meeting

FIELD VISITS

• Bangladesh, Myanmar completed

• South Sudan (May)

• Colombia (June)

QUALITY FRAMEWORK

• 2 approaches developed

Page 6

ACTIVITY REVIEW

• Pre-visit phase• Inception meeting

• Literature review

• Develop concept sketch protocol

• Pre-visit preparation

• Visit each country – Visit 1• Initial meetings

• Make protocol specific

• Perhaps do an assessment

• Between visits• Provide external support on quality monitoring

process in contexts

• Provide tools for the contexts to monitor the process

• Visit each country – Visit 2• Review activities and monitoring data

• Continue to support protocol for quality monitoring

• Do assessment (if not completed during first visit)

• Post-visit phase• Final meeting

• Reports, guidance and Advocacy

CONCEPTUAL MODELS

Page 8

DMAIC

DEFINE: Objectives and processes (WHAT & HOW)

• Strategic operational framework (SOF)

• Humanitarian Response Plan (HRP)

• Contextualised standards document

MEASURE:

• Data collection, sharing

ANALYSE:

• Response level, identify priority issues and root causes,

control/influence/accept

IMPROVE:

• Jointly agree corrective action to control / influence

• Implement improvements

• Continue to MEASURE changes

CONTROL:

• Continue to monitor

• Capture lessons, adaptations to response documents

The Define – Measure – Analyse – Improve – Control (DMAIC) methodology is a structured

problem-solving technique that builds upon the PDCA cycle. It is one of the core tools used

in the Six Sigma quality management approach developed by Motorola in 1980 and widely

used in manufacturing and business processes. The slide shows how this process could be

adapted to the humanitarian WASH sector as the basis of a Quality Assurance System.

THE DONABEDIAN MODEL

Structure Coordination Resourcing EvidenceOrganisational

capacity

Process Needs assessment

Analysis Project design Implementation

Outcomes Activities Outputs Outcomes Impact

Avedis Donabedian was a physician and founder of the study of quality in health care and

medical outcomes research. His approach for evaluating the quality of care is widely used

and conceptualises the measurement of quality into three components: structure, process

and outcomes. In this model, outcome measures are the ultimate validators of quality,

however they are affected by both process factors and structural factors in a causal chain

(examples given in the slide are not exhaustive).

Page 10

THE SERVQUAL MODEL

The SERVQUAL Model is an approach to measuring quality in the service sector, from the

perspective of customers or service users. The model comprises of a conceptual model and

a questionnaire which is administered to users to quantify their expectations and experience

of a service along different dimensions. The model has been applied and adapted widely in

a variety of contexts to measure the quality of services.

The five dimensions of service quality were derived from a systematic analysis of over 100

aspects of quality initially proposed in literature. The model was eventually reduced to five

dimensions found to be independent, stable and robust. They are defined as:

Reliability: the firm’s ability to perform the promise service accurately and dependably

Assurance: knowledge and courtesy of employees and their ability to inspire trust and

confidence

Tangibles: physical facilities, equipment and appearance of personnel

Empathy: caring and individualized attention paid to customers

Responsiveness: the firm’s willingness to help customer and provide prompt service

Page 11

THE “QUALITY CHAIN”

Links the processes in the project cycle with the results chain

The quality of what we do it throughout the project cycle influences what is

achieved throughout the results chain

Impact is the “ultimate validator of quality” but…

The further down the results chain, the more difficult to measure

Measuring outcomes doesn’t diagnose the causes of low quality

The level of process monitoring we do of the project cycle affects our

confidence in the assumptions made in the results chain

The quality chain can be proposed as a way to link ‘quality of process’ with ‘quality of results’

and shows that outcomes are somewhat dependant on the quality of assessment, analysis,

design and implementation.

Page 12

EA

SIE

R T

O M

EA

SU

RE

HIG

HE

R C

ON

FID

EN

CE

ASSUMPTIONS MADE AT

EACH STAGE:

(POTENTIAL ‘QUALITY

GAPS’)

If impact is the ultimate measure of quality, the further down the quality chain we can

measure the more confident we can be in our understanding, because there are fewer

assumptions needed (the measure is more direct). Monitoring results further down the

quality chain, however, becomes more challenging as output-level results typically rely on

making measurements of behaviours and perceptions. Attribution of outcomes to response

activities may also be difficult to demonstrate.

Page 13

THE ‘ENABLING ENVIRONMENT’:

• Resources

• Coordination

• Organisational capacity

• Evidence / Learning

The above diagram illustrates how the enabling environment, project cycle and results

causal chain are related, and how measuring of processes and measuring of results fit into

the system. Process monitoring of the project cycle (right hand side) can be carried out

through the use of checklists as proposed by URD 2018 in the COMPAS handbook (other

process checklists have been developed for example ACF’s Project Quality Management

Tool). These checklists measure to what degree each process in the project cycle has

followed a list of important actions or considerations.

Results are monitored at different stages along the causal chain, from activity monitoring

(e.g. construction supervision) through to impact evaluations.

Information from each level of monitoring should be fed-back to inform and improve different

stages in the project cycle. For example:

Supervision of activities, quality control and output monitoring is fed back to improve

implementation and provide progress updates e.g. if latrines are being constructed poorly,

additional capacity building of daily labour may be required

Outcome monitoring feeds back to inform changes in project design e.g. if latrines

constructed are not being used, changes may need to be made to the siting or design of

superstructures.

Impact monitoring feeds back to inform an understanding of needs e.g. increasing incidence

of diarrhoea may change our prioritisation of public health risks in a particular location

Monitoring of results alone does not necessarily provide sufficient information to design

corrective action but combined with process monitoring using agreed checklists they can

guide the first steps in diagnosing problems.

FIELD VISITS

Page 16

BANGLADESH: COX’s BAZAR

• Visited 9 camps in Kutupalong

• Interviewed 26 KIs (16 orgs)

• Workshop with 40 participants (25

orgs)

DRAFT ACTION PLAN

ADAPTED QUALITY FRAMEWORK

• Limited follow-up from country

THE HUMAN LENS

PARTICIPATION,

INCLUSION,PROTECTION

THE RESOURCE LENS

EFFICIENT, PRIORITISED, SUSTAINABLE

THE TECHNICAL LENS

APPROPRIATE,EFFECTIVE, RESPONSIBLE

SUPPORTING ENVIRONMENT

COORDINATED, SUPPORTED, RESOURCED,

INFORMED

Principle Criteria Example practices from Bangladesh Example indicators for Bangladesh

Participation: The WASH response is based upon meaningful participation

with persons of all gender identities, ages, disabilities and backgrounds

Sector-standard designs for WASH facilities are based on the results

of consultation

% Communities express satisfaction that WASH actors have listened to their feedback

and made changes in programme where possible; % of WASH Committees fulfilling

requirements of minimum representation by people from different groups; # of people

who have been involved in community consultations (disaggregated by gender, age,

disability

Feedback: The WASH stakeholders seek and act upon where necessary ,

feedback and complaints from persons of all gender identities, ages,

disabilities and backgroundsFeedback and complaints collected by different agencies are

summarised at the sector level to inform sector approaches

Time between complaint - action - feedback; Number of complaints by theme and

gender/age/disability; # blocks / camps covered by complaints mechanisms; % people

who can identify an appropriate feedback mechanism;

Communication: The WASH stakeholders communicate clearly and

respectfully with affected people, including those with physical, sensory and

intellectual disabilities, persons with mental health disabilities and older

people WASH field workers are able to receive and refer feedback and

answer frequently asked questions in an appropriate language

# beneficiaries per community volunteer / field staff; % field staff speaking

Rohingya/Chittagonian; % Communities report that key information is clearly

communicated in appropriate languages and reaches all sections of the community using

context specific channelsAccess: The WASH stakeholders provide equitable access to information,

services and resources to all affected persons

Access to WASH services is monitered and results disaggregated by

sex, age, disability so that gaps may be identified Latrine use by gender, age, disability; People per latrine by gender, age, disability;

Safety: The WASH stakeholders prioritise do no harm; actively monitor and

addresse the safety of affected persons

Safety risks and mitigation measures in place are monitered and

results disaggregated by sex, age, disability

Safety perception; use of facilities by day/night; 'safe' standard designs (safety audit

using latrine checklist); % of women and girls who feel safe using a water facility

Adapted: The WASH response is based upon a systematic, objective and

ongoing analysis of the context and an impartial assessment of needs, risks,

vulnerabilities and capacities of different groups

The WASH sector collects and shares needs assessments

conducted by partners;

The WASH response plan includes strategies for responding to the

different needs of vulnerable groups

Vulnerable groups are identified and used to disaggregate data collection; Where access

for some groups is insufficient, or inequitable, action is taken to address specific needs /

barriers; % of women, girls, elderly or mobility impaired people who cannot safely

access latrines, provided with alternative safe sanitation options (adult potties, container-

based sanitation)

Prepared: The WASH response identifies future hazards and acts to mitigate

risks Preparedness plans have been developed at a sector / inter-sector

level for the priority risks identified

WASH sector has identified key hazards (monsoon, cyclone, AWD outbreak etc.) and has

developed plans for preparedness and immediate response, either as part of inter-sector

preparedness planning or individuallyTimely: The WASH stakeholders are responsive to changing needs,

capacities and contexts and make decisions and act without unnecessary

delay.

Alerts (full latrines, lack of water etc.) are addressed quickly;

Decommissioning of latrines is conducted without delay; Time between alert and action for desludging; water point repair; decommissioningStandards: The WASH response meets agreed technical humanitarian

standards based upon relevant evidence, guidance and best practice

(Sphere, national standards, or as appropriate)

Key Sphere / national WASH minimum standards have been

defined for the response, along with standard options for meeting

them and a monitoring framework to measure compliance; Relevant Sphere standardsIntegration: The WASH response appropriately integrates with different

sectors and approaches (health, cash, community engagement...) in order to

achieve programme objectives

Data from health sector is used to monitor WASH disease and

integrated actions are taken in the case of an outbreak; Market-

based approaches are considered for WASH NFIs;

Environmental: The WASH response identifies, communicates and mitigates

the risk of negative impacts to the environment from WASH interventions

Aquifer monitoring - water levels and water quality (F.Coli / chemical)

FSM effluent monitoring - BOD/COD of effluent discharged above

ground

BOD/COD of FSM effluent discharged above ground; Presence of OD 'hotspots'; Static /

dynamic water level at tubewell / borehole; Water quality at deep and shallow aquifer; Controlled: The WASH response includes sufficient quality control

mechanisms to identify deficiencies in design and implementation of WASH

services Supervision of construction activities; Infrastructure audit; % of latrines that pass inspection before being opened to use;

Transition: The WASH response transitions as early as possible away from

emergency approaches towards those that are more efficient based on an

analysis of the life-cycle costs

Identify indicators and define indicators for phasing of approaches

depending on emergency phase in master plans; Ensure key

transition activities (e.g. decommissioning of emergency tube wells

and latrines) is done in consultation with the local community. % of new facilities constructed that meet guidance on transitional approaches

Collaboration: The WASH response is developed and run in collaboration with

local communities and local actors, building capacity where required

Identify government and community stakeholders and ensure they

are included in sector strategy and decision making; Jointly identify

and plan for capacity building of local actors; % of new facilities constructed with a handover plan agreed with local service providersIndependence: The WASH response is designed to reduce dependence on

imported equipment, chemicals and spare parts, fuel supplies, or staff with

specialised skills not available locally

Carry out a market assessment to identify what is available at local /

national level; Identify any key items required for O&M that are not

available locally;

% of water supply systems (boreholes, treatment, pumping, distribution) where O&M

can be handed over to local service providers / authorities

HU

MA

N L

EN

ST

EC

HN

ICA

L L

EN

SR

ES

OU

RC

E L

EN

S

Page 23

MYANMAR: RAKHINE

• Visited 4 camps in Sittwe and

Pauktaw

• Interviewed 36 KIs

• Workshop with 14 participants (11

orgs)

DRAFT ACTION PLAN

COMMITMENT FROM CLUSTER

ADAPTED QUALITY FRAMEWORK

During the Myanmar mission an action plan was developed jointly with the Myanmar WASH Cluster to begin using the framework for Rakhine level reporting from the beginning of the Q3 period. Remote support will be provided to enable the cluster to achieve this.

Page 24

MODULAR APPROACH

CoordinationPublic Health

ProtectionCommunity

Engagement

x y z

FRAMEWORK COMPONENTS

During the Myanmar visit an alternative framework was developed to reduce the number of indicators and to focus measurements on core outcomes. The objective of this framework is to:

• SIMPLIFY getting key information from the field level to coordination

• FOCUS on tracking the most important indicators of success

• CLARIFY the minimum expectations for each stage of the programme cycle This framework will be tested and further developed during the South Sudan and Colombia missions.

Page 26


• Focus area objective

• Key Performance Indicators (KPI):

• What are the most important measures of success?

• What information do we need to design programmes, trigger alerts, take

corrective actions?

• What is useful to track over time?


• Expectations for “good quality” processes, actions:

• Programme phase Assessment Planning Implementation Monitoring

• Based on existing guidance

• Aim to be practical to carry out

• Critical questions to support the KPIs

Page 28

GENERAL FINDINGS

• 'Quality’ is already being measured to some degree

• Limited harmonisation of approaches

• Limited sharing response-level analysis

• Call for strengthened compliance role for cluster / sector

• Different levels of engagement with coordination platform

• Good knowledge of standards, but limited use in monitoring

quality assurance & accountability project · routinely and systematically monitored and...

Documents