data quality in remote monitoring (mona fetouh, unoios)
TRANSCRIPT
Data Quality in Remote Monitoring
A comparative analysis of experiences in Somalia and Eastern Burma
Mona Fetouh (Co-Author and Presenter), Christian Balslev-Olesen (Co-Author), and Volker Hüls (Co-Author)
Remote Monitoring Humanitarian space restricted in many situations; risks
have increased Both delivery and monitoring of programmes affected Remote monitoring and cross-border approaches
required in a number of countries in recent years Increased reliance on local actors Reduced ability to collect and verify information Often results in compromises on data quantity & quality Although remote management is increasingly common,
evidence on best practice is still emerging
Comparable Contexts Somalia
Classic remote management situation Difficulty of access due to insecurity, heavy reliance on
national civil society (NGOs, communities) for aid delivery (still largely managed from neighbouring Kenya)
Risk to external monitors; monitoring relies heavily on local partners – risk of bias
Eastern Burma Hard to impossible to access from capital due to
government restrictions and long-running conflicts between ethnic groups in the East and the government
Heavy reliance on civil society (LNGOs, communities) for aid delivery that is managed from neighbouring Thailand
Little to no access for INGO staff, including national staff Risk to external monitors, monitoring relies heavily on local
partners – risk of bias
Different Data Environments (1)
Somalia:Main concern is availability of data. Little opportunity to collect data regularly,
even for local partnersLocal partners have varied capacity to
produce quality data, severe education gap results in overall low staff capacity
Focus of strengthening objective monitoring was through investment in independent systems (“third party verification”).
Different Data Environments (2) Eastern Burma:
Main concern is management and quality of data Data are abundant, both monitoring data and surveys Local partner capacity is varied but good – strong
ethnic and professional exchange with Thai border area; better access to education
Information remains in technical silos; local NGOs are largely confined to ethnic areas; access opportunities are often limited to a particular sector.
Few mechanisms for independent verification/ triangulation of data.
Focus of strengthening objective monitoring was on the quality and verification of information
Access is improving due to ceasefire agreements
Comparative Analysis Similar contexts require different
approaches Presentation with details on
Third party verification in SomaliaQuality assurance of monitoring information in
Eastern Burma Experiences presented are based on of
work of UNICEF (Somalia) and International Rescue Committee and the Border Consortium (Thailand/Burma)
Somalia – Third Party Verification (1) First Level: Information from partners and
networks Implementing Partner Reports are main source of
primary performance data Reviewed against:
Previous track record of partner / confidence level Specific concerns about partner performance Reporting complete and realistic? Specific issues flagged in or apparent from
requiring follow-up Comparison to occasional information from staff
contacts in the filed (email, telephone)
Somalia – Third Party Verification (2) Second Level: Third Party Verification
Flagged issues are scheduled for third party monitoring
3rd Party Systems use field monitors that are not affiliated with any implementing partner often outside of the aid business
And therefore can move more freely with less risk are not as qualified to judge details of implementation used mostly for verification of easily obtainable information
Third party monitors, are ‘blind’ tasked to avoid fabrication of reports
Were successfully tasked to track leakage of relief goods into markets, including quantities and pricing.
Somalia – Third Party Verification (3) Third Level: Follow up on concerns from Level 1 and 2
Third party information is kept confidential and assessed for the risk level of a particular performance issue.
Depending on risk level, issues are taken up with the partner:
without revealing source e.g. dedicated open monitoring mission at next opportunity
Main reasons for staggering: First level flags issues, but is in itself not sufficient for
reliable data Second level is costly, and can be targeted to only flagged
issues Third level is costly and not timely, and should only be used
with knowledge of problems Low-level third party networks have worked well in
Somalia for other purposes, e.g. for food price monitoring
Eastern Burma – Strengthening Monitoring Quality (1)
Correlating data in geographical information systems Main limitation to data correlation / triangulation is
sector-based systems (Health, Education, Relief information management systems)
Sector IMS are basis for cross-sectoral GIS solution Platform maps service delivery data of all sectors to
village location Allows analysis of performance data from all sectors
per location Allows sectors to engage in cross-monitoring and
data sharing
Eastern Burma – Strengthening Monitoring Quality (2) Regular surveys are expanded in scope
and feed into the information system Key strength of the Eastern Burma
programmes is history of conducting regular (sector) surveys.
All partners are now supporting the expansion of the scope and the coverage of these surveys.
Example: Annual Poverty Survey
Eastern Burma – Strengthening Monitoring Quality (3) Increased linkages between implementing
partners, and improved M&E capacityEfforts in recent years to connect local
organizations across sector and ethnic groupDiscussion on cross-monitoringComprehensive M&E training for local groups Improved community feedback/village
monitoring (also used by Oxfam in Somalia and Tearfund in Afghanistan)
Eastern Burma – Strengthening Monitoring Quality (4) Post-facto review of health centre
logbooks and patient filesSimple but innovative example:Can be done remotely and is not time
sensitive Reveals substantial information about quality
of support and services Initiated by the IRC, and now being expanded
to whole sectorVariations conceivable for other sectors
Eastern Burma – Strengthening Monitoring Quality (5) Increased use of photographic and video
evidence of implementationWhere possible, local NGOs use
photography video
to document their work. Use of video started with success by one local
NGOProvides better representation e.g. of
trainings and public awareness activities.
Lessons learned, and looking towards the future Similar contexts, different data environments Good examples of when similar contexts warrant
different approaches Both contexts are changing rapidly – more access in
Somalia, political reforms in Burma Improved monitoring systems instill long-term effects
to adapt to these changes—stronger information, increased local capacity
Changes in Somalia may make Eastern Burma lessons applicable in near future
Experience in Somalia valuable for similar situations elsewhere, e.g. Syria
Thank you!
For further questions:
Mona Fetouh - [email protected]
Volker Hüls - [email protected]
Christian Balslev Olesen - [email protected]