mics4 survey design workshop multiple indicator cluster surveys survey design workshop the mics3...
TRANSCRIPT
MICS4 Survey Design Workshop
Multiple Indicator Cluster SurveysSurvey Design Workshop
The MICS3 Evaluation
MICS4 Survey Design Workshop
MICS3 Evaluation
Commissioned by UNICEF and carried out byJohn Snow Inc. during 2007-2008
MICS4 Survey Design Workshop
Methodology used
• Global online survey (350 respondents)
• Key informant interviews
• Document review
• In-depth study in 8 countries
• Data quality assessment
MICS4 Survey Design Workshop
Findings
• MICS is seen as a key data source
• Data quality at acceptable levels, on par with other survey programmes (e.g. DHS), in most countries examined
• Tools (manuals, guidelines) technical assistance (workshops, online etc.) of high standard, and highly valued
MICS4 Survey Design Workshop
Findings
• Strong evidence of data use at global level
• Progress in data use at country level, but the link to programme action and policy development not an evident pattern
• Wide-spread perception that MICS has helped to build local capacity
MICS4 Survey Design Workshop
Findings
• Final reports are often delayed for long periods, to the point that the data are no longer seen as timely
• Significant quality lapses and deviations from accepted norms in sampling and fieldwork procedures found in several countries
Quality problems are a result of overlooking basic standards of survey implementation
MICS4 Survey Design Workshop
From Evaluation Report:
“The MICS3 manual and regional workshops give specific guidelines regarding how to conduct the field work and how to maintain quality control. The evaluation found significant deviation from these practices in the countries studied, which occurred despite the fact that most countries received technical assistance from UNICEF and rated it highly…”
MICS4 Survey Design Workshop
From Evaluation Report:
“When countries adhered to the MICS3 data processing protocols and made use of the support provided, data processing appears to have gone well.”
MICS4 Survey Design Workshop
Overlooking basic standards
Deviations from MICS3 recommended practices (examples):– Out-of-date or incomplete household listings– Shortened training in many cases – Ad hoc translation in the field in some cases; no
pre-test– Team composition/supervision/use of editors
MICS4 Survey Design Workshop
Contributing to delays and quality issues
• Over-loaded questionnaires, non-MICS additions, inappropriate use of modules, lack of understanding
• Lack of capacity in sampling, data processing, report writing
• Problems discovered (too) late
MICS4 Survey Design Workshop
From Evaluation Report:
“In some instances, the need for skilled technical advice is only identified after key decisions have been taken and implemented necessitating “rescue” operations of sorts to salvage the effort.”
MICS4 Survey Design Workshop
Some conclusions
To improve data quality; standards must be adhered to. This can only happen by providing:
• Timely and high quality in-country support
• Regional level coordination
• Good communication
• Realistic budgets to avoid short-cuts
MICS4 Survey Design Workshop
Some conclusions
To improve data use;
• Improve quality
• Improve timeliness
• Set realistic expectations – use interim products (preliminary findings report)
MICS4 Survey Design Workshop
MICS4
• Strengthened technical assistance system, with increased in-country support
• Strengthened coordination at regional level
• Increased funding
MICS4 Survey Design Workshop
MICS4
• Continue good practices – manual, workshops• Easier-to-use and customize survey tools