Monitoring the MDGsA catalogue of procedures and an assessment of statistical capacity

Part 1: International study

Overview

• Objective of the study

• The method used

• The main findings

The catalogue For each of the 48 MDG indicators the catalogue specifies:

1. Agency responsible for reporting international statistics2. Definition of the indicator3. Availability and timeliness of data4. Original data sources5. Construction of the indicator and/or process of reporting6. Checks performed on the original sources7. Comparison with other international data sources8. Notes on websites holding the indicator data

This exercise has illustrated various shortcomings of the current dataset, and it also highlighted possible areas for immediate improvement.

Example from the indicator catalogue

Indicator 1 Lead Agency Definition Availability Year Countries 2001 2 2000 38 1999 10 1998 66 1997 14

1987-98 265 national surveys from a total of 83 countries, representing 88% of the total population of the developing world

Percentage of population below $1 (PPP) per day

World Bank World Development Indicators (annual) and www.worldbank.org

The percentage of population below $1 a day comprises those who live on less than $1.08 a day at 1993 international prices (equivalent to $1 in 1985 prices, adjusted for purchasing power parity). The dollar a day is per person and it includes income in kind. Because the poverty line of this indicator has fixed purchasing power across countries or areas, the dollar a day poverty line is often called “absolute poverty line”.

These data are taken from the WDI 2003

Original Sources Household surveys obtained from government statistical offices and World Bank country departments.

Construction of indicator/process of reporting - What type of information is

requested: raw data or estimates - Who is supposed to fill in the

questionnaire - When questionnaires are sent

off /periodicity /reminders - If changes are made are they

sent back for approval

This indicator is calculated using household survey data and produced by the World Bank’s Development Research Group. It is not normally calculated at national level, where national poverty lines are used instead. To begin with, household consumption aggregates (or income aggregate) are estimated. Whenever possible, consumption aggregates are used as the welfare indicator, and otherwise household income data are computed. If only income distribution is available, average income is adjusted to accord with either a survey-based estimate of mean consumption (when available) or an estimate based on consumption data from national accounts (mean income is re-scaled by 1 minus the national saving rate). No adjustment is made for spatial differences in cost of living within countries. Aggregates are then divided by the number of people in the household to calculate per capita consumption per person. The international poverty line is 1.08 dollar per day (equivalent to 32.74 dollars per month). PPP exchange rates are based on the 1993 Internati onal Comparison Project, which covered 110 countries. The PPP conversion factor used for this series is the number of units of a country’s currency required to buy the same amount of goods and services in the domestic market as a US dollar would buy in the United States. After converting the international poverty line in local currency at PPP in 1993 this is again converted to the survey year considering the country-specific official Consumer Price Index. People that live with less than this amount are considered poor. Regional estimates: regional estimates are constructed for specific reference years (usually where a good number of countries had a national surveys) and for countries with poverty estimates in different years estimates are made taking into consideration the growth rate in real per capita private consumption from the national accounts (assuming that Lorenz curve does not change or making a time weighted average of two estimates coming from two different years).

Checks performed on the original sources - Quality criteria: are there cases

where original sources are changed or not published?

- What are the steps taken in order to assure comparability across countries and over time?

Only surveys that meet the following criteria are used: the surveys are nationally representative; include a sufficiently comprehensive consumption or income aggregate (including consumption or income from own production); and allow for the construction of a correctly weighted distribution of consumption or income per ca pita. Comparability problems are eliminated to the extent possible by re-estimating the consumption/income aggregates or if necessary by dropping a survey. Problems The dollar a day indicator is a global one, and like other indicators is not equally relevant in all regions, because different countries have different definitions of poverty. The international poverty measure, therefore, has to be supplemented by indicators more relevant to the country and the region. PPP exchange rates are used because they take into account the local prices of goods and services not traded internationally, but PPP rates were designed not for making international poverty comparisons but for comparing aggregates from national accounts. As a result, there is no certainty that an international poverty line measures the same degree of need or deprivation across countries. Problems also exist in comparing poverty measures within countries, especially regarding the urban/rural differences. The cost of living is typically higher in urban than in rural areas, so the urban monetary poverty line should be higher than the rural poverty. There is also a problem regarding the comparability across surveys: household survey questionnaires can differ widely, and even similar surveys may not be strictly comparable because of the difference in their qualities. Comparisons across countries at different levels of development may also pose a potential problem, because of differences in the relative importance of consumption of non -market goods. It should be noted that even if the surveys are entirely accurate, the measure of poverty can miss some important aspects of individual welfare. Firstly, using household consumption ignores inequalities within households. Secondly, the measure does not reflect people’s feeling about relative deprivation, or their concerns about uninsured risk to their income and health.

Other Sources

There are no other international sources of this indicator.

References: Table 2.6 of WDI 2002 (http://www.worldbank.org/data/wdi2002/pdfs/table%202-6.pdf) Ravallion and Chen (2000): How did the world’s poorest fare in the 1990s? World Bank (downloadable on: www-wds.worldbank.org)

Main shortcomings:

1. Definitional Issues2. Data availability3. Issues that arise from modelling exercises4. Timeliness5. Comparability & Differences in Calculation Methods

Key Issues to Address:

• Appropriate use of available data from household surveys

• Use by lead organisations of data reporting questionnaires

• Use of international population data

• Management of common methodologies and definitions

• Data management practices

Appropriate use of household surveys:

• 1/3 of indicators depend on data from household surveys• Monitoring Goals 1, 4, 5, 6, and 7 depend almost exclusively on

household surveys• However an expansion of HH Survey roles could result

because:– There are surveys escaping the screening system

• Examples – Integrated Household Survey in Malawi; PIHS in Pakistan (p.12)

– There is scope for using HH Surveys to overcome problems of old or missing data (p.13)

• Examples – Education Indicators; Literacy Indicators

• …but…large management & coordination problems to address consistency and timeliness issues.

Lead agency use of questionnaires:

• A number of lead agencies use questionnaires sent to governments for data reporting on the following indicators (p.15):– Education indicators– Water and sanitation indicators– Immunization coverage indicators

• Quality of these data does not always satisfy the required standards because of the: – Inability to control the process – Absence of direct contact

Use of international population data:

• International population data are used in two ways:– For regional and global estimates (weights)– In the construction of country indicators

• The combining different data sources in the construction of single country estimates generates inconsistencies in the following indicators: – Net enrolment ratio– Literacy rates– Water and sanitation – …..an example….

The NER in Albania is:

100%

(according to international sources)

81%

(according to national sources)

What explains this difference?

The population aged 5-9 is:

320 thousands

(according to international sources)

400 thousands

(according to national sources)

80%*400=320!

Management of common methodologies and definitions:

• Lack of clarity in the definition of some indicators:– For example in the HIV/AIDS indicators (footnote 5, p.9)

• Clear guidelines, but lack of coordination in the implementation:– For example in the definition of literacy (p. 18)

• Adjustment of methodologies and definitions to the variety of country realities:– For example: immunization in Vietnam

Proportion of births attended by skilled health personnel

Country WHO UNICEF Original Source

Paraguay 58.1 (1998) 70.9 (1998) ENSMI (1999)

China 89.3 (1995) 70 (1999)

1) Western Pacific Regional Office2) Chinese Health Statistical Digest

Data Management issues:

A number of problems/difficulties have been noticed in the way in which data is presented by lead agencies:

• Absence of detailed explanatory notes on (p.18):– Sources of data on which estimates are based– Explanation of any dramatic changes in the data time series

• Use of year-ranges for data presentation (p.19)

• Coordination in updating web pages, and absence of information on when the last update took place

Further issues to explore:

• MANAGEMENT – Issue of accountability and definition of responsibilities between the various

international agencies and countries

• DATA COLLECTION– Further review of the scope for improving data from agency questionnaires – An assessment of the general quality of databases of household surveys and

hh surveys more generally

• METHODS– Systematic listing of inconsistencies in methodologies– A review of inconsistencies in population data – How current methodologies for education and literacy indicators could integrate

information from household surveys

• ANALYSIS– Review case for broadening scope of re-analysis of raw data