12015

1990 Module 9: Module 9:

Applications of IndicatorsApplications of Indicators

Tools for Civil Society to Understand Tools for Civil Society to Understand and Use Development Data: and Use Development Data:

Improving MDG Policymaking and Improving MDG Policymaking and MonitoringMonitoring

22015

1990

What you will be able to do by What you will be able to do by the end of this module:the end of this module:

• Understand the place that MDG and other Understand the place that MDG and other indicators have in developing and monitoring indicators have in developing and monitoring policy, and in advocacypolicy, and in advocacy

• Know how indicators can be used for these Know how indicators can be used for these purposespurposes

• Understand how proxy indicators can be used Understand how proxy indicators can be used when direct indicators are not availablewhen direct indicators are not available

• Take their strengths and weaknesses into Take their strengths and weaknesses into account when using indicatorsaccount when using indicators

• Use indicators which are consistent with one Use indicators which are consistent with one anotheranother

• Understand the properties of the different levels Understand the properties of the different levels of indicators, and when to use the correct levelof indicators, and when to use the correct level

32015

1990

Why Use MDG Indicators?Why Use MDG Indicators?

• To set targets using empirical dataTo set targets using empirical data

• To ascertain and inform the public and policy To ascertain and inform the public and policy makers on progress on MDGsmakers on progress on MDGs

• To trigger action to meet time-bound targetsTo trigger action to meet time-bound targets

• To contribute to the design of policies based on To contribute to the design of policies based on evidenceevidence

• To create public interest in MDG progress at To create public interest in MDG progress at home and abroadhome and abroad

42015

1990

Use of IndicatorsUse of Indicators

• Where are we today?Where are we today?

• How far away from the target are we?How far away from the target are we?

• What progress is being made?What progress is being made?

52015

1990

Where are We Today?Where are We Today?

The value of an indicator tells us what the current The value of an indicator tells us what the current level is in a given moment and placelevel is in a given moment and place

Example: MDG6, Indicator Example: MDG6, Indicator

““Death rate associated with tuberculosis (per Death rate associated with tuberculosis (per 100,000 people)”100,000 people)”

In Belarus in 2003 the rate was 10.4In Belarus in 2003 the rate was 10.4

In Moldova in 2003 the rate was 16.4In Moldova in 2003 the rate was 16.4

62015

1990

How can this snapshot be used?How can this snapshot be used?

• Nationally:Nationally:

- Gives us a picture of the current status - Gives us a picture of the current status (at different levels of disaggregation if (at different levels of disaggregation if

available)available)

- Tells us the magnitude of the challenge of - Tells us the magnitude of the challenge of reaching targetsreaching targets

- Provides hard evidence to motivate policy - Provides hard evidence to motivate policy makersmakers

- Helps to quantify the intervention needed- Helps to quantify the intervention needed

72015

1990

How can this snapshot be used? (2)How can this snapshot be used? (2)

• RegionallyRegionally

- Shows us how we compare with similar - Shows us how we compare with similar countriescountries

- Feeds into regional policies and - Feeds into regional policies and organizationsorganizations

• GloballyGlobally

- Information for international organizations - Information for international organizations and donorsand donors

- Helps to build a global picture of current - Helps to build a global picture of current statusstatus

82015

1990

How can this snapshot be used? (3)How can this snapshot be used? (3)

Standardized death rate associated with tuberculosis, 2004

10.7

2.8

8.6

18.2

9.2

20.322.6

27.0

Belarus Bulgaria Kazakhstan Lithuania Moldova Romania RussianFederation

Ukraine

Per

100

,000

po

pu

lati

on

Source: Health for All Database, WHO Regional Office for Europe, 2006Source: Health for All Database, WHO Regional Office for Europe, 2006

92015

1990 Moldova. Death rate associated with tuberculosis

18.618.6

22.8

18.6

10.0

18.9

16.816.415.9

15.016.9

0

5

10

15

20

25

2000 2001 2002 2003 2004 2005 2006 2015

Per

100

,000

po

pu

lati

on

Belarus. Death rate associated with tuberculosis

11.111.1

15.1

11.1

5.87.3

6.4

4.4

0

2

4

6

8

10

12

14

16

1990 1995 2000 2004 2015

Per

100

,000

po

pu

lati

on

Position relative to the targetPosition relative to the target

MDG targets specify values or levels to be achieved MDG targets specify values or levels to be achieved by certain datesby certain dates

Examples.Examples.

Trend Trend

Target Target

Sources: National MDG Reports of the Republic of Belarus and the Sources: National MDG Reports of the Republic of Belarus and the Republic of MoldovaRepublic of Moldova

102015

1990

What extra information What extra information does this give us?does this give us?

• Shows whether change is happening, and in Shows whether change is happening, and in which directionwhich direction

• Tells us how big the change has beenTells us how big the change has been

• Allows us to decide whether the intervention is Allows us to decide whether the intervention is having the predicted effecthaving the predicted effect

- Direction- Direction

- Magnitude- Magnitude

112015

1990

What extra information What extra information does this give us? (2)does this give us? (2)

•Allows us to estimate trendAllows us to estimate trend

•Need to be very sure of underlying trendNeed to be very sure of underlying trend

- Form, e.g., linear, exponential- Form, e.g., linear, exponential

- Effects of natural limits, e.g., available - Effects of natural limits, e.g., available resourcesresources

- Unforeseen outside forces- Unforeseen outside forces

WarningWarning

Prediction gets less reliable, the further you Prediction gets less reliable, the further you extrapolateextrapolate

122015

1990

Proxy IndicatorsProxy Indicators

• Not always possible to measure the indicator Not always possible to measure the indicator we’re interested in we’re interested in

• Need to measure something closely related, that is Need to measure something closely related, that is

- Accessible- Accessible

- Consistent with the measure we can’t get- Consistent with the measure we can’t get

- Useful- Useful

132015

1990

Proxy Indicators. ExamplesProxy Indicators. Examples

• Environmental indicators are often proxiesEnvironmental indicators are often proxies

- Use of ozone depleting substances as proxy to - Use of ozone depleting substances as proxy to ozone depletionozone depletion

• Literacy of men and women age 15-24Literacy of men and women age 15-24

- Difficult to measure- Difficult to measure

- In past educational attainment has been used as - In past educational attainment has been used as a proxy measurea proxy measure

- These days, reading and writing tests are - These days, reading and writing tests are recommended instead of educational recommended instead of educational attainmentattainment

142015

1990

Proxy Indicators. Examples (2)Proxy Indicators. Examples (2)

• Access to communication systems. Two proxy Access to communication systems. Two proxy indicators used:indicators used:

- Telephone lines per 1,000 people - Telephone lines per 1,000 people

- Personal computers per 1,000 people- Personal computers per 1,000 people

They measure availability and not access or useThey measure availability and not access or use

152015

1990

ConsistencyConsistency

• Definitions should maintain consistency amongst Definitions should maintain consistency amongst related indicators, that is, use commonrelated indicators, that is, use common

- Definitions (e.g., of primary education)- Definitions (e.g., of primary education)

- Denominators (e.g., defined populations)- Denominators (e.g., defined populations)

- Data (data from the same surveys)- Data (data from the same surveys)

- Time (collected at the same or very similar time - Time (collected at the same or very similar time points)points)

- Analysis methods (where this is necessary to - Analysis methods (where this is necessary to maintain common definitions)maintain common definitions)

• Can provide checks on values and trendsCan provide checks on values and trends

162015

1990

Conditions for correct and effective useConditions for correct and effective use

• Selection of appropriate indicatorsSelection of appropriate indicators - Measures the correct thing- Measures the correct thing - Fit for the purpose (policy development or - Fit for the purpose (policy development or

assessment, advocacy, target audience)assessment, advocacy, target audience) - Consistent with related indicators- Consistent with related indicators• Available data, and information about the dataAvailable data, and information about the data

(metadata)(metadata)• Correct and accurate analysis methodsCorrect and accurate analysis methods• Some measure of reliability/variability/sample sizeSome measure of reliability/variability/sample size• Specification of bias, if anySpecification of bias, if any• TimelinessTimeliness• Effective presentationEffective presentation

172015

1990

Abuse of IndicatorsAbuse of Indicators

• Wrong indicatorWrong indicator - Telephone lines??- Telephone lines??• Wrong interpretationWrong interpretation• Comparing unlike scalesComparing unlike scales - E.g., comparing poverty rates of two countries - E.g., comparing poverty rates of two countries

which use different welfare aggregates and which use different welfare aggregates and poverty lines poverty lines

• Errors in data or analysis methodsErrors in data or analysis methods - Rely on skills of data analysis teams- Rely on skills of data analysis teams• Using out of date values Using out of date values • Inappropriate extrapolationInappropriate extrapolation• Ignoring variabilityIgnoring variability

182015

1990

Indicator LevelsIndicator Levels

•GlobalGlobal

•RegionalRegional

•NationalNational

•Sub-nationalSub-national

192015

1990

Global ComparisonsGlobal Comparisons

• Varying sources for global MDG indicatorsVarying sources for global MDG indicators

• Some taken from national MDG reportsSome taken from national MDG reports

• Some produced by national agenciesSome produced by national agencies

• Some produced by international agencies,Some produced by international agencies,e.g., poverty measures produced by World Banke.g., poverty measures produced by World Bank

202015

1990

Global MDGs vs. National MDGsGlobal MDGs vs. National MDGs

• National MDGsNational MDGs

- Use only national data- Use only national data

- Allow sub-national disaggregation- Allow sub-national disaggregation

• Global MDGsGlobal MDGs

- Often adjusted to make comparisons possible- Often adjusted to make comparisons possible

- May use different analysis methods- May use different analysis methods

- May use different data (particularly UN - May use different data (particularly UN population estimates)population estimates)

212015

1990

Policy and AdvocacyPolicy and Advocacy

• Only national indicator values are useful for Only national indicator values are useful for developing and monitoring national policydeveloping and monitoring national policy

• Global and regional indicator values don’t give a Global and regional indicator values don’t give a direct measure of how well the national policy is direct measure of how well the national policy is doingdoing

• For advocacy, we need direct local evidence to For advocacy, we need direct local evidence to support calls for intervention and action, so need support calls for intervention and action, so need local valueslocal values

222015

1990

Global and National MDG IndicatorsGlobal and National MDG Indicators

• Both are valuableBoth are valuable

• GlobalGlobal

- Position relative to region and world- Position relative to region and world

- Internationally agreed basis for interventions- Internationally agreed basis for interventions

- May be used in place of national estimates, - May be used in place of national estimates, or where none is availableor where none is available

- May not be directly applicable to national - May not be directly applicable to national policiespolicies

232015

1990

Global and National MDG Indicators (2)Global and National MDG Indicators (2)

• NationalNational

- Relate to national and sub-national issues- Relate to national and sub-national issues

- Values used directly as input to national/regional - Values used directly as input to national/regional policy developmentpolicy development

- Show national changes over time- Show national changes over time

- Extra indicators can be used- Extra indicators can be used

• Though different, both can be right depending on Though different, both can be right depending on contextcontext

242015

1990

Global and National MDG Indicators (3)Global and National MDG Indicators (3)

Example. School enrollmentExample. School enrollment

Country, Country, yearyear

IndicatorIndicator National National MDG MDG

ReportReport

World Development World Development Indicators, Indicators,

World Bank 2008World Bank 2008

Belarus, Belarus, 20042004

Net general Net general secondary secondary enrollment ratioenrollment ratio

86.686.6 89.089.0

Moldova, Moldova, 20062006

Gross general Gross general secondary secondary enrollment ratioenrollment ratio

92.492.4 82.382.3

252015

1990

Sub-National IndicatorsSub-National Indicators

• Based on a common national definitionBased on a common national definition

• Data from national datasets, which include sub-Data from national datasets, which include sub-population identifierspopulation identifiers

• Allows for disaggregation of indicatorsAllows for disaggregation of indicators

262015

1990

Use in PracticeUse in Practice

• National examplesNational examples

- Developing and monitoring development - Developing and monitoring development strategiesstrategies

- Assessing the effectiveness of policies- Assessing the effectiveness of policies

- Monitoring impact of economic and social - Monitoring impact of economic and social changechange

272015

1990

Use in Practice (2)Use in Practice (2)

• Sub-national examplesSub-national examples

- Meeting regional and district needs for planning - Meeting regional and district needs for planning and service deliveryand service delivery

- Addressing the problems associated with - Addressing the problems associated with pocketspockets

- Advocating for change- Advocating for change

- Targeting interventions- Targeting interventions

282015

1990

Strengths and LimitationsStrengths and Limitations

• Strengths of indicatorsStrengths of indicators

- Simple- Simple

- Well-defined- Well-defined

- Comparability- Comparability

- Roles in policy development/advocacy are clear- Roles in policy development/advocacy are clear

• LimitationsLimitations

- Availability of data- Availability of data

- Timeliness of data- Timeliness of data

- Comparisons: global/national- Comparisons: global/national

- Changes in external environment- Changes in external environment

292015

1990

SummarySummary

•Purpose and uses of indicatorsPurpose and uses of indicators

•Proxy indicatorsProxy indicators

•ConsistencyConsistency

•LevelsLevels

•Strengths and limitationsStrengths and limitations

302015

1990

Practical 9Practical 9

YearRatio of girls to boys in

secondary schoolRatio of boys 10-14

to girls 10-14

1990 0.67 0.99

1995 0.72 0.98

2000 0.80 0.95

1. How would you interpret the following 1. How would you interpret the following information?information?

2. Is there any further information that would help 2. Is there any further information that would help you interpret the indicators in question 1?you interpret the indicators in question 1?

312015

1990

Practical 9Practical 9

3. 3. Look at all of the MDG indicators included in your Look at all of the MDG indicators included in your National MDG Report. Choose one indicator for National MDG Report. Choose one indicator for which data are present in your report, and which data are present in your report, and describe briefly how it could be used either to describe briefly how it could be used either to support policy development or to advocate for support policy development or to advocate for policy change. What are the strengths and policy change. What are the strengths and limitations of this particular indicator for this limitations of this particular indicator for this purpose? How could it subsequently be used to purpose? How could it subsequently be used to monitor change over time?monitor change over time?