The role of statistics in promoting The role of statistics in promoting human rights:human rights:

Without data, all you are is just Without data, all you are is just another person with an opinionanother person with an opinion

AAAS Science and Human RightsAAAS Science and Human RightsMary W. GrayMary W. Gray

January 11, 2111January 11, 2111

mgray@american.edumgray@american.edu

Why statistics?Why statistics?

There are three kinds of lies …lies, damn lies, and statistics.There are three kinds of lies …lies, damn lies, and statistics.Benjamin DisraeliBenjamin Disraeli

Numbers do not lie, but they have the propensity to tell the truth with intent to Numbers do not lie, but they have the propensity to tell the truth with intent to deceive.deceive.

The death of one man is a tragedy. The death of millions is a statistic. Stalin, The death of one man is a tragedy. The death of millions is a statistic. Stalin, comment to Churchill at Potsdam, 1945comment to Churchill at Potsdam, 1945

If you want to inspire confidence, give plenty of statistics. It does not matter that If you want to inspire confidence, give plenty of statistics. It does not matter that they should be accurate, or even intelligible, as long as there is enough of them. they should be accurate, or even intelligible, as long as there is enough of them.

Lewis Carroll Lewis Carroll

Far better an approximate answer to the right question, than the exact answer to Far better an approximate answer to the right question, than the exact answer to the wrong question.the wrong question.

John TukeyJohn Tukey

Why use statistics?Why use statistics?

What can we say and do to help human rights researchers and What can we say and do to help human rights researchers and activists?activists?

Statistics can provide the information needed to answer questionsStatistics can provide the information needed to answer questionswhere should a new clinic be locatedwhere should a new clinic be located

Statistics can suggest questions that should be askedStatistics can suggest questions that should be askedwhat is the prevalence of infant mortalitywhat is the prevalence of infant mortality

Statistics can tell us how probable certain results are or will beStatistics can tell us how probable certain results are or will beis it credible that figures offered by the government are correctis it credible that figures offered by the government are correct

Statistics can assist us to make predictions or estimatesStatistics can assist us to make predictions or estimatesoutcome of an election, immunization coverageoutcome of an election, immunization coverage

Statistics can help us come to conclusionsStatistics can help us come to conclusions what needs to be done about a problemwhat needs to be done about a problem

Can you believe what you read in the Can you believe what you read in the newspapers?newspapers?

What Happened to Skepticism?What Happened to Skepticism?

New York TimesNew York Times front page headline 21 May 2009: front page headline 21 May 2009: ““1 in 7 Detainees Rejoined Jihad, Pentagon Finds”1 in 7 Detainees Rejoined Jihad, Pentagon Finds”

The first paragraph reported that an unreleased Pentagon study had concluded that about one The first paragraph reported that an unreleased Pentagon study had concluded that about one in seven of the 534 prisoners already transferred abroad from Guantanamo had “returned in seven of the 534 prisoners already transferred abroad from Guantanamo had “returned to terrorism or militant activity.”to terrorism or militant activity.”

The article failed to distinguish between former prisoners suspected of new acts of terrorism—The article failed to distinguish between former prisoners suspected of new acts of terrorism—more than half the cases—and those supposedly confirmed to have rejoined jihad against more than half the cases—and those supposedly confirmed to have rejoined jihad against the West. Had only confirmed cases been considered, 1 in 7 would be changed to 1 in 20. the West. Had only confirmed cases been considered, 1 in 7 would be changed to 1 in 20.

Now the recidivism rate is reduced from 14% to 5%. But is it recidivism at all? Were any of the Now the recidivism rate is reduced from 14% to 5%. But is it recidivism at all? Were any of the released inmates ever engaged in terrorist or militant activity before their imprisonment? released inmates ever engaged in terrorist or militant activity before their imprisonment? Or were they radicalized while in Guantanamo?Or were they radicalized while in Guantanamo?

So, clearly the So, clearly the New York TimesNew York Times reporters and editors should have been skeptical, but so should reporters and editors should have been skeptical, but so should the readers—and in fact, many were and the paper’s Public Editor ended up writing a the readers—and in fact, many were and the paper’s Public Editor ended up writing a critical column on the issue.critical column on the issue.

Could this be why not?Could this be why not?

““DoonesburyDoonesbury” ” Washington Post Washington Post 11 July 2008 11 July 2008

If it seems too good/bad to be true, If it seems too good/bad to be true, it probably isn’tit probably isn’t

The Kuwaiti incubators story.

Were there so many premature babies in Kuwait in 1991 that 312 incubators were in operation?

More skepticism: “The Deadly Toll of More skepticism: “The Deadly Toll of Abortion by Amateurs”Abortion by Amateurs”

This was the headline on 2 June 2009. The story reported that for every 100,000 births in This was the headline on 2 June 2009. The story reported that for every 100,000 births in Tanzania, 950 women die. But nowhere do we see how many of the deaths are Tanzania, 950 women die. But nowhere do we see how many of the deaths are attributable to botched abortions.attributable to botched abortions.

We are told there are 19 million “unsafe” abortions a year and 70,000 deaths attributable to We are told there are 19 million “unsafe” abortions a year and 70,000 deaths attributable to them (presumably worldwide). This is, we learn further, is 13% of all maternal deaths.them (presumably worldwide). This is, we learn further, is 13% of all maternal deaths.

World population 6,800,000,000 World population 6,800,000,000 women 15 t0 49 1,760,000,000women 15 t0 49 1,760,000,000 Rate of “unsafe” abortions worldwide 19/1760 = 1%Rate of “unsafe” abortions worldwide 19/1760 = 1%Is it credible that each year 1 woman in every 100 worldwide has an “unsafe” abortion?Is it credible that each year 1 woman in every 100 worldwide has an “unsafe” abortion?

By extension of the worldwide figures on maternal deaths, we might conjecture that 123 (of By extension of the worldwide figures on maternal deaths, we might conjecture that 123 (of every 100,000 births) of the maternal deaths in Tanzania arise from “unsafe” abortions if every 100,000 births) of the maternal deaths in Tanzania arise from “unsafe” abortions if Tanzania is an “average” country. But is it? Or are all 950 deaths referenced above due to Tanzania is an “average” country. But is it? Or are all 950 deaths referenced above due to unsafe abortions, making the situation particularly dire there?unsafe abortions, making the situation particularly dire there?

Maternal deaths and unsafe abortions, Maternal deaths and unsafe abortions, continuedcontinued

In what may be intended as an explanation of the situation in Tanzania, we are told In what may be intended as an explanation of the situation in Tanzania, we are told that the use of contraception is 25% in Tanzania, but 39% in Kenya, and 60% in that the use of contraception is 25% in Tanzania, but 39% in Kenya, and 60% in South Africa (where abortion is legal). But no mortality rate is given for these South Africa (where abortion is legal). But no mortality rate is given for these countries.countries.

So what are we to conclude about “unsafe” abortion, use of contraception and So what are we to conclude about “unsafe” abortion, use of contraception and death rates?death rates?

Well, “unsafe” abortions are not good, in Tanzania or anywhere else, but without Well, “unsafe” abortions are not good, in Tanzania or anywhere else, but without statistics, I’m just another person with an opinion.statistics, I’m just another person with an opinion.

Advice to researchers: Advice to researchers: Some sources of dataSome sources of data

OnlineOnline

www.worldbank.orgwww.worldbank.org

www.unstats.un.org/demographicwww.unstats.un.org/demographic

www.census.gov, www.bls.gov, www.cdc.gov, www.nih.gov,www.census.gov, www.bls.gov, www.cdc.gov, www.nih.gov,www.stat.can.gc.can. www.statistics.gov.ukwww.stat.can.gc.can. www.statistics.gov.uk

www.who.int/research/en/www.who.int/research/en/

www.nber.org/links/data.htmlwww.nber.org/links/data.html

University data sources, e.g., University of MichiganUniversity data sources, e.g., University of Michiganwww.lib.umich.edu/govdocs/stforeig.html#compwww.lib.umich.edu/govdocs/stforeig.html#compThis source lists general international sources as well as data sources for This source lists general international sources as well as data sources for

individual countriesindividual countries

Other sources of dataOther sources of data

Census dataCensus dataGenerally collected by government agencies (so look for bias)Generally collected by government agencies (so look for bias)Worldwide compilations by IGOs and others Worldwide compilations by IGOs and others

Financial, budget, accounts recordsFinancial, budget, accounts recordsChurch records, Land recordsChurch records, Land records

Survey dataSurvey dataGovernments, IGOs, NGOsGovernments, IGOs, NGOsAcademic researchersAcademic researchersCommercial entitiesCommercial entities

Observational dataObservational dataRetrospective case studiesRetrospective case studies

Experimental dataExperimental dataClinical trialsClinical trials

Considerations when seeking dataConsiderations when seeking data

What is the age of the data? What is the age of the data?

Where did they come from?Where did they come from?

In what medium were they originally produced? In what medium were they originally produced?

What is the geographical coverage of the data?What is the geographical coverage of the data?

Does the data seem logical and consistent?Does the data seem logical and consistent?

In what format are the data kept?In what format are the data kept?

How were the data checked?How were the data checked?

Why were the data compiled?Why were the data compiled?

What is the reliability of the provider?What is the reliability of the provider?

From what population do the data come?From what population do the data come?

When to give upWhen to give up

Consider the possibility that there is no data set that contains exactly the Consider the possibility that there is no data set that contains exactly the information you need in exactly the form you need it.information you need in exactly the form you need it.

In trying to compare data, face the fact that because of different time frames, In trying to compare data, face the fact that because of different time frames, geographic limits, different categories and different definitions of terms, it may geographic limits, different categories and different definitions of terms, it may be impossible to integrate data from various sources.be impossible to integrate data from various sources.

Sample the data sets to check for compatibility, credibility, etc. If there are Sample the data sets to check for compatibility, credibility, etc. If there are glaring inconsistencies, you probably need to abandon the data set.glaring inconsistencies, you probably need to abandon the data set.

Is it possible to construct your own data set?Is it possible to construct your own data set?

Census v. surveyCensus v. survey

In theory, census data includes all of a populationIn theory, census data includes all of a populationUndercountUndercount

OvercountOvercountSurveys rely on sampling to make estimates about a populationSurveys rely on sampling to make estimates about a population

A sampling frame is requiredA sampling frame is requiredProbability samplesProbability samples

Simple random sampleSimple random sampleSystematic sampleSystematic sampleStratified sampleStratified sampleCluster sampleCluster sampleMulti-stage sampleMulti-stage sample

Non-probability samplesNon-probability samplesVolunteer sampleVolunteer sampleConvenience sampleConvenience sample

SamplingSampling

Why sample?Why sample?To estimate something about a populationTo estimate something about a populationTo make a predictionTo make a predictionTo test a hypothesisTo test a hypothesis

A A simple random sample simple random sample of size of size nn is a sample such that any subset of is a sample such that any subset of n n elementselements is is equally likely to be selected.equally likely to be selected.

A A stratified sample stratified sample consists of random samples from several strata of a population.consists of random samples from several strata of a population.

A A cluster sample cluster sample consists of samples from randomly selected subsets of a population.consists of samples from randomly selected subsets of a population.

Sampling and non-sampling errorSampling and non-sampling error

Sampling error in making estimates about the population from a sample or in testing Sampling error in making estimates about the population from a sample or in testing a hypothesis results from the process of sampling itself and can be controlled, a hypothesis results from the process of sampling itself and can be controlled, but not eliminated.but not eliminated.

Non-sampling errorNon-sampling errorSelection biasSelection biasQuestion(er) biasQuestion(er) biasResponse biasResponse bias

Involuntary Involuntary VoluntaryVoluntary

Sample size and variationSample size and variation

Too small a sample: population differences cannot be detected or estimates will not Too small a sample: population differences cannot be detected or estimates will not be very accurate. The smaller the sample the larger the sampling error.be very accurate. The smaller the sample the larger the sampling error.

Too large a sample: costly (in money and in people) and may focus on differences Too large a sample: costly (in money and in people) and may focus on differences too small to be of practical significance.too small to be of practical significance.The larger the sample the larger the sampling error.The larger the sample the larger the sampling error.

If the sample is not too large or too small, the sampling error does not depend on If the sample is not too large or too small, the sampling error does not depend on the size of the population.the size of the population.

But variation within the sample is also critical—the larger the variation ,the larger But variation within the sample is also critical—the larger the variation ,the larger the sampling error and the smaller the variation, the smaller the sampling error.the sampling error and the smaller the variation, the smaller the sampling error.

What might a data set look like?What might a data set look like?ScoreScore gender studygender study1010 00 331010 00 2288 11 2277 00 1188 00 3399 11 5588 11 441111 00 661414 00 551515 00 771414 11 661515 00 881111 11 221111 00 551212 11 441212 11 331212 11 111313 00 661212 00 331313 00 661010 00 221313 00 5577 11 111313 11 331111 00 771414 00 881212 11 331212 00 551010 00 441010 11 331212 00 6699 00 2299 11 331313 00 551010 11 331111 11 44

ScoreScore represents the represents the scores on an examscores on an examadministered to administered to 36 individuals36 individuals

GenderGender is represented is represented by by

0 for males and 0 for males and 1 for females for1 for females forcomputational purposescomputational purposes

StudyStudy represents therepresents thenumber of hours thatnumber of hours thateach individual each individual studied for the examstudied for the exam

Organizing dataOrganizing data

Each line represents an observationEach line represents an observationIndividualsIndividualsGroups of people (averages)Groups of people (averages)CountriesCountriesProvincesProvincesRegionsRegionsCitiesCitiesPeriod of timePeriod of time

Each column represents values of a variableEach column represents values of a variable

Kinds of variablesKinds of variablesCategorical (qualitative)—gender, country, “dummy” variablesCategorical (qualitative)—gender, country, “dummy” variablesOrdered—Likert scales (but be careful)Ordered—Likert scales (but be careful)Quantitative—age, income, number of newspapers readQuantitative—age, income, number of newspapers read

Centrality measuresCentrality measures

MeanMeansum of all observations/number of observations (“average”)sum of all observations/number of observations (“average”)

MedianMedianif there are an odd number of ordered observationsif there are an odd number of ordered observations

median = middle observationmedian = middle observationif there are an even number of ordered observationsif there are an even number of ordered observations

median = mean the two middle observationsmedian = mean the two middle observations

ModeModethe most frequent observationthe most frequent observation

Normal curveNormal curve

Data distributionData distribution

SD

Average tax cutAverage tax cut

1. Bill Gates 1. Bill Gates $999,010$999,010

2. Taxpayer 22. Taxpayer 2 1010. .. .. .. .. .. ... . .

1000. Taxpayer 1000 1000. Taxpayer 1000 10 10 $1,000,000$1,000,000

Average = $1,000,000/1000 = $1000Average = $1,000,000/1000 = $1000

Bill Gates is an “outlier.”Bill Gates is an “outlier.”

From description to inferenceFrom description to inference

From descriptive statistics we would like to make inferencesFrom descriptive statistics we would like to make inferences

What can we say about a population from sample data?What can we say about a population from sample data?

How likely is an event?How likely is an event?Test an hypothesis about a populationTest an hypothesis about a population

Can we make predictions?Can we make predictions?Outcome of election, need for facilitiesOutcome of election, need for facilities

What is the relation between variables?What is the relation between variables?Correlation v. causationCorrelation v. causation

Important considerationsImportant considerations

Distinguish between

Uncertainty is a lack of knowledge about specific factors, models, parameters, measurement, sampling, systematic errors, oversimplification of real world processes, misspecification of model structure, inappropriate proxy variables, descriptive or aggregation errors, misjudgment,incomplete analysis

Variability arises from real heterogeneity, diversity, results of natural random processes

Summary advice: Know when you need a Summary advice: Know when you need a statistician and get one as early as possiblestatistician and get one as early as possible

THE ROLE OF A STATISTICIAN IN A PROJECTParticipate early and often to provide guidance in designing studies and collecting data

Formulate the questions to be askedConsider the data needed to answer the questions

Resist unrealistic expectationsMonitor the execution of studies and collection of data to provide a basis for accountability and cost-effectiveness From existing sources—creating methods to improve the quality of available data Through retrospective case studies Through surveys—sampling planGrapple with the data

Clean the data Design and carry out the analysis

Interpret the data and results and draw conclusions

Pitfalls for which statisticians may share Pitfalls for which statisticians may share responsibilityresponsibility

1) Failure to gain the support of key decision-makers

2) Unrealistic goals and expectations

3) Failure to develop a clear map of the process

4) Building of scenarios that are not credible

5) Inappropriate time frames and scopes

6) Failure to design a survey process to address the questions asked

Useful referencesUseful references

SoftwareSoftwareSPSS SAS STATA RSPSS SAS STATA R

Basic textBasic textMoore, McCabe and Craig, Moore, McCabe and Craig, Introduction to the Practice of Statistics, Introduction to the Practice of Statistics, 6 6thth ed., Freeman, 2009 ed., Freeman, 2009

More advancedMore advancedAgresti and Finlay, Agresti and Finlay, Statistical Methods for the Social Sciences, Statistical Methods for the Social Sciences, 44thth ed., Pearson, 2009 ed., Pearson, 2009Finkelstein and Levin, Finkelstein and Levin, Statistical Methods for Lawyers,Statistical Methods for Lawyers, 2 2ndnd ed., Springer, 2001 ed., Springer, 2001

SamplingSamplingLevy and Lemeshow, Levy and Lemeshow, Sampling of Populations: Methods and Applications, Sampling of Populations: Methods and Applications, 44thth ed., Wiley, ed., Wiley, 20082008

ApplicationsApplicationsAsher, Banks and Scheuren, eds., Asher, Banks and Scheuren, eds., Statistical Methods for Human Rights, Statistical Methods for Human Rights, Springer, 2007Springer, 2007Ball and Asher, “Statistics and Slobodan,” Ball and Asher, “Statistics and Slobodan,” ChanceChance, 15, pp. 17-24, 2002, 15, pp. 17-24, 2002Ball, Scheuren, Seltzer and Spirr, “Multiple or Ball, Scheuren, Seltzer and Spirr, “Multiple or NN-system estimates of the number of political -system estimates of the number of political killings in Guatemala,” killings in Guatemala,” ASA Proceedings in Social Statistics, ASA Proceedings in Social Statistics, pp. 156-160, 1999pp. 156-160, 1999Burnham, Lafta, Doocy and Roberts, “Mortality after the 2003 invasion of Iraq,” Burnham, Lafta, Doocy and Roberts, “Mortality after the 2003 invasion of Iraq,” The The Lancet, Lancet, 368, pp. 1421-1428, 2006368, pp. 1421-1428, 2006Gray, “Statisticians discuss mortality in Iraq survey,” Gray, “Statisticians discuss mortality in Iraq survey,” Amstat News, Amstat News, April 2007, pp. 7-9April 2007, pp. 7-9