statistical analysis of endorsement experiments:...

Statistical Analysis of Endorsement Experiments:Measuring Support for Militant Groups in Pakistan

Kosuke Imai

Department of PoliticsPrinceton University

Joint work with Will Bullock and Jacob Shapiro

May 6, 2011

Kosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 1 / 23

Motivation

Survey is used widely in social sciencesValidity of survey depends on the accuracy of self-reportsSensitive questions =⇒ social desirability, privacy concernse.g., racial prejudice, corruptionsLies and nonresponses

How can we elicit truthful answers to sensitive questions?Survey methodology: protect privacy through indirect questioningStatistical methodology: efficiently recover underlying responses


Survey Techniques for Sensitive Questions

Randomized Response TechniqueMost extensively studied and commonly usedUse randomization to protect privacyDifficulties: logistics, lack of understanding among respondents

List ExperimentsAlso known as block total response and item count techniqueUse aggregation to protect privacyNew estimators to enable multivariate regression analysisNew methods to detect and correct list experiment failures

Endorsement ExperimentsUse randomized endorsements to measure support levelsDevelop a measurement model based on item response theoryApplications:

1 Pakistanis’ support for Islamic militant groups2 Afghanis’ support for Taliban and ISAF (joint with J. Lyall)3 Nigerians’ support for insurgents (joint with G. Blair)


Endorsement Experiments

Measuring support for political actors (e.g., candidates, parties)when studying sensitive questionsAsk respondents to rate their support for a set of policiesendorsed by randomly assigned political actors

Experimental design:

1 Select policy questions

2 Randomly divide sample into control and treatment groups

3 Across respondents and questions, randomly assign political actorsfor endorsement (no endorsement for the control group)

4 Compare support level for each policy endorsed by different actors


The Pakistani Survey Experiment

6,000 person urban-rural sample

Four militant groups:Pakistani militants fighting in Kashmir (a.k.a. Kashmiri tanzeem)Militants fighting in Afghanistan (a.k.a. Afghan Taliban)Al-Qa’idaFirqavarana Tanzeems (a.k.a. sectarian militias)

Four policies:WHO plan to provide universal polio vaccination across PakistanCurriculum reform for religious schoolsReform of FCR to make Tribal areas equal to rest of the countryPeace jirgas to resolve disputes over Afghan border (Durand Line)

Response rate over 90%


Endorsement Experiment Questions: Example

The script for the control groupThe World Health Organization recently announceda plan to introduce universal Polio vaccinationacross Pakistan. How much do you support such aplan?

The script for the treatment groupThe World Health Organization recently announceda plan to introduce universal Polio vaccinationacross Pakistan, a policy that has receivedsupport from Al-Qa’ida. How much do you supportsuch a plan?


Distribution of ResponsesPolio Vaccinations Curriculum Reform FCR Reforms Durand Line

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0

Not At All

A LittleA Moderate Amount

A LotA Great Deal

Control Group

Pakistani militant groups in Kashmir

Afghan TalibanAl−Qaida

Firqavarana Tanzeems

Control Group




Control Group




Control Group




Pun

jab

Sin

dhN

WF

PB

aloc

hist

an


Methodological Challenges and Proposed Solutions

1 How to combine responses from multiple questions?=⇒ item response theory

2 How to recoup loss of statistical efficiency?=⇒ hierarchical modeling

3 What is the key assumption?=⇒ learning vs. support

4 How to select policy questions?Policies should belong to a single dimensionPolicies should be known to respondentsRespondents should not have strong views


Endorsement Experiments Framework

N respondents

J policy questions

K political actors

Yij ∈ {0,1}: response of respondent i to policy question j

Tij ∈ {0,1, . . . ,K}: political actor randomly assigned to endorsepolicy j for respondent i

Yij(t): potential response given the endorsement by actor t

Covariates measured prior to the treatment


The Proposed Model

Quadratic random utility model:

Ui(ζj1, k) = −‖(xi + s∗ijk )− ζj1‖2 + ηij

Ui(ζj0, k) = −‖(xi + s∗ijk )− ζj0‖2 + νij

where xi is the ideal point and s∗ijk is the “influence” of anendorsementThe statistical model (item response theory):

Pr(Yij = 1 | Tij = k) = Pr(Yij(k) = 1)

= Pr(Ui(ζj1, k) > Ui(ζj0, k))

= Pr(αj + βj(xi + s∗ijk ) > εij)

Support level: ∂∂sijk

Pr(Yij = 1 | Tij = k) > 0 where

sijk =

{s∗ijk if βj ≥ 0−s∗ijk otherwise


The Proposed Model (Continued)

Hierarchical modeling:

xiindep.∼ N (Z>i δ, σ

2x )

sijkindep.∼ N (Z>i λjk , ω

2jk )

λjki.i.d.∼ N (θk ,Φk )

“Noninformative” hyper prior on (αj , βj , δ, θk , ω2jk ,Φk )

Interpretation:spacial model vs. factor analysislearning vs. support


Quantities of Interest and Model Fitting

Average support level for each militant group k

τjk (Zi) = Z>i λjk for each policy j

κk (Zi) = Z>i θk averaging over all policies

Standardize them by dividing the (posterior) standard deviation ofideal points

Bayesian Markov chain Monte Carlo algorithmMultiple chains to monitor convergenceImplementation via JAGS (Plummer)


Model for the Division Level Support

Ordered response with an intercept αjl varying across divisionsThe model specification:

xiindep.∼ N (δdivision[i],1)

sijkindep.∼ N (λk ,division[i], ω

2k )

δdivision[i]indep.∼ N (µprovince[i], σ

2province[i])

λk ,division[i]indep.∼ N (θk ,province[i],Φk ,province[i])

Averaging over policiesPartial pooling across divisions within each province


Estimated Division Level Support−

1.0

−0.

50.

00.

51.

0

Sta

ndar

dize

d Le

vel o

f Sup

port

Pakistani militant groups in KashmirMilitants fighting in AfghanistanAl−QaidaFirqavarana Tanzeems

Bah

awal

pur

n=

118

Der

a G

hazi

Kha

n n

=0

Fais

alab

ad

n=31

3

Guj

ranw

ala

n=

403

Laho

re

n=57

9

Mul

tan

n=

495

Raw

alpi

ndi

n=

208

Sar

godh

a n

=13

1

Hyd

erab

ad

n=20

3

Kar

achi

n=

473

Lark

ana

n=

311

Mir

purk

has

n=

0

Suk

kur

n=

293

Ban

nu

n=0

Der

a Is

mai

l Kha

n n

=84

Haz

ara

n=

287

Koh

at

n=50

Mal

akan

d n

=0

Mar

dan

n=

215

Pes

haw

ar

n=28

8

Kal

at

n=10

3

Mak

ran

n=

0

Nas

iraba

d n

=21

0

Que

tta

n=32

0

Sib

i n

=67

Zho

b n

=61

Punjab Sindh NWFP Balochistan


Model with Individual Covariates

Ordered response with an intercept αjl varying across divisionsThe model specification:

xiindep.∼ N (δdivision[i] + Z>i δ

Z ,1)

sijkindep.∼ N (λk ,division[i] + Z>i λ

Zk , ω

2k )

δdivision[i]indep.∼ N (µprovince[i], σ

2province[i])

λk ,division[i]indep.∼ N (θk ,province[i],Φk ,province[i])

Expands upon the division level model to include individual levelcovariates:

gender, urban/rural, education, incomeIndividual level covariate effects after accounting for differencesacross divisionsPoststratification on these covariates using the census


Estimated Effects of Individual Covariates

−0.

2−

0.1

0.0

0.1

0.2

Pakistani militant groups in KashmirS

tand

ardi

zed

Leve

l of S

uppo

rt

●

●

●

●

●

●●

● ● ●

Female Rural Income Education −0.

2−

0.1

0.0

0.1

0.2

Militants fighting in Afghanistan

Sta

ndar

dize

d Le

vel o

f Sup

port

●

●

●

● ●●

●

● ●

●

Female Rural Income Education

−0.

2−

0.1

0.0

0.1

0.2

Al−Qaida

Sta

ndar

dize

d Le

vel o

f Sup

port

●

●

●

●

●

● ●●

●

●

Female Rural Income Education −0.

2−

0.1

0.0

0.1

0.2


Sta

ndar

dize

d Le

vel o

f Sup

port

●

●

●

●

● ●● ●

●

●

Female Rural Income Education

Demographics play a small role in explaining support for groupsKosuke Imai (Princeton) Endorsement Experiments West Coast Experiment 16 / 23

Regional Clustering of the Support for Al-Qaida


Association between Support and Violence

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

0

50

100

150

200

Division−Level Estimated Support (Proposed Method)

Num

ber

of In

cide

nts

correlation = −0.574

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.2 0.0 0.2 0.4 0.6

0

50

100

150

200

Division−Level Estimated Support (Ordered Probit)

Num

ber

of In

cide

nts


●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

0

50

100

150

200


Num

ber

of In

cide

nts


●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.2 0.0 0.2 0.4 0.6

0

50

100

150

200


Num

ber

of In

cide

nts


●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

0

50

100

150

200


Num

ber

of In

cide

nts


●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.2 0.0 0.2 0.4 0.6

0

50

100

150

200


Num

ber

of In

cide

nts

correlation = 0.021

●●

● ●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

0

50

100

150

200


Num

ber

of In

cide

nts


●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.4 −0.2 0.0 0.2 0.4 0.6

0

50

100

150

200


Num

ber

of In

cide

nts



Militants fighting in Afghanistan

Al−Qaida


Strong negativeassociation betweensupport and violence

Much weaker associationfor the standard orderedprobit model (divisiondummies, treatmentvariables interacted withdivision dummies)


Ideology, Support, and Violence

No strong relationship between:ideology and violenceideology and support

●●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

−0.5 0.0 0.5

0

50

100

150

200

Division−Level Estimated Ideal Point

Num

ber

of In

cide

nts


●

●

●

●

● ●●

●

●

●

● ●

●

●

●

●

●

●

●

●

●

−0.5 0.0 0.5

−0.25

−0.20

−0.15

−0.10

−0.05

0.00

0.05

0.10

Division−Level Estimated Ideal Point

Div

isio

n−Le

vel A

vera

ge E

stim

ated

S

uppo

rt fo

r M

ilita

nt G

roup

s correlation = 0.087


Simulation Studies

1 Based on the Pakistani DataSame 2 models plus province-level issue ownership modelTop-level parameters held constant across simulationsSample sizes and distribution same as beforeIdeal points, endorsements and responses follow IRT models

2 Varying sample sizesModel for division-level estimates with no covariatesModel for province-level estimates with no covariates but supportvarying across policiesN = 1000,1500,2000Again, top-level parameters held constant across simulations whileideal points, endorsements and responses follow IRT models

100 simulations under each scenario (3 chains, 60000 iterations)Frequentist evaluation of Bayesian estimators


Monte Carlo Evidence based on the Pakistani Data

Den

sity

−0.10 −0.05 0.00 0.05 0.10

0

10

20

30

40

Den

sity

0.80 0.85 0.90 0.95 1.00

0

5

10

15

20

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Effect Size

Pro

port

ion

Sta

tistic

ally

Sig

nific

ant

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●

Den

sity

−0.10 −0.05 0.00 0.05 0.10

0

10

20

30

40

Den

sity

0.80 0.85 0.90 0.95 1.00

0

5

10

15

20

●●●●●●

●

●●●●●

●

●

●

●●

●●● ●

●

●

●

●

●

●

●

●

●

●●

●●

●●●●

●

●

●

●

●

●

●

●●

●

●

● ●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●●●

●●

●

● ●

●

●

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Effect Size

Pro

port

ion

Sta

tistic

ally

Sig

nific

ant

The

Div

isio

n M

odel

W

ith In

divi

dual

Cov

aria

tes

The

Div

isio

n M

odel

Bias Coverage Rate of the 90% Confidence Intervals

Statistical Power

α level = 0.10


Monte Carlo Evidence with Varying Sample Size

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

N=1000 N=1500 N=2000

−0.2

−0.1

0.0

0.1

0.2

Bias

Bia

s

●

● ●

N=1000 N=1500 N=2000

0.75

0.80

0.85

0.90

0.95

1.00

1.05

Coverage Rate of the 90% Confidence Intervals

Cov

erag

e R

ate

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Statistical Power

Effect Size

Pro

port

ion

Sta

tistic

ally

Sig

nific

ant

●

●●●●●●

●●●●●●●●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●●

●●

●

●

●●●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

N=2000N=1500N=1000

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

N=1000 N=1500 N=2000

−0.2

−0.1

0.0

0.1

0.2

Bia

s

●

● ●

N=1000 N=1500 N=2000

0.75

0.80

0.85

0.90

0.95

1.00

1.05

Cov

erag

e R

ate

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Effect Size

Pro

port

ion

Sta

tistic

ally

Sig

nific

ant

●

●●●●●●

●●●●●●●●

●

●●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●●

●●

●

●

●●●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●

●

●

●

●●

●

●

N=2000N=1500N=1000

The

Div

isio

n M

odel

The

Div

isio

n M

odel

W

ith In

divi

dual

Cov

aria

tes

α level = 0.10


Concluding Remarks

Survey methodology to study sensitive questions

Endorsement ExperimentsMost indirect form of questioningApplicability limited to measuring supportAnalysis based on the ideal points frameworkMultilevel modeling to efficient estimation of spatial patterns

Design considerations:Policy positions should not be well-knownResponse distribution should not be skewedPolicies should belong to a single dimensionMeasure policy positions and political knowledge separately


statistical analysis of endorsement experiments:...

Documents