Capeto

wn

PODER

Lab Experiments in the Field1

7/8

/14

2

Capeto

wn

OUTLINE FOR TODAY:

Leftover from yesterday Multi-site experiments: Solidarity

7/8

/14

3

Capeto

wn

NUTS AND BOLTS

7/8

/14

4

Capeto

wn

SECOND STEPS:(AFTER THE QUESTION/THEORY)

Instrumentation Construct Validity – how will I test what I want to

test? Paper/Pencil or Computer? Timeline of experiment Instructions

Sampling/Randomization What subject pool? How will Treatment be randomized?

Analysis Plan What are the units of analysis Power tests

7/8

/14

SUBJECT SELECTION, I Convenience Samples: students

Students advantages: Convenient, inexpensive and relatively homogeneous

Student disadvantages: May behave differently from target population,

young, educated, and talk to each other (diffusion) Classroom:

Representative sample of students Environment might affect behavior:

Lab: May select certain students Neutral environment

Data: Eckel and Grossman ExEc: Students give more to charity in the classroom than in the

lab Why?

5

Capeto

wn

7/8

/14

SUBJECT SELECTION, II

Specialized Groups:ElderlyProfessionalsMedical casesPoorResidents of hurricane-vulnerable areasPublic officials

Population Samples Pluses: External validity, Heterogeneity Minuses: Costly, risk of decreased control,

heterogeneity6

Capeto

wn

7/8

/14

SUBJECT SELECTION III

Subject selection should suit the question you are asking

Theory testing: Independent of subject characteristics?

Policy (measurement or institutional design): Target group subjects

Examples: WEIRD people (Henrich, et al. 2010) People from other cultures (Barr and Serra

2010)

7

Capeto

wn

7/8

/14

8

Capeto

wn

SECOND STEPS:(AFTER THE QUESTION/THEORY)

Instrumentation Construct Validity – how will I test what I want to

test? Paper/Pencil or Computer? Timeline of experiment Instructions

Sampling/Randomization What subject pool? How will Treatment be randomized?

Analysis Plan What are the units of analysis Power tests

7/8

/14

NUTS AND BOLTS, I Lab log. IRB and Ethics Pilot experiments. Lab set-up Subject registration Experimenter(s) Monitor(s) Randomizing Devices Instructions Subject confidence (non-deception)

9

Capeto

wn

7/8

/14

10

Capeto

wn

LAB BOOK (LUPIA & VARIAN 2010) 1. State your objectives. 2. State a theory. 3. Explain how focal hypotheses are derived from the theory if the

correspondence between a focal hypothesis and a theory is not 1:1. 4. Explain the criteria by which data for evaluating the focal hypotheses were

selected or created. 5. Record all steps that convert human energy and dollars into datapoints. 6. State the empirical model to be used for leveraging the data in the service of

evaluating the focal hypothesis. (a) All procedures for interpreting data require an explicit defense. (b) When doing more than simply offering raw comparisons of observed differences between treatment and control groups, offer an explicit defense of why a given structural relationship between observed outcomes and experimental variables and/or set of control variables is included.

7. Report the findings of the initial observation. 8. If the findings cause a change to the theory, data, or model, explain why the

changes were necessary or sufficient to generate a more reliable inference. 9. Do this for every subsequent observation so that lab members and other

scholars can trace the path from hypothesis to data collection to analytic method to every published empirical claim.

ELNs: OneNote in Microsoft or Growlybird Notes for the Mac (http://www.growlybird.com/GrowlyBird/Notes.html)

7/8

/14

NUTS AND BOLTS, I Lab log. IRB and Ethics Pilot experiments. Lab set-up Subject registration Experimenter(s) Monitor(s) Randomizing Devices Instructions Subject confidence (non-deception)

11

Capeto

wn

7/8

/14

12

ETHICS

IRB keeps us honest (some countries don’t have) Focus on potential harm to subjects Consent, debriefing limit harm, but may impact

sample Balance between potential benefit and risk

Field experiments: No consent process! Unwitting subject, high potential

cost Findley et al 2014. – no consent, no debriefing Correspondence studies on discrimination (more later) Intervention studies: elections, political institutions Facebook study on emotional contagion: no consent,

potential risk, very low potential benefit

7/8

/14

Capeto

wn

NUTS AND BOLTS, I Lab log. IRB and Ethics Pilot experiments. Lab set-up Subject registration Experimenter(s) Monitor(s) Randomizing Devices Instructions Subject confidence (non-deception)

13

Capeto

wn

7/8

/14

NUTS AND BOLTS, II Subject questions “Learning periods” Experiment Recording data Termination of experiment Debriefing Subject payment Bankruptcy Backup plan

14

Capeto

wn

7/8

/14

15

Capeto

wn

WRITEUP AND REPORTING

Biases in published data Registration and CONSORT

7/8

/14

16

Capeto

wn

BIASES IN PUBLISHED DATA

Selective reporting + publication bias => many published studies have p=.05.

Data mining and selective presentation of results have been a concern in economics for a long time

These concerns are not limited to Economics: Medical trials, Ioannidis (2005, “Why most published

research findings are false”) Psychology, Simmons et al. 2011, “False - positive

psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant”)

Political science, Humphreys et al. (2012, “Fishing”) Finds affect millions of people. How to fix?

7/8

/14

17

Capeto

wn

EXAMPLE: (GERBER AND MALHOTRA, AJPS)

7/8

/14

18

Capeto

wn

ECONOMICS, “STAR WARS” (BRODEUR ET AL 2013)

7/8

/14

Capeto

wn

19

CONSORT/REGISTRATION (HUMPHREYS ET AL, 2013)

CONSORT Statement: improve the reporting of a randomized controlled trial (RCT), enabling readers to understand a trial's design, conduct, analysis and interpretation, and to assess the validity of its results. http://www.consort-

statement.org/

7/8

/14

20

Capeto

wn

REGISTRATION

Benefit: Limits selective reporting/fishing Rounds out “body of evidence” Forces researcher to think through design,

statistical analysis

Potential costs Limits exploratory research Serendipitous findings may be hard to publish

But: frees it from the burden of (false) presentation as formal hypothesis testing.

7/8

/14

GENERAL REMARK: LAB V. FIELD

Lab has greater internal validity Lab is cheap, field is costly Lab mistakes can be fixed; often

not so in field Students v. population

Population has higher variance, harder to detect effects

Selection bias is not limited to labGreater monitoring costs to ensure

population sample21

Capeto

wn

7/8

/14

22

Capeto

wn

IF TIME, HERE ARE UDT SLIDES

7/8

/14

THE ULTIMATUM GAME

Task: Two players must divide a fixed amount-$100 Game is played once only

Players Proposer- chooses a split Respondent-accepts or rejects

Payoffs: If accepted, money split as planned If rejected, both players get zero

Game theory: Start with responder. Payoff-maximizer accepts

anything >0 Therefore proposer offers smallest possible amount

$5 ULTIMATUM GAME (ECKEL AND GROSSMAN, RUN IN 1992, PUB. 2000)

Figure 1: Distribution of Offers, All Treatments

0% rejected

0% rejected

0% rejected

1.6%rejected

100%rejected

83.3% rejected

56.5% rejected

20.9%rejected

0

0.1

0.2

0.3

0.4

0.5

0.6

0 0.5 1 1.5 2 2.5 3 3.5 4

Amount Offered

Per

cen

t o

f O

ffer

s Rejected

Accepted

ULTIMATUM GAME RESULTS

Unequal offers are rejected Payoff-maximizing offer is modal offer is

60/40 split Most common split is 60/40 Looks as if proposers are “rational” What about responders?

(Why do they reject?)

BEHAVIOR IS INCONSISTENT WITH STANDARD MODEL:

Why do people offer >0? Value other’s payoff (Altruism) Afraid to be rejected (lose it all)

Risk averse?

Why do people reject? Fairness: think the distribution is not fair overall

Note we see people rejecting really high offers, too. Selfish fairness: care about own relative payoff Low offer means cost of punishment is low

THE DICTATOR “GAME”

Task: two players must divide a fixed amount-$100

Players Proposer- chooses a split Respondent-passively accepts

Dictator game allows player 1 to make an altruistic allocation

We use this game to measure altruism

REAL PEOPLE PLAY DICTATOR GAME : HISTOGRAM OF DECISIONS

Giving in the Dictator Game

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

10 9 8 7 6 5 4 3 2 1

Amount Kept

Pe

rce

nt

of

Su

bje

cts

DICTATOR RESULTS/COMPARISON

2/3 of subjects are selfish Many give away something – altruistic Measure of altruism!

Varies across individuals Reliable Correlated with behavior like volunteering

Different conditions lead to different outcomes “Double blind” (see prev. slide) Identity Charity

MEN AND WOMEN – DICTATORAmount Donated by Men and Women

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 1 2 3 4 5 6 7 8 9 10

Dollars Donated to Anonymous Recipient

Perc

en

t o

f D

ec

isio

ns

Women

MenMen donate $0.82

Women donate $1.60

Source: Eckel and Grossman, Economic Journal, 1998

TRUST GAME

Player A and Player B both begin with $100. (important!)

Player A decides how much, if any, to send to Player B. Any amount sent is tripled on the way to B.

Player B decides how much, if any to send back to Player A.

Game theory: B returns 0, so A sends 0 This game is used to measure trust and

reciprocity.

32

Capeto

wn

TRUST GAME RESULTS:

People send positive amounts Trust (just) pays on average In the field, higher levels of trust and

reciprocity Measure of individualized trust (More on this later).

7/8

/14

TESTING THEORY V. MEASUREMENT

These games violate predictions of standard game theory (assuming payoff-maximizing agents) Led to a huge amount of research (experimental

and theoretical) These games are useful for measuring

preferences and social norms

LAB EXPERIMENTS IN THE FIELD

Lab or field? False dichotomy Lab experiments complement field exps.

Lab experiments in the field (extra-lab) for measurement, etc.

Lab experiments back home to settle methodological issues that arise in the field

Pretest and refine experimental designs before going into the field

Issues: External validity (perpetually)

34

Capeto

wn

35

NEXT: SOLIDARITY

7/8

/14