Planning Sample Size for Randomized EvaluationsJed Friedman, World BankBased on slides from Esther Duflo, J-PAL

Planning Sample Size for Randomized EvaluationsGeneral question: How large does the sample need to be to credibly detect a given effect size? What does Credibly mean here? It means that I can be reasonably sure that the difference between the group that received the program and the group that did not is due to the programRandomization removes bias, but it does not remove noise: it works because of the law of large numbers how large much large be?

Basic set upAt the end of an experiment, we will compare the outcome of interest in the treatment and the comparison groups. We are interested in the difference: Mean in treatment - Mean in control = Effect sizeFor example: mean of the number of bed nets in villages with free distribution v. mean of the number of bed nets in villages with cost recovery

Estimation

But we do not observe the entire population, just a sample

In each village of the sample, there is a given number of bed nets. It is more or less close to the actual mean in the total population, as a function of all the other factors that affect the number of bed nets

We estimate the mean by computing the average in the sample

If we have very few villages, the averages are imprecise. When we see a difference in sample averages, we do not know whether it comes from the effect of the treatment or from something else

Estimation

The size of the sample:Can we conclude if we have one treated village and one non treated village? Can we conclude if we give IPT to one classroom and not the other? Even though we have a large class size?What matter is the effective sample size i.e. the number of treated units and control units (e.g. class rooms). What is it the unit the case of IPT given in the classroom?

The variability in the outcome we try to measure: If there are other many non-measured things that explain our outcomes, it will be harder to say whether the treatment really changed it.

When the Outcomes are Very Precise

Chart4

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

10

30

60

120

180

170

220

130

30

40

00

11

012

012

016

020

013

011

09

03

03

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Less Precision

Chart5

00

00

00

00

00

10

00

10

20

20

20

20

50

10

50

51

40

81

81

80

61

23

75

32

21

45

87

25

47

28

26

06

13

14

17

06

14

02

04

05

00

01

00

00

00

00

01

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Can we conclude?

Chart6

00

00

20

20

20

11

20

21

11

30

20

31

40

40

22

33

61

42

31

23

43

52

20

14

14

40

43

36

47

47

03

23

25

35

23

04

03

24

02

21

01

00

21

02

10

00

03

00

177

178

079

380

081

082

183

084

085

086

087

088

089

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Confidence Intervals

The estimated effect size (the difference in the sample averages) is valid only for our sample. Each sample will give a slightly different answer. How do we use our sample to make statements about the overall population? A 95% confidence interval for an effect size tells us that, for 95% of any samples that we could have drawn from the same population, the estimated effect would have fallen into this interval. The Standard error (se) of the estimate in the sample captures both the size of the sample and the variability of the outcome (it is larger with a small sample and with a variable outcome)Rule of thumb: a 95% confidence interval is roughly the effect plus or minus two standard errors.

Hypothesis TestingOften we are interested in testing the hypothesis that the effect size is equal to zero (we want to be able to reject the hypothesis that the program had no effect)We want to test:

Against:

Two Types of Mistakes First type of error : Conclude that there is an effect, when in fact there are no effect.

The level of your test is the probability that you will falsely conclude that the program has an effect, when in fact it does not.So with a level of 5%, you can be 95% confident in the validity of your conclusion that the program had an effect

For policy purpose, you want to be very confident in the answer you give: the level will be set fairly low. Common level of a: 5%, 10%, 1%.

Relation with Confidence Intervals

If zero does not belong to the 95% confidence interval of the effect size we measured, then we can be at least 95% sure that the effect size is not zero.

So the rule of thumb is that if the effect size is more than twice the standard error, you can conclude with more than 95% certainty that the program had an effect

Two Types of MistakesSecond type of error: you fail to reject that the program had no effect, when it fact it does have an effect.

The Power of a test is the probability that I will be able to find a significant effect in my experiment if indeed there truly is an effect (higher power are better since I am more likely to have an effect to report)Power is a planning tool. It tells me how likely it is that I find a significant effect for a given sample sizeOne minus the power is the probability to be disappointed.

Calculating Power

When planning an evaluation, with some preliminary research we can calculate the minimum sample we need to get to: Test a pre-specified hypothesis: program effect was zero or not zeroFor a pre-specified level (e.g. 5%)Given a pre-specified effect size (what you think the program will do)To achieve a given powerA power of 80% tells us that, in 80% of the experiments of this sample size conducted in this population, if there is indeed an effect in the population, we will be able to say in our sample that there is an effect with the level of confidence desired. The larger the sample, the larger the power.

Common Power used: 80%, 90%

Ingredients for a Power Calculation in a Simple Study

Picking an Effect SizeWhat is the smallest effect that should justify the program to be adopted:Cost of this program v the benefits it bringsCost of this program v the alternative use of the moneyIf the effect is smaller than that, it might as well be zero: we are not interested in proving that a very small effect is different from zeroIn contrast, any effect larger than that effect would justify adopting this program: we want to be able to distinguish it from zeroCommon danger: picking effect size that are too optimisticthe sample size may be set too low!

Standardized Effect SizesHow large an effect you can detect with a given sample depends on how variable the outcomes is. Example: If all children have very similar learning level without a program, a very small impact will be easy to detectThe standard deviation captures the variability in the outcome. The more variability, the higher the standard deviation isThe Standardized effect size is the effect size divided by the standard deviation of the outcomed = effect size/St.dev. Common effect sizes:

d=0.20 (small) d =0.40 (medium) d =0.50 (large)

The Design Factors that Influence PowerThe level of randomization Availability of a BaselineAvailability of Control Variables, and Stratification. The type of hypothesis that is being tested.

Level of RandomizationClustered DesignCluster randomized trials are experiments in which social units or clusters rather than individuals are randomly allocated to intervention groupsExamples:

Reason for Adopting Cluster RandomizationNeed to minimize or remove contaminationExample: In the deworming program, schools was chosen as the unit because worms are contagiousBasic Feasibility considerationsExample: The PROGRESA program would not have been politically feasible if some families were introduced and not others. Only natural choiceExample: Any education intervention that affect an entire classroom (e.g. flipcharts, teacher training).

Impact of ClusteringThe outcomes for all the individuals within a unit may be correlated All villagers are exposed to the same weatherAll patients share a common health practitionerAll students share a schoolmasterThe program affect all students at the same time. The member of a village interact with each otherThe sample size needs to be adjusted for this correlationThe more correlation between the outcomes, the more we need to adjust the standard errors

Example of Group Effect Multipliers________________________________Intra-Class Randomized Group Size Correlation 10 50 100 2000.00 1.00 1.00 1.00 1.000.02 1.09 1.41 1.73 2.230.05 1.20 1.86 2.44 3.310.10 1.38 2.43 3.30 4.57 ____________________________________________

ImplicationsIt is extremely important to randomize an adequate number of groups. Often the number of individual within groups matter less than the number of groups

Think that the law of large number applies only when the number of groups that are randomized increase

You CANNOT randomize at the level of the district, with one treated district and one control district!!!!

Availability of a BaselineA baseline has three main uses:Can check whether control and treatment group were the same or different before the treatmentReduce the sample size needed, but requires that you do a survey before starting the intervention: typically the evaluation cost go up and the intervention cost go downCan be used to stratify and form subgroupsTo compute power with a baseline:You need to know the correlation between two subsequent measurement of the outcome (for example: consumption measured in two years). The stronger the correlation, the bigger the gain.Very big gains for very persistent outcomes such as LFP;

Control VariablesIf we have additional relevant variables (e.g. village population, block where the village is located, etc.) we can also control for them

What matters now for power is ,the residual variation after controlling for those variables

If the control variables explain a large part of the variance, the precision will increase and the sample size requirement decreases.

Warning: control variables must only include variables that are not INFLUENCED by the treatment: variables that have been collected BEFORE the intervention.

Stratified SamplesStratification: create BLOCKS by value of the control variables and randomize within each blockStratification ensure that treatment and control groups are balanced in terms of these control variables. This reduces variance for two reasons: it will reduce the variance of the outcome of interest in each strata the correlation of units within clusters.Example: if you stratify by district for an IRS programAgroclimatic and associated epidemiologic factors are controlled for The common district government effect disappears.

The Design Factors that Influence PowerClustered designAvailability of a BaselineAvailability of Control Variables, and Stratification. The type of hypothesis that is being tested.

The Hypothesis that is being TestedAre you interested in the difference between two treatments as well as the difference between treatment and control? Are you interested in the interaction between the treatments?Are you interested in testing whether the effect is different in different subpopulations?Does your design involve only partial compliance? (e.g. encouragement design?)

Power Calculations Using the OD SoftwareChoose Power v. number of clusters in the menu clustered randomized trials

Cluster SizeChoose cluster size

Choose Significance Level,Treatment Effect, and CorrelationPick a : levelNormally you pick 0.05Pick d :Can experiment with 0.20Pick the intra class correlation (rho) You obtain the resulting graph showing power as a function of sample size.

Power and Sample Size

Conclusions: Power Calculation in PracticePower calculations involve some guess work. At times we do not have the right information to conduct it very properlyHowever, it is important to spend effort on them:Avoid launching studies that will have no power at all: waste of time and moneyDevote the appropriate resources to the studies that you decide to conduct (and not too much).