reading efron's 1979 paper on bootstrap

INTRODUCTIONDESCRIPTION OF METHODS

BOOTSTRAP IN REGRESSION MODELSBAYESIAN BOOTSTRAP

DISCUSSIONBAG OF LITTLE BOOTSTRAP

Bootstrap Methods:Another Look at the Jackknife

Marco Brandi

TSI-EuroBayes StudentUniversity Paris Dauphine

26 November 2012 / Reading Seminar on Classics

Marco Brandi Bootstrap Methods: Another Look at the Jackknife




"To pull oneself up by one is bootstrap"

Rudolph Erich Raspe





OUTLINE

1 INTRODUCTION

2 DESCRIPTION OF METHODSMETHOD 1METHOD 2METHOD 3

3 BOOTSTRAP IN REGRESSION MODELS

4 BAYESIAN BOOTSTRAP

5 DISCUSSION

6 BAG OF LITTLE BOOTSTRAP





Outline

1 INTRODUCTION




5 DISCUSSION






PRESENTING THE PROBLEM

X = (X1, . . . ,Xn)

Xi ∼ F with F completely unspecified

GOAL

⇓

Given R(X,F ) estimate R on the basis of x = (x1, . . . , xn)





INTRODUCTION JACKKNIFE METHOD

θ(F ) parameter of interest and t(X) its estimatorR(X,F ) = t(X)− θ(F )

R(X,F ) = t(X)− ˆBias(t)−θ(F )

ˆ(Var(t))1/2

ˆBias(t) and ˆVar(t) are obtained recomputing t(·) n times , eachtime removing one component of X





BOOTSTRAP METHOD

BOOTSTRAP METHODat x1, x2, . . . , xn put mass 1/n

F̂ is the sample probability distributionX ∗i = x∗i X ∗i ∼ F̂ i = 1, . . . ,nX∗ boostrap sampleR∗ = R(X∗, F̂ )





BOOTSTRAP METHOD

BOOTSTRAP METHODat x1, x2, . . . , xn put mass 1/nF̂ is the sample probability distribution

X ∗i = x∗i X ∗i ∼ F̂ i = 1, . . . ,nX∗ boostrap sampleR∗ = R(X∗, F̂ )





BOOTSTRAP METHOD

BOOTSTRAP METHODat x1, x2, . . . , xn put mass 1/nF̂ is the sample probability distributionX ∗i = x∗i X ∗i ∼ F̂ i = 1, . . . ,n

X∗ boostrap sampleR∗ = R(X∗, F̂ )





BOOTSTRAP METHOD

BOOTSTRAP METHODat x1, x2, . . . , xn put mass 1/nF̂ is the sample probability distributionX ∗i = x∗i X ∗i ∼ F̂ i = 1, . . . ,nX∗ boostrap sample

R∗ = R(X∗, F̂ )





BOOTSTRAP METHOD

BOOTSTRAP METHODat x1, x2, . . . , xn put mass 1/nF̂ is the sample probability distributionX ∗i = x∗i X ∗i ∼ F̂ i = 1, . . . ,nX∗ boostrap sampleR∗ = R(X∗, F̂ )





SIMPLE EXAMPLE

Dichotomous Example

θ(F ) = Pr{X = 1} R(X,F ) = X̄ − θ(F )

{X ∗i = 1 x̄ = θ(F̂ )

X ∗i = 0 1− x̄

⇓

R∗ = R(X∗, F̂ ) = X̄ ∗ − x̄

E∗(X̄ ∗ − x̄) = 0 Var∗(X̄ ∗ − x̄) = x̄(1− x̄)/n





PROBLEM

The complexity on the bootstrap procedure is to calculatethe bootstrap distribution

⇓

3 methods of calculation are possible





METHOD 1METHOD 2METHOD 3

Outline

1 INTRODUCTION




5 DISCUSSION







Method 1

Direct theoretical calculation






ESTIMATING THE MEDIAN 1ST STEP

Initializing the procedure

θ(F ) indicate the median of F

t(X) = X(m)

X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1R(X,F ) = t(X)− θ(F )








θ(F ) indicate the median of Ft(X) = X(m)

X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1R(X,F ) = t(X)− θ(F )









X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1

R(X,F ) = t(X)− θ(F )









X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1R(X,F ) = t(X)− θ(F )







Formalazing the procedureX∗ = x∗

N∗i = #{X ∗i = xi} N∗ = (N∗1 ,N∗1 , . . . .N

∗n)

R∗ = R(X∗, F̂ ) = X ∗(m) − x(m)

Pr∗{R∗ = x(l) − x(m)} =Pr{Bin(n,l − 1

n) ≤ m − 1}−

−Pr{Bin(n,ln

) ≤ m − 1}(1)






RESULTS(1)

for n = 15 and m = 8l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8

(1) .0003 .0040 .0212 .0627 .1249 .1832 .2073

Use E∗(R∗)2 =∑15

l=1[x(l) − x(8)]2Pr∗{

R∗ = x(l) − x(8)}

as an estimate of EF R2 = EF [t(X)− θ(F )]2






RESULTS(2)

Results for bootstrap

limn→∞ nE∗(R∗)2 = 1/4f 2(θ)

Results for the standard jackknife

limn→∞ n ˆVar(R) = (1/4f 2(θ))[χ2

2]2






Outline

1 INTRODUCTION




5 DISCUSSION







METHOD 2 - MONTE CARLO APPROXIMATION

Repeat X∗ B times

x∗1,x∗2, . . . ,x∗B

R(x∗1, F̂ ),R(x∗2, F̂ ), . . . ,R(x∗B, F̂ )

is taken as an approximation of the boostrap distribution






EXAMPLE(1)

Xi ∼ Pois(2) i = 1, . . . ,15

t(X) = E [X]

B = 10000n◦of bootstrap samplesmean = 1.9341se = 0.382






EXAMPLE(1)

Xi ∼ Pois(2) i = 1, . . . ,15

Histogram of bootstrap mean

Bootstrap estimation of mean

De

nsity

0.5 1.0 1.5 2.0 2.5 3.0 3.5

0.0

0.4

0.8

t(X) = E [X]







EXAMPLE(1)

Xi ∼ Pois(2) i = 1, . . . ,15



De

nsity

0.5 1.0 1.5 2.0 2.5 3.0 3.5

0.0

0.4

0.8

t(X) = E [X]

B = 10000n◦of bootstrap samples

mean = 1.9341se = 0.382






EXAMPLE(1)

Xi ∼ Pois(2) i = 1, . . . ,15



De

nsity

0.5 1.0 1.5 2.0 2.5 3.0 3.5

0.0

0.4

0.8

t(X) = E [X]

B = 10000n◦of bootstrap samplesmean = 1.9341

se = 0.382






EXAMPLE(1)

Xi ∼ Pois(2) i = 1, . . . ,15



De

nsity

0.5 1.0 1.5 2.0 2.5 3.0 3.5

0.0

0.4

0.8

t(X) = E [X]







EXAMPLE(2)

t(X) = V [X]







EXAMPLE(2)

Histogram of bootstrap variance

Bootstrap estimation of variance

De

nsity

0 1 2 3 4 5

0.0

0.2

0.4

t(X) = V [X]







EXAMPLE(2)



De

nsity

0 1 2 3 4 5

0.0

0.2

0.4

t(X) = V [X]

B = 10000n◦of bootstrap samples

mean = 2.191se = 0.649






EXAMPLE(2)



De

nsity

0 1 2 3 4 5

0.0

0.2

0.4

t(X) = V [X]

B = 10000n◦of bootstrap samplesmean = 2.191

se = 0.649






EXAMPLE(2)



De

nsity

0 1 2 3 4 5

0.0

0.2

0.4

t(X) = V [X]







R CODE

## s imu la t i on poisson dataset . seed (592)x= rpo i s (15 , lambda=2)B=10000## create the boots t rap f u n c t i o nboots t rap <− f u n c t i o n ( data , nboot , theta , . . . ){

z <− l i s t ( )datab <−

mat r i x ( sample ( data , s i ze= leng th ( data )∗nboot , rep lace=TRUE) , nrow=nboot )estb <− apply ( datab ,1 , theta , . . . )es t <− t he ta ( data , . . . )z$est <− estz$ d i s t n <− estbz$bias <− mean( estb)−estz$se <− sd ( estb )z

}## Est imat ing the meanX1=boots t rap ( x ,B, the ta=mean)h i s t (X1$ d is tn , main=" Histogram of boo ts t rap mean" , prob=T ,x lab=" Boots t rap es t ima t ion o f mean" )mean(X1$ d i s t n )X1$se






Outline

1 INTRODUCTION




5 DISCUSSION







METHOD 3 - RELATIONSHIP WITH THE JACKKNIFE

P∗i = N∗i /n P∗ = (P∗1 ,P∗2 , . . . ,P

∗n)

E∗P∗ = e/n Cov∗P∗ = I/n2 − e′e/n3






USING TAYLOR EXPANSION

R(P∗) = R(X∗, F̂ ) evaluate in P∗ = e/n

R(P∗) = R(e/n) + (P∗ − e/n)U +12

(P∗ − e/n)V(P∗ − e/n)′

U =

...

∂R(P∗)∂P∗

i...

P∗=e/n

V =

...

......

... ∂2R(P∗)∂P∗

i ∂P∗j

......

......

P∗=e/n






DERIVATION OF BOOTSTRAP EXPECTATION ANDVARIANCE

R(P∗) = R(

P∗∑ni=1 P∗i

)eU = 0 eV = −nU′ eVe′ = 0

E∗R(P∗) = R(e/n) +12

trV[I/n2 − e′e/n3

]= R(e/n) +

12n

V̄

Var∗R(P∗) = U′[I/n2 − e′e/n3

]U =

n∑i=1

U2i /n

2






RESULTS

BiasFθ(F̂ ) ≈ 12n V̄

VarFθ(F̂ ) ≈∑n

i=1 U2i /n

2

The results agree with those given by Jaeckel’s infinitesimaljackknife





Outline

1 INTRODUCTION




5 DISCUSSION






REGRESSION MODELS

Xi = gi(β) + εi εi ∼ F i = 1, . . . ,n

Having observed X = x we compute the estimate of β

β̂ = minβn∑

i=1

[xi − gi

(β̂)]2

F̂ : mass1n

at ε̂i = xi − gi

(β̂)





BOOTSTRAP SAMPLE

X ∗i = gi

(β̂)

+ ε∗i ε∗i ∼ F̂

β̂∗ : minβn∑

i=1

[x∗i − gi

(β̂)]2

β̂∗1, β̂∗2, β̂∗3, . . . , β̂∗B





LINEAR MODEL

gi(β) = ciβ C′C = G

β̂ = G−1C′X has mean β and covariance matrix σ2F G−1

β̂∗ = G−1C′X∗ has boostrap mean and variance

E∗β̂∗ = β̂ Cov∗β̂∗ = σ̂2G−1

where σ̂2 =∑n

i=1

[xi − g

(β̂)]2

/n





JACKKNIFE IN LINEAR REGRESSION

Applying the infinitesimal jackknife in a linear regression model,Hinkley derive the approximation of

Cov β̂ ≈ G−1

[n∑

i=1

c′i ci ε̂2i

]G−1

Jackknife methods ignore that the errors εi are assumed tohave the same distribution for every value of i





Outline

1 INTRODUCTION




5 DISCUSSION






DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981)

Bayesian BootstrapIn bootstrap we consider sample cdf is population cdf

Each BB replications generates a posterior probability foreach xi

The posterior probability of each xi is centered at 1n but has

variability





DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin1981)

Bayesian BootstrapIn bootstrap we consider sample cdf is population cdfEach BB replications generates a posterior probability foreach xi

The posterior probability of each xi is centered at 1n but has

variability





BB REPLICATION

BB replication

(n − 1) Unif (0,1) u(0) = 0 e u(n) = 1

gl = u(l) − u(l−1)

Attach the vector (g1, . . . ,gn) to the data X





BB REPLICATION

BB replication

(n − 1) Unif (0,1) u(0) = 0 e u(n) = 1gl = u(l) − u(l−1)

Attach the vector (g1, . . . ,gn) to the data X





CONCEPTUAL DIFFERENCE

Bayesian BootstrapSimulates the posterior distribution of the parameter

Classical BootstrapSimulates the estimated sampling distribution of a statistic





BB EXAMPLE

Dichotomous Example

The parameter is θ = Pr{Xi = 1} and let n1 number of Xi = 1

Call P1 the sum of the n1 probabilities assigned to the xi = 1

(g1, . . . ,gn) ∼ Dirichlet(1, . . . ,1)⇒ P1 ∼ Beta(n1,n − n1)

Note: Beta(n1,n − n1) is the posterior distribution when theprior is P(θ) ∝ [θ(1− θ)]−1





Outline

1 INTRODUCTION




5 DISCUSSION






INFERENCES PROBLEMS

Is it possible that all the values of X have been observed?Is it reasonable to assume a priori independentparameters, constrained only to sum to 1, for these values?

Using the gap to simulate the posterior distributions ofparameters may no longer work

so..BB and bootstrap cannot avoid the sensitivity of inference to

model assumptions





INFERENCES PROBLEMS

Is it possible that all the values of X have been observed?

Is it reasonable to assume a priori independentparameters, constrained only to sum to 1, for these values?



model assumptions





INFERENCES PROBLEMS

Is it possible that all the values of X have been observed?Is it reasonable to assume a priori independentparameters, constrained only to sum to 1, for these values?



model assumptions





CONCLUSION

Knowledge of the context of a data set may make theincorporation of reasonable model constraints obvious and

bootstrap may be useful in particular contexts

In general"There are no general data analytic panaceas thatallow us to pull ourselves up by our bootstraps"

Donald Rubin





Outline

1 INTRODUCTION




5 DISCUSSION






BLB (M. Jordan 2012)

When n gets large computational cost is largeExpected numbers of distinct points in a resample is ∼ 0.632n

BLB ProcedureDivide the dataset in s subset of dimension b, with b < n

From each subset we draw r samples with replacement ofdimension nCompute for each subset the estimator quality assessment(e.g the bias) indicated with ξ







BLB ProcedureDivide the dataset in s subset of dimension b, with b < nFrom each subset we draw r samples with replacement ofdimension n

Compute for each subset the estimator quality assessment(e.g the bias) indicated with ξ







BLB ProcedureDivide the dataset in s subset of dimension b, with b < nFrom each subset we draw r samples with replacement ofdimension nCompute for each subset the estimator quality assessment(e.g the bias) indicated with ξ





BLB IMAGE





FINALLY...

if we choose b = n0.6 ad we have a dataset of 1TB, thesubsamples contains at most 3981 distinct points and have sizeat most 4GB

Like the bootstrapShare bootstrap’s consistencyAutomatic : without knowledge of the internals θ

Beyond the bootstrapCan explicity control b

Generally faster than the bootstrap and requires less totalcomputation





FINALLY...

if we choose b = n0.6 ad we have a dataset of 1TB, thesubsamples contains at most 3981 distinct points and have sizeat most 4GB

Like the bootstrapShare bootstrap’s consistencyAutomatic : without knowledge of the internals θ

Beyond the bootstrapCan explicity control bGenerally faster than the bootstrap and requires less totalcomputation





References I

B. Efron.Bootstrap Methods: Another Look at the Jackknife.The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26.

D.B. Rubin.The Bayesian Bootstrap.The Annals of Statistics, Vol. 9, No.1, pp. 130-134.

M. Jordan.The Big Data Bootstrap.Proceedings of the 29th International Conference onMachine Learning (ICML).





THANK YOU

FOR

YOUR ATTENTION


reading efron's 1979 paper on bootstrap

Documents

bootstrap procedure

x f xi

x x e x x

description of methods

var x x

xn xi f

xixi f i

basis of x