chapter 5 multiple random variablesjwatkins/e5_clt.pdf · chapter 5 multiple random variables...

Classical Central Theorem Motivation The Classical Central Limit Theorem Slutsky’s Theorem

Chapter 5Multiple Random Variables

Central Limit Theorem

1 / 19


Outline

Classical Central Theorem

MotivationBernoulli Random VariablesExponential Random Variables

The Classical Central Limit TheoremExponential Random VariablesBernoulli Trials and the Continuity Correction

Slutsky’s Theorem

2 / 19


Motivation

For the law of large numbers, the sample means from a sequence of independentrandom variables converge to their common distributional mean as the number n ofrandom variables increases.

1

nSn = Xn → µ as n→∞.

Moreover, the standard deviation of Xn is inversely proportional to√n.

Thus, we magnify the difference between the running average and the mean by afactor of

√n and investigate the graph of

√n

(1

nSn − µ

)versus n

3 / 19


Motivation

Does the distribution of the size of these fluctuations have any regular and predictablestructure? Let’s begin by examining the distribution for the sum of X1,X2 . . .Xn,independent and identically distributed random variables

Sn = X1 + X2 + · · ·+ Xn.

What distribution do we see? We begin with the simplest case, Xi Bernoulli randomvariables. The sum Sn is a binomial random variable. We examine two cases.

• keep the number of trials the same at n = 100 and vary the success probability p.

• keep the success probability the same at p = 1/2, but vary the number of trials.

4 / 19


Bernoulli Random Variables

0 20 40 60 80 100

0.00

0.02

0.04

0.06

0.08

0.10

x

probability

0 20 40 60 80 100

0.00

0.02

0.04

0.06

0.08

0.10

x

probability

0 20 40 60 80 100

0.00

0.02

0.04

0.06

0.08

0.10

x

probability

0 20 40 60 80 100

0.00

0.02

0.04

0.06

0.08

0.10

x

probability

Successes in 100 Bernoulli trials with p = 0.2, 0.4, 0.6 and 0.8.5 / 19



0 10 20 30 40 50 60

0.00

0.05

0.10

0.15

x

probability

0 10 20 30 40 50 60

0.00

0.05

0.10

0.15

x

probability

0 10 20 30 40 50 60

0.00

0.05

0.10

0.15

x

probability

Successes in 20, 40, and 80 Bernoulli trials with p = 0.5.6 / 19



The binomial random variable Sn has

mean np and standard deviation√

np(1− p).

Thus, if we take the standardized version of these sums of Bernoulli random variables

Zn =Sn − np√np(1− p)

,

then these bell curve graphs would lie on top of each other.

Now, let’s consider exponential random variables . . . .

7 / 19


Exponential Random Variables

-3 -2 -1 0 1 2 3

0.0

0.1

0.2

0.3

0.4

0.5

x

density

-3 -2 -1 0 1 2 3

0.0

0.1

0.2

0.3

0.4

0.5

x

density

-3 -2 -1 0 1 2 3

0.0

0.1

0.2

0.3

0.4

0.5

x

density

-3 -2 -1 0 1 2 3

0.0

0.1

0.2

0.3

0.4

0.5

x

density

-3 -2 -1 0 1 2 3

0.0

0.1

0.2

0.3

0.4

0.5

x

density

The density of the standardized random variables that result from the sum of 2,4,8,16, and 32

exponential random variables8 / 19


The Classical Central Limit Theorem

To obtain the standardized random variables,

• we can either standardize using the sum Sn having mean nµ and standarddeviation σ

√n, , or

• we can standardize using the sample mean Xn having mean µ and standarddeviation σ/

√n.

This yields two equivalent versions of the standardized score or z-score.

Zn =Sn − nµ

σ√n

=Xn − µσ/√n

=

√n

σ(Xn − µ).

The theoretical result behind these numerical explorations is called the classical centrallimit theorem.

9 / 19



Theorem. Let {Xi ; i ≥ 1} be independent random variables having a commondistribution. Let µ be their mean and σ2 be their variance. Then Zn, the standardizedscores of the sample means, converges in distribution to Z a standard normal randomvariable, i.e., the distribution function FZn converges to Φ, the distribution function ofthe standard normal for every value z .

limn→∞

FZn(z) = limn→∞

P{Zn ≤ z} =1√2π

∫ z

−∞e−x

2/2 dx = Φ(z).

In practical terms the central limit theorem states that

P{a < Zn ≤ b} ≈ P{a < Z ≤ b} = Φ(b)− Φ(a).

The number value is obtained in R using the command pnorm(b)-pnorm(a).

10 / 19


The Classical Central Limit TheoremWe will prove this in the case that the Xi have a moment generating function MX (t)for the interval t ∈ (−h, h) by showing that

limn→∞

MZn(t) = expt2

2

or equivalently, show that the cumulant generating functions KZn(t) = logMZn(t)satisfy

limn→∞

KZn(t) =t2

2Write

Yi =Xi − µσ

then

Zn =1√n

n∑i=1

Yi .

11 / 19



Let MY (t) denote the moment generating function for the Yi .

MZn(t) = E exp(tZn) = E exp

(t√n

n∑i=1

Yi

)=

n∏i=1

E exp

(t√nYi

)=

(MY

(t√n

))n

and

KZn(t) = n logMY

(t√n

)= nKY

(t√n

).

Recall that for the cumulant generating function KY ,

K ′Y (0) = EY1 = 0, K ′′Y (0) = Var(Y1) = 1.

12 / 19



Finally, from two applications of L’Hopital’s rule,

limn→∞

KZn(t) = limn→∞

nKY

(t√n

)= lim

ε→0

KY (εt)

ε2

= limε→0

tK ′Y (εt)

2ε= lim

ε→0

t2K ′′Y (εt)

2.

=t2K ′′Y (0)

2=

t2

2.

Note that this is continuous at t = 0, thus we have convergence in distribution. Thecumulant generating function KZ (t) = t2/2. So, the limiting distribution is a standardnormal.

13 / 19



• The more skewed the distribution, the more observations are need to obtain thebell curve shape.

• Based on simulations with the goal of having adequate confidence intervals,Sugden et al. (2000) provide a minimum sample size from independent identicallydistributed observations.

n∗ = 28 + 25γ21 .

where γ1 is the skewness.

• For the exponential distribution, γ1 = 2 and n∗ = 128.

• For the Bernoulli distribution, γ1 = (1− 2p)/√

p(1− p). For p = 1/3, 2/3,γ = ±1/

√2, and n∗ = 40.5.

14 / 19


Exponential Random VariablesFor exponential random variables, the mean, µ = 1/λ and the standard deviation,σ = 1/λ and therefore

Zn =Sn − n/λ√

n/λ=

Xn − 1/λ

1/(λ√n)

=√n(λXn − n).

Let X144 be the mean of 144 independent with parameter λ = 1. Thus, µ = 1 andσ = 1. Note that n = 144 is sufficiently large for the use of a normal approximation.

P{X144 < 1.1} = P{

12(X144 − 1) < 12(1.1− 1)}

= P{Z144 < 1.2} ≈ 0.8649.

With S144 ∼ Γ(144, 1), we compare.

> pnorm(1.2);pgamma(1.1*144,144,1)

[1] 0.8849303

[1] 0.8829258

15 / 19


Bernoulli Trials and the Continuity CorrectionFor Bernoulli random variables, µ = p and σ =

√p(1− p). The sample mean is

generally denoted by p to indicate the fact that it is a sample proportion.The normalized version of p

Zn =pn − p√p(1− p)/n

,

For 100 trials of a fair coin,

Z100 =p − 1/2

1/20= 20(p − 1/2)

and

P{p ≤ 0.40} = P{p − 1/2 ≤ 0.40− 1/2} = P{20(p − 1/2) ≤ 20(0.4− 1/2)}= P{Z100 ≤ −2} ≈ P{Z ≤ −2} = 0.023.

16 / 19


Bernoulli Trials and the Continuity Correction

We can improve the normal approximation to the binomial random variable byemploying the continuity correction. For a binomial random variable X , thedistribution function

P{X ≤ x} = P{X < x + 1} =x∑

y=0

P{X = y}

can be realized as the area of x + 1 rectangles, height P{X = y}, y = 0, 1, . . . , x andwidth 1. These rectangles look like a Riemann sum for the integral up to the valuex + 1/2.

17 / 19


Bernoulli Trials and the Continuity CorrectionExample. For X ∼ Bin(100, 0.3),P{X ≤ 32} is the area of 33 rectan-gles with right side at the value 32.5.

25 26 27 28 29 30 31 32 33 34

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

number of successes

Figure: Bin(100, 0.3) mass function(black) and approximating densityN(100 · 0.3,

√100 · 0.3 · 0.7).

For the approximating normal Y , this suggestscomputing P{Y ≤ 32.5}.

> pbinom(32,100,0.3)

[1] 0.7107186

> n<-100; p<-0.3;mu<-n*p

> sigma<-sqrt(p*(1-p))

> x<-c(32,32.5,33)

> prob<-pnorm((x-mu)/(sigma*sqrt(n)))

> data.frame(x,prob)

x prob

1 32.0 0.6687397

2 32.5 0.7073105

3 33.0 0.7436546

18 / 19


Slutsky’s TheoremSome useful extensions of the central limit theorem are based on Slutsky’s theorem.

Theorem. Let Xn →D X and Yn →P c , a constant as n→∞. Then

1. YnXn →D cX , and2. Xn + Yn →D X + c.

For example, by the law of large numbers, the sample variance

S2n →a.s. σ2,

the distribution variance as n→∞. Thus,σ

Sn→a.s. 1.

This ratio also converges in probability. So, by Slutsky’s theorem, the t-statistic

Xn − µSn/√n

=σ

Sn· Xn − µσ/√n

=σ

Sn· Zn →D 1 · Z ,

a standard normal as n→∞.19 / 19

chapter 5 multiple random variablesjwatkins/e5_clt.pdf · chapter 5 multiple random variables...

Documents