3. distributions - isye home | isyesman/courses/6739/6739-03...discrete distributions poisson...

3. Distributions

Dave Goldsman

Georgia Institute of Technology, Atlanta, GA, USA

6/10/14

Goldsman 6/10/14 1 / 74

Outline

1 Discrete DistributionsBernoulli and Binomial DistributionsHypergeometric DistributionGeometric and Negative Binomial DistributionsPoisson Distribution

2 Continuous DistributionsUniform DistributionExponential, Erlang, and Gamma DistributionsOther Continuous Distributions

3 Normal DistributionBasicsStandard Normal DistributionSample Mean of Normal ObservationsCentral Limit TheoremExtensions of the Normal Distribution

4 Computer StuffGoldsman 6/10/14 2 / 74

Discrete Distributions Bernoulli and Binomial Distributions

Bernoulli(p) and Binomial(n, p) Distributions

The Bern(p) distribution is given by

X =

{1 w.p. p (“success”)

0 w.p. q (“failure”)

Recall: E[X] = p, Var(X) = pq, and MX(t) = pet + q.

Further, X1, . . . , Xniid∼ Bern(p)⇒ Y ≡

∑ni=1Xi ∼ Bin(n, p).

P (Y = k) =

(n

k

)pkqn−k, k = 0, 1, . . . , n.

Example: Toss 2 dice 5 times. Let Y be the number of 7’s you see.Y ∼ Bin(5, 1/6). Then, e.g.,

P (Y = 4) =

(5

4

)(1

6

)4(5

6

)5−4.

Goldsman 6/10/14 3 / 74

Discrete Distributions Bernoulli and Binomial Distributions

Y ∼ Bin(n, p) implies

E[Y ] = E

[ n∑i=1

Xi

]=

n∑i=1

E[Xi] = np

and, similarly,Var(Y ) = npq.

We’ve already seen that MY (t) = (pet + q)n.

Binomials add up: If Y1, . . . , Yk are indep and Yi ∼ Bin(ni, p), then

k∑i=1

Yi ∼ Bin( k∑i=1

ni, p).

Goldsman 6/10/14 4 / 74

Discrete Distributions Hypergeometric Distribution

Hypergeometric Distribution

You have a objects of type 1 and b objects of type 2.

Select n objects w/o replacement from the a+ b.

Let X be the number of type 1’s selected.

P (X = k) =

(ak

)(b

n−k)(

a+bn

) , k = 0, 1, . . . ,min(a, n).

Goldsman 6/10/14 5 / 74

Discrete Distributions Hypergeometric Distribution

After some algebra, it turns out that

E[X] = n

(a

a+ b

)and

Var(X) = n

(a

a+ b

)(1− a

a+ b

)(a+ b− na+ b− 1

).

Example: 25 sox in a box. 15 red, 10 blue. Pick 7 w/o replacement.

P (exactly 3 reds are picked) =

(153

)(104

)(257

)

Goldsman 6/10/14 6 / 74

Discrete Distributions Geometric and Negative Binomial Distributions

Geometric(p) and Negative Binomial(r, p) Distributions

Definition: Suppose we consider an infinite sequence of indep Bern(p)trials.

Let Z equal the number of trials until the first success is obtained. Theevent Z = k corresponds to k − 1 failures, and then a success. Thus,

P (Z = k) = qk−1p, k = 1, 2, . . . ,

and we say that Z has the Geometric(p) distribution.

Goldsman 6/10/14 7 / 74


The mgf of the Geom(p) is

MZ(t) = E[etZ ] =

∞∑k=1

etkqk−1p

= pet∞∑k=0

(qet)k

=pet

1− qet, for qet < 1.

So

MZ(t) =pet

1− qet, for t < `n(1/q).

Goldsman 6/10/14 8 / 74


Thus,

E[Z] =d

dtMZ(t)

∣∣∣∣t=0

=(1− qet)(pet)− (−qet)(pet)

(1− qet)2

∣∣∣∣t=0

=pet

(1− qet)2

∣∣∣∣t=0

=p

(1− q)2=

1

p.

(We can also prove this directly from the definition of expected value.)

Similarly, after a lot of algebra,

E[Z2] =d2

dt2MZ(t)

∣∣∣∣t=0

=2− pp2

,

so that

Var(Z) = E[Z2]− (E[Z])2 =2− pp2− 1

p2=

q

p2.

Goldsman 6/10/14 9 / 74


Example: Toss a die repeatedly. What’s the prob that we observe a ‘3’for the first time on the 8th toss?

Answer: The number of tosses we need is Z ∼ Geom(1/6).P (Z = 8) = (5/6)7(1/6).

How many tosses would we expect to take?

Answer: E[Z] = 1/p = 6 tosses.

Goldsman 6/10/14 10 / 74


Memoryless Property of Geometric

Theorem: Suppose that Z ∼ Geom(p). Then for positive integers s, t,we have

P (Z > s+ t|Z > s) = P (Z > t).

Why is it called the memoryless property? If an event hasn’t occurredby time s, the prob that it will occur after an additional t time units is thesame as the (unconditional) prob that it will occur after time t — itforgot that it made it past time s!

Goldsman 6/10/14 11 / 74


Proof: First of all, for any t = 0, 1, 2, . . .,

P (Z > t) =

∞∑j=t+1

qj−1p = pqt∞∑j=0

qj =pqt

1− q= qt. (∗)

In addition,

P (Z > s+ t|Z > s) =P (Z > s+ t ∩ Z > s)

P (Z > s)

=P (Z > s+ t)

P (Z > s)

=qs+t

qsby (*)

= qt.

Thus, P (Z > s+ t|Z > s) = P (Z > t). Done.Goldsman 6/10/14 12 / 74


Example: Let’s toss a die until a 5 appears for the first time. Supposethat we’ve already made 4 tosses without success. What’s the probthat we’ll need more than 2 more tosses before we observe a 5?

Let Z be the number of tosses required. By the memoryless prop (withs = 4 and t = 2) and (*), we want

P (Z > 6|Z > 4) = P (Z > 2) = (5/6)2.

Fun Fact: The Geom(p) is the only discrete distribution with thememoryless property.

Not-as-Fun Fact: Some books define the Geom(p) as the number ofBern(p) failures until you observe a success. # failures = # trials − 1.You should be aware of this inconsistency, but don’t worry about it now.

Goldsman 6/10/14 13 / 74


Definition: Suppose we consider an infinite sequence of indep Bern(p)trials.

Now let W equal the number of trials until the rth success is obtained.W = r, r + 1, . . .. The event W = k corresponds to exactly r − 1successes by time k − 1, and then the rth success at time k.

We say that W has the Negative Binomial(r, p) distribution (aka thePascal distrn).

Example: ‘FFFFSFS’ corresponds to W = 7 trials until the r = 2ndsuccess.

Notation: W ∼ NegBin(r, p).

Remark: As with the Geom(p), the exact definition of the NegBindepends on what book you’re reading.

Goldsman 6/10/14 14 / 74


Theorem: If Z1, . . . , Zriid∼ Geom(p), then

W =∑n

i=1 Zi ∼ NegBin(r, p).

Proof: Won’t do it here, but you can use the mgf technique.

Anyhow, it makes sense if you think of Zi as the number of trials afterthe (i− 1)st success up to and including the ith success.

Since the Zi’s are i.i.d., the above theorem gives:

E[W ] = rE[Zi] = r/p,

Var(W ) = rVar(Zi) = rq/p2,

MW (t) = [MZi(t)]r =

(pet

1− qet

)r.

Goldsman 6/10/14 15 / 74


Just to be complete, let’s get the pmf of W . Note that W = k iff getexactly r− 1 successes by time k− 1, and then the rth success at timek. So. . .

P (W = k) =

[(k − 1

r − 1

)pr−1qk−r

]p =

(k − 1

r − 1

)prqk−r, k = r, r + 1, . . .

Example: Toss a die until a 5 appears for the third time. What’s theprob that we’ll need exactly 7 tosses?

Let W be the number of tosses required. Clearly, W ∼ NegBin(3, 1/6).

P (W = 7) =

(7− 1

3− 1

)(1/6)3(5/6)7−3.

Goldsman 6/10/14 16 / 74


How are the Binomial and NegBin Related?

X1, . . . , Xniid∼ Bern(p) ⇒ Y ≡

∑ni=1Xi ∼ Bin(n, p).

Z1, . . . , Zriid∼ Geom(p) ⇒ W ≡

∑ri=1 Zi ∼ NegBin(r, p).

E[Y ] = np, Var(Y ) = npq.

E[W ] = r/p, Var(W ) = rq/p2.

Goldsman 6/10/14 17 / 74

Discrete Distributions Poisson Distribution

Poisson Distribution

We’ll first talk about Poisson processes.

Let N(t) be a counting process. That is, N(t) is the number ofoccurrences (or arrivals, or events) of some process over the timeinterval [0, t]. N(t) looks like a step function.

Examples: N(t) could be any of the following.(a) Cars entering a shopping center (time).(b) Defects on a wire (length).(c) Raisins in cookie dough (volume).

Let λ > 0 be the average number of occurrences per unit time (orlength or volume).

In the above examples, we might have:(a) λ = 10/min. (b) λ = 0.5/ft. (c) λ = 4/in3.

Goldsman 6/10/14 18 / 74


A Poisson process is a specific counting process. . .

First, some notation: o(h) is a generic function that goes to zero fasterthan h goes to zero.

Definition: A Poisson process is one that satisfies the following:

(1) There is a short enough interval of time h, such that, for all t,

P (N(t+ h)−N(t) = 0) = 1− λh+ o(h)

P (N(t+ h)−N(t) = 1) = λh+ o(h)

P (N(t+ h)−N(t) ≥ 2) = o(h)

(2) The distribution of the “increment” N(t+ h)−N(t) only depends onthe length h.

(3) If a < b < c < d, then the two “increments” N(d)−N(c) andN(b)−N(a) are indep RV’s.

Goldsman 6/10/14 19 / 74


English translation of Poisson process assumptions.

(1) Arrivals basically occur one-at-a-time, and then at rate λ/unit time.(We must make sure that λ doesn’t change over time.)

(2) The arrival pattern is stationary — it doesn’t change over time.

(3) The numbers of arrivals in two disjoint time intervals are indep.

Poisson Process Example: Neutrinos hit a detector. Occurrences arerare enough so that they really do happen one-at-a-time. You never getarrivals of groups of neutrinos. Further, the rate doesn’t vary over time,and all arrivals are indep of each other.

Anti-Example: Customers arrive at a restaurant. They show up ingroups, not one-at-a-time. The rate varies over the day (more atdinnertime). Arrivals may not be indep. This ain’t a Poisson process.

Goldsman 6/10/14 20 / 74


Definition: Let X be the number of occurrences in a Poisson(λ)process in a unit interval of time. Then X has the Poissondistribution with parameter λ.

Notation: X ∼ Pois(λ).

Theorem/Definition: X ∼ Pois(λ)⇒ P (X = k) = e−λλk/k!,k = 0, 1, 2, . . ..

Remark: The value of λ can be changed simply by changing the unitsof time.

Example:X = # calls to a switchboard in 1 minute ∼ Pois(3)Y = # calls to a switchboard in 5 minutes ∼ Pois(15)Z = # calls to a switchboard in 10 sec ∼ Pois(0.5)

Goldsman 6/10/14 21 / 74


Theorem: X ∼ Pois(λ)⇒ mgf is MX(t) = eλ(et−1).

Proof:

MX(t) = E[etX ] =

∞∑k=0

etk(e−λλk

k!

)= e−λ

∞∑k=0

(λet)k

k!= e−λeλe

t.

Theorem: X ∼ Pois(λ)⇒ E[X] = Var(X) = λ.

Proof (using mgf):

E[X] =d

dtMX(t)

∣∣∣∣t=0

=d

dteλ(et−1)

∣∣∣∣t=0

= λetMX(t)

∣∣∣∣t=0

= λ.

Goldsman 6/10/14 22 / 74


Similarly,

E[X2] =d2

dt2MX(t)

∣∣∣∣t=0

=d

dt

(d

dtMX(t)

)∣∣∣∣t=0

= λd

dt(etMX(t))

∣∣∣∣t=0

= λ

[etMX(t) + et

d

dtMX(t)

]∣∣∣∣t=0

= λet[MX(t) + λetMX(t)

]∣∣∣∣t=0

= λ(1 + λ).

Thus, Var(X) = E[X2]− (E[X])2 = λ(1 + λ)− λ2 = λ. Done.

Example: Calls to a switchboard arrive as a Poisson process with rate3 calls/min. Let X = number of calls in 40 sec.

So X ∼ Pois(2), E[X] = Var(X) = 2, P (X ≤ 3) =∑3

k=0 e−22k/k!.

Goldsman 6/10/14 23 / 74


Theorem (Additive Property of Poissons): Suppose X1, . . . , Xn areindep with Xi ∼ Pois(λi), i = 1, . . . , n. Then

Y ≡n∑i=1

Xi ∼ Pois( n∑i=1

λi

).

Proof: Since the Xi’s are indep, we have

MY (t) =

n∏i=1

MXi(t) =

n∏i=1

eλi(et−1) = e(

∑ni=1 λi)(e

t−1),

which is the mgf of the Pois(∑n

i=1 λi) distribution.

Example: Cars driven by males [females] arrive at a parking lotaccording to a Poisson process with a rate of 3/hr [5/hr]. All arrivals areindep. What’s the probability of exactly 2 arrivals in the next 30 min?

The total number of arrivals is Pois(8/hr), and so the total in the next30 min is X ∼ Pois(4). So P (X = 2) = e−442/2!.

Goldsman 6/10/14 24 / 74

Continuous Distributions

Outline





Continuous Distributions Uniform Distribution

Uniform(a, b) Distribution

Recall that

f(x) =

{1b−a a < x < b

0 otherwise

Previous work showed that

E[X] =a+ b

2and Var(X) =

(a− b)2

12.

We can also derive the mgf,

MX(t) = E[etX ] =

∫ b

aetx

1

b− adx =

etb − eta

t(b− a).

Goldsman 6/10/14 26 / 74

Continuous Distributions Exponential, Erlang, and Gamma Distributions

Exponential, Erlang, and Gamma Distributions

Recall that the Exponential(λ) distribution has pdf

f(x) =

{λe−λx x > 0

0 otherwise

Previous work showed that the cdf F (x) = 1− e−λx,

E[X] = 1/λ, and Var(X) = 1/λ2.

We also derived the mgf,

MX(t) = E[etX ] =

∫ ∞0

etxf(x) dx =λ

λ− t, t < λ.

Goldsman 6/10/14 27 / 74


Memoryless Property of Exponential

Theorem: Suppose that X ∼ Exp(λ). Then for positive s, t, we have

P (X > s+ t|X > s) = P (X > t).

Similar to the discrete Geometric distribution, the prob that X willsurvive an additional t time units is the (unconditional) prob that it willsurvive at least t — it forgot that it made it past time s!

Proof:

P (X > s+ t|X > s) =P (X > s+ t ∩X > s)

P (X > s)=

P (X > s+ t)

P (X > s)

=e−λ(s+t)

e−λs= e−λt = P (X > t).

Goldsman 6/10/14 28 / 74


Example: Suppose that the life of a lightbulb is exponential with amean of 10 months. If the light survives 20 months, what’s the probthat it’ll survive another 10?

P (X > 30|X > 20) = P (X > 10) = e−λx = e−(1/10)(10) = e−1.

Example: If the time to the next bus is exponentially distributed with amean of 10 minutes, and you’ve already been waiting 20 minutes, youcan expect to wait 10 more.

Remark: The exponential is the only cts distrn with the memorylessproperty.

Remark: Look at E[X] and Var(X) for the Geometric distrn and seehow they’re similar to those for the exponential. (Not a coincidence.)

Goldsman 6/10/14 29 / 74


Definition: If X is a cts RV with pdf f(x) and cdf F (x), then its failurerate function is

S(t) ≡ f(t)

P (X > t)=

f(t)

1− F (t),

which can loosely be regarded as X ’s instantaneous rate of death,given that it has so far survived to time t.

Example: If X ∼ Exp(λ), then S(t) = λe−λt/e−λt = λ. So if X is theexponential lifetime of a lightbulb, then its instantaneous burn-out rateis always λ — always good as new! This is clearly a result of thememoryless property.

Goldsman 6/10/14 30 / 74


The Exponential is also related to the Poisson!

Let X be the amount of time until the first arrival in a Poisson processwith rate λ. Then X ∼ Exp(λ).

Proof:

F (x) = P (X ≤ x) = 1− P (no arrls in [0, x])

= 1− e−λx(λx)0

0!(since # arrls in [0, x] is Pois(λx))

= 1− e−λx.

Amazingly, it can be shown (after a lot of work) that the interarrivaltimes of a Poisson process are all i.i.d. Exp(λ)! See for yourself whenyou take a stochastic processes course.

Goldsman 6/10/14 31 / 74


Example: Suppose that arrivals to a shopping center are from aPoisson process with rate λ = 20/hr. What’s the probability that thetime between the 13th and 14th customers will be at least 4 minutes?

Let the time between custs 13 and 14 be X. Since we have a Poissonprocess, the interarrivals are iid Exp(λ = 20/hr), so

P (X > 4 min) = P (X > 1/15 hr) = e−λt = e−20/15.

=

Goldsman 6/10/14 32 / 74


Definition: Suppose X1, . . . , Xkiid∼ Exp(λ), and let S =

∑ki=1Xi. Then

S has the Erlangk ditribution with parameter λ.

The Erlang is simply the sum of i.i.d. exponentials.

Special Case: Erlang1(λ) ∼ Exp(λ).

The pdf and cdf of the Erlang are

f(s) =λke−λssk−1

(k − 1)!, s ≥ 0,

F (s) = 1−k−1∑i=0

e−λs(λs)i

i!.

Notice that the cdf is the sum of a bunch of Poisson probabilities.(Won’t do it here, but this observation helps in the derivation of the cdf.)

Goldsman 6/10/14 33 / 74


Expected value, variance, and mgf:

E[S] = E[ k∑i=1

Xi

]=

k∑i=1

E[Xi] = k/λ

Var(S) = k/λ2

MS(t) =

(λ

λ− t

)k.

Example: Suppose X and Y are i.i.d. Exp(2). Find P (X + Y < 1).

P (X + Y < 1) = 1−k−1∑i=0

e−λs(λs)i

i!= 1−

2−1∑i=0

e−(2·1)(2 · 1)i

i!= 0.594

Goldsman 6/10/14 34 / 74


Definition: X has the gamma distribution with parameters α > 0 andλ > 0 if it has pdf

f(x) =λαe−λxxα−1

Γ(α), x ≥ 0,

whereΓ(α) ≡

∫ ∞0

tα−1e−t dt

is the gamma function.

Remark: The gamma distrn generalizes the Erlang distrn (where α hasto be a positive integer). It has the same expected value and varianceas the Erlang, with α in place of k.

Remark: If α is a positive integer, then Γ(α) = (α− 1)!.Party trick: Γ(1/2) =

√π.

Goldsman 6/10/14 35 / 74

Continuous Distributions Other Continuous Distributions

Other Continuous Distributions

Triangular(a, b, c) Distribution — good for modeling RV’s on the basisof limited data (min, mode, max).

f(x) =

2(x−a)

(b−a)(c−a) a < x ≤ b2(c−x)

(c−b)(c−a) b < x < c

0 otherwise

E[X] =a+ b+ c

3and Var(X) = mess

Goldsman 6/10/14 36 / 74


Beta(a, b) Distribution — good for modeling RV’s that are restricted toan interval.

f(x) =

{Γ(a+b)

Γ(a)Γ(b)xa−1(1− x)b−1 0 < x < 1

0 otherwise

E[X] =a

a+ band Var(X) =

ab

(a+ b)2(a+ b+ 1)

This distribution gets its name from the beta function, which is definedas

β(a, b) ≡ Γ(a)Γ(b)

Γ(a+ b)=

∫ 1

0xa−1(1− x)b−1 dx.

Goldsman 6/10/14 37 / 74


Weibull(a, b) Distribution — good for modeling reliability models. a isthe “scale” parameter, and b is the “shape” parameter.

f(x) =

{ab(ax)b−1e−(ax)b x > 0

0 otherwise

F (x) = 1− exp[−(ax)b], x > 0.

E[X] = (1/a)Γ(1 + (1/b)) and Var(X) = slight mess

Goldsman 6/10/14 38 / 74


Remark: The exponential is a special case of the Weibull.

Example: Time-to-failure T for a transmitter has a Weibull distrn withrate a = 1/(200 hrs) and parameter b = 1/3. Then

E[T ] = 200Γ(1 + 3) = 1200 hrs.

The prob that it fails before 2000 hrs is

F (2000) = 1− exp[−(2000/200)(1/3)] = 0.884.

Goldsman 6/10/14 39 / 74


Cauchy distribution — good for disproving things!

f(x) =1

π(1 + x2)and F (x) =

1

2+

arctan(x)

π, x ∈ <.

Theorem: The Cauchy distribution has an undefined mean and infinitevariance!

Weird Fact: X1, . . . , Xniid∼ Cauchy⇒

∑ni=1Xi/n ∼ Cauchy. Even you

take the average of a bunch of Cauchys, you’re right back where youstarted!

Goldsman 6/10/14 40 / 74


Alphabet Soup of Other Distrns

χ2 distribution — coming up in the statistics portion

t distribution — coming up

F distribution — coming up

Pareto, LaPlace, Rayleigh, Gumbel distributions

Etc. . .

Goldsman 6/10/14 41 / 74

Normal Distribution

Outline





Normal Distribution Basics

Normal Distribution

So important that we’ll give it an entire section.

Definition: X ∼ Nor(µ, σ2) if it has pdf

f(x) =1√

2πσ2exp

[−(x− µ)2

2σ2

], ∀x ∈ <.

f(x) is “bell-shaped” and symmetric around x = µ, with tails falling offquickly as you move away from µ.

Small σ2 corresponds to a “tall, skinny” bell curve; large σ2 gives a“short, fat” bell curve.

Remark: The Normal distribution is also called the Gaussian distrn.

Examples: Heights, weights, SAT scores, crop yields, and averages ofthings tend to be normal.

Goldsman 6/10/14 43 / 74


Fun Fact (1):∫< f(x) dx = 1.

Proof: Transform to polar coordinates. Good luck.

Fun Fact (2): The cdf is

F (x) =

∫ x

−∞

1√2πσ2

exp

[−(t− µ)2

2σ2

]dt = ??

Remark: No closed-form solution for this. Stay tuned.

Fun Facts (3) and (4): E[X] = µ and Var(X) = σ2.

Proof: Integration by parts or mgf (below).

Fun Fact (5): The mgf is MX(t) = exp(µt+ 12σ

2t2).

Proof: Calculus (or look it up in a table of integrals).

Goldsman 6/10/14 44 / 74


Theorem (Additive property of normals): If X1, . . . , Xn are indep withXi ∼ Nor(µi, σ

2i ), i = 1, . . . , n, then

Y ≡n∑i=1

aiXi + b ∼ Nor( n∑i=1

aiµi + b,

n∑i=1

a2iσ

2i

).

So a linear combination of indep normals is itself normal.

Goldsman 6/10/14 45 / 74


Proof: Since Y is a linear function,

MY (t) = M∑i aiXi+b(t) = etbM∑

i aiXi(t)

= etbn∏i=1

MaiXi(t) (Xi’s indep)

= etbn∏i=1

MXi(ait) (mgf of linear fn)

= etbn∏i=1

exp

[µi(ait) +

1

2σ2i (ait)

2

](normal mgf)

= exp

[( n∑i=1

µiai + b)t+

1

2

( n∑i=1

a2iσ

2i

)t2]. (Done.)

Goldsman 6/10/14 46 / 74


Remark: A normal distrn is completely characterized by its mean andvariance.

By the above, we know that a linear combination of indep normals isstill normal. Therefore, when we add up indep normals, all we have todo is figure out the mean and variance — the normality of the sumcomes for free.

Example: X ∼ Nor(3, 4), Y ∼ Nor(4, 6) and X,Y indep. Find the distrnof 2X − 3Y .

Solution: This is normal with

E[2X − 3Y ] = 2E[X]− 3E[Y ] = 2(3)− 3(4) = −6

andVar(2X − 3Y ) = 4Var(X) + 9Var(Y ) = 70.

Thus, 2X − 3Y ∼ Nor(−6, 70).Goldsman 6/10/14 47 / 74


Corollary (of Theorem):

X ∼ Nor(µ, σ2) ⇒ aX + b ∼ Nor(aµ+ b, a2σ2).

Proof: Immediate from Theorem after noting that E[aX + b] = aµ+ band Var(aX + b) = a2σ2.

Corollary (of Corollary):

X ∼ Nor(µ, σ2) ⇒ Z ≡ X − µσ

∼ Nor(0, 1).

Proof: Use above Cor with a = 1/σ and b = −µ/σ.

Goldsman 6/10/14 48 / 74

Normal Distribution Standard Normal Distribution

Standard Normal Distribution

Definition: The Nor(0, 1) distrn is called the standard normaldistribution, and is often denoted by Z.

The Nor(0, 1) is nice because there are tables available for its cdf.

You can standardize any normal RV X into a standard normal byapplying the transformation Z = (X − µ)/σ. Then you can use the cdftables.

The pdf of the Nor(0, 1) is

φ(z) ≡ 1√2πe−z

2/2, z ∈ <.

The cdf isΦ(z) ≡

∫ z

−∞φ(t) dt, z ∈ <.

Goldsman 6/10/14 49 / 74


Remarks:

P (Z ≤ a) = Φ(a)P (Z ≥ b) = 1− Φ(b)P (a ≤ Z ≤ b) = Φ(b)− Φ(a)

Φ(0) = 1/2Φ(−b) = P (Z ≤ −b) = P (Z ≥ b) = 1− Φ(b)P (−b ≤ Z ≤ b) = Φ(b)− Φ(−b) = 2Φ(b)− 1

Then

P (µ− kσ ≤ X ≤ µ+ kσ) = P (−k ≤ Z ≤ k) = 2Φ(k)− 1.

So the probability that any normal RV is within k standard deviations ofits mean doesn’t depend on the mean or variance.

Goldsman 6/10/14 50 / 74


Famous Nor(0, 1) table values. Or you can use software calls, likeNORMDIST in Excel (which calculates the cdf for any normaldistribution.)

z Φ(z) = P (Z ≤ z)0.00 0.5000

1.00 0.8413

1.28 0.8997 ≈ 0.90

1.645 0.9500

1.96 0.9750

2.33 0.9901 ≈ 0.99

3.00 0.9987

4.00 ≈ 1.0000

Goldsman 6/10/14 51 / 74


By the discussion on the last two pages, the probability that anynormal RV is within k standard deviations of its mean is

P (µ− kσ ≤ X ≤ µ+ kσ) = 2Φ(k)− 1.

For k = 1, this probability is 2(0.8413)− 1 = 0.6826.

There is a 95% chance that a normal observation will be within 2 s.d.’sof its mean.

99.7% of all observations are within 3 standard deviations of the mean!

Goldsman 6/10/14 52 / 74


Famous Inverse Nor(0, 1) table values. Can also use software, suchas Excel’s NORMINV function, which actually calculates inverses forany normal distribution, not just standard normal.

Φ−1(p) is the value of z such that Φ(z) = p.

p Φ−1(p)

0.90 1.28

0.95 1.645

0.975 1.96

0.99 2.33

0.995 2.58

Goldsman 6/10/14 53 / 74


Example: X ∼ Nor(21, 4). Find P (19 < X < 22.5). Standardizing, weget

P (19 < X < 22.5)

= P

(19− µσ

<X − µσ

<22.5− µ

σ

)= P

(19− 21

2< Z <

22.5− 21

2

)= P (−1 < Z < 0.75)

= Φ(0.75)− Φ(−1)

= Φ(0.75)− [1− Φ(1)]

= 0.7734− [1− 0.8413] = 0.6147.

Goldsman 6/10/14 54 / 74


Example: Suppose that heights of men are M ∼ Nor(68, 4) andHeights of women are W ∼ Nor(65, 1).

Select a man and woman independently at random.

Find the probability that the woman is taller than the man.

Note that

W −M ∼ Nor(E[W −M ],Var(W −M))

∼ Nor(65− 68, 1 + 4) ∼ Nor(−3, 5).

Then

P (W > M) = P (W −M > 0)

= P

(Z >

0 + 3√5

)= 1− Φ(3/

√5)

≈ 1− 0.91 = 0.09.

Goldsman 6/10/14 55 / 74

Normal Distribution Sample Mean of Normal Observations

Sample Mean of Normal Observations

The sample mean of X1, . . . , Xn is X̄ ≡∑n

i=1Xi/n.

Corollary (of old Theorem):X1, . . . , Xn

iid∼ Nor(µ, σ2)⇒ X̄ ∼ Nor(µ, σ2/n).

Proof: By previous work, as long as X1, . . . , Xn are i.i.d. something, wehave E[X̄] = µ and Var(X̄) = σ2/n. Since X̄ is a linear combination ofindependent normals, it’s also normal. Done.

Remark: This result is very significant! As the number of observationsincreases, Var(X̄) gets smaller (while E[X̄] remains constant).

In the upcoming statistics portion of the course, we’ll learn that thismakes the sample mean X̄ an excellent estimator for the mean µ,which is typically unknown in practice.

Goldsman 6/10/14 56 / 74


Example: Suppose that X1, . . . , Xniid∼ Nor(µ, 16). Find the sample size

n such thatP (|X̄ − µ| ≤ 1) ≥ 0.95.

How many observations should you take so that X̄ will have a goodchance of being close to µ?

Solution: Note that X̄ ∼ Nor(µ, 16/n). Then

P (|X̄ − µ| ≤ 1) = P (−1 ≤ X̄ − µ ≤ 1)

= P

(−1

4/√n≤ X̄ − µ

4/√n≤ 1

4/√n

)= P

(−√n

4≤ Z ≤

√n

4

)= 2Φ(

√n/4)− 1.

Goldsman 6/10/14 57 / 74


Now we have to find n such that this probability is at least 0.95. . . .

2Φ(√n/4)− 1 ≥ 0.95 iff

Φ(√n/4) ≥ 0.975 iff

√n

4≥ Φ−1(0.975) = 1.96

iff n ≥ 61.47 or 62.

So if you take the average of 62 observations, then X̄ has a 95%chance of being within 1 of µ.

Goldsman 6/10/14 58 / 74

Normal Distribution Central Limit Theorem

Central Limit Theorem

The most important theorem in prob and stats.

Central Limit Theorem: Suppose X1, . . . , Xn are i.i.d. with E[Xi] = µand Var(Xi) = σ2. Then as n→∞,∑n

i=1Xi − nµσ√n

=X̄ − µσ/√n

d−→ Nor(0, 1),

where“ d−→ ” means that the cdf→ the Nor(0, 1) cdf.

Proof: Not in this class.

Goldsman 6/10/14 59 / 74


Remarks: (1) So if n is large, then X̄ ≈ Nor(µ, σ2/n).

(2) The Xi’s don’t have to be normal for the CLT to work! It even workson discrete distributions!

(3) You usually need n ≥ 30 observations for the approximation to workwell. (Need fewer observations if the Xi’s come from a symmetricdistribution.)

(4) You can almost always use the CLT if the observations are i.i.d.

(5) In fact, the CLT is actually a lot more general than the theorempresented here!

Goldsman 6/10/14 60 / 74


Example: Suppose X1, . . . , X100iid∼ Exp(1/1000). Find

P (950 ≤ X̄ ≤ 1050).

Solution: Recall that if Xi ∼ Exp(λ), then E[Xi] = 1/λ andVar(Xi) = 1/λ2.

Further, if X̄ is the sample mean based on n observations, then

E[X̄] = E[Xi] = 1/λ and

Var(X̄) = Var(Xi)/n = 1/(nλ2).

For our problem, λ = 1/1000 and n = 100, so that E[X̄] = 1000 andVar(X̄) = 10000.

Goldsman 6/10/14 61 / 74


So by the CLT,

P (950 ≤ X̄ ≤ 1050)

= P

(950− E[X̄]√

Var(X̄)≤ X̄ − E[X̄]√

Var(X̄)≤ 1050− E[X̄]√

Var(X̄)

)≈ P

(950− 1000

100≤ Z ≤ 1050− 1000

100

)≈ P

(−1

2≤ Z ≤ 1

2

)= 2Φ(1/2)− 1 = 0.383.

Note: This problem can be solved exactly if we have access to theExcel Erlang cdf function GAMMADIST. And what do you know, you endup with exactly the same answer of 0.383!

Goldsman 6/10/14 62 / 74


Example: Suppose X1, . . . , X100 are i.i.d. from some distribution withmean 1000 and standard deviation 1000. Find P (950 ≤ X̄ ≤ 1050).

Solution: By exactly the same manipulations as in the previousexample, the answer ≈ 0.383.

Notice that we didn’t care whether or not the data came from anexponential distrn. We just needed the mean and variance.

Goldsman 6/10/14 63 / 74


Normal Approximation to the Binomial(n, p)

Suppose Y ∼ Bin(n, p), where n is very large. In such cases, weusually approximate the Binomial via an appropriate Normaldistribution.

The CLT applies since Y =∑n

i=1Xi, where the Xi’s are i.i.d. Bern(p).

ThenY − E[Y ]√

Var(Y )=

Y − np√npq

≈ Nor(0, 1).

The usual rule of thumb for the Normal approximation to the Binomialis that it works pretty well as long as np ≥ 5 and nq ≥ 5.

Goldsman 6/10/14 64 / 74


Why do we need such an approximation?

Example: Suppose Y ∼ Bin(100, 0.8) and we want

P (Y ≥ 84) =

100∑i=84

(100

i

)(0.8)i(0.2)100−i.

Good luck with the binomial coefficients (they’re too big) and number ofterms to sum up (it’s going to get tedious). I’ll come back to visit you inan hour.

The next example shows how to use the approximation.

Note that it incorporates a “continuity correction” to account for the factthat the Binomial is discrete while the Normal is cts. If you don’t wantto use it, don’t worry too much.

Goldsman 6/10/14 65 / 74


Example: The Braves play 100 indep baseball games, each of whichthey have prob 0.8 of winning. What’s the prob that they win ≥ 84?

Y ∼ Bin(100, 0.8) and we want P (Y ≥ 84) (as in the last example). . .

P (Y ≥ 84) = P (Y ≥ 83.5) (“continuity correction”)

≈ P

(Z ≥ 83.5− np

√npq

)(CLT)

= P

(Z ≥ 83.5− 80√

16

)= P (Z ≥ 0.875) = 0.1908.

The actual answer (using the true Bin(100,0.8) distribution) turns out tobe 0.1923 — pretty close!

Goldsman 6/10/14 66 / 74

Normal Distribution Extensions of the Normal Distribution

Extensions of the Normal Distribution

(X,Y ) has the Bivariate Normal Distrn if it has pdf

f(x, y) = C exp

−[z2X(x)− 2ρzX(x)zY (y) + z2

Y (y)

]2(1− ρ2)

where

ρ ≡ Corr(X,Y ), C ≡ 1

2πσXσY√

1− ρ2,

zX(x) ≡ x− µXσX

and zY (y) ≡ y − µYσY

.

Goldsman 6/10/14 67 / 74


Pretty nasty joint pdf, eh?

In fact, X ∼ Nor(µX , σ2X) and Y ∼ Nor(µY , σ2

Y ).

Example: (X,Y ) could be a person’s (height,weight). The twoquantities are marginally normal, but positively correlated.

If you want to calculate bivariate normal probabilities, you’ll need toevaluate quantities like

P (a < X < b, c < Y < d) =

∫ d

c

∫ b

af(x, y) dx dy,

which will probably require numerical integration techniques.

Goldsman 6/10/14 68 / 74


Fun Fact (which will come up later when we discuss regression): Theconditional distribution of Y given that X = x is also normal. Inparticular,

Y |X = x ∼ Nor(µY + ρ(σY /σX)(x− µX), σ2Y (1− ρ2)).

Information about X helps to update the distribution of Y .

Example: Consider students at a university. Let X be their combinedSAT scores (Math and Verbal), and Y their freshman GPA (out of 4).Suppose a study reveals that

µX = 1300, µY = 2.3,

σ2X = 6400, σ2

Y = 0.25, ρ = 0.6.

Find P (Y ≥ 2|X = 900).

Goldsman 6/10/14 69 / 74


First,

E[Y |X = 900] = µY + ρ(σY /σX)(x− µX)

= 2.3 + ρ(√

0.25/6400)(900− 1300) = 0.8,

indicating that the expected GPA of a kid with 900 SAT’s will be 0.8.

Second,Var(Y |X = 900) = σ2

Y (1− ρ2) = 0.16.

Thus,Y |X = 900 ∼ Nor(0.8, 0.16).

Now we can calculate

P (Y ≥ 2|X = 900) = P

(Z ≥ 2− 0.8√

0.16

)= 1− Φ(3) = 0.0013.

This guy doesn’t have much chance of having a good GPA.Goldsman 6/10/14 70 / 74


Definition: If Y ∼ Nor(µY , σ2Y ), then X ≡ eY has the lognormal distrn

with parameters (µY , σ2Y ). This distribution has tremendous uses, e.g.,

in the pricing of certain stock options.

Turns Out: The pdf and moments of the lognormal are

f(x) =1

xσY√

2πexp

{− 1

2σ2Y

[`n(x)− µY ]2}, x > 0,

E[X] = exp

{µY +

σ2Y

2

}

Var(X) = exp{2µY + σ2Y }(

exp{σ2Y } − 1

)

Example: Suppose Y ∼ Nor(10, 4) and let X = eY . Then

P (X ≤ 1000) = P

(Z ≤ `n(1000)− 10

2

)= Φ(−1.55) = 0.061.

Goldsman 6/10/14 71 / 74

Computer Stuff

Outline





Computer Stuff

We can use various computer packages such as Excel, Minitab, R,SAS, etc., to calculate pmf’s/pdf’s and cdf’s for a variety of commondistributions. For instance, in Excel, we find the functions:

BINOMDIST = Binomial distributionEXPONDIST = ExponentialNEGBINOMDIST = Negative BinomialNORMDIST and NORMSDIST = Normal and Standard NormalPOISSON = Poisson

Functions such as NORMSINV can calculate the inverse of the standardnormal

And you can use RAND to simulate a Unif(0,1) distribution.How would you simulate other RV’s? Well, this is the subject ofanother course, but here’s a remarkable example...

Goldsman 6/10/14 73 / 74

Computer Stuff

Simulating Normal RV’s

How would you generate a normal RV’s when conducting computersimulation experiments?

Theorem (Box and Muller): If U1, U2iid∼ U(0, 1), then

Z1 =√−2`n(U1) cos(2πU2) and Z2 =

√−2`n(U1) sin(2πU2)

are i.i.d. Nor(0,1).

Remarks: (1) Proof: Not here.

(2) Many other ways to generate Nor(0,1)’s, but this is the easiest.

(3) Cosine and Sine must be calculated in radians, not degrees.

(4) To get X ∼ Nor(µ, σ2), just take X = µ+ σZ.

Goldsman 6/10/14 74 / 74

3. distributions - isye home | isyesman/courses/6739/6739-03...discrete distributions poisson...

Documents