probability theory and mathematical statistics lecture 07 ...the main di culty in using the...

Moment-Generating Function Product Moment Moments of Linear Combinations of Random Variables Conditional Expectations ending

Probability Theory and Mathematical StatisticsLecture 07: Moment-Generating Functions

Chih-Yuan Hung

School of Economics and ManagementDongguan University of Technology

April 10, 2019


5. Moment-Generating Function

The moments of most distributions can be determined directlyby evaluating the necessary integrals or sums.

But some of the integration and sum are hard to find.

An alternative procedure sometimes provides considerablesimplifications. This technique utilizes moment-generatingfunctions.


Definition (6, Moment Generating Function)

The moment generating function of a random variable X ,where it exists, is given by

MX (t) = E(etX)= ∑

x

etx · f (x)

when X is discrete, and

MX (t) = E(etX)=∫ ∞

−∞etx f (x)dx

when X is continuous.

We evaluate this kind of expectation at t near 0. The Maclaurin’sseries expansion of etx is

etx = 1 + tx +t2x2

2!+

t3x3

3!+ ... +

trx r

r !+ ...


The MGT in the form of Maclaurin’s series expansion

MX (t) =E(etX)= ∑

x

etx · f (x)

=∑x

[1 + tx +

t2x2

2!+

t3x3

3!+ ... +

trx r

r !+ ...

]f (x)

=∑x

f (x) + t ∑x

xf (x) +t2

2! ∑x

x2f (x) + ... +tr

r ! ∑x

x r f (x) + ...

=1 + µ · t + µ′2 ·t2

2!+ ... + µ′r

tr

r !+ ...


Example (13)

Find the moment-generating function of the random variablewhose probability density is given by

f (x) =

{e−x for x > 0

0 elsewhere

and use it to find an expression for µ′r .


Solution

By definition

MX (t) = E(etX)=∫ ∞

0etxe−xdx

=∫ ∞

0e−x(1−t)dx

=1

1− tfor t < 1

When |t| < 1,

MX (t) =1 + t + t2 + ... + tr + ...

=1 + 1!t

1!+ 2!

t2

2!+ ... + r !

tr

r !+ ...

and hence µ′r = r ! for r = 0, 1, 2, ...


The Use of MGF

The main difficulty in using the Maclaurin’s series of a MGF isusually not that of finding MGF, but that of expanding it into aMaclaurin’s series.The following theorem help us to find the first few moment easily.

Theorem (9)

d rMX (t)

dtr|t=0 = µ′r


Example (14)

Given that X has the probability distribution f (x) = 18 (

3x) for

x = 0, 1, 2, and 3, find the moment-generating function of thisrandom variable and use it to determine µ′1 and µ′2.

Solution

In accordance with Definition 4.6,

MX (t) = E (etX ) =1

8·

3

∑x=0

etX(

3

x

)=

1

8(1 + 3et + 3e2t + e3t)

=1

8·

3

∑x=0

(3

x

)etX13−x

=1

8(1 + et)3


Solution

Then, by Theorem 4.9,

µ′1 = M ′X (0) =3

8(1 + et)2|t=0 =

3

2

and

µ′2 = MX”(0) =3

4(1 + et)|t=0 = 3


Theorem (10)

If a and b are constant, then

1 MX+a(t) = E[e(X+a)t

]= eat ·MX (t);

2 MbX (t) = E[e(bX )t

]= MX (bt);

3 MX+ab(t) = E

[e(

X+ab )t

]= e

ab t ·MX (

tb ).

An application you may have seen already:Let a = −µ, b = σ, we standardize a normal random variable as

X − µ

σ

The MGF of this standardized r.v. is

MX−µσ(t) = e−

µtσ ·MX

( tσ

)


Product Moment

Consider two random variables, the product moments is defined as:

Definition (7, Product Moments about the Origin)

The rth and sth product moment about the origin of therandom variables X and Y , denoted by µ′r ,s , is the expected valueof X rY s ; symbolically,

µ′r ,s = E (X rY s) = ∑x

∑y

x ry s · f (x , y)

for r = 0, 1, 2, ... and s = 0, 1, 2, ... when X and Y are discrete,and

µ′r ,s = E (X rY s) =∫ ∞

−∞

∫ ∞

−∞x ry s · f (x , y)dxdy

when X and Y are continuous.


Note that µ′1,0 = E (X ) = µX and µ′0,1 = E (Y ) = µY , we have

Definition (8, Product Moments about the Mean)

The rth and sth product moment about the means of therandom variables X and Y , denoted by µr ,s , is the expected valueof (X − µX )

r (Y − µY )s ; symbolically,

µr ,s =E [(X − µX )r (Y − µY )

s ]

=∑x

∑y

(X − µX )r (Y − µY )

s · f (x , y)

for r = 0, 1, 2, ... and s = 0, 1, 2, ... when X and Y are discrete,and

µr ,s =E [(X − µX )r (Y − µY )

s ]

=∫ ∞

−∞

∫ ∞

−∞(X − µX )

r (Y − µY )s · f (x , y)dxdy

when X and Y are continuous.


You may know that µ1,1 is of special important, which is worth togive it a name.

Definition (9, Covariance)

µ1,1 is called the covariance of X and Y , and it is denoted byσXY , cov(X ,Y ), or C (X ,Y ).

It is indicative of the relationship, if any, between the value of Xand Y .

If the joint probability has the tendency that large X withlarge Y, then the covariance will be positive;

If there is a high probability that large values of X will go withsmall values of Y , and vice versa, the covariance will benegative.


Theorem (11)

σXY = µ′1,1 − µX µY

Proof.

Using the various theorems about expected values, we can write

σXY =E [(X − µX )(Y − µY )]

=E (XY − XµY − Y µX + µX µY )

=E (XY )− µYE (X )− µXE (Y ) + µX µY

=E (XY )− µY µX − µX µY + µX µY

=µ′1,1 − µX µY


Example (15)

The joint and marginal probability of X and Y , the number ofaspirin and sedative caplets among two caplets drawn at randomfrom a bottle containing three aspirin, two sedative, and fourlaxative caplets, are recorded as follows:

xf (x , y) 0 1 2

y

0 16

13

112

712

1 29

16

718

2 136

136

512

12

112

Find the covariance of X and Y


Solution

µ′1,1 =E (XY )

=0 · 0 · 1

6+ 0 · 1 · 2

9+ 0 · 2 · 1

36+ 1 · 0 · 1

3+ 1 · 1 · 1

6+ 2 · 0 · 1

12

=1

6

Also,

µX = E (X ) = 0 · 5

12+ 1 · 1

2+ 2 · 1

12=

2

3

and

µY = E (Y ) = 0 · 7

12+ 1 · 7

28+ 2 · 1

36=

4

9

It follows

σXY =1

6− 2

3· 4

9= − 7

54


Example (16)

Find the covariance of the random variables whose joint probabilitydensity is given by

f (x , y) =

{2 for x > 0, y > 0, x + y < 1

0 elsewhere


Solution

Evaluating the necessary integrals, we get

µX =∫ 1

0

∫ 1−x

02xdydx =

1

3

µY =∫ 1

0

∫ 1−y

02ydxdy =

1

3

µ′1,1 =∫ 1

0

∫ 1−y

0x · y2dxdy =

1

12

It follows that

σXY =1

12− 1

3

1

3= − 1

36


If X and Y are statistical independent, we have:

Theorem (12)

If X and Y are independent, then E (XY ) = E (X ) · E (Y ) andσXY = 0.

Proof.

For the discrete case we have, by definition,

E (XY ) = ∑x

∑y

xy · f (x , y)

Since X and Y are independent, we can writef (x , y) = g(x) · h(y), where g(x) and h(y) are the values of themarginal distribution of X and Y , and we get


Proof.

E (XY ) =∑x

∑y

xy · g(x)h(y)

=

[∑x

x · g(x)] [

∑y

y · h(y)]

=E (X ) · E (Y )

Hence,

σXY =µ′1,1 − µX µY

=E (X ) · E (Y )− E (X ) · E (Y )

=0


Independence ⇒ zero covarianceIndependence : zero covariance

Example (17)

If the joint probability distribution of X and Y is given by

xf (x , y) -1 0 1

y

-1 16

13

16

23

0 0 0 0 01 1

6 0 16

13

13

13

13

Show that their covariance is zero but not independent.


Solution

µX =(−1) · 1

3+ 0 · 1

3+ 1 · 1

3= 0

µY =(−1) · 2

3+ 0 · 0 + 1 · 1

3= −1

3

and

µ′1,1 =(−1)(−1) · 1

6+ (−1)1 · 1

6+ 1(−1) · 1

6+ 1 · 1 · 1

6=0

Thus, σXY = 0− 0(13

)= 0.

But f (x , y) 6= g(x) · h(y).for example, x = −1, y = −1,f (−1,−1) = 1

6 6=13 ·

23 = g(−1) · h(−1).


Linear Combination of n Random Variables

One of the form of r.v.s is of special important in statisticalinference, linear combination.

Theorem (14)

If X1,X2, ...,Xn are random variables and Y = ∑ni=1 aiXi , where

a1, a2, ..., an are constants, then

E (Y ) =n

∑i=1

aiE (X )

and

Var(Y ) =n

∑i=1

a2i · var(Xi ) + 2 ∑ ∑i<j

aiajcov(XiXj )


Corollary (3)

If the random variables X1,X2, ...,Xn are independent andY = ∑n

i=1 aiXi , then

Var(Y ) =n

∑i=1

a2i Var(Xi )

Example (18)

If the random variables X , Y , and Z have the meansµX = 2, µY = −3 and µZ = 4, the variance σ2

X = 1, σ2Y = 5 and

σ2Z = 2, and the covariance cov(X ,Y ) = −2, cov(X ,Z ) = −1,

and cov(Y ,Z ) = 1, find the mean and variance ofW = 3X − Y + 2Z .


Solution

By Theorem 14, we get

E (W ) =3E (X )− E (Y ) + 2E (Z )

=3 · 2− (−3) + 2 · 4=17

and

Var(W ) =9Var(X ) + Var(Y ) + 4Var(Z )

− 6cov(X ,Y ) + 12cov(X ,Z )− 4cov(Y ,Z )

=9 · 1 + 5 + 4 · 2− 6(−2) + 12(−1)− 4 · 1=18


Theorem (15)

If X1,X2, ...Xn are random variables and

Y1 =n

∑i=1

aiXi and Y2 =n

∑i=1

biXi

where a1, a2, ..., an and b1, b2, ..., bn are constants, then

cov(Y1,Y2) =n

∑i=1

aibi · Var(Xi ) + ∑ ∑i<j

(aibj + ajbi ) · cov(Xi ,Xj )


Corollary (4)

If the random variables X1,X2, ...Xn are independent,

Y1 =n

∑i=1

aiXi and Y2 =n

∑i=1

biXi ,

then

cov(Y1,Y2) =n

∑i=1

aibi · Var(Xi )


Example (19)

If the random variables X , Y , and Z have the meansµX = 3, µY = 5 and µZ = 2, the variances σ2

X = 8, σ2Y = 12 and

σ2Z = 18, and the covariances cov(X ,Y ) = 1, cov(X ,Z ) = −3,

and cov(Y ,Z ) = 2, find the covariance of

U = X + 4Y + 2Z and V = 3X − Y − Z

Solution

By Theorem 15, we get

cov(U,V ) =cov(X + 4Y + 2Z , 3X − Y − Z )

=3var(X )− 4var(Y )− 2var(Z )

+ 11cov(X ,Y ) + 5cov(X ,Z )− 6cov(Y ,Z )

=3 · 8− 4 · 12− 2 · 18 + 11 · 1 + 5(−3)− 6 · 2=− 76


Conditional Expectation

Definition (10)

If X is a discrete random variable, and f (x |y) is the value of theconditional probability distribution of X given Y = y at x , theconditional expectation of u(X ) given Y = y is

E [u(X )|y ] = ∑x

u(x) · f (x |y)

correspondingly, if X is a continuous random variable, and f (x |y)is the value of the conditional probability density of X given Y = yat x , the conditional expectation of u(X ) given Y = y is

E [u(X )|y ] =∫ ∞

−∞u(x) · f (x |y)dx


Let u(X ) = X , we obtain the conditional mean of the randomvariable X given Y = y , which we denote by

µX |y = E (X |y)

Corresponding the conditional variance of the random variable Xgiven Y = y is

σ2X |y =E

[(X − µX |y )

2|y]

=E (X 2|y)− µ2X |yw

where E (X 2|y) is the result of u(X ) = X 2 in definition 10.


Example (20)

If the joint probability density of X and Y is given by

f (x , y) =

{23 (x + 2y) for 0 < x < 1, 0 < y < 1

0 elsewhere

find the conditional mean and the conditional variance of X givenY = 1

2 .

Solution

First we solve the conditional probability of X given Y = y ,

f (x |y) ={

2x+4y1+4y for 0 < x < 1,

0 elsewhere


Solution

So, when Y = 12 ,

f

(x

∣∣∣∣12)=

{23 (x + 1) for 0 < x < 1,

0 elsewhere

Thus, µX | 12is given by

E

(X

∣∣∣∣12)=∫ 1

0

2

3x(x + 1)dx =

5

9

Next we find

E

(X 2

∣∣∣∣12)=∫ 1

0

2

3x2(x + 1)dx =

7

18

and it follows that σ2X | 12

= 718 −

(59

)2= 13

162


Appendix

Example

Consider example 12 in chapter 3 again.

xf (x , y) 0 1 2

y

0 16

13

112

712

1 29

16

728

2 136

136

512

12

112

find the conditional mean of X given Y = 1.


Appendix

Solution

First we solve the conditional probability of X given Y = y ,

f (x |y) = f (x ,y )h(y )

:

xf (x |y) 0 1 2 h(y)

y

0 27

47

17

712

1 47

37

718

2 1 136

Therefore,

E (X |1) = 0 · 4

7+ 1 · 3

7=

3

7

Also find E (X |0) and E (X |2).


Appendix

Consider this example again,

xf (x , y) 0 1 2

y

0 16

13

112

712

1 29

16

728

2 136

136

512

12

112

We can find E (X ) by

1 E (X ) = ∑x g(x) = 0 · 512 + 1 · 12 + 2 · 1

12 = 23

2 Ey [E (X |y)] = ∑y E (X |y)h(y) = 67 ·

712 +

37 ·

718 + 0 · 1

36 = 23

The second method is called iterated expectation.


Appendix

Theorem (Iterated Expectation/Law of Total Expectation)

if X is a random variable whose expected value E (X ) is defined,and Y is any random variable on the same probability space, then

E (X ) = E (E (X |Y ))

That is the expected value of the conditional expected value of Xgiven Y is the same as the expected value of X .


Appendix

Example

Suppose that two factories supply light bulbs to the market.Factory X ’s bulbs work for an average of 5000 hours, whereasfactory Y ’s bulbs work for an average of 4000 hours. It is knowthat factory X supplies 60 percent of the total bulbs available.What is the expected length of time that a purchased bulb willwork for?

Solution

Applying the law of total expectation, we have:

E (L) = E (L|X )P(X ) + E (L|Y )P(Y ) = 5000 · 0.6 + 4000 · 0.4

Thus each purchased light bulb has an expected lifetime of 4600hours.


Homework

Work with your partner (in group)

hand in the homework to the editor group on duty before17:00, Sunday.

Group editor on duty shall organize the final answers and sendthe file of final answer to [email protected] before nextTuesday

HW: please see the attached file


Questions??

probability theory and mathematical statistics lecture 07 ...the main di culty in using the...

Documents