tch-prob 1

Chapter 4. Multiple Random Variables

Ex. 4.1. Select a student’s name from an urn.

: height

: weight

: age

H

W

A

S

1 2, , , nX X XX

In some random experiments, a number of different quantities are measured.

tch-prob 2

A vector random variable X is a function that assigns a vector of real numbers to each outcome in S, the sample space of the random experiment.

1 2

Each event involving an -dimensional random variable

, , , has a corresponding region in an

-dimensional real space.n

n

X X X

n

X

The vector ( ( ), ( ), ( )) is a vector random variable.H W A

4.1 Vector Random Variables

tch-prob 3

Event Examples• Consider the two-dimensional random variable X=(X,Y).

Find the region of the plane corresponding to events

2 2

10 ,

min( , ) 5 ,

100 .

A X Y

B X Y

C X Y

tch-prob 4

Product Form• We are particularly interested in events that have the

product form

1 1 2 2

...

where is a one-dimensional event that involves only.n n

k k

A X in A X in A X in A

A X

1 2 2 2( , ) ( , )x y x y

1 2 2{ } { }x X x Y y 1 2 1 2{ } { }x X x y Y y

x1 x2

y1

y2

tch-prob 5

Product Form• A fundamental problem in modeling a system with a

vector random variable involves specifying the probability of product-form events

• Many events of interest are not of product form.

• However, the non-product-form events can be approximated by the union of product-form events.

1 1 2 2

1 1 2 2

[ ] ...

in , in ,..., in

n n

n n

P A P X in A X in A X in A

P X A X A X A

5 and 5 and 5B X Y X Y Ex.

tch-prob 6

4.2 Pairs of Random variables

A. Pairs of discrete random variables

- Let X=(X,Y) assume values from

- The joint pmf of X is

, , 1, 2,..., 1, 2,... .j kS x y j k

, ,

, 1, 2,..., 1, 2,...

X Y j k j k

j k

p x y P X x Y y

P X x Y y j k

, in

[ in ] ,j k

XY j kx y A

P A p x yX

It gives the probability of the occurrence of the pair ,j kx y

- The probability of any event A is the sum of the pmf over the outcomes in A:

1 1

, 1XY j kj k

p x y

- When A=S,

tch-prob 7

Marginal pmf• We are also interested in the probabilities of events involvi

ng each of the random variables in isolation.

• These can be found in terms of the Marginal pmf.

• In general, knowledge of the marginal pmf’s is insufficient to specify the joint pmf.

,1

,1

, anything

,

,

x j j j

X Y j kk

y k k X Y j kj

p x P X x P X x Y

p x y

p y P Y y p x y

tch-prob 8

Ex. 4.6. Loaded dice: A random experiment consists of tossing two loaded dice and noting the pair of numbers (X,Y) facing up. The joint pmf

, ( , )X Yp j k

1 2 3 4 5 6

1 2/42 1/42 1/42 1/42 1/42 1/42

2 1/42 2/42 1/42 1/42 1/42 1/42

3 1/42 1/42 2/42 1/42 1/42 1/42

4 1/42 1/42 1/42 2/42 1/42 1/42

5 1/42 1/42 1/42 1/42 2/42 1/42

6 1/42 1/42 1/42 1/42 1/42 2/42

j

k

The marginal pmf P[X=j]=P[Y=k]=1/6.

tch-prob 9

Ex. 4.7. Packetization problem: The number of bytes N in a message has a geometric distribution with parameter 1-p and range SN={0,1,2,….}. Suppose that messages are broken into packets of maximum length M bytes.Let Q be the number of full packets and let R be the number of bytes left over. Find the joint pmf and marginal pmf’s of Q and R.

tch-prob 10

joint cdf of X and Y

, 1 1 1 1, ,X YF x y P X x Y y

(x1,y1)

y

x

The joint cdf of X and Y is defined as the probability of the product-form event

1 1{ } { }:X x Y y

, 1 1 , 2 2 1 2 1 2

, 1 , 1

,

,

( ) ( , ) ( , ) if and

( ) ( , ) ( , ) 0

( ) ( , ) 1

( ) ( ) ( , ) [ , ] [ ]

( ) [ ]

X Y X Y

X Y X Y

X Y

X X Y

Y

i F x y F x y x x y y

ii F Y F X

iii F

iv F x F x P X x Y P X x

F y P Y y

marginal cdf

tch-prob 11

joint cdf of X and Y

, ,x a

, ,b

1 2 1 2

, 2 2 , 2 1 , 1 2 , 1 1

( ) lim ( , ) ( , )

lim ( , ) ( , )

( ) [ , ]

( , ) ( , ) ( , ) ( , )

X Y X Y

X Y X Yy

X Y X Y X Y X Y

v F x y F a y

F x y F x b

vi P x X x y Y y

F x y F x y F x y F x y

(x1,y1)

y

x

tch-prob 12

),(),(],[ 11,12,121 yxFyxFyYxXxP YXYX

x1 x2

y1

(x1,y1) (x2,y1)

y

x

y

(x2,y2)

(x1,y1)

xx1 x2y2

y1

B B

A

tch-prob 13

joint pdf of two jointly continuous random variables

,

, ,

,,

1 21 1 2 2 ,

1 2

,

1 ( ', ') ' '

( , ) ( ', ') ' '

2 ( , ) ( , )

, , ( ', ') ' '

, ( ', ')

X Y

X Y X Y

X YX Y

X Y

X Y

f x y dx dyyxF x y f x y dx dy

F x yf x y

x y

b bP a X b a Y b f x y dx dy

a a

y dyx dxP x X x dx y Y y dy f x y dxx y

,

' '

( , ) X Y

dy

f x y dx dy

, ( , )X Yf x yX and Y are jointly Continuous if the probabilities of events involving (X,Y) can be expressed as an integral of a pdf, .

, ( ', ') ' ' X Y

A

P in A f x y dx dyX

tch-prob 14

Marginal pdf: obtained by integrating out the variables that are not of interest.

,

,

( ) ( , ') '

( ) ( ', ) '

X X Y

Y X Y

f x f x y dy

f y f x y dx

,

marginal

X Y

d x f (x', y')dy' dx'dx

cdf

tch-prob 15

Ex. 4.10. A randomly selected point (X,Y) in the unit square has uniform joint pdf given by

,

,

1 0 1 and 0 1( , )

0 elsewhere.

Find ( , ).

X Y

X Y

x yf x y

F x y

v

iv

iii

ii

i 1

1

tch-prob 16

Ex. 4.11 Find the normalization constant c and the marginal pdf’s for the following joint pdf:

,

0( , )

0 elsewhere.

x y

X Y

ce e y xf x y

tch-prob 17

Ex. 4.12

Find [ 1] in Example 4.11.P X Y

10

tch-prob 18

Ex. 4.13 The joint pdf of X and Y is

2 2 2

, 2

( 2 )/2(1 )1( , ) , .

2 1X Y

x xy yf x y x ye

We say that X and Y are jointly Gaussian. Find the marginal pdf’s.

tch-prob 19

4.3 Independence of Two Random Variables

X and Y are independent random variables if any event A1 defined in terms of X is independent of any event A2 defined in terms of Y;

P[ X in A1, Y in A2 ] = P[ X in A1 ] P[ Y in A2 ]

Suppose that X,Y are discrete random variables, and suppose we are interested in the probability of the event where A1 involves only X and A2 involves only Y.

“”If X and Y are independent, then A1 and A2 are independent events.

Let

, ( , ) ,

( ) ( ) for all and .

X Y j k j k

j k

X j Y k j k

p x y P X x Y y

P X x P Y y

p x p y x y

1 2 ,A A A

1 2 and j kA X x A Y y

tch-prob 20

“”

,If ( , ) ( ) ( ) for all and ,X Y j k X j Y k j kp x y p x p y x y

1 2

1 2

j 1 k 2

, in in

in in

x in y in

1 2

then ( , )

= ( ) ( )

= ( ) ( )

=

j k

j k

X Y j kx A y A

X j Y kx A y A

X j Y kA A

P A p x y

p x p y

p x p y

P A P A

tch-prob 21

In general, X, Y are independent iff

If X and Y are independent r.v. ,then g(X) and h(Y) are also independent.

,

,

( , ) ( ) ( )

or ( , ) ( ) ( ) if , are jointly continuous.X Y X Y

X Y X Y

F x y F x F y

f x y f x f y X Y

( ) in A, ( ) in B in A', in B'

in A' in B'

( ) in A ( ) in B .

P g X h Y P X Y

P X P Y

P g X P h Y

# A and A’ are equivalent events; B and B’ are equivalent events.

tch-prob 22

Ex.4.15 In the loaded dice experiment in Ex. 4.6, the tosses are not independent.

Ex. 4.16 Q and R in Ex. 4.7 are independent.

Ex.4.17 X and Y in Ex. 4.11 are not independent, even though the joint pdf appears to factor.

,

2 0( , )

0 elsewhere.

x y

X Y

e e y xf x y

[ , ] (1 )

[ ] (1 )( ) , 0,1,2,...

[ ] (1 ) /(1 ), 0,1,2,..., 1.

qM r

M M q

r M

P Q q R r p p

P Q q p p q

P R r p p p r M

2( ) 2 (1 ) ( ) 2x x yX Yf x e e f y e

tch-prob 23

4.4 Conditional Probability and Conditional Expectation

Many random variables of practical interest are not independent. We are interested in the probability P[Y in A] given X=x?

conditional probability

A. If X is discrete, can obtain conditional cdf of Y given X=xk

The conditional pdf, if the derivative exists, is

][

],in []in [

xXP

xXAYPxXAYP

[ , ]( ) , for 0.

[ ]k

Y k kk

P Y y X xF y x P X x

P X x

)()( kYkY xyFdy

dxyf

in A

in A | ( | )k Y kyP Y X x f y x dy

tch-prob 24

If X and Y are independent

- If X and Y are discrete

If X and Y are independent

)()(

)()(

][][],[

yfxyf

yFxyF

xXPyYPxXyYP

YY

YY

kk

)(

),(

][

],[)(

kX

jkXY

k

jkkjY xP

yxP

xXP

yYxXPxyP

)()( jYkjY yPxyP

tch-prob 25

B. If X is continuous, P[ X = x] = 0

conditional cdf of Y given X = x

conditional pdf.

, ,

,

( ', ') ' ' ( , ') 'lim

( )0 ( ') '

( , ') '( )

X Y X Y

XX

X Y

X

y x h yf x y dx dy f x y dy hxx h f x hh f x dxx

y f x y dyf x

,( ) lim ( | ) lim0 0Y Y

P Y y x X x h

P x X x hF y x F y x X x h

h h

,( , )

( )( )

X YY

X

f x yf y x

f x

tch-prob 26

Discrete

continuous

discrete

continuous

,in Aall

in A in Aall all

( , )

( | ) ( ) ( ) ( | )

[ in A ] ( )

[ in A]

all

X Y k jyx jk

Y j k X k Y j kX ky yx xj jk k

k X k

k

p x y

p y x p x p x p y x

P Y X x p x

P Y

x

,,

( , )( ) ( , ) ( ) ( )

( )X Y

Y X Y Y XX

f x yf y x f x y f y x f x

f x

[ in A] [ in A ] ( )X

P Y P Y X x f x dx

,,

( , )( ) ( , ) ( ) ( ).

( )X Y k j

j k X Y k j Y j k X kX k

Ypp x y

y x p x y p y x p xp x

Theorem on total probability

tch-prob 27

Ex 4.22. The total number of defects on a chip is a

Poisson random variable with mean . Suppose that

each defect has a probability of falling in a specific

region R and that the location of each

X

p

defect is

independent of the locations of all other defects. Find

the pmf of the number of defects that fall in the region .Y R

[ ] , 0,1,2,...!

k

P X k e kk

(1 ) 0[ | ]

0

j k jkp p j k

P Y j X k j

j k

0

( )[ ] [ | ] [ ] ...

!

jp

k

pP Y j P Y j X k P X k e

j

tch-prob 28

Ex. 4.23 The number of customers that arrive at a service station during

a time is a Poisson random variable with parameter . The time

required to service each customer is an exponential random

t t variable with

parameter . Find the pmf for the number of customers that arrive

during the service time of a specific customer. Assume that the customer

arrivals are independent of the customer

N

T

service time.

( )[ | ] , 0,1,2,...

!

ktt

P N k T t e kk

( ) , 0tTf t e t

0[ ] [ | ] ( )

, 0,1,2,...

T

k

P N k P N k T t f t dt

k

tch-prob 29

Ex. 4.24 The random variable is selected at random from

the unit interval; the random variable is then selected at

random from the interval (0, ). Find the cdf of .

X

Y

X Y

|

1/ 0( | )

0Y x

x y xf y x

otherwise

( ) 1, 0 1Xf x x

/ 0[ | ]

1

y x y xP Y y X x

x y

1

0( ) [ ] [ | ] ( )Y XF y P Y y P Y y X x f x dx

1

0( ) 1 ' ' ln

'

y

Y y

yF y dx dx y y y

x

tch-prob 30

Conditional Expectation The conditional expectation of Y given X=x is or if X,Y are discrete.

( )j Y j

j

py y xy

[ | ] ( )Y

E Y x yf y x dy

The conditional expectation | can be viewed as defining

a function of : ( ) | .

( ) can be used to define a random variable ( ) | .

What is ( ) | ?

E Y x

x g x E Y x

g x g X E Y X

E g X E E Y X

|

| ( ) discrete

[ | ] ( ) continuous

X

k X kxk

E Y X

E Y x p x X

E E Y x f x dx X

tch-prob 31

can be generalized to

[ | ] ( )

( ) ( )

( , )

( )[ ]

X

Y X

XY

Y

E Y x f x dx

yf y x dy f x dx

y f x y dxdy

yf y dyE Y

]][[)]([ XYEEXgE

]])([[)]([ XYhEEYhE

tch-prob 32

[ X Y ] [ 0,0 ] 0.1 [ 1,0 ] [ 1,1 ] [ 2,0 ] [ 2,1 ] [ 2,2 ] [ 3,0 ] [ 3,1 ] [ 3,2 ] [ 3,3 ]

E[Y] = 1 E[X] = 2.0

x

y

1( , )

10XYp x y 30 xyfor

0.4

0.3( )

0.2

0.1

Yp y

0.1

0.2( )

0.3

0.4

Xp x

0y

3

2

1

0x

3

2

1

tch-prob 33

1( 2) 3

0Yp y x

1( 1) 2

0Yp y x

1,0y

2,1,0y

00.1]0[2

11.

2

10.

2

1]1[

12.3

11.

3

10.

3

1]2[

2

3)3210(

4

1]3[

xYE

xYE

xYE

xYE

0.11.03.06.0

1.002.02

13.014.0

2

3][ YE

x

y

tch-prob 34

Ex. 4.25 Find the mean of Y in Ex. 4.22 using conditional expectation.

Ex. 4.26 Find the mean and variance of the number of customer arrivals N during the service time T of a specific customer in Ex. 4.23.

0

|k

E Y E Y X k P X k

2

2

|

|

E N T t

E N T t

E N

E N

[ ]pE X p

0k

Pp kk X

t2( )t t

0 0[ | ] ( ) ( ) [ ] /T TE N T t f t dt tf t dt E T

2 2 2

0 0

2 2 2 2

[ | ] ( ) ( ) ( )

[ ] [ ] / 2 /

T TE N T t f t dt t t f t dt

E T E T

tch-prob 35

4.5 Multiple Random Variables

Extend the methods for specifying probabilities of pairs of random variables to the case of n random variables.

We say that are jointly continuous random variables if

1 2

, , , 1 2 1 2 in A 1 2

, , , 1 2( ', ', , ')

( ', ', , ') ' ' '

where is the joint pdf function.

,........ in A1

n

X X X n nX n

nX X Xf x x x

P f x x x dx dx dxX Xn

1

, , , 1 2 , , , 1 2 1 21 2 1 2' ' ' ' ' '

The joint cdf is given by

( , , , ) ( , , , )n

X X X n X X X n nn nf dx xF x x x x x x dx x dx

1 2

, , , 1 2 1 1 2 21 2

, ,The joint cdf of is defined as

( , , , ) , ,...,X X X n n nn

XX

F x x x P X x X x X x

nXXX ,...,, 21

tch-prob 36

1

1 , , , 1 2 21 1 2

1 2 1

, , , 1 2 1, , , 1 2 1 1 21 2 1

The marginal pdf of is

( ) ' ' ' '

The marginal pdf for , , , is

( , , , , ') '

( , , , )

( , , , )

X X X X n nn

n

X X X n n nX X X n nn

X

f x f dx dx

X X X

f f x x x x dx

x x x

x x x

1 1

, , 111 1

, , 1 11 1

The conditional pdf of given the values of , , is

( , , )( , , )

( , , )

n n

X X nnX n nn X X nn

X X X

f x xf x x x

f x x

11 1

( , , ) 0, , nn

if f x xX X

, , , 1 2 1 1 1 1 2 2 1 11 2 1 2 1

Repeated applications of above( , , ) ( , , ) ( ) ( ).( , , , )

X X X n X n n X n n X Xn n nf f x x x f x x x f x x f xx x x

, , , 1 2 , , , 1 21 2 1 21

The joint pdf, if it exists, is given by

( , , , ) ( , , , )X X X n X X X nn n

n

fn

x x x F x x xx x

tch-prob 37

1 2 3

2 2 21 2 1 2 3

, , 1 2 31 2 3

1 3

EX. 4.29 Random variables , , and have joint Gaussain pdf

( 2 /2)( , , ) .

2Find the marginal pdf of and .

X X X

X X X

x x x x xef x x x

X X

23

1

2

, 1 31 3

2 21 3

1 2 2 2( 2 )32 1 2 1 2 ( , )

22 2 21 12 2[( ) ]

2 1 12222 2 2

1 2( )1 12 2 2 123 12 222 2 2 2

1 1 1 12 2 ' 2( ) 21 32 2 1 2 2'2 2 2 2 2

x

X X

x x x x xe ef x x dx

x x xe e dx

x xx xe e e dx

x x x xx xe e e e edx

X1 and X3 are independent zero-mean, unit-variance Gaussian r.v.s.

tch-prob 38

Independence

1 2

, , , 1 2 1 21 2 1 2

, , , 1 2 11 2 1

1 11

, , , are independent if and only if

or ( ) ( ) if continuous

( , , ) ( ) ( ) if disc

( , , , ) ( ) ( ) ( )

( , , , )

n

X X X n X X X nn n

X X X n X X nn n

n X X nn

X X

f f x f x

p x x p x p x

XF x x x F x F x F x

x x x

rete

tch-prob 39

4.6 Functions of Several Random Variables Quite often we are interested in one or more functions of random variables involved with some experiment. For example, sum, maximum or minimum of X1, X2, …,Xn.

1 2

Let random variable be defined as ( , , , ).

n

ZZ g X X X

1' ' ' '

, , 1 11

The cdf of ( ) [ ]

[{ ( , , ) such that g( ) z)}]

( , , ) in eqv.event

pdf ( ) ( )

Z

n

X X n nn

Z Z

ZF z P Z z

P x x

f x x dx dx

df z F zdz

x x

x

tch-prob 40

Example 4.31 Z=X+Y

,

,

( ) [ ]

[ ]'z-x ' ' ' ' ( , )- -

' ' ' ( ) ( ) ( , )

Z

X Y

Z Z X Y

F z P Z z

P X Y z

f x y dy dx

df z F z f x z x dxdz

Superposition integral

y

x

y=-x+z

If X and Y are independent r.v.,

( ) ( ') ( ') 'Z X Y

f z f x f z x dx convolution integral

tch-prob 41

Example 4.32 Sum of Non-Independent r.v.s Z=X+Y , X,Y zero-mean, unit-variance with correlation coefficient

2

1

,

2 2 2( 2 ) 2(1 )1( , ) - x, y22 1

X Yx xy yf x y e

,

2

( ) ( ', ') '

2' ' ' ' 2[ 2 ( ) ( ) ]/2(1 )1 '22 1-

Z X Yf z f x z x dx

x x z x z xe dx

2

2

2 2 2 2' ' ' 2 ' ' ' 2 ' ' 2 ( ) ( ') 2 2 22 1' ' 2 ' 2 2 22(1 ) 2(1 ) 2(1 )[ ]

21' 2 22(1 )[ ]

2

x x z x z x x x z x z zx xzx x z z x z z

zx z

tch-prob 42

(1 ) 2' 22 ( ) (1 )4(1 ) '2( )

22 12

1' 2( ) 24(1 ) '2 2 22 12

1' 2( ) 24(1 ) 1 '2 2 2 1 2 12

22 2

4(1 ) 2 2(1 )

2 1 2 2 2(

Z

zzxef z e dx

zzxe e dx

zzxe e dx

z ze e

1 2- 22 21 )

ze

Sum of these two non-independent Gaussian r.v.s is also a Gaussian r.v.

tch-prob 43

Ex.4.33 A system with standby redundancy. Let T1 and T2 be the lifetimes of the two components. They are independent exponentially distributed with the same mean.

The system lifetime is

1T

2T

21 TTT

1

2 2

( )0

x 0 ( )

0 x 0

( ) x 0 x z( ) ( )0 x 0 0 x z

( )

T

T T

xz z x

xef x

z xxe ef x f z x

f z e e dxT

2 2

0 zz ze dx ze

Erlang m=2

tch-prob 44

Let Z = g (X,Y).Given Y = y, Z = g (X,y) is a function of one r.v. X.

Can first find from then find

)( yYzZf )(xXf

( ) ( ') ( ') 'Z Z Y

f z f z y f y dy

The conditional pdf can be used to find the pdf of a function ofseveral random variables.

tch-prob 45

Example 4.34 Z = X/Y X,Y indep., exponentially distributed with mean one. Assume Y = y, Z = X/y is a scaled version of X

( ) ( )

( )

Z X

X

dxf z y f x y x yz dzy f yz y

,( ) ' ( ' ') ( ') ' ' ( ' , ') '

' ' ' ( ' ) ( ') ' ' '0 0'(1 ) ' '0

' '(1 ) '(1 )1 '00 1(1 )1 1

1 1

Z X Y X Y

X Y

f z y f y z y f y dy y f y z y dy

y z yy f y z f y dy y e e dy

y zy e dyy y z y ze e dy

zz

z z

'(1 )0

1 02(1 )

y ze

zz

'(1 )' 'y '(1 )1'

(1 )

y ze dy

y zdy ez

tch-prob 46

y

x

(z>0)x yzx yz

x yzyz x0y if

yz xoy if

zy

x(z<0)

y

xx yz

y yz

0

0

0

0

, ,

, ,

, ,

,

00

Z X Y X Y

X Y X Y

Z X Y X Y

X Y

yzF (z) f (x,y)dxdy f (x,y)dxdyyzyz f (yz,y)d(yz)dy f (yz,y)d(yz)dyyz

f (z) yf (yz,y)dy yf (yz,y)dy

y f (yz,y)dy

tch-prob 47

0 0

0 0

0

, ,

,

,

,2

X Y X Y

Z X Y

X Y

X Y

if f (x,y) f ( x, y)yzF (z) f (x, y)dxdyyzyz yz f ( x, y)dxdyyz f (x,y)dxdy

tch-prob 48

,

Ex. min( , )

If , are independent,

( ) ( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( , )

( ) ( ) ( ) ( )

Z X Y X Y X Y

Z X Y X Y

Z X Y X Y

Z X Y

X Y

f z f z f z f z F z F z f z

F (z) F z F z F z z

F (z) F z F z F z F z

(z, z)

tch-prob 49

,

Ex. max( , )

If , are independent,

( ) ( ) ( ) ( ) ( )

( , )

( ) ( )

Z X Y X Y

Z X Y

Z X Y

Z X Y

X Y

f z f z F z F z f z

F (z) F z z

F (z) F z F z

(z, z)

tch-prob 50

Transformation of Random Vectors

Joint cdf of ),,2,1( nZZZ

, , 1 1 11( , , ) [ ( ) , , ( ) ]

Z Z n n nnF z z P g z g z X X

1 1 1 2

2 2 1 2

1 2

( , , , )

( , , , )

( , , , )

n

n

n n n

Z g X X X

Z g X X X

Z g X X X

tch-prob 51

Example 4.35 W = min (X,Y) , Z = max (X,Y)

If z>w

If z<w

,( , ) [{min( , ) } {max( , ) }]

W ZF w z P X Y w X Y z

, , , , , ,

, , ,

( , ) ( , ) { ( , ) ( , ) ( , ) ( , )}

( , ) ( , ) ( , )W Z X Y X Y X Y X Y X Y

X Y X Y X Y

F w z F z z F z z F w z F z w F w w

F w z F z w F w w

, ,( , ) ( , )

min( , ) max( , ) z min( , ) z max( , ) z

W Z X YF w z F z z

X Y z X Y X Y X Y

(z,z)

(w,w)

x

y

tch-prob 52

pdf of Linear Transformation

Linear Transformation V = a X + b Y W = c X + e Y

assume

Y

XA

Y

Xa

W

V

e c

b

0 bcaeA

w

vA

y

x 1

x

y(x,y+dy) (x+dx,y+dy)

(x,y) (x+dx,y)

v

w

(v+bdy,w+edy)(v+adx+bdy,w+cdx+edy)

(v,w)(v+adx,w+cdx)

dP

Equivalent event

tch-prob 53

stretch factor

(a,b)

ba

(c,e)

(a,b)V2

V1

, ,

,,

( , ) ( , )

( , )( , )

X Y V W

X YV W

f x y dxdy f v w dP

f x yf v w

dPdxdy

1 2 1 2 sin

v v v v

ae bc

dP = ?

tch-prob 54

o

(bdy,edy)

h

(adx,cdx)

dxdybcae

xdcxdahdP

xdcxda

adx

xdaxdc

cdxedybdyh

ba

a

ba

b

ba

b

ba

aec

)(

2222

2222,

2222),(

a)(-b,on e)(c, of Projection22

,22

e)(c,

larPerpendicu 0b)(a,a)(-b,

b)(a,on e)(c, of Projection22

,22

),(

,,

b

c e

( ( , ), ( , )) ( , )

X YV W

adP ae bc Adxdy

f x v w y v wf v w

A

1( )For n-dimensional vector , ( )

f AA f z

A

X

Z

ZZ X

tch-prob 55

Example 4.36 X,Y jointly Gaussian

2 2( 2 )22(1 )1( , ), 22 1

1 11-1 12

1

1 -111 12

( ) ( ) 2 2

x xy y

f x y eX Y

V X XA

W Y Y

A

X V

Y W

V W V WX Y

tch-prob 56

)1(22

12

1)1(22

12

1

)1(22

)1(22

2-12

1

)21(2]2

22)(

2)(22)

2[(

2-12

1

1

)2

,2

(,),(,

w

e

v

e

wv

e

wvwvwvwv

e

wvwvYXf

wvWVf

V, W are independent , zero mean, Gaussian r.v.s with variance , and , respectively. see Fig 4-16 Contours of equal value of the joint pdf of XY

1

1

tch-prob 57

Pdf of General Transformation

1

2

1

2

( , )

( , )

Assume that ( , ) and ( , ) are invertiable, i.e.,

( , )

( , )

V g X Y

W g X Y

v x y w x y

v g x y

w g x y

invertible1

2

( , )

( , )

x h v w

y h v w

Fig 4.17a

11 1

1

( , ) ( , )

gg x dx y g x y dx

xg

v dxxv

v dxx

ey

wc

x

w

by

va

x

v

2 ( , )w

g x dx y w dxx

tch-prob 58

y

x

(x,y) (x+dx,y)

(x,y+dy) (x+dx,y+dy)

x

y

(g1(x,y),g2(x,y))

(g1(x+dx,y),g2(x+dx,y))

(g1(x+dx,y+dy),g2(x+dx,y+dy))

(g1(x,y+dy),g2(x,y+dy)

1 2( , )g g

v dx w dxx x

1 1 2 2( , )g g g g

v dx dy w dx dyx y x y

1 2( , )g g

v dy w dyy y

( , )v w

1

2

( , )

( , )

v g x y

w g x y

tch-prob 59

, 1 2,

( ( , ), ( , ))( , ) X Y

V W

f h v w h v wf v w

v vx y

w wx y

Jacobian of the transformation

Jacobian of the Inverse Transformation

Can be shown that

, , 1 2

1

( , ) ( ( , ), ( , ))

V W X Y

v vx xx yv w

y y w w

v w x y

x x

v wf v w f h v w h v wy y

v w

tch-prob 60

Example 4.37 X,Y zero mean , unit-variance , indep. Gaussian r.v.s

12 2 2( ) radius

( , ) angle in (0,2 )

cos , sin

cos sin

sin cos

V X Y

W X Y

x v w y v w

x xw v wv w v

y y w v w

v w

tch-prob 61

,

221 1 22( , )2 2

2 2 2 2cos 2 sin 2 2

21 2

2

V Wyxf v w e e v

v v w v we e

vve

V,W independent

20

0

w

v

Linear transformation method can be used even if we are interested in only one function of random variables.-by defining an “auxiliary” r.v.

uniform

Rayleigh

( ) ( )W Vf w f v

tch-prob 62

Ex. 4.38 X: zero-mean , unit-variance Gaussian Y: Chi-square r.v. with n degrees of freedom X and Y are independent find pdf of

Let W=Y, then

nY

XV

2

0 1

WX V nY W

x x vwv w n wny y

v w

wn

tch-prob 63

)2(2

)]21(2[21

)2(

)2(2

212)2(

2

22

),(,

)2(2

212)2(

2

22

),(,

nn

nvwe

nw

nw

n

we

nwn

wve

wvWVf

n

ye

nyxe

yxYXf

tch-prob 64

0

2

0

21 121 2

22 2

12

12 2

1 1'2' '

21

2 2 11

2

2

V

V

w vnnw

f (v) e dwnnπ Γ( )

w v' Let wn

nv

nn wf (v) (w ) e dwnnπ Γ( )

nv n

Γn

nnπ Γ( )

Student's t - distribution

tch-prob 65

4.7 Expected Value of Function of Random Variables Z=g(X,Y)

,

( , ) ( , ) , jointly continuous,[ ]

( , ) ( , ) , discretei n X Y i ni n

g x y f x y dxdy X YX YE Z

g x y p x y X Y

Ex. 4.39 Z=X+Y

,

, ,

[ ] [ ]

( ' ') ( ', ') ' '

' ( ', ') ' ' ' ( ', ') ' '

' ( ') ' ' ( ') '

[ ] [ ]

X Y

X Y X Y

X Y

E Z E X Y

x y f x y dx dy

x f x y dy dx y f x y dx dy

x f x dx y f y dy

E X E Y

X, Y need not be independent

][]1[]21[ nXEXEnXXXE In general,

tch-prob 66

Ex. 4.40. X,Y independent r.v.s and let

The jkth joint moment of X and Y is

when j=1 , k=1

E[XY]: the correlation of X and Y

If E[XY]=0 , then X and Y are orthogonal.

1 2

1 2

1 2

1 2

1 2

( , ) ( ) ( )

[ ( , )] [ ( ) ( )]

( ') ( ') ( ') ( ') ' '

( ') ( ') ' ( ') ( ') '

[ ( )] [ ( )]

X Y

X Y

g X Y g X g Y

E g X Y E g X g Y

g x g y f x f y dx dy

g x f x dx g y f y dy

E g X E g Y

,

,

( , ) , jointly continuous[ ] ( , ) , discrete

j kX Yj k

j ki n X Y i n

i n

x y f x y dxdy X YE X Y x y p x y X Y

tch-prob 67

The jkth central moment of X and Y

When j=1 , k=1E[(X-E[X])(Y-E[Y])]=COV(X,Y) covariance of X and Y

COV(X,Y)=E[XY-XE[Y]-YE[X]+E[X]E[Y]] =E[XY]-2E[X]E[Y]+E[X]E[Y] =E[XY]-E[X]E[Y]

Ex. 4.41. X,Y independent COV(X,Y)=E[(X-E[X])(Y-E[Y])] =E[X-E[X]]E[Y-E[Y]] =0

]])[(])[[( kYEYjXEXE

tch-prob 68

The correlation coefficient of X and Y

X,Y are uncorrelated if

If X,Y are independent , then COV(X,Y)=0 , , X, Y uncorrelated.

X,Y uncorrelated does not necessarily imply X,Y are independent.

( , ) [ ] [ ] [ ]

where ( ) , ( ) are the standard deviation of and .

XYX Y X Y

X Y

COV X Y E XY E X E Y

Var X Var Y X Y

0XY

0XY

,

2

, ,

1 1.

[ ] [ ]pf : 0

1 2 1 2(1 )

X Y

X Y

X Y X Y

X E X Y E YE

tch-prob 69

X,Y uncorrelated does not necessarily imply X,Y are independent.

Ex. 4.42 : uniformly distributed in (0,2 )

=cos and =sinX Y

tch-prob 70

Joint characteristic Function

If X and Y are independent r.v.s

1 1 2 2

1 2

1 2

( ), ,..., 1 2

( ), 1 2

1 2,

( , , , )

Consider the case =2:

( , )

( ) ( , )

n n

n

j X X XX X X n

j X YX Y

X Y

E e

n

E e

j x yf x y e dxdy

1 2 1 2

1 2

( ), 1 2

1 2

( , )

( ) ( )

j X Y j X j YX Y

j X j YX Y

E e E e e

E e E e

tch-prob 71

If Z=aX+bY

If , are independent, ( ) ( ) ( )Z X YX Y w a b

,( ) ( )( ) [ ] [ ] ( , )Z X Y

j aX bY j aX bYE e E e a b

, 1 21 2

1 2

The th joint moment of , is

1[ ] ( , ) 0, 0

i ki k

X Yi ki k

ik X Y

E X Yj

tch-prob 72

4.8 Jointly Gaussian Random Variables

X,Y are said to be jointly Gaussian if

2 2

1 1 2 2,2

1 1 2 2,

, 21 2 ,

1exp 2

2 1

2 1

X Y

X Y

X Y

X Y

x m x m y m y mρ

σ σ σ σρf x, y

σ ρ

,x y

Contours of constant pdf

1 1 2 2,1 1 2 2

2 22 constantX Y

x m x m y m y m

, 1 22 21 2

21 arctan2X Y

tch-prob 73

Marginal p.d.f.

2 21 1

2 22 2

/ 2

1

/ 2

2

2

2

x m

X

x m

Y

ef x

ef y

Conditional pdf

, ,| X Y

XY

f x yf x y f y

2 2

, 1

2 21 ,

1, 2 12

21ex

1

p2

2

1 X Y

X

X Y

Y

yx m m

11 , 2

2

2 21 ,and conditional variance .

is Gaussian with conditional mean

1

X Y

X Y

m y m

, 0, , are independent.If |X Y X X X Yf x y f x For , jointly Gaussian, , uncorrelated , independent. X Y X Y X Y *

tch-prob 74

1 2

1 2 .

,

|

Cov X Y X m Y m

X m Y m Y

211 2 , 2

2

21, 2

2

21, 2

2

, 1 2

|

( , )

X Y

X Y

X Y

X Y

X m Y m Y Y m

Cov X Y Y m

Y m

We now show that is indeed the correlation coefficient.,X Y

1 2

2 1

2 1

12 , 2

2

|

|

|

X Y

X m Y m Y y

y m X m Y y

y m X Y y m

y m y m

,

1 2

ov ,X Y

C X Y

CorrelationCoefficient

tch-prob 75

1 2

1

, ,..., 1 2 / 2 1/ 2

1exp 2, ,...,2n

T

X X X n n

Kf f x x x

K

X

x m x mx

jointly Gaussian Random Variablesn

1 2, ,..., are jointly Gaussian ifnX X X

11 1

22 2where , and is the covariance matrix

nn n

E Xx m

E Xx mK

E Xx m

x m

1 1 2 1

2 1 2 2

1 2

( ) ( , ) ( , )

( , ) ( ) ( , )

( , ) ( , ) ( )

n

n

n n n

VAR X COV X X COV X X

COV X X VAR X COV X X

COV X X COV X X VAR X

K

tch-prob 76

The pdf of the jointly Gaussian random variables is completely specified by the individual means and variances and the pairwise covariances.

Ex. 4.46 Verify that (4.83) becomes (4.79) when n=2.

Ex. 4.48

1 2

1 2

, ,..., are jointly Gaussian.

If ( , ) 0, , , ,..., independent.n

i j n

X X X

COV X X i j X X X

tch-prob 77

Linear Transformation of Gaussian Random Variables

1f Af

A

X

Y

yy

1 1 1

/ 2 1/ 2

1exp 2

2

T

n

A K A

A K

y m y m

1 1

2 2Let be jointly Gaussian, and define by =A ,

n n

x y

x y

x y

X Y Y X

1 1

1 1T TT

A A A

A A A

y m y m

y m y m

From elementary properties of matrices,

tch-prob 78

1 1 1

/ 2 1/ 2

1exp 2

2

TT

n

A A K A Af

A K

Y

y m y my

11 1 1Since ,

let , .

T T

T

A K A AKA

C AKA A

n m 2det det det detTC AKA A K

1

/ 2 1/ 2

1( ) ( )2

2

T

n

Ce

fC

Y

y n y n

y

Thus, Y jointly Gaussian with mean n and covariance matrix C.

tch-prob 79

TAKA If we can find a A s.t. , a diagonal matrix

1

/ 2 1/ 2

12

( )2

T

n

ef

Y

y n y n

y

2

1/ 2

1 2

2

1/ 21

1exp /2 1

2 2 ..... 2

1exp /2

2

i i i

n

n i i i

i i

ny n

i

y n

1 2, ,...... are independent.nY Y Y

If we can select matrix that diagonalize with 1,

then the linear transformation corresponds to a rotation.

A K A

tch-prob 80

cos sin

sin cos

V X

W Y

1 2 1 2 cos sin sin cos

Cov V,W Ε V Ε V W Ε W

E X m Y m X m Y m

1 2

1 2

cos sin

sin cos

E V m m

E W m m

Ex. 4.49, 1 2

2 21 2

21 arctan2X Y

tch-prob 81

Ex.4.50

1 1 2 2 .... n nZ a X a X a X

2 2 3 3 nLet Z , Z ,..., Z .nX X X

2Define , ,..., , thennZ Z ZZ

1 2

0 1 0where .

0 0 1

na a a

A

is jointly Gaussian with mean An mand covariance matrix TC AKA

1 2, ,..., are jointly Gaussian.nX X X

AZ X

1

11

1

,1 1

i

i j i j

nE Z n a E Xii

n nVAR Z C a a COV X X

i j

tch-prob 82

Joint Characteristic Function of n jointly Gaussian random variables

1 2, ,..... isnX X X

1 2, ,.... 1 2

1 ,21 1 1, ,...,

n

i i i k i k

X X X n

n n nj m COV X Xi i ke

12

T Tj Ke

ω m ω ω

tch-prob 83

4.9 Mean Square Estimation

022

2min

2222min

aYda

aYa

d

aYaYaYa

Ya

2

. . .m s e Y a VAR Y

We are interested in estimating the value of an inaccessible random variable Y

in terms of the observation of an accessible random variable X.

The estimate for Y is given by a function of X, g(X).

1. Estimating a r.v. Y by a constant a so that the mean square error (m.s.e) is minimized :

tch-prob 84

2. Estimating Y by g(X) = a X + b

2min,

Y aX ba b

best is b b Y aX Y a X

2best is by mina Y Y a X X

a

Differentiate w.r.t. a

XXXXaYY 2

2 , 0COV X Y aVAR X

,

,X Y

COV X Y YaVAR X X

tch-prob 85

Minimum mean square error (mmse) linear estimator for Y

,X Y YX

Y a X b

X E XE Y

Zero-mean, unit-variance version of X

error of the best linear estimator observation

0Y Y a X X X X

Orthogonality condition

In deriving a*, we obtain

tch-prob 86

Mean square error of best linear estimator

2 E Y Y a X X

E Y Y a X X Y E Y

a E Y Y a X X X X

, ,

2,

2

,

1

YX Y X Y X YX

X Y

Y Y a X X Y Y

VAR Y a COV X Y

VAR Y

VAR Y

tch-prob 87

3. Best mmse estimator of Y is in general a non-linear function of X, g(X)

2

.min

gY g X

XXgYXgY |22

dxxXfxXXgY |2

constant when X x

The constant that minimizes is 2|E Y g X X x

|g x E Y X x Regression curve

is the estimator for Y in terms of X that yields the smallest m.s.e. XY |

tch-prob 88

Ex. 4.51 Let X be uniformly distributed in (-1,1) and let Y=X .

Find the best linear estimator and best estimator of Y in terms of X.

Ex. 4.52 Find the mmse estimator of Y in terms of X when X and Y are

jointly Gaussian random variables.

2