ct111 introduction to communication systems …intranet.daiict.ac.in/.../ct111lecture9prob.pdfct111...

../CourseOutline/collegelogo1

Average Value Probability A Vectorial Point of View Characterization of RVs Examples of Discrete RVs Examples of Continuous RVs

CT111 Introduction to Communication SystemsLecture 10: A Review of Probability Theory

Yash M. Vasavada

Associate Professor, DA-IICT, Gandhinagar

9th February 2018

Yash M. Vasavada (DA-IICT) CT111: Intro to Comm. Systems 9th February 2018 1 / 74



Overview of Today’s Talk1 Average Value2 Probability3 A Vectorial Point of View4 Characterization of RVs5 Examples of Discrete RVs6 Examples of Continuous RVs




Taking an Average

We have earlier seen this operation on a time domain signal w(t):

〈w(t)〉




Taking an Average

We have earlier seen this operation on a time domain signal w(t):

〈w(t)〉 def= limT→∞

1

T

∫ T/2

−T/2w(t) dt

Let us reflect on this a bit more




Taking an Average

Let us say that in a quiz of 10 marks, the following is the observedperformance of 30 groups:

→ 8, 9, 1, 10, 6, 1, 3, 6, 10, 10, 1, 10, 10, 5, 8, 1, 4, 10, 8, 10, 7, 0, 9,10, 7, 8, 8, 4, 7, 1

Suppose we want to know the average score. One way is simple:

→ Average score: (8+9+1+10+6+1+3+6+10+10+1+10+10+5+8+1+4+10+8+10+7+0+9+10+7+8+8+4+7+1)/30 = 6.4




Taking an Average

Let us write this mathematically:

→ Let k denote the index of the group, where k = 1, 2, . . . ,N→ N = 30 is the total number of groups→ Let x(k) denote the mark obtained by k th group. Thus, x1 = 8, x2 = 9

and so on

With this, the average marks can be written as follows:

X̄ =1

N

N∑k=1

x(k)

Here X is a notation for the Random Variable (or RV), and X̄denotes the average




Taking an Average

Let us write the average operation mathematically:

X̄ =1

N

N∑k=1

x(k)

X is a notation for the Random Variable (or RV)

. An RV is a number, it can be an integer (in which case the RV iscalled a Discrete RV) or a real number (in which case it is called acontinuous-valued RV) or a complex number (Complex RV)

In our example, X stands for the score obtained by different groups

. Note that X is inherently random; we don’t know what marks differentgroups are going to get in the next quiz

x(k) denotes kth realization of XX̄ denotes the average




Random Variables in Communication Systems

Random variables are extremely useful to model the communicationsystems. These let us talk about quantities and signals which are notknown in advance:

Data sent through the communication channel is best modeled as asequence of random variablesMessage signal value at any given time is also an RV (e.g., considerthe speech signal)Noise, interference and fading affecting this data transmission are allsignals whose values at any given time is given by an RVReceiver performance is measured probabilistically and by means ofan RV




Taking an Average

Let us say that in a quiz of 10 marks, the following is the observedperformance of 30 groups:

→ 8, 9, 1, 10, 6, 1, 3, 6, 10, 10, 1, 10, 10, 5, 8, 1, 4, 10, 8, 10, 7, 0, 9,10, 7, 8, 8, 4, 7, 1

Suppose we want to know the average score. One way is simple:

→ Average score: (8+9+1+10+6+1+3+6+10+10+1+10+10+5+8+1+4+10+8+10+7+0+9+10+7+8+8+4+7+1)/30 = 6.4

There is another way possible:

→ Average score:((1× 0) + (5× 1) + (0× 2) + (1× 3) + (2× 4) + (1× 5)+(2× 6) + (3× 7) + (5× 8) + (2× 9) + (8× 10))/30 = 6.4




Taking an Average

Let us write the average operation mathematically in an alternateway:

X̄ =1

N

N∑k=1

x(k) =1

N

M∑m=0

nxmxm

Here, we have introduced two different notations:

. In our case, X is a discrete RV, and its value can be any one integerfrom {0, 1, . . . , 10}, where 10 is the maximum score possible in thequiz.

. Integer xm denotes the values that the RV can take and M is themaximum value.

. Finally, nxm denotes the number of times xm occurs in the sample set




Understanding the Probability

Suppose someone says that the probability of the quiz score of 10 outof 10 is 5%, on what basis he/she made this statement?One answer is that it is his or her belief.How did he/she arrive at this belief?Because he/she saw that on average only 5% of the groups got fullmarksHowever, what does this mean mathematically?

pxmdef= lim

N→∞

nxmN




Probabilityand Averages

Probability that a random variable X takes a value of xm can bewritten as:

pxmdef= lim

N→∞

nxmN

Here, nxm is the number of times xm occurs in a total of NexperiementsWith this definition of the probability, the average can be written as

X̄ =1

N

N∑k=1

x(k) =1

N

M∑m=0

nxmxm =M∑

m=0

pxmxm




ProbabilityAverages and Expected Value for Discrete RVs

Expected value µX of an RV X is defined as

E (X ) = µX =∑X

pX (x)x

Technically, there is a subtle difference between the expected valueµX and the average value X̄

. X̄ =1

N

N∑k=1

x(k) is the time domain average, does not require

knowledge of the probabilities pX (x), but it requires statisticalexperiments

. E (X ) = µX =∑X

pX (x)x is the statistical average, requires knowledge

of the probabiilties, but does not require experimentation. As N →∞, X̄ → E (X )




Expected Values

The definition on the prior slide is for the discrete RV. It generalizesto the continuous RV:

→ Mean: E (X ) = mX =

∫ ∞−∞

x pX (x) dx

Thus, there is yet another name; average or expected value of the RVis also called the mean of the RVNote when X stands for a random signal (also called as randomprocess), mX can be thought of as the DC component of the signal.

→ When the DC component is zero, the random signal is called azero-mean random signal (or process)

Can we know what the mean value is if we are given the FourierTransform of a signal?




ProbabilityAverages and Expected Value for Continuous RVs

Expected value µX of an RV X is defined as

E (X ) = µX =

∫XpX (x)x dx

Technically, there is a subtle difference between the expected valueµX and the time-domain average value X̄

. X̄ = 〈x(t)〉 is the time domain average, does not require knowledge ofthe probabilities pX (x), but it requires statistical experiments

. E (X ) = µX =∑X

pX (x)x is the statistical average, requires knowledge

of the probabiilties, but does not require experimentation. As N →∞, X̄ → E (X )




Expected ValuesMean-Squared

There is one type of mean which is quite useful:

→ Mean-Squared: E (X 2) = mX 2 =

∫ ∞−∞

x2 pX (x) dx

Why is E (X 2) a useful mean value?

→ E (X 2) can be equated to the average power in the random variable X→ Recall RMS (Root Mean Squared). We have defined it earlier for

deterministic signals. For random signals and random variables, theRMS is defined as

√E (X 2)




Expected ValuesVariance

There is another type of mean which is quite useful:

→ Variance: σ2 def= E

((x −mX )2

)=

∫ ∞−∞

(x −mX )2 p(x) dx

→ Variance measures the power of the truly random component, afterremoving the constant or the DC component of the signal given by mX

→ It can be proven that E (X 2) = m2X + σ2

X .




Expected ValuesCorrelation

For random signals (or random processes), there is yet another typeof mean value which is very important:

→ Define a product signal Y (t, τ) = X (t)X ∗(t + τ)→ Correlation: RX (τ) = E [Y (t, τ)]→ Correlation: 〈Y (t, τ)〉 → RX (τ) as the length of the time domain

averaging increases→ Correlation measures how fast the random process changes with time t.




Example Random Process: 1X1(t)




Y1(t, τ) = X1(t)X∗1 (t + τ) for τ = 5 samples for

Example Random Process: 1




RX1(τ) = E [Y1(t, τ)] for all τ





Wiener Khinchine Theorem

P(f ) = E[|X (f )|2

]is called the Power Spectral Density or PSD

Weiner Khinchine Theorem: R(τ) P(f )

→ Correlation and the PSD form a Fourier Transform Pair→ A question: what is P(f ) if R(τ) can be approximated by δ(τ)?→ Answer: spectrally flat PSD over f ∈ [−∞,+∞]




Dot Product

Correlation operation can be thought of as taking a dot productWe have seen the dot product operation earlier:

1 Fourier Transformation2 Convolution Operation (in Filtering)

Dot Product is an extremely powerful concept, which can be bestunderstood by means of vector algebra




Vectorial Form

Let us define vectors as a collection of numbers:

h = [h1, h2, . . . , hN ]T

r = [r1, r2, . . . , rN ]T

h · r = hT r = h1r1 + h2r2 + . . .+ hN rN

Here, the last term h · r = hT r is known as the dot product between thevectors h and r.




Vectorial FormNotations

Transposition

. h = [h1, h2, . . . , hN ] denotes a row vector with 1 row and N columns(i.e., of size 1× N)

. hT = [h1, h2, . . . , hN ]T denotes a column vector with N rows and 1column (size N × 1)

→ Think of the row vector as a horizontal vector sleeping on a couch, andthe column vector as a vertical vector standing tall

→ Here (·)T denotes the transposition operation: turns row vectors intocolumn vectors and vice versa




Vectorial FormNotations

A rule of vector product:

. Cannot multiply a 1× N vector with 1× N vector. The internaldimensions don’t match

. Can multiply a 1× N vector with N × 1 vector. The internaldimensions do match.

Therefore, in h · r = hT r, the first vector is a horizonal vector of size1× N and the second vector is a tall vector of size N × 1

→ If the vectors have complex-valued coordinates, the dot productrequires Hermitian Transpose (transpose and conjugation): h · r = hHr




VectorsTwo Key Properties

Following are two key properties of the dot products.

1 Dot product of any vector x with itself is the square of the vectorlength.

2 Dot product of vectors x and y is directly related to the cosine of theangle between the vectors x and y.

We examine each of these two properties next.




Dot Product between Two VectorsFirst Key Property: Length of any Vector

Length of a vector x is denoted as ‖x‖.Length of a vector x = [x1, x2] in 2D plane is the length of hypotenuseof a right angled triangle whose bases have lengths x1 and x2.

The square of length of this hypotenuse is given by Pythagorastheorem:

‖x‖2 = x21 + x22 = xTx




Dot Product between Two VectorsFirst Key Property: Length of any Vector

This can be extended to three dimensional space.

‖x‖2 = x21 + x22 + x23 = xTx

Therefore, we conclude, generalizing to N dimensional space, that thedot product of any N-dimensional vector x with itself gives the squareof the length of the vector.

xTx = x21 + x22 + x23 + . . .+ x2N = ‖x‖2




First Key Property: Length of any Vector

What does ‖x‖2 = x21 + x22 + x23 + . . .+ x2N = xTx representphysically?

. Approaches N × E[X 2]

as N →∞Thus, the power of the signal E

[X 2]

can be thought of as the

normalized length ‖x‖2 /N of the signal vector




Dot Product between Two VectorsSecond Key Property: Angle between Two Vectors

Now, let us consider the dot product of vectors x and y.

Length ‖x‖ is the hypotenuse in the triangle OxQ, and the sine andconsine of α are

sinα =x2‖x‖

, cosα =x1‖x‖





Similarly, for angle β for vector y,

→ sinβ =y2‖y‖

, and cosβ =y1‖y‖

.

Now,

cos θ = cos (β − α)

= cosβ cosα + sinβ sinα

=x1y1 + x2y2‖x‖ ‖y‖

=xTy

‖x‖ ‖y‖





dot product is proportional to the cosine of the angle between thevectors x and y:

xTy = cos θ × ‖x‖ ‖y‖

If x and y are unit length vectors, dot product equals the cosine ofthe angle between the vectors.

xTy = cos θ

If x and y point in the same direction, θ = 0 and the dot product ismaximized.

xTy = cos(θ = 0) = 1

If x and y are perpendicular, θ = 90o and the dot product becomeszero.

xTy = cos(θ = 90o) = 0




Second Key Property: Alignment between Vectors

Therefore, whenever we are taking dot products between two vectors(in discrete time) or two signals (in continuous time), think (orvisualize) this as if

. we are measuring how well the two signals are aligned with each otherin a multi-dimensional space

In h · r, one of the two vectors, let us say h, is sometimes fixed (i.e., itmakes up a reference), and r varies

. The dot product h · r will be high when the variable vector r happensto be well-aligned to our reference vector h

. If the dot product turns out to be zero or near zero, think of this as rbeing orthogonal to h (i.e., pointing in a direction that is 90o offset tothe direction of h)




Probability Mass Functions or PMFs

For discrete random variables, the PMF is the collection of all

probabilities for different values of X , i.e., p(x)def= PX (X = x)

Properties of the PMFs:1 p(x) ≥ 02

∑X

p(x) = 1

3 P(a ≤ X ≤ b) =b∑

x=a

p(x)




Probability Density Function (PDF)

PDF is the generalization of the PMF for continuous RVs.PDF measures how likely a random variable is to lie at a particularvalueProperties:

1 p(x) ≥ 0

2

∫ ∞−∞

p(x)dx = 1

3 P(a ≤ X ≤ b) =

∫ b

a

p(x)dx




Cumulative Distribution Function (CDF)

Definition: FX (x) = F (x) = P(X ≤ x) =

∫ x

−∞pX (y)dy

CDF is the integration of PDF; PDF is the derivative of the CDFProperties:

1 F (x) is monotonically nondecreasing2 F (−∞) = 03 F (∞) = 14 P(a < X ≤ b) = F (b)− F (a)




Example PMFsBinary Distribution

Outcome of the toss of a fair coin: p(x) =

{1/2, x = 0(head),

1/2, x = 1(tail)

→ Mean: mx =∑x

x p(x) = 0× 1/2 + 1× 1/2 = 1/2

→ Variance: σ2x = 1/4

If X1 and X2 are independent binary random variables,PX1X2(x1 = 0, x2 = 0) = PX1(x1 = 0)PX2(x2 = 0)




Example PMFsBinomial Distribution

Suppose a biased coin comes up heads with probability p = 0.3 whentossed. What is the probability of achieving 0, 1, . . . , 6 heads after sixtosses?Let heads represent 0 and tails represent 1.




Example PMFsBinomial Distribution

Let Y =n∑

i=1

Xi , where {Xi} ; i = 1, . . . ,N are independent binary

RVs with p(x) =

{1− p, x = 0(head),

p, x = 1(tail)In this case, RV Y follows the Binomial Distribution given asPY (y) =

(Ny

)py (1− p)N−y

Mean my = N × pVariance σ2y = N × p × (1− p)




Example PMFsBinomial PMF




Example PMFsUnderstanding Binomial Distribution

Suppose I perform an experiment N times, where each outcome is aBinary RV, with probability of 1 equal to p and 0 equal to 1− pQuestions:

. What are all the possible outcomes?

. What is the sum of N binary outcomes?

Let us look at the cases N = 2, 3 and 4.





1 Outcomes of N experiments form a binary “string” of length N bits2 When these N bits are added up, we get a variable Y , which is the

Binomial RV3 Value of Y can be any integer between 0 (all-zeros binary string) to

N (all-ones string)4 There is a total of 2N possible strings (these form a set of all possible

outcomes)5 A total of

(Ny

)strings have y ones and N − y zeros. When any of

these strings occur, the variable Y will take a value y .6 Due to independence between N experiments, probability of

occurrence of any one of(Ny

)string is the same, and it is given as

py (1− p)N−y .7 Therefore, the total probability that Y = y is given as(N

y

)py (1− p)N−y .





You are told:

. At a bicycle company, 70% of the newly made cycles pass the qualityinspection and 30% fail.

Question:

. What are the chances that 3 of 10 cycles that you’re inspecting willfail?

Answer:

. N = 10, p = 0.1, y = 3.

.(N=10y=3

)= 120

. py (1− p)N−y = 0.330.77 = 0.0022235661

.(N=10y=3

)py (1− p)N−y = 120× 0.0022235661 = 0.2668

Therefore, the chances that 30% of N = 10 cycles you’re inspectingwill fail are ≈ 27%.What are the chances that 2 cycles will fail, or 5 will fail, or all 10 willfail?

. Binomial PMF gives the answers to all such questions.




Example PMFsBinomial PMF: An observation

Although the failure rate is 30%, there is a good amount of chancethat only one out of 10 cycles fails (10% failure rate) or even thatnone fail (0% failure rate)This happens because N = 10 is not large enough as the number ofindependent experiments

. Let us see what happens as we increase N




Example PMFsBinomial PMF: An observation

As N →∞, we will indeed (as expected) see with a very highprobability that 0.3× N cycles failingAlternatively, the chances that the number of cycles failing is differentfrom 0.3× N will diminish and become near zero




Example RVsUniform PDF

p(x) =

1

b − a, a ≤ x ≤ b

0, else




Example PDFsUniform CDF

F (x) =

0, x ≤ 0x − a

b − a, a ≤ x ≤ b

1, x ≥ b




Example PDFsUniform RV

Mean: mx =

∫ b

ax p(x) dx =

1

b − a

∫ b

ax dx =

a + b

2

Variance: σ2x =

∫ b

a(x −mx)2 p(x) dx =

(b − a)2

12Probability:

P(a1 ≤ x < b1) =

∫ b1

a1

p(x) dx =b1 − a1b − a

, a < a1, b1 < b




Example RVsGaussian PDF

p(x) =1√

2πσ2xexp

(−(x −mx)2

2σ2x

)




Example RVsGaussian CDF





http:

//turtleinvestor888.blogspot.in/2012_01_01_archive.htmlYash M. Vasavada (DA-IICT) CT111: Intro to Comm. Systems 9th February 2018 65 / 74

http://turtleinvestor888.blogspot.in/2012_01_01_archive.html

http://turtleinvestor888.blogspot.in/2012_01_01_archive.html



Example RVsCentral Limit Theorem

Let X1,X2, . . . ,XN be N independent RVs with identical PDFs

Let Y =N∑i=1

Xi

A theorem of probability theory called Central Limit Theorem or CLT:as N →∞, distribution of Y tends to a Gaussian distribution

→ In practice, N = 10 is sufficient to see the tendency of Y to follow theGaussian PDF

Importance of CLT:

→ Thermal noise results from random movements of many electrons, andit is well modeled by the Gaussian PDF

→ Interference from many identically distributed interferers in a CDMAsystem tends toward the Gauassian PDF




Example RVsCentral Limit Theorem: A Uniform Distribution




Example RVsCentral Limit Theorem. Average of N = 2 identically distributed Uniform RVs





An application of Gaussian PDFs: signal level at the output of adigital communications receiver can often be given as r = s + n,where

→ r is the received signal level,→ s = −a is the transmitted signal level, and→ n is the Gaussian noise with mean 0 and variance σ2

n

Probability that the signal level −a can be mistaken by the receiver asthe signal level +a is given as:

P(r > 0) =

∫ ∞0

1√2πσ2x

exp

(−(x + a)2

2σ2n

)dx = Q

(a

σn

)

Definition: Q(x) =1√2π

∫ ∞x

exp

(−v2

2

)dv




Example RVsRayleigh PDF

Suppose r =√x21 + x22 , where x1 and x2 are Gaussian with zero

mean and variance σ2

PDF p(r) =r

σ2exp− r2

2σ2is the Rayleigh PDF

Used to model fading when no line of sight is present


ct111 introduction to communication systems …intranet.daiict.ac.in/.../ct111lecture9prob.pdfct111...

Documents