Rochester Institute of TechnologyRIT Scholar Works

Theses Thesis/Dissertation Collections

2012

Signal to noise ratio estimation using theExpectation Maximization AlgorithmAhmed Almradi

Follow this and additional works at: http://scholarworks.rit.edu/theses

This Thesis is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusionin Theses by an authorized administrator of RIT Scholar Works. For more information, please contact ritscholarworks@rit.edu.

Recommended CitationAlmradi, Ahmed, "Signal to noise ratio estimation using the Expectation Maximization Algorithm" (2012). Thesis. Rochester Instituteof Technology. Accessed from

SignaltoNoiseRatioEstimationusingtheExpectationMaximizationAlgorithm

By

AHMEDMALMRADI

AThesisissubmittedinpartialfulfillmentoftherequirementsforthedegreeof

MASTEROFSCIENCEinElectricalEngineering

APPROVEDBY:

…………………………………………………………

Dr.SohailDianat,Thesisadvisor

…………………………………………………………

Dr.ErnestFokoué,ThesisCommitteemember

………………………………………………………….

Dr.ChanceM.Glenn,ThesisCommitteemember

………………………………………………………….

Dr.VincentAmuso,ThesisCommitteemember

………………………………………………………….

Dr.SohailDianat,Departmenthead

DEPARTMENTOFELECTRICALANDMICROELECTRONICENGINEERING

KATEGLEASONCOLLEGEOFENGINEERING

ROCHESTERINSTITUTEOFTECHNOLOGY

ROCHESTER,NEWYORK

MARCH2012

SignaltoNoiseRatioEstimationusingtheExpectationMaximizationalgorithm

By:

AHMEDALMRADI

Thesisadvisor:

Dr.SOHAILDIANAT

Abstract

Signal to Noise Ratio (SNR) estimationwhen the transmitted symbols are unknown is acommonprobleminmanycommunicationsystems,especiallythosewhichrequireanaccurateSNRestimation. For instance, modern wireless communication systems usually require accurateestimate of SNR without knowledge of the transmitted symbols. In addition, SNR estimation isrequiredinordertoperformefficientsignaldetection,powercontrol,andadaptivemodulationInthis study, Non data Aided (NDA) SNR estimation for Binary Phase Shift Keying (PBSK) andQuadrature Phase Shift Keying (QPSK) using the Expectation Maximization (EM) algorithm isdeveloped. The assumption here is that the received data samples are drawn from amixture ofGaussiansdistributionandtheyareindependentandidenticallydistributed i. i. d. .Thequalityofthe proposed estimator is examined via the Cramer‐Rao Lower Bound (CRLB) of NDA SNRestimator.Itisalsoassumedthatthechannelgainisconstantduringeachsymbolinterval,andthenoiseisAdditiveWhiteGaussian(AWGN).MaximumLikelihoodestimatorisbeingusedifwehaveaccesstothecompletedata,inthiscasetheproblemwouldbemucheasiersincewegettheexactclosed form solution, but when the observed data are incomplete or partially available, the EMalgorithmwillbeused.Thisapproachisaniterativemethodtogetanapproximatedresultwhichiseitheranapproximatedglobalmaximumorlocalmaximum.However,intheNDASNRestimation,we only have a global maximum since our assumption is that the distribution is a mixture ofGaussians.This isbeing investigated forthecasesofSingle InputSingleOutput(SISO)andSingleInputMultipleOutput (SIMO).Themainconcernabout thereceivediversity is thecost, size,andpower,thatiswhyweresorttothetransmitdiversitysuchasMultipleInputsingleOutput(MISO)with space time block codes (STBC). The base station usually serves hundreds to thousands ofremoteunitswhich is thesolereasonofusingtransmitdiversityat thebasestation insteadofateveryremoteunitcoveredbythebasestation.Itismoreeconomicalinthiscasetoaddequipmenttothebasestationinsteadoftheremoteunits.Alamoutiusedasimpletransmitdiversitytechniqueandassumedinhispaperthatthereceiverhasperfectknowledgeofthechanneltransitionmatrix.However,thisassumptionmayseemhighlyunrealistic.Oneofourcontributionsistoestimatethechannelinformation,aswellasthenoisevariancewhichwouldbeusedinestimatingtheSNRandderiving the CRLB for both DA and NDA case. The performance of our estimator would beempiricallyassessedusingMonte‐Carlosimulations,withCRLBasaperformancemetric.

  

ii  

TABLEODCONTENTS

Listoffigures......................................iv

Listofsymbolsandabbreviations............................v

Acknowledgments....................................viii

Abstract.........................................ix

1. Introduction.....................................1

1.1Estimation....................................2

1.2MultivariateGaussiandistribution.........................5

1.3MinimumVarianceUnbiasedEstimator(MVUE)..................7

1.4CramerRaoLowerBound(CRLB)...........................8

2. MixturesofGaussians(MoG).............................10

2.1TheBasicdefinitionofMoG...........................10

2.2TheLikelihoodfunction..............................12

3.TheEMalgorithm....................................15

3.1Introduction....................................15

3.2Jensen’sEquality..................................16

3.3Kullback‐Leibler(KL)Divergence..........................17

3.4TheEMDerivation.................................18

3.5 MixtureofGaussiansandtheEMalgorithm.....................

22

4.SISONDASNRestimationusingtheEMalgorithmintheLiterature...............27

4.1TheBPSKcase...................................27

4.1.1Thesystemmodel...............................27

4.1.2TheCRLBforBPSK.............................28

4.1.3SISONDASNRestimationusingtheEMalgorithmfortheBPSKcase.....31

  

iii  

4.2TheQPSKcase...................................38

4.2.1TheCRLBforQPSK..............................41

4.2.2 SISONDASNRestimationusingtheEMalgorithmfortheQPSKcase....43

5.MISOwithSTBCNDASNRestimationusingtheExpectationMaximizationalgorithm.....51

5.1ConventionalReceivediversity(SIMO)model......................51

5.2Transmitdiversity(MISO)model............................52

5.3MIMOmodel.......................................53

5.4ThetransmitdiversityforBPSKcase...........................54

5.5TheCRLBforthetransmitdiversityintheBPSKcase..................58

5.6STBCNDASNRestimationusingtheEMalgorithm....................65

6.Conclusions..........................................70

7.Futurework......................................70

8.References.........................................71

  

iv  

LISTOFFIGURES

Figure1.1Normaldistributionwith 0and 1............................3

Figure2.1MixturesoftwoGaussians......................................11

Figure2.2Samplesandtheircontourofthepreviousdistribution...................11

Figure3.1mixturesoffourGaussiansdataandtheirfitwhenC 0.35I................26

Figure3.2mixturesoffourGaussiansdataandtheirfitwhenC 0.05I................26

Figure4.1TheMSE&CRLBindB2vs.SNRindBfortheBPSKwhenN=128..............35

Figure4.2TheMSE&CRLBindB2vs.SNRindBfortheBPSKwhenN=512..............36

Figure4.3TheMSE&CRLBindB2vs.SNRindBfortheBPSKwhenN=1024.............37

Figure4.4TheErrorindB2vs.SNRindBforBPSKatdifferentNs....................37

Figure4.5TheMSEindB2vs.N..........................................38

Figure4.6TheMSE&CRLBindB2vs.SNRindBfortheQPSKwhenN=128.............47

Figure4.7TheMSE&CRLBindB2vs.SNRindBfortheQPSKwhenN=512..............48

Figure4.8TheMSE&CRLBindB2vs.SNRindBfortheQPSKwhenN=1024.............49

Figure4.9TheErrorindB2vs.SNRindBforQPSKatdifferentNs....................49

Figure5.1SIMO/Receivediversitysystemmodel..............................51

Figure5.1MISO/Transmitdiversitysystemmodel.............................52

Figure5.3MIMOsystemmodel.........................................53

Figure5.4.TheMSE&CRLBindB2vs.SNRindBforSTBCwhenN=128................67

Figure5.5.TheMSE&CRLBindB2vs.SNRindBforSTBCwhenN=512................67

Figure5.6.TheMSE&CRLBindB2vs.SNRindBforSTBCwhenN=1024...............68

Figure5.7.TheMISOwithSTBCBPSKcase...........................69

  

v  

LISTOFSYMBOLSANDABBREVIATIONS

Symbols

SignaltoNoiseRatioindB

EstimatedSignaltoNoiseRatioindB

NonDataAided

DataAided

BinaryPhaseShiftKeying

QuadraturePhaseShiftKeying

ExpectationMaximization

. . .IndependentandIdenticallyDistributed

CramerRaoLowerBound

AdditiveWhiteGaussianNoise

MaximumLikelihood

SingleInputSingleOutput

SingleInputMultipleOutput

MultipleInputSingleOutput

MultipleInputMultipleOutput

Space‐TimeBlockCodes

MinimumVarianceUnbiasedEstimator

MixturesofGaussians

GaussianMixtureModel

MaximumLikelihoodEstimate

  

vi  

MonteCarlo

ProbabilityDensityFunction

ӨParameterswhichneedtobeestimated

ӨTheEstimatedparameterValues

μ, ∑ RealGaussiandistributionwithmeanμandCovariance∑

μ, ∑ ComplexGaussiandistributionwithmeanμandCovariance∑

Ө Likelihoodfunction

Ө LogLikelihoodfunction

Gradientwithrespecttotheparameters

Realof

Imaginaryof

μMean

Variance

μEstimatedmean

Estimatedvariance

TheExpectedvalue

Ө FisherInformationMatrix

Ө ElementijoftheFisherInformationMatrix

MeanSquareError

Ө Bias

Mixingproportion

Ө , Ө Auxiliaryfunction

  

vii  

Kullback‐Leibler

SignaltoNoiseratio

⊗KronckerProduct

  

viii  

Acknowledgments

I introduce my deep gratitude to my advisor, Dr. Sohail Dianat, for his constant guidance and

supporttillwemadethisworkvisibleandvaluable.Hisvisionandideasmakeminegrowgradually

during my work in finishing this project. I am thankful for the opportunity in being under his

guidanceandsupport.

I also would like to send my thanks to Dr. Ernest Fokoue, from the center for quality and

appliedstatistics,forhisappreciatedandvaluablediscussionwhichhelpedmeshapemyprojectin

a betterway.His knowledge filled in the gapwhich I had in some statistical concepts regarding

densityestimationwhenincompletedataispresent.

Finally, special thanks go tomy family;mymother (Mabrooka) andmy father (Mohammed)

whowerealwayssupportiveandhelpful.Ialsothankmywife(Afaf)andmydaughter(Shahad)for

theirpatiencewhileIambusyworkingonthisproject.Theirsupportandsmilehashelpedmein

completingthisworkproperly.

  

ix  

Abstract

Signal toNoiseRatio (SNR)estimationwhenthe transmittedsymbolsareunknown isacommon

problem in many communication systems, especially those which require an accurate SNR

estimation. In this study, Non data Aided (NDA) SNR estimation for Binary Phase Shift Keying

(PBSK) and Quadrature Phase Shift Keying (QPSK) using the Expectation Maximization (EM)

algorithmisdeveloped.Theassumptionhere is that thereceiveddatasamplesaredrawn froma

mixtureofGaussiansdistributionandtheyareindependentandidenticallydistributed . . . .The

quality of the proposed estimator is examined via the Cramer‐Rao LowerBound (CRLB) ofNDA

SNRestimator.Itisalsoassumedthatthechannelgainisconstantduringeachsymbolinterval,and

thenoise isAdditiveWhiteGaussian(AWGN).MaximumLikelihoodestimator isbeingused ifwe

haveaccesstothecompletedata,inthiscasetheproblemwouldbemucheasiersincewegetthe

exact closed formsolution,butwhen theobserveddataare incompleteorpartiallyavailable, the

EM algorithmwill be used. This approach is an iterativemethod to get an approximated result

which is either an approximated globalmaximum or localmaximum.However, in theNDA SNR

estimation, we only have a global maximum since our assumption is that the distribution is a

mixtureofGaussians.ThisisbeinginvestigatedforthecasesofSingleInputSingleOutput(SISO);

[3],[4],[10],andSingleInputMultipleOutput(SIMO);[5],[9].Themainconcernaboutthereceive

diversityisthecost,size,andpower,thatiswhyweresorttothetransmitdiversitysuchasMultiple

Input single Output(MISO)with space time block codes (STBC). The base station usually serves

hundredstothousandsofremoteunitswhichisthesolereasonofusingtransmitdiversityatthe

basestation insteadofateveryremoteunitcoveredbythebasestation.It ismoreeconomical in

thiscasetoaddequipmentstothebasestation insteadof theremoteunits.Alamouti [12]useda

simple transmit diversity technique and assumed in his paper that the receiver has perfect

knowledgeofthechanneltransitionmatrix.However,thisassumptionmayseemhighlyunrealistic.

Oneofourcontributionsistoestimatethechannelinformation,aswellasthenoisevariancewhich

would be used in estimating the SNR and deriving the CRLB for both DA and NDA case. The

performanceofourestimatorwouldbeempiricallyassessedusingMonte‐Carlosimulations,with

CRLBasaperformancemetric.

 

  

  1    

1Introduction

ManymodernwirelesscommunicationsystemsrequiretheSignaltoNoiseratio(SNR)estimation

without the knowledge of the transmitted data symbols in Non‐Data Aided (NDA)manner. SNR

estimatorsmaybedivided to twocategories,DataAided (DA)andNonDataAided (NDA). InDA

estimators,thetransmittedsymbolsareusedorknownatthereceiverandusedintheestimation

process, in NDA estimators; the estimation process is being donewithout the knowledge of the

transmittedsymbols,justbasedonthereceivedsamples.TheCramer‐RaoLowerBound(CRLB)for

NDA SNR estimation,which gives theminimumvariance of unbiased estimators,will be used in

evaluating our estimator [5]. The CRLB for NDA SNR estimation for the BPSK andQPSKwill be

derived and plotted against the EM‐ML NDA SNR estimator. Here, the transmitted symbols are

treatedasunknownparameters,butnuisanceparameter;unknownandunwanted.Thisestimation

procedurewouldbeclose to theCRLB for theSNRof interest;atsufficientlyhighSNR.Themain

purposeofthisstudyistoderiveaclosedformapproximationfortheNDACRLBintheBPSKand

QPSK case. Besides, the derivations of the EM algorithm for NDA SNR estimation which will

iterativelymaximizethelikelihoodfunctiontillreachapproximatelytheglobalmaximum.

The structure of the rest of this research is as follows, in this first chapter, we introduce the

problem of estimation especially for a Gaussian distributed data, followed by the Minimum

Variance Unbiased Estimator (MVUE) which will then lead us to the Cramer‐Rao Lower Bound

(CRLB).Inchaptertwo,MixturesofGaussians(MoG)willbeintroducedfollowedbytheMaximum

Likelihood estimate (MLE) of the MoG model. In chapter three, we introduce the Expectation

Maximization (EM) algorithm, and then we solve one of its applications which is mixtures of

Gaussiansparameterestimation.Inchapterfour,weintroduceSISONDASNRestimationusingthe

EMalgorithmintheliterature,besidessolvingthismodelfortheBPSKandQPSKcase.AlsoCRLB

forbothBPSKand/orQPSKwillbederivedandplotted.Inchapterfive,MISOwithspaceandtime

block codes (STBC) NDA SNR estimation using the EM algorithm will be addressed. NDA SNR

estimatorwill be derived and plotted in both cases and for different sample sizeN usingMonte

Carlo simulations. Improvements in SNR estimation accuracy will be proven when exploiting

diversity method due to the use of mutual information. The last chapter will talk about the

conclusionandsummaryoftheresearch.

  

  2    

1.1 Estimation

Theproblemofparameterestimationisacommonproblemthatiswidelyfacedinmanyareassuch

as Radar, speech, image analysis, biomedicine, communications, control, and much more. The

problemwearefacedwithisestimatingsomeparametersbasedonsomeseenorobservedsamples

due to the use of digital communication systems or digital computers [2]. Mathematically, we

observeNdatapointsatthereceiverside 1 , 2 , . . . . . , N whichsampledfromanunknown

probability density function ; Ө parameterized by an unknown parameter Ө, which is the

parameter of interest. Unfortunately, we don’t really get to see the distribution of the received

sampleY.Weonlygetthatsample,Y,whichwethenusetoestimateourparameterӨ.Asaresultof

that, our estimatorwon’t give us the exact Ө, but an estimated version of it. Our estimatorwill

dependhighlythroughsomefunctionofthereceivedsampleYas,

Ө 1 , 2 , . . . . . , N 1.1

Thisestimatormightbeusedtoestimatethecarrierfrequency,channelphase,signalpower,noise

power,….etc.Sincethereceiveddataarerandom,wecandescribeitsbehavioraccordingtoitsPDF

function, 1 , 2 , . . . . . , N ; Ө .ThesemicolonisusedtoindicatethatthedistributionofPDF

isparameterizedbytheparameterӨ.Andit isconstant.Thiskindofestimationiscalledclassical

parameter estimation because the parameters of interest are assumed to be deterministic and

needs to be estimated. Since we are assuming the noise which corrupted our received signal is

additivewhiteGaussiannoise,AWGN, thedistributionof thePDFof thereceivedsampleswillbe

Gaussiandistribution.Forexample,ifwereceivedonesampleandtheparameterwhichneedstobe

estimatedisthemeanandvariance.Figure1.1showsanormaldistributionofzeromeanandunite

variance. This is just an illustrative example which shows us the parameters which we will be

estimating.AssumethatwehaveasignalwhichhappentobeconstantcorruptedbyAWGN,w(n),

ourmodelwillbeasfollows,

μ

Theparameterswhichneedtobeestimatedherearetwo, theDCcomponentoraverage,andthe

varianceofourGaussiannoise.

Ifonesamplehasbeenobserved,

1 μ 1

  

  3    

Figure1.1Normaldistributionwithmu=0,sigma=1

Ө μ

then,

1 ; Ө1

√2

1 μ2

1.2

Theassumptionisthatthenoise 1 ~ 0, .Oringeneral,eachsampleof hasthesame

PDF as 1 . Since these samples are uncorrelated with each other, uncorrelated means

independentifthedistributionisGaussian,wecannowgeneralizethePDFofthereceivedsamples

YofsizeNasfollows,

1 , 2 , . . . . . , N ; Ө ; Ө ; Ө Ө/

-1.5 -1 -0.5 0 0.5 1 1.5

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Normal distribution

Y

P(Y

)data 1

mu

sigma

  

  4    

1

√2

n μ2

1

2

12

n μ 1.3.1

Equation(1.3)iscalledthelikelihoodfunctionofthereceiveddata,whichwouldbeusedtofindthe

MaximumLikelihoodestimate(MLE)of themeanandvariance incaseofourreceiveddatawere

generatedbysingleGaussian.Ijustwantedtomakethisclearbecauseinthenextchapterwewill

betalkingaboutreceivedsamplewhichhasbeengeneratedbymorethanoneGaussian;Mixtureof

Gaussians. To find the MLE of equation (1.3), we need to find the values of μ and which

maximizesthelikelihoodfunction Ө .Wecaninsteadmaximizetheloglikelihoodfunctionsince

thelogarithmisamonotonicfunction.MLEhastheasymptoticpropertyofbeingunbiased.Itisalso

provedthatMLEachievestheCRLBforhighSNR.

Ө2

212

n μ 1.3.2

ӨӨ

Ө

whichcanbefoundbysettingthegradientof Ө withrespecttoӨequaltozero;

Ө Ө 0.

TheMLsolutionisgivenas,

μ1

1.4

1μ 1.5

Tochecktheunbiasedness,wefindtheexpectedvalueoftheestimatedparametersandcompare

themwiththetrueparameters.Itcanbeeasilyshownas,

  

  5    

μ1

μ μ 1.6

1μ

1 2 1

1 1.7

So,weseethattheMLEofthevarianceisbiased,whiletheMLEofthemeanisunbiased.TheMLE

distributionmaybeapproximatedas,

Ө Ө, Ө~ 1.8

Where, ~ asymptotically distributed according to, Ө is the fisher informationwhich is

givenby,

Ө; Ө

Ө 1.9

AcomputersimulationcouldbeperformedtodefinehowlargethedatalengthNhadtobeinorder

togetthisasymptoticresult.MonteCarlomethodcouldbeusedforvariousrealizations(sayM)of

ourestimateӨ,Themeanandvarianceoftheestimatorcouldthenbecalculatedas,

Ө1

Ө 1.10

Ө1

Ө Ө 1.11

Thechoiceofabetterestimatorcouldbedoneas;amongalltheestimatorswehave,wechoosethe

onewhichgivesacloserestimate(onaverage) to theglobal truevalue,andhashigherprecision

(lowervariance)[2].

  

  6    

1.2 MultivariateGaussiandistribution

SincethefocusonthisresearchwillbeonGaussiandistribution,itisworthintroducingthegeneral

multivariateGaussiandistribution,itsbehaviorandproperties.

SupposewehavearandomvariablethatfollowsamultivariateGaussiandistribution.Thisrandom

variablemaybesignalcorruptedbynoise.

…

; μ …

Itscovariancematrixis; ∑ Y μ Y μ

Then,thedistributionofthisrandomvariablewillbegivenby,

1

|2 ∑ |

12

μ ∑ μ 1.12

IftheCovariancematrix∑ ,independentrandomvariableswithequalvariances,eachofthe

received samples dimension in equation (1.12)will return to equation (1.2). Let usmention an

importantpropertywhichistheeffectoflineartransformationonGaussiandistribution,let,

where,Aandbareconstantsmatrixandvectorrespectively,andXisavectorofrandomvariable.

, μ μ

, ∑ ∑

TheMLE in themultivariate casewould be as follows, themodel parameterswhich need to be

estimatedis,

Ө μ ∑ ,assumingi.i.dsamples,

; Ө1

|2 ∑ |

12

μ ∑ μ

|2 ∑ | 12

μ ∑ μ

  

  7    

Ө ; Ө2

|2 ∑ |12

μ ∑ μ

TofindtheMLEofthisparameter,weneedtogothroughthesameprocessofdifferentiatingthe

likelihoodofequation(1.12)andequatingittozerowegetthefollowing,

ӨӨ

, firstwedifferntiatewithrespecttoμ asӨμ

, weget,

μ1

1.13

anddifferntiatingwithrespecttothecovariancematrixӨ∑

, weget,

∑1

μ μ 1.14

The parameter estimate in this case is straight forward computed in a closed form since the

assumption thatY’ssamplesarebeingdrawn fromonlyoneGaussiandistribution.However, this

won’tbethecasewhenwehaveamixtureofGaussians.Theproposedalgorithmwillbeamixture

of multivariate Gaussian distribution since we are dealing with PBSK and QPSK constellation

systemsandthetransmitteddatasymbolsareunknown.

1.3 MinimumVarianceUnbiasedEstimator(MVUE)

Thesearchforgoodestimatorsofunknowndeterministicparametersisn’talwaysaneasytask.Our

focuswillbeonestimatorswhichonaverageyield the trueparametervalue,and thenamongall

thoseestimators,welookfortheonewhichhastheleastvariance.Foranestimatortobeunbiased,

Ө Ө 1.15

Thebiasofanestimatormaybedefinedas,

Ө Ө Ө

Tofindanoptimalestimator,weneedtofindacriterionlikethemeansquareerror(MSE),

  

  8    

Ө Ө Ө 1.16

Ө Ө Ө 1.17

GoodestimatorshavesmallMSEorsmallvarianceif thebias iszero.Equations(1.16)and(1.17)

areunrealizableestimatorssincetheyarefunctionofthetrueparametervalue,sowecannotassess

theperformanceoftheestimatorinthiscase.

MSEiscomposedtoerrorduetothebiasandvariance.MVUEdoesnotalwaysexist,andif itdid,

sometimesitisnoteasytobefound[2].OnewayoffindingitisusingtheCramerRaoLowerBound

(CRLB).WhenCRLBsatisfieswithequalityforallӨ,the .

1.4 CramerRaoLowerBound(CRLB)

Finding a lower bound on the variance of an unbiased estimatorwill be extremely important in

practice.Thiswillserveasabenchmarkincomparingthegoodnessofanestimator.CRLBallowsus

toassurethatanestimatoristheminimumvarianceunbiasedestimator(MVUE),whichwillbethe

casewhentheestimatorattainstheCRLBforallvaluesoftheunknownparameter.

If ; Ө satisfiesregularityconditions,thevarianceofunbiasedestimatorisdefinedas,

Ө1 ; ӨӨ

1.18

AnestimatormaybefoundthatattainstheboundforallӨifandonlyifthescorefunctioncanbe

writtenas,

; ӨӨ

Ө Ө

ForsomefunctionsgandI,thatestimatorwhichistheMVUEisgivenby,

Ө

Andtheminimumvarianceis

Ө1Ө

  

  9    

Thenthisestimatorissaidtobeefficient.

LetusextendtheCRLBtoavectorparameter,

Ө Ө Ө Ө …Ө

Ө Ө 1.19

where Ө isthe Fisherinformationmatrix.

Ө ; ӨӨ Ө

1.20

Forexampleif

Ө μ

Then,

Ө

; Өμ

; Өμ

; Өμ

; Ө

Thismatrixissymmetricandpositivedefinite.

Ifweareestimatingafunctionoftheparameterasintheresearchstudy,if

Ө

Then,

ӨӨ

ӨӨӨ

  

  10    

2 MixturesofGaussians(MoG)

WhenitisnotpossibleforoneGaussiandistributiontorepresentourmodel,weresortthemixture

ofGaussiansmethod.Itisacombined(mixture)ofsomeknowndistributionsthatarecombinedto

formourmodel.Inourestimationproblem,ourassumptionisthatwearegivenareceivedsamples

YoflengthN,whicharedrawnfromamixtureofGaussiansdistribution.ThenumberofGaussians

isdeterminedby theconstellationorderweuse.Forexample, inPBSK,wehave twomixturesof

Gaussiansperantennawithmeans{‐1,+1},andinQPSK,wehavefourmixturesofGaussiansper

antenna with means {√,

√}. Gaussian Mixture Model (GMM) can be used in modeling

complicated probability density functions (PDFs)which have complicated shapes that cannot be

modeledusingasingleGaussian.

2.1 ThebasicdefinitionofMoG

ThegeneralformofaGaussianmixturemodelsis

/Ө , /Ө

/ , Ө

; μ , ∑ 2.1

where isthecomponentpriorofeachGaussiancomponent.Thereisanecessaryconditionfor

to be a valid probability density function that, ∑ 1 and 0. The parameters

whichneedtobeestimatedare , ,...., ,Ө ,Ө ,....,Ө .Increasingthenumberof

componentswouldresult inmoreparameterswhichneedtobeestimated,whichleadstoamore

accurateclassconditionalprobabilitydistributionfunction.Weneedtobecarefulaboutfindingthe

correct number of mixture components to avoid singularities, when using theML estimation to

estimate the parameters [1]. Figure 2.1 below shows the probability distribution function of a

mixture of two Gaussians, and figure 2.2 shows the contours of those two Gaussians with the

samplesdrawnfromthesamedistribution. In fact thisexactdistributionwillnotbegiven inour

caseofstudy.WhatisgivenisjustareceivedsampleofsizeNwhichisdrawninfigure2.2.

  

  11    

Figure2.1.MixturesoftwoGaussians.

Figure2.2.SamplesandContoursofthepreviousdistribution.

-10-5

05

10

-10

-5

0

5

100

0.01

0.02

0.03

0.04

0.05

x

Mixture of two Gaussians

y

p(x,

y)

-10 -5 0 5-10

-5

0

5

x

y

  

  12    

2.2 TheLikelihoodfunction

Thelikelihoodfunctionisafunctionoftheparametersofastatisticalmodelgiventheobservation

data. The maximum likelihood solution selects values for the model parameters that give a

distributionwhichgivestheobserveddatathehighestprobability.LetusintroducetheMaximum

Likelihoodestimate(MLE)ofaGaussianMixtureModel(GMM).Maximizingthelikelihoodfunction

isthesameasmaximizingtheLogLikelihoodfunctionsincethelikelihoodfunctionisamonotonic

function.This,inthecaseofoneGaussianorwhenthereisnohiddenvariableswouldsimplifythe

matha lotsincethelikelihoodfunctionisexponential function,sothelogfunctionwillcancelthe

exponential andwewould be leftwith the exponent. The Log likelihood function for the data is

givenby,

Ө /Ө

/ , Ө

; μ , ∑ 2.2

Where, it is clear here that the likelihood function depends on the PDF at a given mixture

component. Intheestimationproblemwe’veassumedthatthenoiseisAWGN,so∑ ,but

∑ in general may take any covariance matrix form. Therefore the log Likelihood function

becomes,

Ө 2

‖ μ ‖

2 2.3

ӨӨ

Ө 2.4

ThismaybesolvedbydifferentiatingthelogLikelihoodfunctionandequateittozeroasfollows,

Ө Ө ,

The maximum likelihood solution for this likelihood function is given by setting the partial

derivativeoftheloglikelihoodfunction, Ө withrespecttotheparameterofinteresttozeroand

solvingfortheparameterofinterest.Consideringaparticularparameter,Ө ,whichisrelatedto

themthmixturecomponent.

  

  13    

ӨӨ /Ө

/ , ӨӨ

0

Where,Ө μ , ∑ intheGaussiandistributioncase,thedistributionofinterest,

LetusfirstletӨ μ then,

Өμ

/μ , ∑∑ /μ , ∑

∑ μ 0

, / , Ө/μ , ∑

∑ /μ , ∑

,μ∑ / , Ө∑ / , Ө

2.5

Similarly,letӨ ∑ then,

Ө∑ ∑ /μ , ∑

/μ , ∑∑

0

Ө∑ ∑ /μ , ∑

12

/μ , ∑ ∑ μ ∑ μ ∑

, ∑∑ / , Ө μ μ

∑ / , Ө 2.6

We conclude that μ , ∑ estimates will only be correct when using the correct posterior

distribution / , Ө , but the posterior distribution / , Ө depends on them both. So,

there isno compact formsolution for theseparameters.We then turn to an iterative solution to

findingthemaximumlikelihoodestimatewhichwillbeourtargetinthisresearchwherewewillbe

using the ExpectationMaximization as an iterative approach in finding theMaximumLikelihood

solution.

Letusnowmaximizetheloglikelihoodfunctionwithrespecttothemixingcoefficients, .Thisisa

constraintoptimizationproblem,soLagrangemultiplierwillbeusedtosolvethisproblem.

  

  14    

Ө, /Ө 1

Thennowdifferentiatethisfunction, Ө, ,withrespecttoboth,themixingcoefficients, ,and

theLagrangemultiplier, ,wegettheestimatedmixturecomponentasbelow,

1N

/ , Ө 2.7

Finally,we realize thatwithout the knowledge of the class or component of each drawn sample

/ , Ө ,wecannot findthemaximumlikelihoodestimateoftheparametersof interest.The

estimatesμ and∑ willbeoneiterationinthedirectionofthemaximumlikelihoodestimateof

theEMalgorithmwhichwouldbeclearinthenextchapter.

  

  15    

3 TheExpectationMaximization(EM)Algorithm

3.1 Introduction

The EM algorithm was invented and implemented by several researchers till Dempster [6]

collectedtheirideastogether,assuredconvergence,andstatedtheterm“EMalgorithm.“which

hasbeenusedsincethen.OneofthemainapplicationareasoftheEMalgorithmisestimating

parameters ofmixture distributionwhich is the aim in this research. The EM algorithm has

broadareasofapplicationssomeofwhichisingenetics,econometric,clinical,andsociological

studies.InsignalprocessingareaslikeMaximumLikelihoodtomographicimagereconstruction,

trainingofhiddenMarkovmodels inspeechrecognition[7]. Theadventof theEMalgorithm

hascometosolvingtheproblemoflatent(hidden)(unobserved)variableswhichMLEcouldn’t

afford. Ifwe introducea jointdistributionoverboth theobservedandhiddenvariables, then

the corresponding distribution (the distribution of the observed variables alone) is given by

marginalization.Thiswouldhelpputtingcomplexdistributionsoverobservedvariablestobein

amoretractableformofdistributionformedbybothobservedandlatentvariables.Themain

focusinthisstudywillbeonEMalgorithminMixturesofGaussians(MoG).Thelatentvariable

inourcasewouldbetheclasslabel.Wewouldbegivenabunchofdatawhicharenotlabeled,

and need to estimate some parameters out of it. Herewe have two problems, the first is to

estimate the latent ormissing data, and thenwe come to our goal of estimating the desired

parameterswhichwouldnotbeestimatedwithoutestimatingthemissing(latent)classes.The

latent variable will be discrete and constant. For example, in the case of Mixture of two

Gaussians,wehavetwolatentvariables.Itwilleventuallytendtoaclassificationproblem.The

classwill thenbebelonging to either classoneor twodependingon theobserveddata.This

wouldgivewhichoftheGaussianmixturecomponentsisassociatedwitheachvector(sample)

intheobserveddata.TheEMalgorithmwouldbethemethodofchoicewhendirectmaximum

likelihood (ML) parameter estimation is not possible without the knowledge of the latent

variables. ThegoaloftheEMalgorithmistofindtheMaximumLikelihood(ML)solutionsfor

modelswhichhavelatentvariables.LetusdenotethesetofobserveddatabyY,anddenotethe

latentvariablesbyZ.Thesetofallmodelparameter isdenotedbyӨ,and then the likelihood

functionwouldbegivenby,

/Ө , /Ө 3.1

  

  16    

Thisisfordiscreterandomvariables,thesameistruewhenworkingwithcontinuousrandom

variables except switching the summation in equation (3.1) in to integration over the latent

variableZ.Thebadnewshereinequation(3.1)isthatthesummationoverthelatentvariableZ

is insidethelogarithmwhichmadeitclearthatthesolutionofthisequationwon’tbeaneasy

task.Wecanseethatsincethelogarithmandexponentialswon’tbecanceledwitheachotheras

we’ve seen in the previous chapter. In the previous chapter the summationwas outside the

logarithmwhichmadeiteasytobecancelledwiththeexponentialoftheGaussiandistribution.

LetusassumethatforeveryobservedsampleinY,weknowthecorrespondinglatentvariable

Z. Then we can call , as the complete data set. We will refer to Y as incomplete data.

Maximizationofthecompletedataloglikelihoodfunction , isstraightforwarddone

byMLE.However,wedon’treallygettoseethelatentvariablesZ,butonlytheincompletedata

Y.weknowZbytheposteriordistribution / , Ө .Sincewecannotusethecompletedatalog

likelihood, we instead take its expected value under the posterior distribution of the latent

variable.This stepcorresponds to theEstepof theEMalgorithm.TheMstep thenwouldbe

maximizingthisexpectationwithrespecttotheparametersofinterest.Ifwedenotethecurrent

estimateasӨ ,thentheEandMstepswouldresultinabetterestimateӨ .Thesetwosteps

will then be repeated till we get some precision degree [1]. We start our EM algorithm by

choosinganinitialpointӨ fortheparameterstostartwith.IntheEstep,weusethecurrent

parametervalueӨ tofindtheposteriordistributionofthelatentvariables / , Ө .Wethen

use this posterior distribution in finding the expectation of the complete data log likelihood

computedatӨ ,wedenotethisexpectationas Ө , Ө ,whichcalledauxiliaryfunction,

isgivenby,

Ө , Ө / , Ө , /Ө 3.2

Ө isthencomputedintheMstepas,

ӨӨ

Ө , Ө 3.3

LetusprovethoseequationsalongsidewiththeconvergenceoftheEMalgorithmregardlessof

anyinitialstartingpoint.Todothat,weneedtointroducesomedefinitions.

3.2 Jensen’sInequality

Thisinequalitywillbeusefulinthederivationoftheupdateformulasinthemixturemodels.Itsays

that,

  

  17    

3.4

Wherefisanyconcavefunctionand∑ 1,and0 1.

Also,

or,

1 1

3.3 Kullback‐Leibler(KL)Divergence

Thishelpsus findthedivergencebetweentwoprobabilitydistribution functions.Letustaketwo

distributions,p(y)andq(y).ThentheKLdivergencebetweenthemisgivenby,

,

Usingthefactthat 1,wethenget,

1

0

Thenweseethat , 0.

Thisgivesthefollowing,

Orindiscreteversion,

3.5

  

  18    

LetustrytowritetheKLdivergenceinthecaseoftwoGaussians,

Let, ; μ , ∑ and, ; μ , ∑ then,

,12

∑ ∑ μ μ ∑ μ μ |∑ ||∑ |

3.6

3.4 TheEMDerivation

If we consider a mixture of Gaussians distribution, the parameters values which need to be

estimated are the means, , , . . , covariances, ∑ , ∑ ,… , ∑ and component priors,

, , … , . Thosevalueswill change from iteration to iteration,Ө toӨ till they reach the

optimumvalueorstabilizeatsomevalues.OncetheӨ goestoӨ ,thePDFwillalsoiteratefrom

/Ө to /Ө .Theincreaseintheloglikelihoodwillbe,

Ө Ө /Ө /Ө

/Ө/Ө

3.7

In the case of mixture model, we need to introduce the latent variables as being one of the M

mixtures.

Ө Ө1/Ө

, /Ө

1/Ө

/ , Ө , /Ө/ , Ө

Sincethelogfunctionisstrictlyconcave,wecanapplyJensen’sInequalityasfollows,

Ө Ө / , Ө , /Ө

/Ө / , Ө 3.8

We’veused / , Ө inJensen’sinequality.

  

  19    

Also,we’veseenfrom(3.2)thattheauxiliaryfunctionmaybewrittenas,

Ө , Ө / , Ө , /Ө

Wecanthenrewriteequation(3.7)as,

Ө Ө Ө , Ө Ө , Ө

Thedifference in theauxiliary functiongivesa lowerboundonthe increase in the log likelihood.

Wecanseethat Ө , Ө dependsonthecurrentparameters,soifwejustmaximizetheauxiliary

function Ө , Ө , the likelihood functionwillalsobemaximizedasa resultof that.Nowour

aimistomaximizethefunction Ө , Ө togettotheparametersofinterest.Wecandothatby

differentiating the auxiliary function Ө , Ө with respect to the new parameters Ө and

equatetheresulttozero.Weneedtomakesurethatwhenmaximizingthefunctionwithrespectto

themixingproportion,weneedtouseLagrangemultiplier.

Ө Ө , Ө 0.

WhatweneedtoguaranteeisthatasweiteratefromӨ toӨ ,weneedtomakesurethatweare

in fact increasing the log likelihood function or elsewe are not improving or going towards the

optimumsolution.Weneed,

Ө Ө or,

Ө Ө 0.

Letusintroducetheposteriordistributionofthelatentvariable / , Ө ,

/Ө /Ө / , Ө /Ө /Ө

Since,

/ , Ө 1

Fromthedefinitionofconditionalprobability,

  

  20    

/Ө, /Ө/ , Ө

Wecanthenwritethefollowing,

/ , Ө /Ө / , Ө , /Ө/ , Ө

Similarlyforthesecondterm,

/ , Ө /Ө / , Ө , /Ө/ , Ө

Combiningthelasttwoequationsweget,

Ө Ө / , Ө , /Ө / , Ө / , Ө

/ , Ө , /Ө / , Ө / , Ө

FromthedefinitionoftheKLdivergence,

/ , Ө , / , Ө / , Ө / , Ө/ , Ө

0

UsingthisKLdivergenceequationinthelastequationweget,

Ө Ө / , Ө , /Ө / , Ө , /Ө

Where,thedifferencebetweentheleftandrighthandsidesistheKLdivergence.Ifwecanassure

thattherighthandsideispositive,thenthelefthandsidewillbealsopositive.

Sufficestoshowthat,

/ , Ө , /Ө / , Ө , /Ө

Whichwillassurethat,

Ө Ө .

  

  21    

Letusrecalltheauxiliaryfunction,

Ө , Ө / , Ө , /Ө

Iftheauxiliaryfunctionincreases,thenthelikelihoodwillalsoincrease.If,

Ө , Ө Ө , Ө

Then,

Ө Ө .

We can see the difference between the likelihoods and the auxiliary functions would be the KL

divergence,asfollows,

Ө Ө Ө , Ө Ө , Ө / , Ө , / , Ө 3.9

Theincreaseintheauxiliaryfunctionwouldbealowerboundontheincreaseoftheloglikelihood.

We noted thatmaximizing the auxiliary function once, doesn’t result in the optimumparameter

values, we need to iterate till convergence is being achieved. We can now summarize the EM

algorithmintwosteps:

⤇ The Expectation step which is done after guessing a starting point Ө and calculating the

posterior distribution of the latent variable, / , Ө , by taking the expected value of the log

likelihood of the complete data in terms of the new parameter values Ө , , /

Ө , Ө with respect to the posterior distribution of the latent variables , / , Ө . As

follows,

Ө , Ө / ,Ө , /Ө , Ө 3.10

⤇TheMaximizationstepwhichmaximizestheauxiliaryfunction, Ө , Ө ,withrespecttothe

newparametersӨ .

ӨӨ

Ө , Ө

The major problem with the initial parameter values that if there is at least one local maxima

besidestheglobalmaxima,thentheEMalgorithmmightonlygetalocalmaximainsteadofgetting

  

  22    

the globalmaxima.Oneway of trying to overcome this problem is by starting atmanydifferent

initialpointsandthenchoosestheonewhichgivesthehighestlikelihood.

3.5 MixtureofGaussiansandtheEMalgorithm

One of the applications of the EM algorithm is to find the MLE of the Mixture of Gaussians

parametersintheexistingofunobservedvariablesordata.We’veseenthattheparameterscouldn’t

be foundby a directMLE.An iterative procedureneeds to be done in order for us to get to the

optimum results for the parameters. Themajor problemwhichwe encounter here is thatwhich

Gaussiancomponentwasresponsibleforgeneratingthatspecificsample.Weknowthatthelatent

variables are discrete which determines the number of mixtures which have been used in

generating the observed data. If we have prior knowledge about the components which were

responsible for generating the data, then theproblemwould become easier and could be solved

usingadirectMLE.Sonowourmajorproblemisestimatingthecomponentwhichwasresponsible

forgeneratingeachdatapointinmyobserveddata.Onceweknowtheposteriordistributionofthe

hidden variables / , Ө , the problem then become like a directMLE problem in the auxiliary

functionsense.

Assumethat, 1iftheobserveddata wasgeneratedbycomponent0otherwise

Let us have a look at a single received sample which we suppose that it was generated by

component .Wecanwrite,

, /Ө / , Ө

/ , Ө 3.11

InadditionofYbeing . . .samples,Zalsoare . . .samples,

, /Ө , /Ө

In the Expectation step of the EM algorithm we need to compute iteratively the posterior

distribution of the latent variable / , Ө in order to find the auxiliary function. Then we

maximizetheauxiliaryfunction.

  

  23    

Ө , Ө / , Ө , /Ө

/ , Ө , /Ө

/ , Ө , /Ө

/ , Ө , /Ө

/ , Ө / , Ө

/ , Ө / , Ө

/ , Ө

Ө , Ө / , Ө / , Ө

/ , Ө 3.12

Weknowthattheloglikelihoodfunctionforacomponentmixtureis,

; μ , ∑12

2 |∑ | μ ∑ μ

Thisisthensubstitutedintheauxiliaryfunctionasfollows,

  

  24    

Ө , Ө / , Ө12

μ ∑ μ

/ , Ө12

2 ∑

/ , Ө 3.13

We now need to estimate the parameters of interest of component at iteration 1 by

differentiatingtheauxiliaryfunctionwithrespecttooneoftheparametersandequatingtozero,

Ө Ө , Ө 0

Inthecaseofestimatingthemeanofthe mixturecomponent,

Ө , Ө 0

Thiswillresultintheupdatingmeanformula,

μ∑ / , Ө∑ / , Ө

3.14

∑ Ө , Ө 0

ThiswillresultintheupdatingCovarianceformula,

∑∑ / , Ө μ μ

∑ / , Ө 3.15

Equations(3.13)to(3.15)willbeiteratedtillconvergenceisachieved.

Tosumup,theEMalgorithmforMixtureofGaussiansmaybesummarizedasfollows,

1. Initialize the parameters mean μ , Covariance ∑ , and mixing coefficient

,thencalculatetheinitialvaluefortheloglikelihoodfunction.

2. IntheEstep,weneedtocalculatetheresponsibilities, / , Ө ,

/ , Ө/μ , ∑

∑ /μ , ∑

3. IntheMstep,weneedtoreestimatetheparameters,usingthecurrentresponsibility,

  

  25    

∑ / , Ө∑ / , Ө

∑∑ / , Ө

∑ / , Ө

∑ / , Ө

4. Evaluate the log likelihood checking for increasing the log likelihood and stopping the

iterationprocessifagivenprecisionhasbeenachieved.

/ , , ∑ /μ , ∑

Iftheprecisionhasn’tbeenachieved,wegobacktopointnumber2anditerateonceagain.

Wedothesameproceduretillweachievetheprecisioncondition.

For instance, let us consider an example showing a mixture of four Gaussians as an

illustrationexample.Iassumedthatthemeansasthevaluesofthefoursentsymbolsinthe

QPSKcase, ∈ 1, 1 ,andhaveequalcovarianceof∑ 0.35 ,whereIisa2×2matrix.

Figure 3.1 shown shows these generated data with its Gaussian fit using the previous

algorithm,myassumptionmodelwaslike,

Where isthereceiveddatawhichcorruptedbyadditivewhiteGaussiannoiseAWGN, of

covariance,∑,bothofsizeN,and 1, 1 .Thisisdonebygenerating800samples

drawn from eachmixture of four Gaussians and then pretending likewe don’t know the

parametersfromwhichthosesamplesweredrawnandtryingtoestimatethoseparameters

byfindingthebestcontourmatchandestimatingtheparametersusingtheEMalgorithm.

Aswecansee fromthecoming figures, themorewe increasethevariances, themorethe

inference become difficult and we might not get the true parameters as a result. As we

decreasethevariance,themixtureswouldbecomemorecircularandseparatedandeasyto

be classified. Figure 3.2 shows the mixture of four Gaussians exactly as in the previous

figure except that we decreased the variances to 0.05. We clearly see the difference in

estimatingthecontoursofthosemixtures.Inthisfigureclassificationcouldbedoneeasily

without encountering toomuch error. K‐means plot is also drawn to see that it is just a

specialcaseoftheEMalgorithmwhenthecovarianceapproacheszero.

  

  26    

Figure3.1.mixturesoffourGaussiansdataandtheirfitwhen 0.35

Figure3.2.mixturesoffourGaussiansdataandtheirfitwhen 0.05

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

y1(n)

y 2(n)

Scatter plot and its PDF Contours estimate using the EM algorithm

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

y1(n)

y 2(n)

Scatter plot with noise Covariance 0.35I

-3 -2 -1 0 1 2 3-4

-3

-2

-1

0

1

2

3

4

y1(n)

y 2(n)

Cluster 1

Cluster 2Cluster 3

Cluster 4

Com

pone

nt 1

Pos

terio

r P

roba

bilit

y

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

y1(n)

y 2(n)

partitioned data using the K-means

Cluster 1

Cluster 2

Cluster 3Cluster 4

Centroids

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

y1(n)

y 2(n)

Cluster 1

Cluster 2Cluster 3

Cluster 4

Com

pone

nt 1

Pos

terio

r P

roba

bilit

y

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

y1(n)

y 2(n)

partitioned data using the k-means

Cluster 1

Cluster 2

Cluster 3Cluster 4

Centroids

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

y1(n)

y 2(n)

Scatter plot with noise Covariance 0.05I

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

y1(n)

y 2(n)

Scatter plot and PDF Contours estimate using the EM algorithm

  

  27    

4 SISONDASNRestimationusingtheEMalgorithmintheLiterature

4.1 BPSKcase

In the binaryphase shift keying,wehave amixtureof two classes, ∈ 1, 1 .Those are the

transmittedsymbols.

4.1.1 Thesystemmodel

We are assuming that there is no error in timing, phase deviation, or frequency deviation. The

matchedfilteroutputmaybegivenby,

, 1, … . . , 4.1

where, isthereceivedsample, isthetransmittedsymbol, isthechannelgain,and isa

zeromeanAWGNwithvariance .

Wecanwriteequation(4.1)inavectorformas,

4.2

where, . . . . , . . . . , . . . . . The transmitted

symbols may be modeled as realizations of . . . binary random variables. The probability

distributionfunctionmaybewrittenas,

/ , Ө1

√2

2 4.3

Assumingequallylikelysymbols,wehave,

; Ө / , Ө /Ө

1

√2

12 2

12

2

where,Ө .NowforNreceived . . .sampleswehave,

; Ө ; Ө

  

  28    

; Ө1

√2

12 2

2

4.4

This is the likelihood functionof thereceiveddata.Nowgiven theobservationdata,estimate the

parametervaluesӨ andthensubstitutetofindtheSNR,

10

4.1.2 TheCRLBforBPSK

TheCramer‐RaoLowerBound(CRLB) isa lowerbound for theMeanSquareError (MSE)of any

unbiasedestimatorwhichsatisfiestheregularitycondition[2].InourcaseofSNRestimation,the

CRLBcanbewrittenas,

ӨӨ

Ө 4.5

Where Ө isthefisherinformationmatrixgivenby,

Ө

; Ө ; Ө

; Ө ; Ө 4.6

And,

Ө20

10

1010

4.7

FortheNDAcase,

Theloglikelihoodfunctionisgivenas,

Ө /Ө

2

212

4.8

  

  29    

After differentiating this function with respect to the channel gain and the variance we get the

following,

; Ө

; Ө

; Ө ; Ө

; Ө2

1 2

Ө

1

12

4.9

where,

2√2

2

4.10

Wecannowsubstitutethisresultintoequation(4.5)andgetthefollowing,

20 10

10

10

1

12

20 10

1010

20010

21

1 2 4.11

  

  30    

FortheDAcase,

Ө / , Ө

12

22

where isgiventobeeither+1or‐1.

After differentiating this function with respect to the channel gain and the variance we get the

following,

; Ө

; Ө

; Ө ; Ө

; Ө2

1

Ө0

02

4.12

Wemaynowsubstitutethisresultintoequation(4.5)andgetthefollowing,

20 10

10

10

0

02

20 10

1010

20010

2

1 4.13

  

  31    

4.1.3 NDASNRestimationusingtheEMalgorithm

WenowintroducethemethodofEMalgorithminestimatingtheSNR.Ourassumptionsarethatthe

data was drawn from a mixture of two Gaussians, and are uncorrelated with each other. The

samplesaredrawninan . . .manner.Theparameterswhichweneedtoestimatenoware,

Ө

Ө

; Ө

10 4.14

TheEMalgorithmwillbeusedtoiterativelyestimatethechannelgainandthenoisevariancethen

usethemtoestimatethesignaltonoiseratio.Thereceiveddatawillbeseenasincompletedataset

sincewedon’tknowitsclass(transmittedsymbol).Theauxiliaryfunctioninthiscasemaybegiven

as,

Ө , Ө / ,Ө , /Ө , Ө

Ө , Ө / , Ө / , Ө

/ , Ө 4.15

ӨӨ

Ө , Ө 4.16

Now solving those two equations will give us the E & M steps. Since we know the mixing

proportion ,wedon’tneedtoestimateit.

Theauxiliaryfunctionmaybegivenas,

Ө , Ө 1 / , Ө / , Ө

  

  32    

Ө , Ө / , Ө12

∑

/ , Ө12

2 ∑ / , Ө

/ , Ө12

22

Ө , Ө / , Ө

20

/ , Ө 0

/ , Ө 0

/ , Ө

/ , Ө / , Ө

, / , Ө / , Ө 1

/ , Ө 1 / , Ө

2 / , Ө 1

12 / , Ө 1

  

  33    

2 / , Ө 1 2

2

2

2

1

2

2

2

2

2

2

2

2

2

2

1 4.17

Solvingforthevariance,wehave,

Ө , Ө

/ , Ө / , Ө / , Ө / , Ө 0

/ , Ө1

2 2/ , Ө

1

2 20

/ , Ө1

2 21 / , Ө

1

2 2

0

/ , Ө2

1

2 2/ , Ө

2

0

  

  34    

2 / , Ө1

2 20

4 / , Ө 2 0

2 2 / , Ө 1 0

2 2

0

2 0

1 4.18

10

1 4.19

1 4.20

Wedon’tneedtoworryaboutthemixingcoefficientsinceweknowthattheyareequallylikely.The

hyperbolictangentinupdatingthechannelgainwillserveasasoftthreshold.Whenthehyperbolic

tangent argument is large, this thresholdwould becomemore like a signum function threshold.

After Starting with initial values for the channel gain and noise variance, equations (4.17) and

(4.18)willbeusedtoiterativelyfindtheestimatestotheparameterswhichwouldbethenusedto

  

  35    

estimate the signal to noise ratio, , as given above. The will be found for different

estimatedvaluesofSNR.ThenthisMSEwillbeassessedusingtheNDACRLBinunitsofdB2.The

estimationprocesswillbedoneforarangeofSNRfrom1dBto20dB,andfordifferentvaluesof

receivedsamplesize,N,128,512and1024.Figure4.1shownexplainsthreethingswhenBPSKis

usedas the transmittedsymbols.Theredcurve is theCRLB forNDAsignal, thebluecurve is the

CRLBforDAsignal,andtheblackcurveshowstheMSEestimatedusingtheEMalgorithm.TheEM

algorithm estimate is repeated 10000 times using Monte Carlo simulations to get 10000

realizationsof theSNRestimates, thenaveragingthemtogettheestimatedSNR,thenfindingthe

variance to get the MSE for that specific SNR estimate value. We then repeat this process for

different values of SNR, say from 0 dB to 20 dB and their resultant MSE. Those estimates then

assessedwiththeCRLBasshownbelow.WeseethatforlowSNRestimatevalues,theMSEislarge

comparedwiththeNDACRLB.ThehighertheSNRbecome,thelowertheMSEwegettillthey(the

CRLB andMSE) bothmatch. This was for sample size of N=128. The next step we would do is

increaseNandseewhatisgoingtohappentoourestimationaccuracy.

Figure4.1TheMSE&CRLBindB2vs.SNRindBwhenN=128fortheBPSKcase

0 2 4 6 8 10 12 14 16 18 200

0.5

1

1.5

2

2.5

3

SNR (dB)

MS

E &

CR

LB (

dB)2

N=128

CRLBNDA

CRLBDAMSE-EM

  

  36    

Figure4.2TheMSE&CRLBindB2vs.SNRindBwhenN=512fortheBPSKcase

Figure4.2showstherelationbetweenMSEandCRLBvs.SNRforN=512.Whatwerealizehereis

that theMSE&CRLBaredecreased comparedwith the casewhenN=128.CRLB itselfdecreased

from0.3dB2forhighSNRwhenNwas128,tolessthan0.1dB2forhighSNRwhenNis512.Figure

4.3showsthesamethingbutforN=1024.Figure4.3showsthattheprecisionoftheMSEandCRLB

isintheorderoflessthan0.05dB2forSNRsgreaterthan8dB.Figure4.4showstheerrorwhichis

the difference between theMSE and CRLB in dB2 and the SNRs for the previous three cases of

differentNs,128,512,and1024.ThisfigureprovesthatasNincreases,theerrordecreaseswhich

meansthattheMSEbecomecloserandclosertotheCRLBuntiltheybothmatchatsomeSNRvalue

and beyond. From the SNR equationwe see that the SNR is inversely proportional to the noise

variance,whichmeansasweincreasethenoisevariance,theSNRwilldecreaseasaresultofthat,

andthetwomixturesthenwillintervenewitheachotherwhichmakesitdifficultforclassification.

Ontheotherhandhowever,aswedecreasethenoisevariance,theSNRwillincreaseasaresultof

thatandthetwomixturesthenwillbefarawayfromeachotherwhichmakesclassificationbecome

easierandresultsinalmostnoerror..

0 2 4 6 8 10 12 14 16 18 200

0.2

0.4

0.6

0.8

1

1.2

SNR (dB)

MS

E &

CR

LB (

dB)2

N=512

CRLBNDA

CRLBDAMSE-EM

  

  37    

Figure4.3TheMSE&CRLBindB2vs.SNRindBwhenN=1024fortheBPSKcase

Figure4.4TheErrorindB2vs.SNRindBfordifferentNs.

0 2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

SNR (dB)

MS

E &

CR

LB (

dB)2

N=1024

CRLBNDA

CRLBDAMSE-EM

0 2 4 6 8 10 12 14 16 18 200

0.5

1

1.5

2

2.5

SNR (dB)

Err

or in

(dB

)2

N=128

N=512

N=1024

  

  38    

Figure4.5TheMSEindB2vs.N.

The last figure4.5which shows the changes in theMSEaswe increaseN. this is easily couldbe

inferred from the previous figures, but we want it to be clear that as we increase N, MSE will

decreaseasaresult.WevaryNfrom128to4096andseetheresultantMSEwhenSNRhasbeen

chosenconstantinthiscasetobe4dB.TheMSEdecreasedfrom0.45dB2whenNwas128toabout

0.025dB2whenNis4096.OurexpectationisthatasNtendstoinfinity,MSEwilltendtozeroasa

result

4.2 TheQPSKcase

In the case ofQPSK case, it is assumed that the received samplesweredrawn independently an

identicallyfromamixtureoffourcircularsymmetriccomplexGaussiandistribution.Ourmodelin

thiscasewouldbeexactlythesameexceptforthetransmittedsymbolsnowfourinsteadoftwo.We

areassumingthatthereisnoerrorintiming,phasedeviation,orfrequencydeviation.Thematched

filteroutputmaybegivenas,

, 1, … . . , 4.21

500 1000 1500 2000 2500 3000 3500 40000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

N

MS

E in

dB

2

MSE as we increase N for SNR=4dB

  

  39    

where, isthereceivedsample, ∈√ √

isthetransmittedsymbol, isthechannelgain,

and isazeromeanAWGNwithvariance .

Wecanwriteequation(4.21)inavectorformas,

4.22

where, . . . . , . . . . , . . . . . The transmitted

symbolsmaybemodeledasrealizationsof . . .circularsymmetriccomplexGaussiandistribution.

Theprobabilitydistributionfunctionmaybewrittenas,

/ , Ө1

exp1| | 4.23

Assumingequallylikelysymbols,

; Ө / , Ө /Ө

1 1

41 1

√2

1

√2

14

1 1

√2

1

√2

14

1 1

√2

1

√2

14

1 1

√2

1

√2

1 1

41

√21

1

√21

1

√21

1

√21

| | ∗

| | | | 2 ∗

  

  40    

; Ө1 1

41

4

1

4

1

4

1

4

; Ө1 1

4

| |

2

4

2

42

4

2

4

; Ө1 1

4

| | 2

24

22

4

; Ө1 1

2

| |

24

24

; Ө1

| |

√2

√2

TheprobabilitydistributionfunctionofN . . .receivedsamplesof willbeasfollows,

; Ө1

| |

√2

√2

4.24

TheLogLikelihoodfunctionisthendefinedasfollows,

Ө /Ө

  

  41    

Ө 1

| |

√2

√2

1 | | √2

√2

4.25

ThisistheLogLikelihoodfunctionofthereceiveddataof ,nowtheproblemofestimationwould

beasfollows,giventheobservationvector, ,estimatetheparameters,Ө .Thenusethem

tofindtheSNRestimate,

10

4.2.1 TheCRLBforQPSK

TheCramer‐RaoLowerBound(CRLB) isa lowerbound for theMeanSquareError (MSE)of any

unbiasedestimatorwhichsatisfiestheregularitycondition[2].InourcaseofSNRestimation,the

CRLBcanbegivenas,

ӨӨ

Ө

Where Ө isthefisherinformationmatrixgivenby,

Ө

; Ө ; Ө

; Ө ; Ө

And,

Ө20

10

1010

FortheNDAcase,

Theloglikelihoodfunctionisgivenby,

  

  42    

Ө1 | | √2

√2

ThisprocedureissimilartotheonewhichbeingdonebyAlagha[8],

Ө

2 1 2 4

4 1 4

4.26

where,

2√

2 4.27

Wecannowsubstitutethisresultinequation(4.5)andgetthefollowing,

20 10

10

10

2 1 2 4

4 1 4

20 10

1010

10010

12

2

1 2 4 4.28

FortheDAcase,

Theloglikelihoodfunctionisgivenby,

Ө / , Ө

Ө 1

exp| |

| |

Thensubstitutingtofindthefisherinformationmatrix,

  

  43    

Ө

20

0

20 10

10

10

20

0

20 10

1010

10010

21 4.29

4.2.2 NDASNRestimationusingtheEMalgorithm

Weuse theEMalgorithmnow inestimating theSNR, itsMSEandcompareourestimatedvalues

with the CRLB values. Our assumptions are that the data was drawn from a mixture of four

Gaussians,andareuncorrelatedwitheachother.Thesamplesaredrawninan . . .manner.The

parameterswhichweneedtoestimatenoware,

Ө

Ө

; Ө

10 4.30

TheEMalgorithmwillbeusedtoiterativelyestimatethechannelgainandthenoisevariancethen

usethemtoestimatethesignaltonoiseratio.Thereceiveddatawillbeseenasincompletedataset

sincewedon’tknowtheclassofeachtransmittedsymbol.Theauxiliaryfunctioninthiscasemaybe

givenby,

Ө , Ө / ,Ө , /Ө , Ө

  

  44    

Ө , Ө / , Ө / , Ө

/ , Ө

Ө , Ө / , Ө12

∑

/ , Ө12

2 ∑

/ , Ө

/ , Ө 1

Ө , Ө / , Ө

ӨӨ

Ө , Ө 4.31

NowsolvingthosetwoequationswillgiveustheE&Msteps.

Whensolvingforthechannelgainweget,

Ө , Ө / , Ө 0

Ө , Ө / , Ө 0

/ , Ө 2 2 ∗ 0

/ , Ө / , Ө ∗

  

  45    

/ , Ө exp 2 1 /4

/ , Ө exp /4 / , Ө exp 3 /4

/ , Ө exp 5 /4 / , Ө exp 7 /4

/ , Ө 1 / , Ө / , Ө / , Ө

/ , Ө exp /4 / , Ө exp 3 /4

/ , Ө exp 5 /4

1 / , Ө / , Ө / , Ө exp 7 /4

/ , Ө exp /4 exp 7 /4

/ , Ө exp 3 /4 exp 7 /4

/ , Ө exp 5 /4 exp 7 /4 exp 7 /4

√2 / , Ө / , Ө 1 / , Ө

12

1

√2 / , Ө / , Ө

12

/ , Ө / , Ө12

4.32

Whensolvingforthevarianceweget,

Ө , Ө / , Ө 0

  

  46    

/ , Ө1

0

/ , Ө

/ , Ө / , Ө

/ , Ө

1 / , Ө / , Ө / , Ө

/ , Ө 2 ∗ 2 ∗

/ , Ө 2 ∗ 2 ∗

/ , Ө 2 ∗ 2 ∗

2 / , Ө exp /4 exp 7 /4

2 / , Ө exp 3 /4 exp 7 /4

2 / , Ө exp 5 /4 exp 7 /4

2 / , Ө √2 2 / , Ө 2 1

2 / , Ө √2

2√2 / , Ө / , Ө 1

/ , Ө1

2√2 7 /4

  

  47    

2√2 / , Ө / , Ө

12

/ , Ө / , Ө12

1

2√2| |

1

2√2

1| | 4.33

Wedon’t need toworry about themixing coefficient sincewe know that they are equally likely.

After Starting with initial values for the channel gain and noise variance, equations (4.27) and

(4.28)willbeusedtoiterativelyfindtheestimatestotheparameterswhichwouldbethenusedto

estimatethe asgivenabove.The willbefoundfordifferentestimatedvaluesofdifferent

SNRvaluesandwillbeassessedsingtheCRLB.

Figure4.6TheMSE&CRLBindB2vs.SNRindBwhenN=128

ThenthisMSEperformancewillbecomparedtothefundamentallimitoftheCRLBinunitsofdB2at

eachestimatedSNRvalue.TheestimationprocesswillbedoneforarangeofSNRfrom0dBto20

0 2 4 6 8 10 12 14 16 18 200

0.5

1

1.5

2

2.5

3

SNR (dB)

MS

E &

CR

LB (

dB)2

N=128

CRLBNDA

CRLBDAMSE-EM

  

  48    

dB, and for different values of received sample size N, 128, 512, and 1024. Figure 4.6 shown

explainsthreethingswhenQPSKisusedasthetransmittedsymbols.TheredcurveistheCRLBfor

NDA signal, theblue curve is theCRLB forDA signal, and theblack curvewhich shows theMSE

estimatedusingtheEMalgorithm.TheEMalgorithmSNRestimateisrepeated10000timesusing

MonteCarlosimulationstoget10000realizationsoftheSNRestimates,thenaveragingthemtoget

theestimatedSNR,andfindtheirvariancetogettheMSEforthatspecificSNRestimatevalue.

Figure4.7TheMSE&CRLBindB2vs.SNRindBwhenN=512

WethenrepeatthisprocessfordifferentvaluesofSNR,sayfrom0dBto20dBandtheirresultant

MSE. Those estimates then comparedwith the CRLB as shown above.We see that for low SNR

estimatevalues; theMSE is largecomparedwiththeNDACRLB.Thehigher theSNRbecome, the

lowertheMSEwegettillthey(theCRLBandMSE)bothmatch.ThiswasforsamplesizeofN=128

infigure4.6inthepreviouspage.ThenextstepwewoulddoisincreaseNandseewhatisgoingto

happentoourestimationaccuracy.Figure4.7showstherelationbetweenMSEandCRLBvs.SNR

forN=512.WhatwerealizehereisthattheMSEisdecreased,alsoCRLBisdecreasedfromwhatit

wasbeforewhenN=128.CRLBitselfdecreasedfrom2whenNwas128andSNRof0dB,tolessthan

0.5whenNis512atthesameSNR.

0 2 4 6 8 10 12 14 16 18 200

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

SNR (dB)

MS

E &

CR

LB (

dB)2

N=512

CRLBNDA

CRLBDAMSE-EM

  

  49    

Figure4.8.TheMSE&CRLBindB2vs.SNRindBwhenN=1024

Figure4.9.TheErrorindB2vs.SNRindBfordifferentNs.

0 2 4 6 8 10 12 14 16 18 200

0.05

0.1

0.15

0.2

0.25

SNR (dB)

MS

E &

CR

LB (

dB)2

N=1024

CRLBNDA

CRLBDAMSE-EM

0 2 4 6 8 10 12 14 16 18 200

0.5

1

1.5

SNR (dB)

Err

or in

(dB

)2

N=128

N=512

N=1024

  

  50    

Figure4.8showsthesamethingFigure4.7shows,butforN=1024.TheprecisionoftheMSEand

CRLBisintheorderoflessthan0.025dB2forSNRsgreaterthan6dB.Figure4.9showstheerror

whichisthedifferencebetweentheMSEandNDACRLBindB2atdifferentvaluesofSNRsshown

for the previous three cases of different Ns, 128, 512, and 1024. This figure proves that as N

increases, the error decreases whichmeans that theMSE become closer and closer to the NDA

CRLBuntiltheybothmatchatsomeSNRvalueandbeyond.FromtheSNRequationweseethatthe

SNRisinverselyproportionaltothenoisevariance,whichmeansasweincreasethenoisevariance,

theSNRwilldecreaseasaresultofthatandthefourmixturesthenwillintervenewitheachother

whichmakesitdifficultforustoclassifyeachclass.Ontheotherhandhowever,aswedecreasethe

noisevariance,theSNRwillincreaseasaresultofthatandthefourmixturesthenwillbefaraway

fromeachother(classesareseparatedfromeachother)whichmakesclassificationbecomeeasier

andresultsinalmostnoerror.

  

  51    

5. MISOwithSTBCNDASNRestimationusingtheExpectationMaximizationalgorithm

Modern communication systems require accurate estimation of the Signal to Noise Ratio

SNR for optimal usage of radio resources [5], [9]. The knowledge of the SNR is a

requirementinmostapplicationsinordertomakeoptimalsignaldetection,powercontrol.

MIMO systems gives a significant enhancement to the accuracy of estimating the SNR

besidesmanyotherimprovementsindatarate,channelcapacity,andbeingabletoreduce

Multipath fading at the remote units. Different diversity modes are being used, time

diversity, frequency diversity, spatial diversity. A MIMO system usually consists of M

transmitandNreceiveantennas.Attentionwillbegiventothetransmitdiversityforsome

reasons, which then will be compared with SISO and SIMO. Transmit diversity or MISO

wouldimproveoursystem’sperformancewithoutthecost,size,andpowerthatSIMOwill

have inorder to improve thequalityof thereceivedsignal [12].For that reason,antenna

diversitytechniquesareutilizedattheBasestation.Asimpletransmitdiversitytechnique

willbeexplainedherewhichbeendonebyAlamoutiin[12].Space‐timecodesareusedto

generate a redundant signal. The signal copy is transmitted not only from a different

antenna,butatadifferenttime.Thesignals1ands2aresentinthefirstsymbol,thenasignal

replicationof–s2*ands1*isaddedtocreatetheAlamoutispace‐timeblockcode.

5.1ConventionalReceivediversity(SIMO)model:

                       

 

                                                                                              

                               

                                         

 

                           

 

                     

   

Figure5.1SIMO/ReceiveDiversitysystemmodel

h

h2

hNR

TX#1 

RX#1 

RX#2 

RX#NR 

Combining schem

e 

  

  52    

h1

h2

hNT

wherei=1,2,.....,NR.andn=1,2,.....,N.

/ , Ө1

exp

/ , Ө1

exp

5.2Transmitdiversity(MISO)model:

Figure5.2MISO/Transmitdiversitysystemmodel

wherei=1,2,.....,NT.andn=1,2,.....,N.

/ , Ө1

exp∑ 1

/ , Ө1

exp∑ 1

TX#1 

TX#2 

TX#N

RX#1 

  

  53    

5.3MIMOmodel:

Figure5.3MIMOsystemmodel

wherei=1,2, . . . . . ,NT, j=1,2, . . . . . ,NR ,andn=1,2, . . . . . ,N,andhij isthechannelgain

betweenthetransmitantennaiandthereceiveantennaj.

/ , Ө1

exp

/ , Ө1

exp

TX#1 

TX#2 

TX#N

RX#1 

RX#2 

RX#NR 

h11

hNT NR

  

  54    

5.4 ThetransmitdiversityforBPSKcase:

ThisapproachisbeingdonebyAlamouti[12]wheretwotransmitantennasandonereceive

antennahasbeenused.Atagiventime,n,twosignalsaresimultaneouslysentfromthetwo

antennas.Denotingthesignalsentbyantennaoneby,S1,andthesignalsentbyantennatwo

by, S2. Then we repeat sending – S2* from the first antenna, then S1* from the second

antenna,thisorderingwillbedonetoallofourdata.Thisiscalledspaceandtimecoding.

Ourmodelinthiscasewilllooklike,

∗ ∗

Wethenmaywriteitinamatrixformas,

∗

∗ ∗ , 1,2, … . . , /2. 5.1

⊗

whereR(n)isthereceivedsamplesfrombothantennasattimen,Histhecomplexchannels

gains matrix, S(n) are the sent symbols at time n from the two antennas, W(n) are the

CircularlysymmetriccomplexGaussiannoisesamplesattimen,Risthevectorofreceived

samples,Iis IdentitymatrixwhereNisthetotalnumberofreceivedsamples,Sisthe

sentvectorofsymbols,Wisthenoisevectoratthereceiver,⊗isthekronckerproduct,and

Hisasfollows;

∗ ∗

The probability distribution function for the received data at a given time n, given the

transmitteddata,givenby,

/ , Ө1

exp‖ ‖

5.2

; Ө / , Ө /Ө

Supposeanequallylikelydatasymbols.

; Ө1

4exp

‖ ‖

  

  55    

; Ө1

4exp

| | | |

FortheBPSKcase, 1, 1 , 1, 1 .

where ∈ 11, 1

1, 1

1, 1

1.

Wecanthenrewrite ; Ө asfollows,

; Ө1

4exp

| | | |

exp| | | |

exp| | | |

exp| | | |

; Ө

14

exp| | | | 2 ∗ 2 ∗ | | | |

exp| | | | 2 ∗ 2 ∗ | | | |

exp| | | | 2 ∗ 2 ∗ | | | |

exp| | | | 2 ∗ 2 ∗ | | | |

; Ө1

4exp

| | | | | | | |∗

  

  56    

exp2 ∗ 2 ∗

exp2 ∗ 2 ∗

exp2 ∗ 2 ∗

exp2 ∗ 2 ∗

; Ө1

4exp

| | | | | | | |

∗ 2 cosh2 ∗ 2 ∗

2cosh2 ∗ 2 ∗

Weknowthat,

cosh cosh 2cosh2

cosh2

; Ө1

2exp

| | | | | | | |

∗ 2 cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

; Ө1

exp| | | | 2| | 2| |

∗ cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

5.3

ForN . . .receivedsamplesofR(n),theJointPDFdistributionwillbewrittenasfollows,

; Ө1

exp| | | | 2| | 2| |

/

∗ cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

5.4

TheLogLikelihoodfunctionwillthenbegivenby,

  

  57    

Ө ; Ө

/

Ө 1

exp| | | | 2| | 2| |

/

∗ cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

Ө

/| | | | 2| | 2| |

cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

5.5

Now given the received sample R, estimate the parameters h1, h2, and . However, h1, h2 are

complexvariables,soweneedtoestimatetheirrealandimaginarypartsseparately.

Let

Nowourparametersareasfollows,

Ө ̂

Ө

; Ө

10 5.6

  

  58    

5.5 TheCRLBforthetransmitdiversityintheBPSKcase.

FortheDAcase:

InthiscaseweareassumingthatweknowthetransmitteddataSandwouldliketofinda lower

boundonthevarianceinthiscase.WeknowthatCRLBisgivenby,

ӨӨ

Ө

where Ө isthefisherinformationmatrixgivenby,

Ө ; ӨӨ Ө

And,

Ө20

10

2010

20

10

2010

10

10

where | | | | .

/ , Ө1

exp‖ ‖

/ , Ө1

exp‖ ‖

/

Ө

/‖ ‖

/| | | |

/

| | | |

  

  59    

/

| | | ∗ |

/

∗ ∗

Ө 1/

2 2 ∗

Ө 1/

2 2

1

/

2 2

1

Ө 1/

42

ӨӨ

ӨӨ

ӨӨ

Ө

Ө 2, Ө

Ө

Ө Ө

Ө Ө0

  

  60    

Ө Ө

20000

02000

00200

00020

00001

ӨӨ

Ө

2010

20

10

2010

20

10

1010

20000

02

000

002

00

0002

0

0000

2010

2010

2010

2010

1010

 

100

log 102

1 5.7

FortheNDAcase:

Ө

/| | | | 2| | 2| |

cosh2 ∗ 2 ∗

cosh2 ∗ 2 ∗

WeneedtofindthefisherinformationmatrixforthisLikelihoodfunctioninordertofindtheCRLB

fortheNDAcase.

Ө Ө

Ө Ө, and Ө Ө for ,

  

  61    

Ө Ө

, Ө Ө

, Ө Ө

,

Ө Ө

, Ө Ө

, Ө Ө

,

Ө Ө

, Ө Ө

, Ө Ө

,

ӨӨ

, Ө Ө

, Ө Ө

Ө Ө

, Ө Ө

, Ө Ө

.

Ө 4/

2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2

Ө/

2 ∗ 2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2 2

Ө/

2 ∗ 2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗

Ө/

2 ∗ 2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗

  

  62    

Ө 4 2 ∗ 2 ∗/

2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗ 2 ∗

Ө 4/

2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2

Ө/

2 ∗ 2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗

Ө/

2 ∗ 2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗

Ө 4 2 ∗ 2 ∗/

2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 2 ∗ 2 ∗

  

  63    

Ө 4/

2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 ∗

Ө/

2 ∗ 2 ∗ 2 2

2 ∗ 2 ∗ 2 ∗ 2 ∗

Ө 4 2 ∗ 2 ∗/

2

2 ∗ 2 ∗ 2 2 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗ 2 ∗

Ө 4/

2 ∗ 2 ∗ 2

2 ∗ 2 ∗ 2 ∗

Ө 4 2 ∗ 2 ∗/

2

2 ∗ 2 ∗ 2 2 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗ 2 ∗

  

  64    

Ө 2/

2| | | ∗ | 2| | 2| |

2 ∗ 2 ∗

22 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗

2 ∗ 2 ∗

22 ∗ 2 ∗

2 ∗ 2 ∗ 2 ∗ 2 ∗

Numericalintegrationisusedtofindthestatisticalexpectationofeachentryinthefollowingfisher

informationmatrix.

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

Ө

ThisisthenusedtofindtheCRLBfortheSNRasfollows,

ӨӨ

Ө

5.6STBCNDASNRestimationusingtheEMalgorithm.

InthecomingSNRestimation,BPSKdatawillbeused,butonemaychooseanysignalingscheme.

TheEMalgorithmwillbeusedtoestimatetheSNR.TheEMalgorithmwilliterativelyestimatethe

channelsandthenoisevariancewhichthenbeusedtoestimatetheSNR.Theproblemis,giventhe

received samples R without the knowledge of the transmitted symbols, estimate the unknown

parameters.Ateachtime,wereceiveasample,wedon’tknowtowhichofthefourmixturesdoes

thissamplebelongsto.Ourreceiveddatawillbeincompletewhichwon’tbeeasyfortheMLEtobe

  

  65    

usedtoestimateitsparameters;weresorttoaniterativemethodliketheEMalgorithm.Westartby

writingtheauxiliaryfunction,

Ө , Ө / ,Ө , /Ө , Ө

Ө , Ө / , Ө

/

/ , Ө

/ , Ө

/

Ө , Ө / , Ө ∑

/

/ , Ө 2 ∑

/

/ , Ө

/

/ , Ө 1

Ө , Ө / , Ө

/

ӨӨ

Ө , Ө

Ө , Ө / , Ө

/

1 / , Ө

/

∗ 5.8

Ө , Ө / , Ө

/

  

  66    

1 / , Ө

/

∗ 5.9

Ө , Ө / , Ө

/

1 / , Ө

/

∗ 5.10

Ө , Ө / , Ө

/

1 / , Ө

/

∗ 5.11

Ө , Ө / , Ө

/

1 / , Ө

/

5.12

whereM=4 in our PBSK case, and N is the received sample size. Also, themixtures classes are

assumed equally likely. Equations (5.8)‐(5.12)will be used to update the parameters iteratively

starting from initial values for those parameters. Luckily that ourmodel has one global optimal

parameters,whichkeepusawayfromthinkingthatthevalueswhichweestimatedmightbelocal

maxima.TheMSE&CRLBwillbedrawnfordifferentvaluesofN.

  

  67    

Figure5.4.TheMSE&CRLBindB2vs.SNRindBwhenN=128.

Figure5.5.TheMSE&CRLBindB2vs.SNRindBwhenN=512.

0 2 4 6 8 10 12 14 16 18 200.1

0.2

0.3

0.4

0.5

0.6

SNR (dB)

MS

E &

CR

LB (

dB)2

Alamouti for N=128

MSE-EM

CRLBDACRLBNDA

0 2 4 6 8 10 12 14 16 18 200.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

SNR (dB)

MS

E &

CR

LB (

dB)2

Alamouti for N=512

CRLBDA

MSE-EMCRLBNDA

  

  68    

Figure5.6.TheMSE&CRLBindB2vs.SNRindBwhenN=1024.

Convergencetime,τoftheEMAlgorithm.(SNR=8dB):

Datasize,N Convergencetime,τinsecondsforonerun

Convergencetime,τ insecondsfor2000MonteCarlosimulations

Convergencetime,τinsecondsfor10000MonteCarlosimulations

τ1 τ2000 τ10,000

N=128 0.131456 6.881052 34.264368

N=512 0.140788 27.668970 139.303882

N=1024 0.173774 91.254040 456.952999

ThesimulationswereperformedusingMATLABversion7.13.0.564(R2011b),withprocessorIntel(R)Core(TM)i7CPU860 @ 2.80GHzwithRAM4GB.

0 2 4 6 8 10 12 14 16 18 200.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

SNR (dB)

MS

E &

CR

LB (

dB)2

Alamouti for N=1024

CRLBDA

MSE-EMCRLBNDA

  

  69    

Fig.5.7.TheMISOwithSTBCBPSKcase.

Fig.5.8.TheSISOBPSKcase.

0 2 4 6 8 10 12 14 16 18 200

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

SNR (dB)

MS

E &

CR

LBs

in (

dB)2

NDA CRLB, N=128

DA CRLB, N=128MSE-EM, N=128

NDA CRLB, N=512

DA CRLB, N=512

MSE-EM, N=512

NDA CRLB, N=1024DA CRLB, N=1024

MSE-EM, N=1024

0 2 4 6 8 10 12 14 16 18 200

0.1

0.2

0.3

0.4

0.5

0.6

SNR (dB)

MS

E &

CR

LBs

in (

dB)2

NDA CRLB, N=128

DA CRLB, N=128MSE-EM, N=128

NDA CRLB, N=512

DA CRLB, N=512

MSE-EM, N=512

NDA CRLB, N=1024DA CRLB, N=1024

MSE-EM, N=1024

  

  70    

6. Conclusions

Inthisresearch,transmitdiversityisusedasawayofimprovingtheSNRestimationaccuracy.The

SNRwasestimatedusingtheEMalgorithminaNDAmannerusingspace‐timeblockcoding(STBC)

which was done by Alamouti [12]. Also, the CRLB for the DA and NDA case were derived and

plottedwhichthenbeusedtoassesstheperformanceofourestimator.ThiswasdonefortheBPSK

constellationsystem.Itisbeingdonefordifferentsamplesizes,N;128,512,and1024.We’veseen

thattheCRLBandMSEareinverselyproportionaltothesamplesize,N.Inaddition,asweincrease

theSNR,CRLBandMSEbecomecloserandclosertoeachothertill theymatchatsomehighSNR

point.Ingeneral,thedetectionproblemdependshighlyontheSNRandN.Itisatradeoffbetween

thesetwofactors.Forexample,iftheSNRislow,thenweneedtoincreaseNinordertodecrease

the errorprobability and theMSEestimate.On theotherhandhowever, if the SNR is high, then

smallNwillwork forus. This techniquewas thencomparedwith theSISO[3], [4], [8], [10]and

SIMO[5],[9]models.SIMOsystemswouldhavesomedrawbacksespeciallywhenbeingappliedfor

mobile phones like, size and cost of the phone, and power consumption. Instead of having the

diversitydoneateachremoteunit,instead,itisbeingdoneatthebasestation.Forexample,having

twoantennasatthebasestationcoveringhundredsofremoteunits(havingonereceiveantenna)

wouldbemoreeconomicalthanhavinghundredsofremoteunits(eachhavingtwoantennas)and

one transmit antenna at the base station. This is the main reason of choosing MISO with STBC

systemsoverSIMOsystems.

7. Futurework

1. SNRestimationusingtheEMalgorithmwithotherlinearlymodulatedsignalsotherthan

BPSKlikeM‐QAMandM‐PSK.

2. BayesianSNRestimation.

3. MIMOSNRestimationusingtheEMalgorithm.

4. SNRestimationinnon‐Gaussiannoise.

  

  71    

8.References:

[1]C.M.Bishop,Patternrecognitionandmachinelearning.Springer2006.

[2]S.M.Kay,Fundamentalsofstatisticalsignalprocessing.PrenticeHall,1993.

[3] A. Wiesel, J. Goldberg, and H. Messer, “Non‐data‐aided signal‐to‐noise‐ratio estimation,”

Communications,2002.ICC2002.IEEEInternationalConferenceon,vol.1,pp.197‐201,2002.

[4] A. Das, “NDA SNR estimation: CRLBs and EM based estimators,” in proc. TENCON 2008,

Hyderabad,India,Nov.2008,pp.1‐6.

[5]M. A. Boujelben, F. Bellili, S. Affes, and A. Stephenne, “EM algorithm for non‐data‐aided SNR

estimation of linearly‐modulated signals over SIMO channels,” in proc. Communications IEEE

GLOBCOM2009,pp.1‐6.

[6]A.P.Dempester,N.M.Laird,andD.B.Rubin,“Maximumlikelihoodfromincompletedataviathe

EMalgorithm,”JournaloftheRoyalStatisticalSociety.SeriesB,no.1,pp.1‐38,1977.

[7]T.K.Moon,“Theexpectation‐maximizationalgorithm,”IEEESignalprocess.Mag.,vol.13,pp.47‐

60,Nov.1996.

[8]N.S.Alagha,“Cramer‐raoboundsofSNRestimatesforBPSKandQPSKmodulatedsignals,”IEEE

Communicationsletters,vol.5,no.1,pp.10‐12,January2001.

[9]M.A.Boujelben,F.Bellili,S.Affes,andA.Stephenne,“SNRestimationoverSIMOchannelsfrom

linearlymodulatedsignals,”inproc.SignalprocessingIEEEBritishCrown,vol.58,n0.12,December

2010,pp.6017‐6028.

[10]A.DasandM.Miller,“Remotenon‐linearitydetectionviaburstpowerditheringandEMbased

SNR estimation,” International workshop on Satellite and space communications. IWSSC, sept.

2007.

[11] F. Rice, B. Cowley, and M. Rice, “Cramer‐rao lower bounds for QAM phase and frequency

estimation,”IEEETrans.Communications,vol.49,pp.1582‐1591,sept.2001.

[12] S. M. Alamouti, “A simple transmit diversity technique for wireless communications,” IEEE

Trans.Communications,vol.16,pp.1451‐1458,sept.1998.

  

  72    

[13 ]N. Celandroni, E. Ferro, andF. Potorti, “Quality estimation of PSKmodulated signals,” IEEE

Comm.Mag.,pp.50‐55.July1997.

[14] K. Balachandran, S. R. Kadaba, and S. Nanda, “Channel Quality estimation and rate

adaptationforcellularmobileradio,”IEEEJ.SelectAreasComm.,vol.17,pp.1244‐1256,July1999.

[15]F.Bellili,S.Affes,andA.Stephenne,“Cramer‐RaoBoundforNDASNRestimatesofSquareQAM

ModulatedSignals,”IEEEinWCNC2009,pp.1‐5.

[16] F. Bellili, S. Affes, andA. Stephenne, “Cramer‐RaoBounds forNDA SNR estimates of Square

QAMModulatedTransmissions,”IEEEinComm.2010,pp.3211‐3218.

[17]S.Caiyao,Y.Hongyi,andZ.Tianzhong,“Cramer=RaoboundsofSNREstimationforBPSKand

QPSKsignalsoverSIMOChannels,”IEEEICSPS2010,pp.647‐650.

[18]W.Gappmair,R. Lopez‐Valcare, andC.Mosquera, “MLandEMalgorithm fornon‐data‐aided

SNRestimationoflinearlymodulatedsignals,”inProc.IEEEsymp.Commun.Syst.,Graz,Austria,Jul.

2008,pp.530‐534.

[19]R.Lopez‐Valcare,C.Mosquera,andW.Gappmair,“IterativeenvelopbasedSNRestimationfor

nonconstantmodulusconstellation,”inProc.IEEESPAWC’07,Helsinki,Finland,Jun.2007,pp.1‐5.

[20]W.Gappmair,R.Lopez‐Valcare,andC.Mosquera,“Cramer‐RaolowerboundandEMalgorithm

forenvelope‐basedSNRestimationofnonconstantmodulusconstellation ,” IEEETrans.Commun.

Vol.57,no.6,June2009,pp.1622‐1627.