statistical inference and regression analysis: stat-gb.3302.30, stat-ub.0015.01

Statistics

Professor William GreeneStern School of BusinessIOMS Department Department of EconomicsStatistical Inference and Regression Analysis: Stat-GB.3302.30, Stat-UB.0015.01

#/124Part 2 Expectations of Random Variables1Part 2 AExpectations of Random Variables2-A Expectations of Random Variables2-B Covariance and Correlation2-C Limit Results for Sums

#/124Part 2 Expectations of Random Variables2Expected Value of a Random VariableWeighted average of the values taken by the variable

#/124Part 2 Expectations of Random VariablesDiscrete UniformX = 1,2,,JProb(X = x) = 1/JE[X] = 1/J + 2/J + + J/J = J(J+1)/2 * 1/J = (J+1)/2Expected toss of a die = 3.5 (J=6)Expected sum of two dice = 7. Proof?

#/124Part 2 Expectations of Random VariablesPoisson ()

#/124Part 2 Expectations of Random VariablesPoisson (5)

#/124Part 2 Expectations of Random VariablesThe St. Petersburg ParadoxCoin toss game. If first heads comes up on nth toss, you win $2nEntry fee to play a game is $CExpected value of the game = E[Win] -C + ()21 + ()222 + + ()k2k Game has infinite value. Noone would pay very much to play. Why not?

#/124Part 2 Expectations of Random VariablesContinuous Random Variable

#/124Part 2 Expectations of Random VariablesGamma Random Variable

#/124Part 2 Expectations of Random VariablesGamma Function: (1/2)=

#/124Part 2 Expectations of Random VariablesExpected Value of a Linear TranslationZ = aX+bE[Z] = aE[X] + bProof is trivial using the definition of the expected value and the fact that the density integrates to 1 to have E[b]=b.

#/124Part 2 Expectations of Random VariablesNormal(,) VariableFrom the definition of the random variable, is the mean.

Proof in Rice (119) uses the linear translation.

If X ~ N[0,1], X + ~ N(,)

#/124Part 2 Expectations of Random VariablesCauchy Random Variablesf(x)=(1/) 1/(1+x2)Mean does not exist. No higher moments exist.If X~N[0,1] and Y ~ N[0,1] then X/Y has the Cauchy distribution.Many applications obtain estimates of interesting quantities as ratios of estimators that are normally distributed.

#/124Part 2 Expectations of Random VariablesCauchy Random Sample

#/124Part 2 Expectations of Random VariablesExpected Value of a Function of XY=g(X)One to one caseE[Y] = expected value of Y(X) find the distribution of the new variableE[g(X)] = x g(x)f(x) will equal E[Y]Many to one case similar argument. Proceed without the transformation of the random variable.E[g(X)] is generally not equal to g(E[X]) if g(X) is not linear

#/124Part 2 Expectations of Random VariablesLinear TranslationZ = aX+bE[Z] = E[aX+b]E[Z] = aE[X] + bProof is trivial using the definition of the expected value and the fact that the density integrates to 1 to E[b]=b.

#/124Part 2 Expectations of Random VariablesPowers of x - MomentsMoment = E[Xk] for positive integer xRaw moment: E[Xk]Central moment: E[(X E[X])k]Standard notationE[Xk] = kE[(X E[X])k] = kMean = 1 =

#/124Part 2 Expectations of Random VariablesVariance as a g(X)Variance = E[(X E[X])2]Standard deviation = square root of variance is usually more interesting

#/124Part 2 Expectations of Random VariablesVariance of a Translation: Y = a + bXVar[a] = 0

Var[bX] = b2Var[X]

Standard deviation of Y = |b|S.D.(X)

#/124Part 2 Expectations of Random VariablesShortcutVar[X] = E[X2] - {E[X]}2

#/124Part 2 Expectations of Random VariablesBernoulliProb(X=1)=; Prob(X=0)=1- E[X] = 0(1- ) + 1 = E[X2] = 02(1- ) + 12 = Var[X] = - 2 = (1- )

#/124Part 2 Expectations of Random VariablesPoisson: Factorial Moment

#/124Part 2 Expectations of Random VariablesNormal Moments

#/124Part 2 Expectations of Random VariablesGamma Random Variable

#/124Part 2 Expectations of Random VariablesChi Squared [1]Chi squared [1] = Gamma(, ) P = , =

Mean = P/ = ()/() = 1

Variance = P/ 2 = ()/[()2] = 2

#/124Part 2 Expectations of Random VariablesHigher MomentsSkewness: 3.0 for all symmetric distributions (not just the normal)Standardized measure 3/3Kurtosis: 4. Standardized 4/4.Compare to normal, 3Degree of excess = 4/4 3.

#/124Part 2 Expectations of Random VariablesSymmetric and Skewed Distributions

#/124Part 2 Expectations of Random Variables Kurtosis: t[5] vs. Normal

Kurtosis of normal(0,1) = 3, Excess = 0Excess Kurtosis of t[k] = 6/(k-4); for t[5] = 6/(5-4) = 6.

#/124Part 2 Expectations of Random VariablesApproximations for g(X)g(X) = continuous function g() existsContinuous first derivative not equal to zero at Taylor series approximation around mu g(X) = g() + g()(X- ) + g()(X- )2 (+ higher order terms)

#/124Part 2 Expectations of Random VariablesApproximation to the Meang(X) ~ g() + g()(X- ) + g()(X - )2E[g(X)] ~ E[approximation] = g() + 0 + g() E[(X - )2] = g() + g() 2

#/124Part 2 Expectations of Random VariablesExample: N[, ]. g(X)=exp(X). True mean = exp( + 2/2). Approximation: = exp() + exp() 2

Example: =0, s = 1, True mean = exp(.5) = 1.6487Approximation = exp(0) + .5*exp(0)*1 = 1.5000

#/124Part 2 Expectations of Random VariablesDelta method: Var[g(X)]Use linear approximationg(X) ~ g() + g()(X - ) Var[g(X)] ~ Var[approximation] = [g()]22Example: Var[X2] ~ (2)22

#/124Part 2 Expectations of Random VariablesDelta Method x ~ N[, 2]y = g(x) = exp(x) ~ lognormalExactE[y] = exp( + 2)Var[y] = exp(2 + 2)[exp(2) 1]ApproximateE*[y] = exp() + exp() 2V*[y] = [exp()]2 2 N[0,1], exact mean and variance are exp(.5) =1.648 and exp(1)(exp(1)-1) = 4.671. Approximations are 1.5 and 1 (!)

#/124Part 2 Expectations of Random VariablesMoment Generating FunctionLet g(X) = exp(tX)M(t) = E[exp(tX)] = the moment generating function for random variable X.

#/124Part 2 Expectations of Random VariablesMGF BernoulliP(x) = (1-) for x=0 and for x=1

E[exp(tX)] = (1- )exp(0t) + exp(1t) = (1 - ) + exp(t).

#/124Part 2 Expectations of Random VariablesMGF Poisson

#/124Part 2 Expectations of Random VariablesMGF Gamma

#/124Part 2 Expectations of Random VariablesMGF NormalMX(t) for X ~ N[0,1] is exp( t2)

MY(t) for Y = X + isexp(t)MX(t) = exp[t + 2t2]

This is the moment generating function for N[,2]

#/124Part 2 Expectations of Random VariablesGenerating the Momentsrth derivative of M(t) evaluated at t = 0 gives the rth raw moment, r M(r)(t) = drM(t)/dtr |t=0 = equals rth raw moment.

#/124Part 2 Expectations of Random VariablesPoisson MGFM(t) = exp((exp(t) 1)); M(0)=1M(t) = M(t) * exp(t); M(0)= = M(0)=1 1 = 2 = E[X2] = M(0) = M(0) exp(0) + exp(0)M(0) = 2 + Variance = 2 - 2 =

#/124Part 2 Expectations of Random VariablesUseful PropertiesMGF of X = MX(t) and y = a+bX then

MY(t) for y is exp(at)MX(bt)

For independent X and Y, MX+Y (t) = is MX(t)MY(t)

The sequence of moments does not uniquely define the distribution

#/124Part 2 Expectations of Random VariablesSide ResultsMGF MX(t) = E[exp(tx)] does not always exist.Characteristic function E[exp(itx)] always exists. Used to prove central limit theoremsCumulant generating function logMX(t)is sometimes useful. Cumulants are functions of moments. First cumulant is the mean, second is the variance.

#/124Part 2 Expectations of Random VariablesPart 2 BCovariance and Correlation

#/124Part 2 Expectations of Random Variables43CovarianceRandom variables X,Y with joint discrete distribution p(X,Y) or continuous density f(x,y).Covariance = E({X E[X]}{Y-E[Y]}) = E[XY] E[X] E[Y].(Note, Covariance of X,X = Var[X].Connection to joint distribution and covariation

#/124Part 2 Expectations of Random VariablesCorrelation and Covariance

#/124Part 2 Expectations of Random VariablesCorrelated Populations

#/124Part 2 Expectations of Random VariablesCorrelated VariablesX1 and X2 are independent with means 0 and standard deviations 1.Y = aX1 + bX2. Choose a and b such thatX1 and Y have means 0, standard deviation 1 and correlation rho.Var[Y] = a2 + b2 = 1Cov[X1,Y] = a = . b = sqr(1 2)

#/124Part 2 Expectations of Random VariablesConditional Distributionsf(y|x) = f(y,x) / f(x)Conditional distribution of y given a realization of xConditional mean = mean of the conditional random variable = regression function Conditional variance = variance of conditional random variable = scedastic function

#/124Part 2 Expectations of Random VariablesLitigation Risk AnalysisForm probability tree for decisions and outcomesDetermine conditional expected payoffs (gains or losses)Choose strategy to optimize expected value of payoff function (minimize loss or maximize (net) gain.

#/124Part 2 Expectations of Random Variables49Litigation Risk Analysis: Using Probabilities to Determine a StrategyTwo paths to a favorable outcome. Probability =(upper) .7(.6)(.4) + (lower) .5(.3)(.6) = .168 + .09 = .258.How can I use this to decide whether to litigate or not?Suppose the cost to litigate = $1,000,000 and a favorable outcome pays $3,000,000. What should you do?

P(Upper path) = P(Causation|Liability,Document)P(Liability|Document)P(Document) = P(Causation,Liability|Document)P(Document) = P(Causation,Liability,Document) = .7(.6)(.4)=.168. (Similarly for lower path, probability = .5(.3)(.6) = .09.)

#/124Part 2 Expectations of Random Variables50Joint Normal Random Variables

#/124Part 2 Expectations of Random VariablesConditional Normal

#/124Part 2 Expectations of Random Variables

Y and Y|XX X Y

#/124Part 2 Expectations of Random Variables Application: Conditional Expected Profits and RiskYou must decide how many copies of your self published novel to print . Based on market research, you believe the following distribution describes X, your likely sales (demand). x P(X=x) 25 .10 (Note: Sales are in thousands. Convert your final result to 40 .30 dollars after all computations are done by multiplying your 55 .45 final results by $1,000.) 70 .15Printing costs are $1.25 per book. (Its a small book.) The selling price will be $3.25. Any unsold books that you print must be discarded (at a loss of $1.25/copy). You must decide how many copies of the book to print, 25, 40, 55 or 70. (You are committed to one of these four 0 is not an option.)A. What is the expected number of copies demanded.B. What is the standard deviation of the number of copies demanded.C. Which of the four print runs shown maximizes your expected profit? Compute all four.D. Which of the four print runs is least risky i.e., minimizes the standard deviation of the profit (given the number printed). Compute all four.E. Based on C. and D., which of the four print runs seems best for you?




#/124Part 2 Expectations of Random VariablesExpected Profit Given Print Run



Run=25,000Run=70,000Run=40,000Run=55,000


Run=25,000Run=70,000Run=40,000Run=55,000


Run=25,000Run=70,000Run=40,000Run=55,000



#/124Part 2 Expectations of Random VariablesUseful Theorems - 1E[Y] = EX[EY[Y|X]]Expectation over X of EY[Y|X]Law of Iterated Expectations

#/124Part 2 Expectations of Random VariablesExample: Hierarchical Model

#/124Part 2 Expectations of Random VariablesUseful Theorems - 2Decomposition of varianceVar[Y] = Var[E[Y|X]] + E[Var[Y|X]]

#/124Part 2 Expectations of Random VariablesBivariate Normal

#/124Part 2 Expectations of Random VariablesUseful Theorems - 3Cov(X,Y)=Cov(X,E[Y|X])

In the hierarchical model, E[y|x]=x soCov(X,Y)=Cov(X, X)= Var[X]= /2

#/124Part 2 Expectations of Random VariablesMean Squared Error

#/124Part 2 Expectations of Random VariablesMinimum MSE Predictor

#/124Part 2 Expectations of Random VariablesVariance of the Sum of X and YVar[X+Y] = EYEX[ {(X+Y) - (X + Y) }2 ] = EYEX[ {(X- X) + (Y- Y)}2] = EYEX[ (X - X)2] + EYEX[(Y- Y )2] + 2 EYEX[(X- X)(Y- Y)] = EX[ (X - X)2] + EY[(Y- Y )2] + 2 EYEX[(X- X)(Y- Y)] = Var[X] + Var[Y] + 2Cov(X,Y)

#/124Part 2 Expectations of Random VariablesVar[aX+bY] = Var[aX] + Var[bY] +2Cov(aX,bY) = a2Var[X] + b2Var[Y] + 2ab Cov(X,Y).Also, Cov(X,Y) is the numerator in xy, so Cov(X,Y) = xy x y.

Variance of Weighted Sum

#/124Part 2 Expectations of Random Variables73Application - PortfolioYou have $1000 to allocate between assets A and B. The yearly returns on the two assets are random variables rA and rB.The means of the two returns are E[rA] = A and E[rB] = BThe standard deviations (risks) of the returns are A and B.The correlation of the two returns is ABYou will allocate a proportion w of your $1000 to A and (1-w) to B.

#/124Part 2 Expectations of Random Variables74Risk and ReturnYour expected return on each dollar is E[wrA + (1-w)rB] = wA + (1-w)BThe variance your return on each dollar is Var[wrA + (1-w)rB] = w2 A2 + (1-w)2B2 + 2w(1-w)ABAB The standard deviation is the square root.

#/124Part 2 Expectations of Random Variables75Risk and Return: ExampleSuppose you know A, B, AB, A, and B (You have watched these stocks for many years.)The mean and standard deviation are then just functions of w.I will then compute the mean and standard deviation for different values of w.For example, A = .04, B, = .07 A = .02, B,=.06, AB = -.4 E[return] = w(.04) + (1-w)(.07) = .07 - .03w SD[return] = sqr[w2(.022)+ (1-w)2(.062) + 2w(1-w)(-.4)(.02)(.06)] = sqr[.0004w2 + .0036(1-w)2 - .00096w(1-w)]

#/124Part 2 Expectations of Random Variables76

#/124Part 2 Expectations of Random VariablesMean and Variance of a Sum

#/124Part 2 Expectations of Random VariablesExtension: Weighted Sum

#/124Part 2 Expectations of Random VariablesMore General Portfolio Problem

#/124Part 2 Expectations of Random VariablesOptimal Portfolio?

#/124Part 2 Expectations of Random VariablesSums of Independent VariablesSuppose P is sales of a store. The accounting period starts with total sales = 0On any given day, sales are random, normally distributed with mean and standard deviation . For example, mean $100,000 with standard deviation $10,000Sales on any given day, day t, are denoted t 1 = sales on day 1,2 = sales on day 2,Total sales after T days will be 1+ 2++ T Therefore, each t is the change in the total that occurs on day t.

#/124Part 2 Expectations of Random Variables82Behavior of the TotalLet PT = 1+ 2++ T be the total of the changes (variables) from times (observations) 1 to T.The sequence isP1 = 1P2 = 1 + 2P3 = 1 + 2 + 3And so onPT = 1 + 2 + 3 + + T

#/124Part 2 Expectations of Random Variables83SummingIf the individual s are each normally distributed with mean and standard deviation , thenP1 = 1 = Normal [ , ]P2 = 1 + 2 = Normal [2, 2]P3 = 1 + 2 + 3= Normal [3, 3]And so on so thatPT ~ N[T, T]

#/124Part 2 Expectations of Random Variables84This Defines a Random WalkThe sequence isP1 = 1P2 = 1 + 2P3 = 1 + 2 + 3And so onPT = 1 + 2 + 3 + + TIt follows thatP1 = 1P2 = P1 + 2P3 = P2 + 3And so onPT = PT-1 + T

#/124Part 2 Expectations of Random Variables85A Model for Stock PricesPreliminary: Consider a sequence of T random outcomes, independent from one to the next, 1, 2,, T. ( is a standard symbol for change which will be appropriate for what we are doing here. And, well use t instead of i to signify something to do with time.)t comes from a normal distribution with mean and standard deviation .

#/124Part 2 Expectations of Random Variables86A Model for Stock PricesRandom Walk Model: Todays price = yesterdays price + a change that is independent of all previous information. Start at some known P0 so P1 = P0 + 1 and so on.Assume = 0 (no systematic drift in the stock price).

#/124Part 2 Expectations of Random Variables87Random Walk Simulations Pt = Pt-1 + t, t = 1,2,,100 Example: P0= 10, t Normal with =0, =0.02

#/124Part 2 Expectations of Random Variables88Random Walk?Dow Jones March 27 to May 26, 2011.

#/124Part 2 Expectations of Random VariablesUncertaintyExpected Price = E[Pt] = P0+TWe have used = 0 (no systematic upward or downward drift).Standard deviation = T reflects uncertainty or risk.Looking forward from now = time t=0, the uncertainty increases the farther out we look to the future.

#/124Part 2 Expectations of Random Variables90Expected Range

#/124Part 2 Expectations of Random Variables91Part 2-C Sums of Random Variables

#/124Part 2 Expectations of Random Variables92Sequences of Independent Random Variablesx1,x2,,xn = a set of n random variables Same (marginal) probability distribution, f(x)Finite identical mean and variance 2Statistically independentIID = independent identically distributedThis is a random sample from the population f(x).

#/124Part 2 Expectations of Random VariablesThe Sample Mean

#/124Part 2 Expectations of Random VariablesConvergence of a Random Variable to a Constant

#/124Part 2 Expectations of Random VariablesConvergence in Mean Square

If E[Xn ] = and Var[Xn ] 0 as n Then Xn converges in mean square to Slightly Broader ExtensionIf E[Xn ] as n Var[Xn ] 0 as n Then Xn converges in mean square to .

Xn + (1/n) converges in mean square to .


Convergence in Mean Square: The top figure is a histogram for 1,000 means of samples of 4; the center is for samples of 9, the lowest one is for samples of 15. The vertical bars go through 7, 10 and 13 on all three figures.

#/124Part 2 Expectations of Random Variables97Convergence of Means98

#/124Part 2 Expectations of Random VariablesProbability Limits: Plim xn

#/124Part 2 Expectations of Random Variables99Probability Limits and ExpectationsWhat is the difference between E[xn] and plim xn?Consider: X = n with prob(X=n)=1/n X = 1 with prob(X=1)=1 1/n E[X]=2 1/n 2 Plim(X) = 1

#/124Part 2 Expectations of Random Variables100The Slutsky TheoremAssumptions: If xn is a random variable such that plim xn = . For now, we assume is a constant.g(.) is a continuous function with continuous derivatives. g(.) is not a function of n.Conclusion: Then plim[g(xn)] = g[plim(xn)] assuming g[plim(xn)] exists. Works for probability limits. Does not work for expectations.

#/124Part 2 Expectations of Random Variables101Multivariate Slutsky TheoremPlim xn = a, Plim yn = bg(xn,yn) is continuous, has continuous first derivatives and exists at (a,b).Plim g(xn,yn) = g(a,b)Generalizes to K functions of M random variables

#/124Part 2 Expectations of Random VariablesMonte Carlo Integration

#/124Part 2 Expectations of Random VariablesMonte Carlo Integration

#/124Part 2 Expectations of Random VariablesApplicationNormal probability from -1 to +1.5 is0.3413 + 0.4332 = .7745.[a = -1, b = +1.5, g(z)=(z).]Compute 10,000 random draws on x from U[0,1]. Compute z = a + (b-a)x = -1+2.5*xAverage the 10,000 draws on (z) then multiply by the average by (b-a) = (1.5 (-1)) = 2.5. Gives .773641 in my experiment.

#/124Part 2 Expectations of Random VariablesApplicationFor Normal(2,1.52), E[exp(x)] = exp(2 + 1.52) = 22.76Draw 10,000 random U(0,1) draws. Transform to x ~ N(0,1) then z = 2 + 1.5*xCompute q=exp(z) and average 10,000 draws on q. My result was 22.87944.

#/124Part 2 Expectations of Random VariablesLimit ResultsMean converges in probability to . Variance goes to zero.If n is finite, what can be said about its behavior?Objective: characterize the distribution of the mean when n is large but finiteStrategy: find a limit result then use it to approximate for finite n.

#/124Part 2 Expectations of Random VariablesA Finite Sample Distribution

Means of 1000 samples of 8 observations from U[0,1].

#/124Part 2 Expectations of Random VariablesCentral Limit Theorems

#/124Part 2 Expectations of Random VariablesLimiting DistributionsXn has probability density fXn(Xn) and cdf FXn(Xn).If FXn(Xn) F(X) as n , then F(X) is the limiting distribution. (At points where F(X) is continuous in X.)

#/124Part 2 Expectations of Random VariablesLindeberg Levy Central Limit Theorem

#/124Part 2 Expectations of Random VariablesOther Central Limit TheoremsLindeberg Levy for i.i.d.Lindeberg Feller heteroscedastic. Variances may differLyupanov: distributions may differExtensions - time series with some covariance across observations.

#/124Part 2 Expectations of Random VariablesRough Approximations

#/124Part 2 Expectations of Random VariablesNormal Approximation to BinomialBinomial (n,p) equals the sum of n Bernoullis with parameter p.Each Bernoulli X has = p and 2 = p(1-p).Sum of n variables is approximately normal with mean np and variance np(1-p).

#/124Part 2 Expectations of Random VariablesApproximation to binomial with n = 48, p=.25

#/124Part 2 Expectations of Random Variables115Demoivres Normal Approximation

The binomial density function has n=48, =.25, so = 12 and = 3. The normal density plotted has mean 12 and standard deviation 3.

#/124Part 2 Expectations of Random Variables116Using deMoivres Approximation

The binomial has n=48, =.25, so = 12 and = 3. The normal distribution plotted has mean 12 and standard deviation 3. 8 0.057905 9 0.08578510 0.11152011 0.12841712 0.13198413 0.12183214 0.10152615 0.076709Total 0.815678

P[8 < x < 15]=P[(8-12)/3

statistical inference and regression analysis: stat-gb.3302.30, stat-ub.0015.01

Documents