1

STATISTICAL INFERENCEPART II

SOME PROPERTIES OF ESTIMATORS

SOME PROPERTIES OF ESTIMATORS

• θ: a parameter of interest; unknown

• Previously, we found good(?) estimator(s) for θ or its function g(θ).

• Goal:

• Check how good are these estimator(s). Or are they good at all?

• If more than one good estimator is available, which one is better?

2

3

SOME PROPERTIES OF ESTIMATORS

• UNBIASED ESTIMATOR (UE): An estimator is an UE of the unknown parameter , if

ˆE for all

Otherwise, it is a Biased Estimator of .

ˆ ˆBias E

Bias of for estimating

If is UE of , ˆ 0.Bias

4

SOME PROPERTIES OF ESTIMATORS

• ASYMPTOTICALLY UNBIASED ESTIMATOR (AUE): An estimator is an AUE of the unknown parameter , if

ˆ ˆ0 lim 0n

Bias but Bias

5

SOME PROPERTIES OF ESTIMATORS

• CONSISTENT ESTIMATOR (CE): An estimator which converges in probability to an unknown parameter for all is called a CE of .

ˆ .p

• MLEs are generally CEs.

For large n, a CE tends to be closer to the unknown population parameter.

6

EXAMPLE

For a r.s. of size n,

E X X is an UE of . By WLLN,

pX

X is a CE of .

7

MEAN SQUARED ERROR (MSE)

• The Mean Square Error (MSE) of an estimator for estimating is

22ˆ ˆ ˆ ˆMSE E Var Bias

If is smaller, is a better estimator of .

ˆMSE

1 2ˆ ˆ ,For two estimators, and of if

1 2ˆ ˆ ,MSE MSE

1 is better estimator of .

8

MEAN SQUARED ERROR CONSISTENCY

• Tn is called mean squared error consistent (or consistent in quadratic mean) if E{Tn}20 as n.

Theorem: Tn is consistent in MSE iff

i) Var(Tn)0 as n.

) lim .nnii E T

• If E{Tn}20 as n, Tn is also a CE of .

9

EXAMPLES

X~Exp(), >0. For a r.s of size n, consider the following estimators of , and discuss their bias and consistency.

Which estimator is better?

1n

X

T,n

X

T

n

1ii

2

n

1ii

1

10

SUFFICIENT STATISTICS

• X, f(x;),

• X1, X2,…,Xn

• Y=U(X1, X2,…,Xn ) is a statistic.

• A sufficient statistic, Y, is a statistic which contains all the information for the estimation of .

11

SUFFICIENT STATISTICS

• Given the value of Y, the sample contains no further information for the estimation of .

• Y is a sufficient statistic (ss) for if the conditional distribution h(x1,x2,…,xn|y) does not depend on for every given Y=y.

• A ss for is not unique:

• If Y is a ss for , then any 1-1 transformation of Y, say Y1=fn(Y) is also a ss for .

12

SUFFICIENT STATISTICS• The conditional distribution of sample rvs

given the value of y of Y, is defined as

1 21 2

; , , ,, , ,

;n

n

L x x xh x x x y

g y

1 21 2

, , , , ;, , ,

;n

n

f x x x yh x x x y

g y

• If Y is a ss for , then

1 21 2 1 2

; , , ,, , , , , ,

;n

n n

L x x xh x x x y H x x x

g y

ss for may include y or constant.

Not depend on for every given y.

• Also, the conditional range of Xi given y not depend on .

13

SUFFICIENT STATISTICS

EXAMPLE: X~Ber(p). For a r.s. of size n, show that is a ss for p.

n

1iiX

14

SUFFICIENT STATISTICS

• Neyman’s Factorization Theorem: Y is a ss for iff

1 2 1 2; , , , nL k y k x x x

where k1 and k2 are non-negative functions.

The likelihood function Does not depend on xi

except through y

Not depend on (also in the range of xi.)

15

EXAMPLES

1. X~Ber(p). For a r.s. of size n, find a ss for p if exists.

16

EXAMPLES

2. X~Beta(θ,2). For a r.s. of size n, find a ss for θ.

17

SUFFICIENT STATISTICS

• A ss may not exist.

• Jointly ss Y1,Y2,…,Yk may be needed. Example: Example 10.2.5 in Bain and Engelhardt (page 342 in 2nd edition), X(1) and X(n)

are jointly ss for

• If the MLE of exists and unique and if a ss for exists, then MLE is a function of a ss for .

18

EXAMPLE

X~N(,2). For a r.s. of size n, find jss for and 2.

MINIMAL SUFFICIENT STATISTICS

• If is a ss for θ, then,

is also a ss

for θ. But, the first one does a better job in data reduction. A minimal ss achieves the greatest possible reduction.

19

))x(s),...,x(s()x(S~

k~

1~

))x(s),...,x(s),x(s()x(S~

k~

1~

0~

*

20

MINIMAL SUFFICIENT STATISTICS• A ss T(X) is called minimal ss if, for any

other ss T’(X), T(x) is a function of T’(x).• THEOREM: Let f(x;) be the pmf or pdf of

a sample X1, X2,…,Xn. Suppose there exist a function T(x) such that, for two sample points x1,x2,…,xn and y1,y2,…,yn, the ratio

is constant with respect to iff T(x)=T(y). Then, T(X) is a minimal sufficient statistic for .

1 2

1 2

, , , ;

, , , ;n

n

f x x x

f y y y

21

EXAMPLE

• X~N(,2) where 2 is known. For a r.s. of size n, find minimal ss for .

Note: A minimal ss is also not unique. Any 1-to-1 function is also a minimal ss.

22

RAO-BLACKWELL THEOREM

• Let X1, X2,…,Xn have joint pdf or pmf f(x1,x2,…,xn;) and let S=(S1,S2,…,Sk) be a vector of jss for . If T is an UE of () and (T)=E(TS), then

i (T) is an UE of () .ii (T) is a fn of S, so it is also jss for .iii) Var((T) ) Var(T) for all . (T) is a uniformly better unbiased estimator

of () .

RAO-BLACKWELL THEOREM

• Notes:(T)=E(TS) is at least as good as T.

• For finding the best UE, it is enough to consider UEs that are functions of a ss, because all such estimators are at least as good as the rest of the UEs.

23

Example

• Hogg & Craig, Exercise 10.10

• X1,X2~Exp(θ)

• Find joint p.d.f. of ss Y1=X1+X2 for θ and Y2=X2.

• Show that Y2 is UE of θ with variance θ².

• Find φ(y1)=E(Y2|Y1) and variance of φ(Y1).

24

25

ANCILLARY STATISTIC

• A statistic S(X) whose distribution does not depend on the parameter is called an ancillary statistic.

• Unlike a ss, an ancillary statistic contains no information about .

Example

• Example 6.1.8 in Casella & Berger, page 257:

Let Xi~Unif(θ,θ+1) for i=1,2,…,n

Then, range R=X(n)-X(1) is an ancillary statistic because its pdf does not depend on θ.

26

27

COMPLETENESS• Let {f(x; ), } be a family of pdfs (or pmfs)

and U(x) be an arbitrary function of x not depending on . If

requires that the function itself equal to 0 for all possible values of x; then we say that this family is a complete family of pdfs (or pmfs).

0 for all E U X

0 for all 0 for all .E U X U x x

i.e., the only unbiased estimator of 0 is 0 itself.

28

EXAMPLES

1. Show that the family {Bin(n=2,); 0<<1} is complete.

29

EXAMPLES

2. X~Uniform(,). Show that the family {f(x;), >0} is not complete.

30

COMPLETE AND SUFFICIENT STATISTICS (css)

• Y is a complete and sufficient statistic (css) for if Y is a ss for and the family

; ;g y is complete. The pdf of Y.

1) Y is a ss for .

2) u(Y) is an arbitrary function of Y. E(u(Y))=0 for all implies that u(y)=0 for all possible Y=y.

31

BASU THEOREM• If T(X) is a complete and minimal sufficient

statistic, then T(X) is independent of every ancillary statistic.

• Example: X~N(,2).

: the mss for X

(n-1)S2/ 2 ~2

1n Ancillary statistic for

By Basu theorem, and S2 are independent.X

S2

statisticcompleteaisX

.familycompleteis)n/,(Noffamilyand)n/,(N~X 22

BASU THEOREM

• Example:

• Let T=X1+ X2 and U=X1 - X2

• We know that T is a complete minimal ss.

• U~N(0, 2) distribution free of T and U are independent by Basu’s Theorem

32

X1, X2~N(,2), independent, 2 known.

THE MINIMUM VARIANCE UNBIASED ESTIMATOR

• Rao-Blackwell Theorem: If T is an unbiased estimator of , and S is a ss for , then (T)=E(TS) is– an UE of , i.e.,E[(T)]=E[E(TS)]= and

– the MVUE of .

33

34

LEHMANN-SCHEFFE THEOREM

• Let Y be a css for . If there is a function Y which is an UE of , then the function is the unique Minimum Variance Unbiased Estimator (UMVUE) of .

• Y css for .

• T(y)=fn(y) and E[T(Y)]=.

T(Y) is the UMVUE of . So, it is the best estimator of .

35

THE MINIMUM VARIANCE UNBIASED ESTIMATOR

• Let Y be a css for . Since Y is complete, there could be only a unique function of Y which is an UE of .

• Let U1(Y) and U2(Y) be two function of Y. Since they are UE’s, E(U1(Y)U2(Y))=0 imply W(Y)=U1(Y)U2(Y)=0 for all possible values of Y. Therefore, U1(Y)=U2(Y) for all Y.

Example

• Let X1,X2,…,Xn ~Poi(μ). Find UMVUE of μ.

• Solution steps:– Show that is css for μ.

– Find a statistics (such as S*) that is UE of μ and a function of S.

– Then, S* is UMVUE of μ by Lehmann-Scheffe Thm.

36

n

1iiXS

Note

• The estimator found by Rao-Blackwell Thm may not be unique. But, the estimator found by Lehmann-Scheffe Thm is unique.

37