bounded normal mean minimax estimation

SECTION 1SECTION 2

Estimating a Bounded Normal Mean

Jacopo Primavera

TSI-EuroBayes StudentUniversity Paris Dauphine

21 November 2011 / Reading Seminar on Classics

Jacopo Primavera Estimating a Bounded Normal Mean

SECTION 1SECTION 2

"ALL MODELS ARE WRONG, BUT SOME AREUSEFUL"

G. E. P. Box

SECTION 1SECTION 2

PRESENTING THE PROBLEM

One observation x ∼ N(θ,1)

θ ∈ [−m,m] ⊂ RSquared loss (θ − δ(x))2

R(θ, δ) = MSE(δ) = BIAS(δ) + VAR(δ)

MINIMAX ESTIMATOR δMM = argminδ

[supθ

R(θ, δ)]

SECTION 1SECTION 2

SUMMARY

SECTION 1SECTION 2

SUMMARY

SECTION 1SECTION 2

Outline

1 SECTION 1THE CANDIDATES2-POINTS PRIOR

2 SECTION 2AND WHEN m GETS LARGE?CLOSING THE LOOP

SECTION 1SECTION 2

THE CANDIDATES2-POINTS PRIOR

Outline

SECTION 1SECTION 2

SAMPLE MEAN

δSM = x

Main characteristicDOES NOT INVOLVEPRIOR INFORMATION

PropertiesΘ BOUNDED⇒ NOT Minimax

SECTION 1SECTION 2

δMLE (x)−m for x ≤ −mx for x ∈ (−m,m)

m for x ≥ m

Main characteristicA SELECTOR ESTIMATOR

PropertiesΘ BOUNDED⇒ δMLE dominates δSM

SECTION 1SECTION 2

DECISION THEORY

FORMALIZING THE CHOICE

Frequentist approachRESTRICT ∆

U = SET UNBIASED δ

CHOOSE UMVE

Decision-oriented approach(i) K OPTIMAL CRITERIA(ii) CHOOSE δ

MINIMIZING R(θ, δ)W.R.T K

SECTION 1SECTION 2

BAYES ESTIMATOR

(i) PROBABILITY MEASURE τ ON Θ

(ii) CRITERIA = Eτ

(iii) Eτ [R(θ, δ)] = r(τ, δ)

(iv) minδ

[r(τ, δ)] = r(τ, δB) = r(τ)

(v) δB BAYES RULE

Bayes method and Decision theoryNATURAL ORDERING CRITERIATHE VERSATILITY OF τ MAKES BAYESIAN METHODCOHERENT WITH DECISION THEORYWHICH PRIOR INDUCES MINIMAXITY ?

SECTION 1SECTION 2

BAYES ESTIMATOR

(i) PROBABILITY MEASURE τ ON Θ

(ii) CRITERIA = Eτ

(iii) Eτ [R(θ, δ)] = r(τ, δ)

(iv) minδ

[r(τ, δ)] = r(τ, δB) = r(τ)

(v) δB BAYES RULE

Bayes method and Decision theoryNATURAL ORDERING CRITERIATHE VERSATILITY OF τ MAKES BAYESIAN METHODCOHERENT WITH DECISION THEORYWHICH PRIOR INDUCES MINIMAXITY ?

SECTION 1SECTION 2

GAME THEORY

Two-person zero-sum game

Θ Set of all possible strategies player 1A Set of all possible strategies player 2L Gain function (pl. 1) and loss function (pl.2)

RATIONAL PLAYER LOOK FOR A GUARANTEEWHATEVER OPPONENT’S MOVEMINIMAX STRATEGY ARISE NATURALLYMINIMAX STRATEGY FOR PLAYER TWO ≡ MAXIMINSTRATEGY FOR PLAYER ONE

SECTION 1SECTION 2

GAME THEORY

Two-person zero-sum game

Θ Set of all possible strategies player 1A Set of all possible strategies player 2L Gain function (pl. 1) and loss function (pl.2)

RATIONAL PLAYER LOOK FOR A GUARANTEEWHATEVER OPPONENT’S MOVEMINIMAX STRATEGY ARISE NATURALLYMINIMAX STRATEGY FOR PLAYER TWO ≡ MAXIMINSTRATEGY FOR PLAYER ONE

SECTION 1SECTION 2

THE LINK

PLAYER II

PLAYER IMAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIAN

NATURELEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION

SECTION 1SECTION 2

THE LINK

PLAYER IIPLAYER I

MAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIANNATURE

LEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION

SECTION 1SECTION 2

THE LINK

PLAYER IIPLAYER IMAXIMIN STRATEGY

GAIN-ORIENTEDRATIONALITY

STATISTICIANNATURELEAST FAVORABLESTATE OF NATURE

LEAST FAVORABLEDISTRIBUTION

SECTION 1SECTION 2

THE LINK

PLAYER IIPLAYER IMAXIMIN STRATEGYGAIN-ORIENTEDRATIONALITY

STATISTICIANNATURELEAST FAVORABLESTATE OF NATURELEAST FAVORABLEDISTRIBUTION

SECTION 1SECTION 2

SIMPLE EXAMPLE

UNDER MINIMAXCOMPARE sup

δ1 OPTIMAL

UNDER BAYES

τ p.d.f. on Θ

COMPARE Eτ [R]

τ SUCH THAT δ1 � δ1

SUFFICIENT BIAS TOθ0

SECTION 1SECTION 2

SIMPLE EXAMPLE

δ1 OPTIMAL

UNDER BAYES

τ p.d.f. on Θ

COMPARE Eτ [R]

SECTION 1SECTION 2

SIMPLE EXAMPLE

δ1 OPTIMAL

UNDER BAYESτ p.d.f. on Θ

COMPARE Eτ [R]

SECTION 1SECTION 2

LEAST FAVORABLE PRIOR

LEAST FAVORABLE FOCUSES ON θ’s MAXIMAL RISKPOINTS FOR A GENERIC BAYES RULE

LEAST FAVORABLE MAXIMIZE THE BAYES RISK

Lemmar(τ, δB

τ ) ≥ R(θ, δBτ ) ∀θ ∈ Θ ⇒

δBτ MINIMAX

τ LEAST FAVORABLE

SECTION 1SECTION 2

Lemmar(τ, δB

τ ) ≥ R(θ, δBτ ) ∀θ ∈ Θ ⇒

δBτ MINIMAX

τ LEAST FAVORABLE

SECTION 1SECTION 2

Lemmar(τ, δB

τ ) ≥ R(θ, δBτ ) ∀θ ∈ Θ ⇒

δBτ MINIMAX

τ LEAST FAVORABLE

SECTION 1SECTION 2

GUESSING MINIMAX DECISION

GUESS MAX. RISK PTS.

Suppose θ = +mLIKELY SAMPLES∈ [m − 1,m + 1]

HIGHLY BIASEDINTERVAL

SECTION 1SECTION 2

Suppose θ = +m

LIKELY SAMPLES∈ [m − 1,m + 1]

SECTION 1SECTION 2

Outline

SECTION 1SECTION 2

A BAYES RULE

CONCENTRATING ONTHE BOUNDS

τ◦m TWO-POINTS

PRIORδ◦m(x) =

m × tanh(m × x)

SECTION 1SECTION 2

A BAYES RULE

CONCENTRATING ONTHE BOUNDSτ◦m TWO-POINTS

δ◦m(x) =

m × tanh(m × x)

SECTION 1SECTION 2

A BAYES RULE

CONCENTRATING ONTHE BOUNDSτ◦m TWO-POINTS

PRIORδ◦m(x) =

m × tanh(m × x)

SECTION 1SECTION 2

SHRINKING TO THE BOUNDS

SECTION 1SECTION 2

CONDITIONS FOR MINIMAXITY

minimaxity of δ◦m depends on the interval width

Theorem

x ∼ N(θ,1)

θ ∼ [−m,m]

m ≤ m0

L Gaussian loss

δ◦m minimaxτ◦m least favorable

Numerical solution for m0

1.056742

SECTION 1SECTION 2

Theorem

x ∼ N(θ,1)

θ ∼ [−m,m]

m ≤ m0

L Gaussian loss

1.056742

SECTION 1SECTION 2

Theorem

x ∼ N(θ,1)

θ ∼ [−m,m]

m ≤ m0

L Gaussian loss

1.056742

SECTION 1SECTION 2

Theorem

x ∼ N(θ,1)

θ ∼ [−m,m]

m ≤ m0

L Gaussian loss

1.056742

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

SKETCHING THE PROOF - 1ST STEP

Prove that r(τ◦m, δ

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′ At most 3 sign chg(−+)(+−)(−+)

R′(0) = 0

R′(θ) = −R′(−θ)

Extremum for θ > 0 is(−+)

R even function⇒ Maximum attainedat 0 or at the bounds

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′ At most 3 sign chg

(−+)(+−)(−+)

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

R even function

⇒ Maximum attainedat 0 or at the bounds

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

R′(0) = 0

R′(θ) = −R′(−θ)

SECTION 1SECTION 2

SKETCHING THE PROOF - CONCLUSION

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

∃m0 such that R(m)≥ R(0) ∀m ≤ m0

r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

= implies ≥⇒ Theorem is proved

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

= implies ≥

⇒ Theorem is proved

SECTION 1SECTION 2

◦m) ≥ R(θ, δ

◦m) ∀θ ∈ Θ

r(τ◦m, δ

◦m) = 1

2R(−m) +12R(m) = R(m, δ

SECTION 1SECTION 2

AND WHEN m GETS LARGE?CLOSING THE LOOP

Outline

SECTION 1SECTION 2

HOW TO PROCEED

AS LONG AS Θ IS COMPACTR(θ, δτ ) analytic and 6= cost

THE NUMBER OF MAXIMAL RISK θ’s ISFINITESτ =SUPPORT OF τLS FINITE

IN GENERALcard{Sτ} INCREASES AS m INCREASES

SECTION 1SECTION 2

HOW TO PROCEED

⇓THE NUMBER OF MAXIMAL RISK θ’s ISFINITE

Sτ =SUPPORT OF τLS FINITE

SECTION 1SECTION 2

HOW TO PROCEED

⇓THE NUMBER OF MAXIMAL RISK θ’s ISFINITESτ =SUPPORT OF τLS FINITE

SECTION 1SECTION 2

HOW TO PROCEED

⇓THE NUMBER OF MAXIMAL RISK θ’s ISFINITESτ =SUPPORT OF τLS FINITE

SECTION 1SECTION 2

GUESSING MINIMAX DECISION m>1

GUESS THE NEXT MAX.RISK PT.

Suppose θ = 0LIKELY SAMPLES∈ [−1,1]

LARGE RANGE (= 2)suppose θ 6= 0 and6= ±mSMALLER RANGE(≤ 2)

SECTION 1SECTION 2

Suppose θ = 0

LIKELY SAMPLES∈ [−1,1]

SECTION 1SECTION 2

LARGE RANGE (= 2)

suppose θ 6= 0 and6= ±mSMALLER RANGE(≤ 2)

SECTION 1SECTION 2

LARGE RANGE (= 2)suppose θ 6= 0 and6= ±m

SMALLER RANGE(≤ 2)

SECTION 1SECTION 2

3-POINTS PRIOR

the expression

δαm(x) = (1−α)mtanh(mx)

1−α+αexp(m22 )sech(mx)

when 1.4 ≤ m ≤ 1.6∃α∗ such that δαm(x) is minimax and ταm is least favorable

SECTION 1SECTION 2

3-POINTS PRIOR

the expression

δαm(x) = (1−α)mtanh(mx)

1−α+αexp(m22 )sech(mx)

when 1.4 ≤ m ≤ 1.6∃α∗ such that δαm(x) is minimax and ταm is least favorable

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

NUMERICAL EVIDENCE

SECTION 1SECTION 2

4-PTS PRIOR

θ’s max risk pts w.r.t a generic bayes rule

MAX BIAS → BOUNDSMAX VARIANCE → PTS FAR FROM BOUNDS

Due to normal symmetryθMS ’s pop up pairwise around zero

SECTION 1SECTION 2

4-PTS PRIOR

SECTION 1SECTION 2

4-PTS PRIOR

SECTION 1SECTION 2

Outline

SECTION 1SECTION 2

GENERALIZED BAYES RULE

Generalizing the approach

(τn) SEQUENCE OFPROPER PRIORSlim−→ δn = δ0

δ0 GENERALIZEDBAYES RULE

δ0 Generalized bayesrulelim−→[rn] ≥ Rδ0(θ) ∀θ<∞

δ0 MINIMAX RULE

SECTION 1SECTION 2

(τn) SEQUENCE OFPROPER PRIORS

lim−→ δn = δ0

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 Generalized bayesrule

lim−→[rn] ≥ Rδ0(θ) ∀θ<∞

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 Generalized bayesrulelim−→[rn] ≥ Rδ0(θ) ∀θ

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 MINIMAX RULE

SECTION 1SECTION 2

δ0 MINIMAX RULE

SECTION 1SECTION 2

NO MORE BOUNDS

Generalized bayesian approach

τn ∼ N(0,n)τn(θ|x) ∼ N( xn

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

τn ∼ N(0,n)

τn(θ|x) ∼ N( xnn+1 ,

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1

R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1

⇒ δ0 MINIMAX

SECTION 1SECTION 2

NO MORE BOUNDS

n+1 ,n

δn = xnn+1 → x = δ0

r(τn) =n

n+1 → 1R(θ, δ0) = 1⇒ δ0 MINIMAX

SECTION 1SECTION 2

EVERY GOOD ESTIMATOR IS BAYES

Berger and Srinivasan 1978

θ natural parameter exponential familyquadratic lossEVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950

Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS

SECTION 1SECTION 2

θ natural parameter exponential family

quadratic lossEVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950

SECTION 1SECTION 2

θ natural parameter exponential familyquadratic loss

EVERY ADMISSIBLE ESTIMATOR = GENERALIZEDBAYES ESTIMATOR

Wald 1950

SECTION 1SECTION 2

Wald 1950

SECTION 1SECTION 2

Wald 1950

SECTION 1SECTION 2

Wald 1950Θ COMPACT

RISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS

SECTION 1SECTION 2

Wald 1950Θ COMPACTRISK SET R CONVEX

ALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS

SECTION 1SECTION 2

Wald 1950Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION

⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS

SECTION 1SECTION 2

Wald 1950Θ COMPACTRISK SET R CONVEXALL ESTIMATORS HAVE CONTINUOUS RISKFUNCTION⇒ BAYES ESTIMATORS’ SET = COMPLETE CLASS

SECTION 1SECTION 2

AS m GOES TO INFINITY

Bickel 1981Minimize Fisherinformation w.r.t. anypriorcos2(π/2)× x , |x | ≤ 1

ρ(m) = 1− π2

m2 + o(m−2)

as m→∞

SECTION 1SECTION 2

THANK YOU

YOUR ATTENTION

bounded normal mean minimax estimation

section 2outline1 section

candidates section

points prioroutline1

smjacopo primavera estimating

t kjacopo primavera

points priorsample mean

points priorgame theory

points priordecision

Education

gaussian estimation: sequence and wavelet...

nonparametric density estimation for multivariate bounded...

minimax estimation for mixtures...

scm minimax

bounded kalman filter method for motion-robust, non ... ·...

the minimax teacher

minimax optimal rates of estimation in functional … ·...

admissible and minimax multiparameter estimation in...

state estimation in wall-bounded ﬂow systems. part 2...

minimax 100 manual

adaptive minimax regression estimation over sparse -hulls

minimax rates of estimation for high-dimensional linear...

on minimax estimation problems for periodically correlated...

estimation of the derivatives of a digital function with a...

estimation of a k monotone density, part 1 ... ·...

local asymptotic minimax estimation of ......local...

minimax adaptive estimation of nonparametric hidden markov...

minimax lower bounds: the fano and le cam methods ·...

near minimax line spectral estimation

minimax estimation in linear regression under...