CHAPTER 12CHAPTER 12 SSTATISTICAL TATISTICAL MMETHODS FOR ETHODS FOR

OOPTIMIZATION IN PTIMIZATION IN DDISCRETE ISCRETE PPROBLEMSROBLEMS

•Organization of chapter in ISSO–Basic problem in multiple comparisons

•Finite number of elements in search domain

–Tukey-Kramer test

–“Many-to-one” tests for sharper analysis •Measurement noise variance known

•Measurement noise variance unknown (estimated)

–Ranking and selection methods

Slides for Introduction to Stochastic Search and Optimization (ISSO) by J. C. Spall

12-2

BackgroundBackground

• Statistical methods used here to solve optimization problem– Not just for evaluation purposes

• Extending standard pairwise t-test to multiple comparisons

• Let {1, 2, …, K} be finite search space (K possible options)

• Optimization problem is to find the j such that = j

• Only have noisy measurements of L(i)

12-3

Applications with Monte Carlo Applications with Monte Carlo SimulationsSimulations

• Suppose wish to evaluate K possible options in a real system– Too difficult to use real system to evaluate options

• Suppose run Monte Carlo simulation(s) for each of the K options

• Compare options based on a performance measure (or loss function) L() representing average (mean) performance represents options that can be varied– Monte Carlo simulations produce noisy measurement of

loss function L at each option

12-4

Statistical Hypothesis TestingStatistical Hypothesis Testing

• Null hypothesis: All options in {1, 2, …, K} are effectively the same in the sense that L(1) = L(2) = … = L(K)

• Challenge in multiple comparisons: alternative hypothesis is not unique– Contrasts with standard pairwise t-test

• Analogous to standard t-test, hypothesis testing based on collecting sample values of L(1), L(2), and L(K), forming sample means , ,...,1 2 KL L L

12-5

Tukey–Kramer TestTukey–Kramer Test• Tukey (1953) and Kramer (1956) independently developed popular multiple

comparisons analogue to standard t-test

• Recall null hypothesis that all options in {1, 2, …, K} are effectively the same in the sense that L(1) = L(2) = … = L(K)

• Tukey–Kramer testTukey–Kramer test forms multiple acceptance intervals for K(K–1)/2 differences ij – Intervals require sample variance calculation based on samples at all K

options

• Null hypothesis is accepted if evidence suggests allall differences ij lie in their respective intervals– Null hypothesis is rejected if evidence suggests at least one ij lies outside its

respective interval

i jL L

12-6

Example: Widths of 95% Acceptance Example: Widths of 95% Acceptance Intervals Increasing with Intervals Increasing with KK in Tukey–Kramer in Tukey–Kramer

Test (Test (nn11==nn22=…==…=nnKK=10)=10)

12-7

Example of Tukey–Kramer TestExample of Tukey–Kramer Test (Example 12.2 in (Example 12.2 in ISSOISSO))

• Goal: With K = 4, test null hypothesis L(1) = L(2) = L(3) = L(4) based on

10 measurements at each i

• All (six) differences ij must lie in acceptance intervals [–1.23, 1.23]

• Find that 34 = 1.72

– Have 34 [–1.23, 1.23]

• Since at least one ij is not in acceptance interval, rejectreject null hypothesis

• Conclude at least one i likely better than others

– Further analysis required to find i that is better

i jL L

12-8

Multiple Comparisons Against One CandidateMultiple Comparisons Against One Candidate• Assume prior information suggests one of K points is optimal, say m

• Reduces number of comparisons from K(K–1)/2 differences ij = to only K–1 differences mj

• Under null hypothesis, L(m) L(j) for all j

• Aim to reject null hypothesis

– Implies that L(m) < L(j) for at least some j

• Tests based on critical values < 0 for observed differences mj

• To show that L(m) < L(j) for all j requires additional analysis

i jL L

mj

12-9

Example of Many-to-One Test with Example of Many-to-One Test with Known Variances (Example 12.3 in Known Variances (Example 12.3 in ISSOISSO))

• Suppose K = 4, m = 2 Need to compute 3 critical values , , and for acceptance regions

• Valid to take

• Under Bonferroni/Chebyshev:

• Under Bonferroni/normal noise:

• Under Slepian/normal noise:• Note tighter (smaller) acceptance regions when assuming

normal noise

21 23 24

242321

96.3

10.1

09.1

}{ 22 jj

Widths of 95% Acceptance Intervals (< 0) Widths of 95% Acceptance Intervals (< 0) for Tukey-Kramer and Many-to-One Tests for Tukey-Kramer and Many-to-One Tests

((nn11==nn22=…==…=nnKK=10)=10)

12-11

Ranking and Selection:Ranking and Selection:Indifference Zone MethodsIndifference Zone Methods

• Consider usual problem of determining best of K possible options, represented 1 , 2 ,…, K

• Have noisy loss measurements yk(i )

• Suppose analyst is willing to accept any i such that L(i) is in indifference zone [L(), L() + )

• Analyst can specify such that

P(correct selection of = ) 1

whenever L(i) L() for all i

• Can use independent sampling or common random numbers (see Section 14.5 of ISSO)