allocating information gathering efforts for selection ......stochastic simulation optimization: an...
TRANSCRIPT
Dennis D. Leber
National Institute of Standards and Technology
Jeffrey W. Herrmann
University of Maryland
Science of Test Workshop β Springfield, VA
April 4, 2017
Allocating Information Gathering
Efforts for Selection Decisions
of 15
Selecting a Radiation Detection System
1
Performance Operational Impact Cost
Pd SNM (e.g., WGPu, HEU)
Pd Industrial (e.g., 137Cs, 57Co)
Pd Medical (e.g., 131I, 201Tl )
Probability of false alarm
of 15
Challenge
2
How do we allocate our limited and fixed budget for
information gathering to maximize the probability
of selecting the true best alternative?
of 15
Selection Decision Terminology
3
Alternatives
a1: βtownhouseβ
a2 : βfarmhouseβ
a3 : βcountry houseβ
Attributes (value)
1: House size (mi1)
2: Lot size (mi2)
3: Cost (mi3)
4: Distance to work (mi4)
5: Quality of school (mi5)
Preferences and Decision Model
Mike Mulligan and His Stream Shovel
Virginia Lee Burton (1939)
1 1 1 2 2 2 3 3 3 4 4 4 5 5 5i i i i i iv v v v v m m m m m
of 15
Best Alternative
4
Alternative that provides the largest decision value
Function of:
1. Decision-makerβs preferences
2. True attribute values
Knowledge of decision values is uncertain
Uncertainty in attribute values is a function of amount of information gathered
1 1 1 2 2 2i i i k k ikv v v m m m
1 1 1 2 2 2Λ Λ Λ Λi i i k k ikv v v m m m
-------------- Estimate --------------
of 15
Selecting an Alternative
5
Probability that alternative ai has largest decision value: ππ = π ππ > ππ, βπ = 1,β¦ ,π
Select as, where π = argmaxπ
ππ
Define Probability of Correct Selection: PCSDM = ππ
of 15
Information Gathering Layout
6
Assume equal cost for all observations
B is observation budget
Performance
Measure 1
Performance
Measure 2β¦
Performance
Measure k
Alternative a1 π11 n12 n1k
Alternative a2 π21 n22 n2k
β¦ β¦ β¦ β¦
Alternative am ππ1 nm2 nmk
of 15
Uniform Allocation
7
Performance
Measure 1
Performance
Measure 2β¦
Performance
Measure k
Alternative a1 π11 =π΅
πππ12 =
π΅
πππ1π =
π΅
ππ
Alternative a2 π21 =π΅
πππ22 =
π΅
πππ2π =
π΅
ππ
β¦ β¦ β¦ β¦
Alternative am ππ1 =
π΅
ππππ2 =
π΅
πππππ =
π΅
ππ
of 15
Proportional Allocation
8
Performance
Measure 1
Performance
Measure 2β¦
Performance
Measure k
Alternative a1 π11 = π1π΅
ππ12 = π2
π΅
ππ1π = ππ
π΅
π
Alternative a2 π21 = π1π΅
ππ22 = π2
π΅
ππ2π = ππ
π΅
π
β¦ β¦ β¦ β¦
Alternative am ππ1 =π1
π΅
πππ2 =
π2π΅
ππππ = ππ
π΅
π
1 1 1 2 2 2i i i k k ikv v v m m m
of 15
Sequential Information Gathering
9
of 15
Ranking and Selection
10
Statistics
Bechhofer, R. E. (1954). A Single-sample Multiple Decision Procedure for
Ranking Means of Normal Populations with Known Variances. The Annals of
Mathematical Statistics, 25, 16-39.
Bechhofer, R. E., Santer, T. J., & Goldsman, D. M. (1995). Design and Analysis of
Experiments for Statistical Selection, Screening, and Multiple Comparisons. New
York: John Wiley and Sons, Inc.
Computer Simulation Kim, S.-H., & Nelson, B. L. (2006). Selecting the Best System. In S. G.
Henderson, & B. L. Nelson (Eds.), Handbook in Operations Research and Management Science (Vol. 13, pp. 501-534). Oxford: Elsevier.
Chen, C.-H., & Lee, L. H. (2011). Stochastic Simulation Optimization: An Optimal Computing Budget Allocation. Hackensack, NJ: World Scientific Publishing Co. Pte. Ltd.
of 15
Amalgamation
11
πππ π‘
Prior knowledge of
attribute values, πππβs Knowledge of
Decision-maker
preferences, ππUpdated knowledge of
attribute values, πππβs with collected data, πΏ π
Prior knowledge of ability
to measure attribute
values, U ΖΈπππ = ππβs
Update knowledge of ability
to measure attribute values,
ππβs with collected data, πΏ π
of 15
Sequential Allocation Procedure
12
Data collected thus far for alternative ai and attribute j: π±ππ π‘ = π₯ππ1, β¦ , π₯πππππ π‘
Knowledge of value of attribute j for alternative ai: π πππ|π±ππ π‘
Knowledge of decision value for alternative ai: π ππ|π±π π‘
Calculate PCSDM π‘ = π ππ > ππ , βπ = 1,β¦ ,π|π π‘
of 15
Allocation Decision
13
Next sample
Posterior predictive distribution
Allocate sample to ππ, π with
largest Expected PCSπ·πππ
π‘ + 1
11
ijijDMijn t
x PCS t
1|
ijijijn t
p x t
x
1 11 1 11, , ,
1, ,
1 max , , | , |ij ij ij ij
q r
DM m m iijn t ijn t ijn tq m r q
r m
E PCS t p t x d d p x t dx
X x
of 15
Allocation Procedure Performance
14
Evaluation Experiment:
50,000 decision cases
Concave efficient frontier
π = 5 alternatives
π = 2 attributes
100 β€ πππ β€ 200
19 decision models, π1, π2 pairs
π£π πππ = πππ Gaussian measurement error (known)
Bayesian prior on attribute values
π πππ = π 150, 352
Experimental budget π΅ = 50
Observed frequency of correct selection
of 15
Summary
15
Allocating a fixed experimental budget across multiple attributes and
alternatives in a selection decision where the results of the experimental
evaluations lead to uncertain estimates of the true attribute values
Allocation approaches:
1. Uniform
2. Proportional
3. Sequential
Allocation does impact the probability of selecting the true best alternative
Importance for projects focused on a selection decision to be managed so
that the decision modeling and the experimental planning are done jointly
rather than in isolation
BACKUP
16
of 15
How should we allocate a fixed budget across multiple attributes
and alternatives to maximize the probability of correct selection?
Assumptions and Problem Statement
17
1. Finite and distinct set of alternatives
attributes; true value
Separate and independent attribute measurement processes
with known
2. Decision model is provided
Linear
3. Fixed experimental budget
B sample measurements; cost equivalent
1, , ma a
1 1, ,
k
i i ik j j ijjf v m m m
2k ijm
ijl ij ijlX m
2~ 0,ijl jN 2
j
j ij ijv m m1
1k
jj
of 15
Bayesian Estimation
18
Estimate by estimating each
Decision-makerβs prior knowledge
Data
Posterior distribution on
Posterior distribution on
ijm
2
0 0,ij ijN m i
1, ,ijij ijnx x
ijm
i
2 2 2 2
0 0 0
1 2 2 2 2
0 0
| , , ~ ,ij
j ij ij ij ij j ij
ij ij ijn
j ij ij j ij ij
n xx x N
n n
m m
2 2 2 2
0 0 02
2 2 2 210 0
| ~ ,k k
j ij ij ij ij j ij
i i j j
j i jj ij ij j ij ij
n xN
n n
m
x
of 15
Results
19