![Page 1: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/1.jpg)
Automating estimation of warm-up length
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC08
The AutoSimOA ProjectA 3 year, EPSRC funded project in collaboration with SIMUL8 Corporation.
http://www.wbs.ac.uk/go/autosimoa
![Page 2: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/2.jpg)
Research Aim
• To create an automated system for dealing with the problem of initial bias, for implementation into simulation software.
• Target audience: non- (statistically) expert simulation users.
![Page 3: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/3.jpg)
The Initial Bias Problem
• Model may not start in a “typical” state.
• Can cause initial bias in the output.
• Method used: Deletion of the initial transient data by specifying a warm-up period (or truncation point).
• How do you estimate the length of the warm-up period required?
![Page 4: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/4.jpg)
• Literature search – 44 methods
• Short-listing of methods• Accuracy & robustness
• Ease of automation
• Generality
• Computer running time
• Preliminary Testing – 6 methods
• MSER-5 most accurate and robust method.
![Page 5: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/5.jpg)
MSER-5 warm-up method
0
0.002
0.004
0.006
0.008
0.01
0.012
0.014
0.016
0.018
0.02
0 50 100 150 200 250 300 350 400
Truncation Point
Tes
t S
tatis
tic
0
1
2
3
4
5
6
Bat
ch M
eans
MSER-5 test statistic
Rejection zone
Estimated warm-up period
Estimated truncation point, Lsol
Output data (batched means values)
![Page 6: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/6.jpg)
Further Testing of MSER-5
1. Artificial data – controllable & comparable initial bias functions steady state functions
2. Full factorial design.
3. Set of performance criteria.
![Page 7: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/7.jpg)
Parameters Levels
Data Type Single run
Data averaged over 5 reps
Error type N(1,1), Exp(1)
Auto-correlation
None, AR(1), AR(2), MA(2), AR(4), ARMA(5,5)
Bias Severity 1, 2, 4
Bias Length 0%, 10%, 40%, 100% (of n = 1000)
Bias direction Positive, Negative
Bias shape 7 shapes
1. Artificial Data Parameters
![Page 8: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/8.jpg)
• Mean Shift:
• Linear:
• Quadratic:
• Exponential:
• Oscillating (decreasing):
Quadratic ExponentialLinear
![Page 9: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/9.jpg)
Add Initial Bias to Steady state:
Superpostion: Bias Fn, a(t), added onto end of steady state function:
e.g.
2. Full factorial design
3048 types of artificial data set
MSER-5 run with each type 100 times
...
)(1
etc
taXY
XX
tt
ttt
![Page 10: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/10.jpg)
i. Coverage of true mean.
ii. Closeness of estimated truncation point (Lsol) to true truncation point (L).
iii. Percentage bias removed by truncation.
iv. Analysis of the pattern & frequency of rejections of Lsol (i.e. Lsol > n/2).
3. Performance Criteria
![Page 11: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/11.jpg)
MSER-5 Results
Does the true mean fall into the 95% CI for the estimated mean?
Non-truncated data sets
Truncated data sets
% of cases
yes yes 7.7%
no yes 72.5%
no no 19.8%
yes no 0%
i. Coverage of true mean.
![Page 12: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/12.jpg)
-70
-50
-30
-10
10
30
50
0 20 40 60 80 100run
Lsol -
L
Quadratic bias Mean-shift bias
ii. Closeness of Lsol to L.
• Wide range of Lsol values.
e.g.
(Positive bias functions, single run data, N(1,1) errors, MA(2) auto-correlation, bias severity value of 2 and true L = 100.)
![Page 13: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/13.jpg)
iii. Percentage bias removed by truncation.
0
5
10
15
20
25
300-4
0
40-5
0
50-6
0
60-7
0
70-8
0
80-9
0
90-9
5
95-9
9
99-1
00
100+
% bias removed
% o
f to
tal v
alid
runs
All valid runs
![Page 14: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/14.jpg)
Effect of data parameters on bias removal
No significant effect: Error type Bias direction
Significant effect: Data type Auto-correlation
type Bias shape Bias severity Bias length
![Page 15: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/15.jpg)
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s Single run
Averaged replications
More bias removed by using averaged replications rather than a single run.
![Page 16: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/16.jpg)
0
50
1000
-40
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cu
mu
lative
% o
f va
lid
ca
se
s no a-c AR(1)
AR(2) AR(4)
MA(2) ARMA(5,5)
The stronger the auto-correlation, the less accurate the bias removal.
Effect greatly reduced by using averaged data.
![Page 17: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/17.jpg)
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cu
mu
lative
% o
f va
lid
ca
se
s
mean-shift Linear
Quad Exp
OscL OscQ
OscE
The more sharply the initial bias declines, the more likely MSER-5 is to underestimate the warm-up period and to remove increasingly less bias.
![Page 18: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/18.jpg)
0
50
1000
-40
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s 1
2
4
As the bias severity increases, MSER-5 removes an increasingly higher percentage of the bias.
![Page 19: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/19.jpg)
0
50
100
0-4
0
40
-50
50
-60
60
-70
70
-80
80
-90
90
-95
95
-99
99
-10
0
10
0+
reje
ctio
ns
% of bias removed
cum
ula
tive
% o
f va
lid c
ase
s
10%
40%
Longer bias removed slightly more efficiently than shorter bias.
Shorter bias - more overestimations - partly due to longer bias overestimations being more likely to be rejected.
![Page 20: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/20.jpg)
0
100
200
300
400
500
600
700
800
900
x=
0
0<
x≤1
1<
x≤5
5<
x≤1
0
10
<x≤2
0
20
<x≤4
0
40
<x≤6
0
60
<x≤8
0
80
<x≤1
00
x = no. of Lsol rejections
no
. o
f ca
se
s
ARMA(5,5)
MA(2)
AR(4)
AR(2)
AR(1)
No auto-correlation
Rejections caused by: high auto-correlation, bias close to n/2, smooth end to data = ‘end point’ rejection.
Averaged data slightly increases probability of getting ‘end point’ rejection but increases probability of more accurate L estimates.
iv. Lsol rejections
![Page 21: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/21.jpg)
0
10
20
30
40
50
1000 1100 1200 1300 1400 1500 1600 1700 1800n
Lso
l re
ject
ion
co
un
t
+ meanshift
+ linear
+ quadratic
+ exp
+ osclinear
+ oscquad
+ oscexp
Giving more data to MSER-5 in an iterative fashion produces a valid Lsol value where previously the Lsol value had been rejected.
e.g. ARMA(5,5)
![Page 22: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/22.jpg)
Lsol values Percentage of cases
Lsol = 0 71%
Lsol ≤ 50 93%
Testing MSER-5 with data that has no initial bias.
Want Lsol = 0
Lsol > 50 mainly due to highest auto-correlated data sets - AR(1) & ARMA(5,5).
Rejected Lsol values: 5.6% of the 2400 Lsol values produced. 93% from the highest auto-correlated data ARMA(5,5).
![Page 23: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/23.jpg)
Testing MSER-5 with data that has 100% bias.
Want 100% rejection rate: Actual rate = 61%
0
1020
30
4050
60
70
8090
100
Line
ar
Qua
d
Exp
Osc
Line
ar
Osc
Qua
d
Osc
Exp
Bias shape
Per
cent
age
of L
sol
reje
ctio
ns
0
10
20
30
40
50
60
70
80
90
M1 M2 M4
Bias severity
Per
cent
age
of L
sol
rej
ectio
ns
Single data Averaged data
![Page 24: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/24.jpg)
Summary
• MSER-5 most promising method for automation– Not model or data type specific. – No estimation of parameters needed. – Can function without user intervention. – Shown to perform robustly and effectively
for the majority of data sets tested. – Quick to run. – Fairly simple to understand.
![Page 25: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/25.jpg)
Heuristic framework around MSER-5
Run k (= 5) replications of length, n ≥ 100
Create averaged
data
Batch data into b batches of length m, where number of
batches = bmn and n* =
b×m ≤ n
MSER-5 returns Lsol value
Produce more data to create
batches of no. orig of %10 or a user specified
number.
Dynamic graph of batched data; single reps, or
MSER-5 statistic
Graph of batched data; single reps,
or MSER-5 statistic with valid Lsol value shown.
Input data into MSER-5 algorithm.
Yes
Yes
No
No
Does User wish to keep running with more data? END
Lsol valid.
Lsol invalid.
Is Lsol ≤ (n* - (m × 5))/2
?
Yes
Have there been 10 invalid Lsol
values in a row?
No
Yes No
Does User wish to keep running with more data?
Produce more data to create
batches of no. orig of %10
Iterative procedure for procuring more data when required.
‘Failsafe’ mechanism - to deal with possibility of data not in steady state; insufficient data provided when highly auto-correlated.
Being implemented in SIMUL8.
![Page 26: Automating estimation of warm-up length Katy Hoad, Stewart Robinson, Ruth Davies Warwick Business School WSC08 The AutoSimOA Project A 3 year, EPSRC funded](https://reader038.vdocuments.us/reader038/viewer/2022110322/56649d3b5503460f94a16460/html5/thumbnails/26.jpg)
ACKNOWLEDGMENTSThis work is part of the Automating Simulation Output
Analysis (AutoSimOA) project (http://www.wbs.ac.uk/go/autosimoa) that is funded by
the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in
collaboration with SIMUL8 Corporation, who are also providing sponsorship for the project.
Katy Hoad, Stewart Robinson, Ruth DaviesWarwick Business School
WSC08