using computational grids for effective...

USING COMPUTATIONAL GRIDS FOR

EFFECTIVE SOLUTION OF

STOCHASTIC PROGRAMS

by

Udom Janjarassuk

Presented to the Graduate and Research Committee

of Lehigh University

in Candidacy for the Degree of

Doctor of Philosophy

in

Industrial and Systems Engineering

Lehigh University

2009

Approved and recommended for acceptance as a dissertation in partial fulfillment of the

requirements for the degree of Doctor of Philosophy.

Date

Robert H. Storer

Chairman

Accepted Date

Committee:

Robert H. Storer, Co-Advisor

Jeff T. Linderoth, Co-Advisor

Jitamitra Desai

Theodore K. Ralphs

*

ii

Acknowledgements

First, I would like to thank my advisor Dr. Jeff Linderoth who brought me this dissertation

topic and gave me tremendous help for the research. Also, I would like to thank my co-

advisor Dr. Robert Storer who devoted a significant amount of his time to this dissertation.

I am also thankful to my other dissertation committee members for the helpful advices,

especially Dr. Theodore Ralphs for the dissertation writing and grammar correction. Thanks

to my family and friends who provide supports during my study at Lehigh University. Finally,

I would like to thank the Thai Government and Thai people who pays taxes for the funding.

iii

Contents

Acknowledgements iii

Contents iv

List of Tables viii

List of Figures xi

Abstract 1

1 Introduction 3

1.1 Introduction to Stochastic Programming . . . . . . . . . . . . . . . . . . . . 3

1.2 Computational Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.4 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.5 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.6 Thesis Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Thesis Preliminaries 13

2.1 Two-stage Stochastic Linear Programs . . . . . . . . . . . . . . . . . . . . . 13

2.2 Decomposition Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.2.1 L-Shaped Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.2.2 Multi-cut L-shaped Method . . . . . . . . . . . . . . . . . . . . . . . 20

2.2.3 Regularized Decomposition Method . . . . . . . . . . . . . . . . . . 23

iv

2.3 Sample Average Approximation . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.3.1 Lower Bound Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.3.2 Upper Bound Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.3.3 Bias of Sample Average Estimates . . . . . . . . . . . . . . . . . . . . 26

2.4 Bootstrap Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.5 Grid Computing Environments and Software Tools . . . . . . . . . . . . . . 29

2.5.1 Condor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.2 MW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.3 SUTIL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.5.4 ATR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.6 Performance Profiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.7 Applications of Two-stage Stochastic Programming . . . . . . . . . . . . . . 33

3 Warm-starting and Dynamic Cut Aggregation in the L-Shaped Method 39

3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.2 Warm-starting Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2.1 Pre-sampling Method . . . . . . . . . . . . . . . . . . . . . . . . . . 40

3.2.2 Scenario-Partitioning Method . . . . . . . . . . . . . . . . . . . . . . 42

3.3 Dynamic Cut Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.3.1 Justification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3.3.2 Cut Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

3.3.3 Cut Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.3.4 Cut Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

3.4 Computational Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

3.4.1 Computational Results for Warm-starting . . . . . . . . . . . . . . . . 51

3.4.2 Computational Results for Dynamic Cut Management . . . . . . . . . 59

3.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4 Nonlinear Programming Methods for Stochastic Programs 69

4.1 Trust-region Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

v

4.1.1 Classical Trust-region Update Method . . . . . . . . . . . . . . . . . 71

4.1.2 Selection of an Initial Trust-region Radius . . . . . . . . . . . . . . . 72

4.1.3 Trust-region Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.2 Parallel Search Methods within a Trust-region . . . . . . . . . . . . . . . . . 79

4.2.1 Line Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.2.2 Zigzag Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.2.3 Dogleg Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82


4.3.1 Initial Trust-region Radius . . . . . . . . . . . . . . . . . . . . . . . . 83

4.3.2 Computational Results for Trust-region Scaling . . . . . . . . . . . . 86

4.3.3 Computational Results on Parallel Search . . . . . . . . . . . . . . . 89

4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5 Bootstrap Estimate of Bias in Sample Average Approximation 97

5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97

5.2 Bootstrap Estimate of Lower Bound Bias . . . . . . . . . . . . . . . . . . . . 98

5.3 Bootstrap-adjusted Estimate of the Optimal Value . . . . . . . . . . . . . . . 100


5.4.1 Estimating the True Optimal Solution Values . . . . . . . . . . . . . . 102

5.4.2 Bias and Bias Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.4.3 Upper Bound Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

5.4.4 Bootstrap Method with Limited Computational Budget . . . . . . . . 111

5.4.5 Comparison of Bootstrap Method and ATR Method . . . . . . . . . . 113

5.4.6 Choosing the value of M and B . . . . . . . . . . . . . . . . . . . . . 117

5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

6 Case Study: Solving Large Stochastic Programs 121

6.1 Computational Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

6.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

vi

7 Conclusion 131

7.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

7.2 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

7.2.1 Inexact Master Solution . . . . . . . . . . . . . . . . . . . . . . . . . 133

7.2.2 Updating the Trust-region Radius . . . . . . . . . . . . . . . . . . . . 134

7.2.3 m Out of n Bootstrap . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Appendices 136

A Tables 136

A.1 Statistics on Test Instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

A.2 Statistics for Search Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 138

A.3 Statistics for Bootstrap Method . . . . . . . . . . . . . . . . . . . . . . . . . 150

Vita 181

vii

List of Tables

2.1 Test Instance Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

3.1 Average statistics on different numbers of clusters . . . . . . . . . . . . . . . 44

3.2 Average percentage from optimality for different starting solutions . . . . . 53

3.3 Average optimality gaps after warm-starting . . . . . . . . . . . . . . . . . . 54

3.4 Decomposition time and warm-starting time in seconds . . . . . . . . . . . . 55

3.5 Average wall clock time for warm-starting . . . . . . . . . . . . . . . . . . . 57

3.6 Average number of iterations for warm-starting . . . . . . . . . . . . . . . . 57

3.7 Average solution time and number of iterations by varying the size of the

partitioned instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

3.8 Average statistics on cut aggregation for large number of clusters . . . . . . 62

3.9 Average statistics on cut aggregation for medium number of clusters . . . . 63

3.10 Average statistics on cut aggregation for small number of clusters . . . . . . 64

4.1 Average wall clock time and number of iterations using different initial trust-

region radius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.2 Average wall clock time for different scaling methods . . . . . . . . . . . . . 87

4.3 Average number of iterations for different scaling methods . . . . . . . . . . 88

4.4 Statistics on percentage of the number of the nth point being evaluated and

being accepted for different search methods . . . . . . . . . . . . . . . . . . 95

5.1 Average running times(in seconds) for bootstrap by solving the deterministic

equivalent problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

viii

5.1 Average running times(in seconds) for bootstrap by solving the deterministic

equivalent problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.2 Upper bound, lower bound, and optimality gap with 95% confidence interval 105

5.3 Count of the instances that the bias is within a specified value for bootstrap

estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.4 Head count of the instances that the bias is within a specified value for small

N . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5.5 The value of B for minimum MSE at different sample sizes . . . . . . . . . . 112

5.6 Comparison of lower bound estimate between the bootstrap method and the

ATR method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

6.1 Total computation time for solution of large-scale instances . . . . . . . . . 127

6.2 Master problem solution time for large-scale instances . . . . . . . . . . . . 128

6.3 Number of iterations for solution of large-scale instances . . . . . . . . . . . 129

6.4 Number of average workers for solution of large-scale instances . . . . . . . 130

A.1 Average of the objective value and the first-stage cost . . . . . . . . . . . . . 137

A.2 Average wall clock time for line search . . . . . . . . . . . . . . . . . . . . . 138

A.3 Average wall clock time for zigzag search . . . . . . . . . . . . . . . . . . . 139

A.4 Average wall clock time for dogleg search . . . . . . . . . . . . . . . . . . . 140

A.5 Average master problems solving time for line search . . . . . . . . . . . . . 141

A.6 Average master problems solving time for zigzag search . . . . . . . . . . . 142

A.7 Average master problems solving time for dogleg search . . . . . . . . . . . 143

A.8 Average number of master problems solved for line search . . . . . . . . . . 144

A.9 Average number of master problems solved for zigzag search . . . . . . . . . 145

A.10 Average number of master problems solved for dogleg search . . . . . . . . 146

A.11 Average number of points evaluated for line search . . . . . . . . . . . . . . 147

A.12 Average number of points evaluated for zigzag search . . . . . . . . . . . . . 148

A.13 Average number of points evaluated for dogleg search . . . . . . . . . . . . 149

ix

A.14 Lower bound and upper bound biases with 95% confidence interval for prob-

lems of small sample sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

A.15 Bootstrap estimate of bias for all tested instances . . . . . . . . . . . . . . . 157

A.16 Mean Square Error for Bootstrap Estimate with Different Number of Replica-

tions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166

x

List of Figures

2.1 Computation Time in Solution of the Extensive Form vs the L-shaped Method 15

2.2 Optimality Cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.1 Cut management feedback loop diagram . . . . . . . . . . . . . . . . . . . . 46

3.2 Performance profile of wall clock time and number of iterations for warm-

starting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.3 Performance profile for cut management with large number of clusters . . . 60

3.4 Performance profile for cut management with medium number of clusters . 60

3.5 Performance profile for cut management with small number of clusters . . . 61

3.6 Performance profile of wall clock time for all settings . . . . . . . . . . . . . 65

3.7 Master problem solution time and number of cuts in the master problem

during computation with large number of clusters . . . . . . . . . . . . . . . 67


during computation with medium number of clusters . . . . . . . . . . . . . 67


during computation with small number of clusters . . . . . . . . . . . . . . 67

4.1 The scaled trust-region based on the subgradient of the current point . . . . 75

4.2 BFGS approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

4.3 The scaled trust-region based on BFGS approximation . . . . . . . . . . . . 78

4.4 Zigzag Search Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

4.5 Dogleg Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

xi

4.6 Performance profile of wall clock time and number of iterations for different

initial trust-region radius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

4.7 Performance profile of wall clock time and number of iterations for trust-

region scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.8 Performance profile of wall clock time for search methods and varying num-

bers of points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4.9 Performance profile of total master problems solution time for search meth-

ods and varying numbers of points . . . . . . . . . . . . . . . . . . . . . . . 91

4.10 Performance profile of number of master problems solved for search methods

and varying numbers of points . . . . . . . . . . . . . . . . . . . . . . . . . 92

4.11 Performance profile of number of points evaluated for search methods and

varying numbers of points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

5.1 Scatter plot of the bias and bias estimate in percentage of the objective value 108

5.2 Stemplot of the bias and bias estimate in percentage . . . . . . . . . . . . . 109

5.3 CDF plots for chem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5.4 CDF plots for semi2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5.5 SAA sample mean and bootstrap sample mean plots by varying B and M . . 119

6.1 Performance profile plots of wall clock time for large-scale instances . . . . . 123

6.2 Performance profile plots of master problem solution time for large-scale in-

stances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

6.3 Performance profile plots of number of iterations for large-scale instances . 124

xii

Abstract

Stochastic programming is a mathematical tool for decision-making under uncertainty.

However, it remains largely unused as a practical tool for decision making. One reason

for its light use in practice may be that practical instances are of larger size than can be

solved by available software tools on existing computational platforms. In this thesis, we

focus on algorithms for solving large-scale two-stage stochastic programs on a parallel dis-

tributed computer platform known as a computational grid. An overarching theme of this

work is to make use of hundreds of loosely-coupled processors over long time periods to

effectively solve stochastic programs. We study how grid computing can be employed for

effective solutions of stochastic programs in the following areas. First, we study a paral-

lel implementation of the L-Shaped decomposition method for solving a sample average

approximation(SAA) of large stochastic programs. We propose warm-starting methods for

the L-Shaped decomposition method by using pre-sampling and scenario partitioning. Fur-

ther, a method for managing cuts in the L-Shaped method is also discussed. Second, since

evaluating the objective function is a computational task that can be easily and efficiently

distributed on a computational grid, we study enhancements to traditional techniques that

can help us exploit this fact. In particular, we propose a line search method and trust-region

scaling methods to enhance the convergence rate. Finally, in order to estimate the bias of

solution value estimates using the SAA method, we propose using a statistical technique

known as the bootstrap method. The bootstrap estimate requires solution of a large number

of bootstrap sample problems, a process that can be easily parallelized by using the power

of grid computing. In all cases, extensive computational experiments were performed. We

1

conclude with a case study demonstrating the application of our developed techniques to

solve very large-scale instances.

2

Chapter 1

Introduction

1.1 Introduction to Stochastic Programming

Mathematical programming is a well-known, systematic tool for decision-making. However,

an inaccurate model can lead to wrong or ineffective decisions. One important factor in the

accuracy of the model is the correctness of the information. Unfortunately, information is

not always known with complete certainty. Often, people gain information by gathering

historical data and providing estimates of the information via a data distribution. A sim-

ple way to handle the uncertainty is to replace uncertain values with their mean value.

Unfortunately, doing so generally leads to a solution that is far from optimal.

Stochastic programming is a tool for modeling decision problems under uncertainty. The

uncertainty is represented by random variables. In stochastic programming, we assume that

the distribution of the random variables is known. A stochastic program can be formulated

in terms of expectation as

minx∈X

f(x) def= EP G(x, ω),

where X ⊂ Rn is the set of feasible solutions and ω is an element of a probability space

(Ω,F , P ) [13].

The random variables may have either continuous or discrete distributions. Continuous

distributions can be approximated by discretization. In our research, we focus on stochastic

3

1.1. INTRODUCTION TO STOCHASTIC PROGRAMMING

programs with discrete random variables. A possible outcome of the uncertainty is known

as a scenario. In practice, the number of scenarios can be extremely large, and this makes

the resulting problem very difficult to solve.

A widely studied class of stochastic programming models is the two-stage stochastic pro-

gram. In this model, actions are taken in two stages. In the first stage, a first-stage decision

has to be made before the uncertainties are resolved. In the second stage, the outcome of

the first-stage decision becomes realized after the random event occurs, and recourse ac-

tions can then be taken. The ability that allows decision makers to take corrective actions is

known as recourse in stochastic programming. The random event and decision making may

occur in multiple periods, in which information becomes available in each period and the

corresponding actions are made successively after each period. In such case, it is formulated

as a multi-stage stochastic program.

Stochastic programs can be solved in many ways. Small stochastic programs may be

solved directly by using linear programming software. Large-scale stochastic programs re-

quire more sophisticated methods. The decomposition approach is one of the most promis-

ing method for solution of large-scale problems. The key idea of the decomposition ap-

proach is to break the problem into many smaller pieces that are easier to manage. In so

doing, one can either break the problem into subproblems according to time stages or break

it into subproblems according to scenarios. A bonus of the decomposition approach is that

solution of many of these subproblems can be parallelized.

Sampling is another approach to handle large-scale stochastic programs. Sampling is

employed when the problem has an extremely large number of scenarios, and it is impos-

sible to solve the problem exactly with existing methods. In the sample average approxi-

mations (SAA) method, a subset of samples are drawn from the original sample space, and

the sampled instance is then solved to get an approximate solution. Sampling techniques

can also be incorporated with the decomposition method for solution of large sampled in-

stances.

4

1.2. COMPUTATIONAL GRIDS

1.2 Computational Grids

A computational grid is a type of parallel computer. Unlike traditional parallel computers,

however, computational grids are made up of resources from disparate distributed comput-

ers, such as desktop computers, which work together via a communication network. One

of the key features of grids is that grids typically utilize the unused computer power of pro-

cessors making up the grid, and grid software provides the ability to migrate tasks to other

computers when a person reclaims the computer.

Grid computing provides users a low-cost and powerful computational platform and

allows more researchers to be able to use parallel computing capabilities. Computational

grids are typically used for solution of large-scale problems in which tasks can be divided

into smaller problems and distributed to individual computers. Grid computing has been

widely used in many areas such as financial management [32], structural biology [67],

climate modeling [7], and astrophysics [55].

Despite their low cost and high computational power, computational grids have their

downsides, including dynamic behavior and unreliability. The computers making up the

grids can be very different in terms of processing speed, amount of memory, and type of

operating system. The numbers of processors available at a specific time can also vary

considerably, as the resources may disappear at anytime during the computation.

Several key components are necessary to make use of the computational power of the

grids efficiently. Among them are grid management software, software library tools, and

parallel algorithms. The management software allows jobs to be distributed efficiently

across the grids. The software library and parallel algorithm are essential for the devel-

opment of software and the efficiency of solving a particular problem.

1.3 Literature Review

The following review focuses mainly on surveys in which parallel computers were used to

solve large-scale stochastic programs. Perhap, the most well-known method for solution

5

1.3. LITERATURE REVIEW

of large-scale stochastic linear programs is the Benders decomposition method. The Ben-

ders decomposition method is an iterative algorithm. It breaks the problem down into a

master problem and a series of subproblems, either by nodes, according to stages, or by

scenarios. Each subproblem is then solved independently, and the result provides informa-

tion used for solving the master problem. The well known L-shaped method, introduced

by Van Slyke and Wets [66] in 1969, is an application of Benders-decomposition method

to stochastic programming. The multi-cut variant of the L-shaped method for solution of

two-stage stochastic linear programs was proposed by Birge and Louveaux [34].

Parallel computing is well-suited for solution of stochastic programs because the com-

putation can easily be done in parallel by using decomposition methods. Dantzig [19]

and Wets [68] were the first to suggest that the function evaluation step in the L-shaped

method can be performed in parallel, and a parallel version of the algorithm was first im-

plemented and tested by Ariyawansa and Hudson [5] for solution of two-stage stochastic

linear programs. Ariyawansa and Hudson provided results of a good speedup on random

test instances. However, they were only able to solve medium-size instances with their

implementation. Later, Qi and Zenios [53] developed the L-shaped method on the Connec-

tion Machine CM-2. Their data-parallel implementation solved both the master problem

and subproblems in parallel by using interior-point method. Their results on medium-size

two-stage instances showed that the algorithm is scalable. Further, Nielsen and Zenios [49]

extended the algorithm on the parallel CM-5 computer. They reported results on larger-

scale instances whose extensive form up to 2 million constraints and 13.8 million variables.

Multi-stage stochastic linear programs have also been studied by many researchers in

the context of parallel computing. The nested decomposition method [9] is a generalization

of the L-shaped method, and can be used efficiently for solution of multi-stage stochastic

linear programs. The parallel implementation of the nested decomposition method is de-

scribed in Birge et.al [12] and in Dempster and Thompson [21]. Birge et.al showed a

significant speedup compared to their serial implementation of the algorithm, and the par-

allel version was able to solve much larger-sized instances of up to 1.5 million constraints

and 5 million variables. Dempster and Thompson [21] also employed stage aggregation

6


to reduce the communication process between processors and therefore improve the paral-

lel efficiency. Their results show different levels of speedup on different tested instances.

Fragniere, Gondzio, and Vial [29] also used a decomposition approach to solve large-scale

multi-stage stochastic programming problems. They showed that a multi-stage stochastic

program can be reduced into a two-stage stochastic program and solved efficiently on a

distributed computer cluster of 10 PCs. They applied their procedure to a simple financial

model with over one million scenarios. Gondzio and Kouwenberg [30] also used stage ag-

gregation to reduce a multi-stage stochastic program into a two-stage stochastic program

and then solved the reduced problem using a decomposition method on a distributed com-

puter cluster. They employed the interior point solver HOPDM to both the master problem

and the subproblems. By using this technique, a large-scale real-life asset liability manage-

ment problem with 12.5 million constraints and 25 million variables was solved on a cluster

of 13 processors within 5 hours.

Other variants of the decomposition method were proposed to further improve the al-

gorithm efficiency. Ruszczynski [56] proposed a parallel decomposition method for multi-

stage stochastic linear programs by extending the regularized decomposition method to

multi-stage stochastic programs. The decomposed subproblems was solved in parallel

and informations was exchanged in an asynchronous manner. Computational experiments

showed a substantial speedup over the sequential method. The parallel implementation of

this algorithm was also described in Korycki [37]. Later, Rosa and Ruszczynski [54] pro-

posed two decomposition approaches using the augmented Lagrangian method. The first

approach decomposes the problem into scenarios, and the second approach decomposes the

problem into nodes corresponding to stages. They showed that the scenario decomposition

approach is less complicated from the modeling point of view, and the convergence rate is

also better. Linderoth and Wright [40] proposed an asynchronous versions of the L-shaped

method and the trust-region method for solution of two-stage stochastic linear programs

with recourse on a computational grid. The trust-region method provides advantage in

controlling the step size of the iterates. They showed that it is possible to solve very large

instances with 10 millions scenarios.

7


Sampling techniques can also be combined with the decomposition method to approx-

imate solutions of large-scale stochastic programs. Dantzig and Glynn [20] applied the

importance sampling technique combined with the nested decomposition method for so-

lution of multi-stage stochastic linear programs. Their parallel implementation was based

on IBM 3090 multi-Vector processor and Encore parallel processor at Stanford University.

They showed that parallel computing can be used for solution of such problems, as well as

for computing estimates of the upper bound and lower bound. They also showed that the

importance sampling technique reduced the sample size required for obtaining the same

degree of confidence as used by the Monte Carlo sampling method. A large-scale facilities

planning problem under uncertainty, the Electric-Power System Model problem was tested

under this implementation. A similar implementation on an Intel iPSC/2 was also described

in Dantzig, Ho, and Infanger [17]. Linderoth, Shapiro, and Wright [42] also used sampling

techniques for solution of two-stage stochastic linear programs. They used sample average

approximation method to estimated the optimal values of five instances by constructing the

upper and lower bounds. The Monte Carlo and Latin Hypercube sampling methods were

used for comparison. The sample average approximation problems were solved in paral-

lel on a distributed computational grid using the trust-region algorithm described in [40].

They also provided several methods for testing the quality of the candidate solutions.

The bootstrap is a well-known resampling method for statistical inference. It was in-

troduced by Efron [25] in 1979, and further developed by Efron and Tibshirani [26]. The

bootstrap method is commonly used to estimate the confidence interval, standard error, or

bias for an estimator. However, there is very little literature of using the bootstrap method

for solution of stochastic programs. Albeanu et.al [3] proposed the usage of the bootstrap

method for solution of single period and multi-period stochastic problems. The bootstrap

method was used to generate the scenario tree using historical data for financial analysis.

8

1.4. MOTIVATION

1.4 Motivation

Although stochastic programming has been studied for decades, it still remains largely

unused as a practical decision-making tool. One reason is that many practical stochastic

programs involve a massive number of scenarios, which can only be solved by using de-

composition and requiring intensive computation. Although computers have significantly

improved in processor speed, the computational power required to solve practical stochastic

programs is still far more than that a single processor can provide. A computational grid is

a new computational platform that provides affordable access to significant computational

power if one is willing to use it in a flexible manner. As grid computing becomes more

widespread and provides access to more processors, it is a good idea to reconsider the way

in which we solve stochastic programs given this new type of resources.

One of the difficulties in solution of large-scale stochastic programs is that they may

require a large number of function evaluation which could be computationally expensive.

One possible solution is to utilize computational grids for parallelizing the function evalua-

tions. In so doing, the function evaluation becomes less of a bottleneck as we increase the

amount of resources, which enables the possibility of using more intensive search mecha-

nisms to find better candidate points. Another way to overcome this issue is to seek ways

to make the existing decomposition algorithm more efficient, which reduces the number of

iterations and therefore reduces the total number of function evaluations.

The necessity of improving the efficiency of the decomposition algorithm is that the

master problem is often the bottleneck for solution of difficult problems. While solution

of the subproblems can be parallelized, solution of the master problem is typically done

sequentially. If solution of the master process takes too long, then it will significantly reduce

the parallel efficiency. One reason for the inefficiency in solving the master problem is

that it is difficult to achieve a good balance in the amount of recourse information that

needs to be aggregated. On the one hand, too much aggregation leads to an increase

of the number of iterations. On the other hand, too little aggregation leads to a more

difficult master problem. This provides an open avenue for improvement by aggregating

9

1.5. CONTRIBUTIONS

the recourse information in a more effective manner.

While large-scale stochastic programs remain practically difficult to solve, small stochas-

tic programs can be formulated equivalently as linear programs and solved efficiently by

using modern LP solvers. Given this fact, we ask the question of whether it is possible to

use information obtained from solution of a small version of the instance to help solve the

large instance.

The sample average approximation method has been proven to be a very effective

method for solving stochastic programs, and it often provides solutions of high quality

even when using a small sample size. However, it can take significant computational effort

to demonstrate that the solutions are near-optimal. As such, the method only provides a

statistical estimate of the true objective value and it requires solving the sampled instances

many times to obtain a reasonable statistical confidence. This may not be very practical on

a conventional computational platform. Another issue in using the sample average approx-

imation method is that the estimate of the optimal solution value is often biased. In order

to reduce the bias, the conventional method is to increase the sample size, which means

solving many more difficult instances. This raises the question: is solving a large sampled

instance always necessary for reducing the bias and obtaining an accurate estimate of the

objective value? Are there any other methods that can be used to obtain a good estimate of

the optimal solution value without increasing the sample size, and how can we utilize the

computational grids for such methods?

1.5 Contributions

In this thesis, our main concentration is on using computational grids for solution of prac-

tical stochastic programs. Part of the thesis builds on the decomposition method and pro-

poses and tests several algorithmic enhancements. Our proposed approaches include using

multi-level aggregations of the recourse information and using non-linear programming

techniques to improve the efficiency of the decomposition algorithm. These greatly reduce

the master problem solution time, which is often the bottleneck in solving difficult instances.

10

1.6. THESIS OUTLINE

In addition, we introduced warm-starting methods for the decomposition algorithm. Our

warm-starting methods solve many small sampled instances to obtain information, and then

use this information to help solve the original instance. In so doing, the number of iteration

required for solution of the original instance is greatly reduced.

We also introduced a statistical approach known as the bootstrap method for estimat-

ing the bias in using the sample average approximation method. This method uses a re-

sampling technique and repeatedly solves the re-sampled instances to obtain the bias es-

timate of the objective value. The bias estimate is then used to adjust the objective value

obtained from the sample average approximation method in order to provide a better esti-

mate of the optimal value.

In this thesis, we made contributions on improving the existing methods as well as in-

troducing new techniques for solution of stochastic programs, making stochastic program-

ming a better practical tool for decision making under uncertainty. We demonstrated the

effectiveness of our approaches in solving larger-scale stochastic programs using a large col-

lection of stochastic programming instances. In addition, our approaches provided insight

in utilizing the computational grids for stochastic programming.

1.6 Thesis Outline

The thesis is divided as follows: In Chapter 2, we present the preliminaries of stochastic

programming, computational tools for stochastic programming on grids, and examples of

stochastic programming applications. In Chapter 3, we present warm-starting methods

and cut-manipulation methods for improving the L-shaped decomposition method, with

which large-scale stochastic programs can be solved. In Chapter 4, we propose sophisticated

mechanisms for adjusting the trust-region and provide line search methods for finding an

incumbent solution, which further reduces the computation time. In Chapter 5, we describe

a bootstrap method for estimating the bias of statistical estimators for the optimal objective

value of large-scale stochastic programs. Experimental results are provided at the end of

11

1.6. THESIS OUTLINE

Chapters 3, 4, and 5. In addition, we provide a case study dealing with the solution of large-

scale stochastic programs in Chapter 6. We finish with conclusions and future research ideas

in Chapter 7.

12

Chapter 2

Thesis Preliminaries

In this chapter, we provide the necessary background for properly understanding the later

chapters. We first introduce two-stage stochastic linear programs and the methodologies

used for solving them. Then we describe the environment used in our computational exper-

iments and the software tools used for our implementation. Finally, we provide examples of

stochastic programming applications with short problem descriptions. The stochastic pro-

gramming community suffers from a lack of available test instances, so many researchers

may find the problem descriptions of independent interest. These examples were used as

our test instances.

2.1 Two-stage Stochastic Linear Programs

A two-stage stochastic linear program involves decisions made in two stages. In the first

stage, decisions are made before knowing certain information. In the second stage, the

information becomes realized, and a recourse action can be taken. In general, the objective

is to minimize the cost of the first-stage decisions plus the expected cost of the recourse

decisions.

Formally, a two-stage stochastic linear program with recourse can be formulated as

13

2.1. TWO-STAGE STOCHASTIC LINEAR PROGRAMS

z∗ = minx≥0

f(x) := cT x +Q(x, ξ)

s.t. Ax = b,

(2.1a)

where Q(x, ξ) = Eξ[Q(x, ξ)] and Q(x, ξ) is the value of the optimal solution value of the

second-stage recourse problem

Q(x, ξ) = miny≥0

q(ω)T y

s.t. T (ω)x + W (ω)y = h(ω).(2.1b)

Here, x is the first-stage decision variable and y is the second-stage decision variable. The

elements of the vectors h, q and the matrices T, W depend on a random event ω, which is

defined on a probability space (Ω,F , P ). The random vector ξ = (q(ω), T (ω),W (ω), h(ω))

denotes the realization.

When the random vector ξ has a discrete distribution, that is, ξ takes values from a

sample space Ξ = [(q1, T1,W1, h1), (q2, T2,W2, h2), .., (qK , TK , WK , hK)] with probability

p1, p2, .., pK respectively, we may write the deterministic equivalent of (2.1) in the exten-

sive form as

min cT x +K∑

k=1

pkqTk yk

s.t. Ax = b

Tkx + Wyk = hk, ∀k = 1, .., K

x ≥ 0

yk ≥ 0, ∀k = 1, .., K.

(2.2)

The deterministic equivalent problem (2.2) can be solved using a standard LP solver

such as CPLEX [58], XPRESS-MP [6], or Clp [28]. However, for many applications, the

number of scenarios is potentially very large. For example, a small instance with a random

vector size of 50, where the components of the vector are independent random variables, in

which each element can take only two possible values will have a total number of scenarios

K = 250 ≈ 1.13× 1015. Therefore, solution of the deterministic equivalent of many stochas-

tic programs exactly is practically intractable. Figure 2.1 shows the computation time for

14

2.2. DECOMPOSITION METHODS

0 50 100 150 200 250 3000

10

20

30

40

50

60

70

number of scenarios

Tim

e

Cplex/Extensive Form

L−shaped

Figure 2.1: Computation Time in Solution of the Extensive Form vs the L-shaped Method

solution of problems in the extensive form (2.2) as the number of scenarios increases. The

times in the figure are compared to the L-shaped decomposition method, which is described

subsequently and has a much slower rate of increase with respect to the number of scenar-

ios. Also note that for instances with a small K, solving the deterministic equivalent is faster

than a decomposition approach.

2.2 Decomposition Methods

In practice, the number of scenarios can be very large. Solving the problem in an extensive

form directly can require too much memory or computation time. Decomposition methods

decompose the problem into a master problem and many subproblems that can be solved

more easily.

The recourse functionQ(x) in (2.1) is a convex function of the first-stage variables [10].

Assuming that the second-stage problem Q(x, ξ) has a finite optimal solution for any given

x in any scenario, formulation (2.1) can be replaced by

15


minx

cT x + θ

s.t. Ax = b

θ ≥ Q(x)

x ≥ 0

(2.3)

where Q(x) = Eξ[Q(x, ξ)].

For a given x and ξk, the second-stage problem associated with scenario k is

Q(x, ξk) = minqTk y|Tkx + Wky = hk, y ≥ 0,

= max(hk − Tkx)T µ|W Tk µ ≤ q,

(2.4)

where the last equation follows from linear programming duality, in which µ is the dual

multiplier for the constraints Tkx + Wky = hk.

2.2.1 L-Shaped Method

The L-Shaped method is an application of Benders’ decomposition method [66]. The al-

gorithm solves an approximation of the problem (2.1) by using outer linearizations of the

recourse function Q(x) at a sequence of points. At the beginning of the algorithm, a master

problem has been solved to obtain a first-stage solution. Given this first-stage solution, the

second-stage problems associated with each realization are solved to verify feasibility and

optimality. If any of the second-stage problems is infeasible or optimality of the problem

(2.3) has not been reached, then either a corresponding feasibility cut or an optimality cut

is generated and added to the master problem. The procedure is repeated until optimality

has been reached. We describe the generation of feasibility and optimality cuts here when

the first-stage solution x is obtained.

Feasibility Cuts

If the subproblem (2.4) is unbounded (x is infeasible for the primal problem), then there

exists a dual multiplier σ such that W T σ ≤ 0 and (h− T x)T σ > 0. On the other hand, the

16


constraints y ≥ 0 and Tx + Wy = h must hold for any feasible x. Multiplying both sides of

Tx + Wy = h by σT , we have

σT (h− Tx) = σT Wy.

Because σT W ≤ 0 and y ≥ 0, we obtain the feasibility cut

σT (h− Tx) ≤ 0,

which cuts off the infeasible solution x.

Optimality Cuts

If the solution to each subproblem (2.4) is finite for all possible scenarios, then we have

Q(x, ξk) = (hk − Tkx)T πk for k = 1, ..,K, where πk is the optimal dual multiplier of the

constraints Wky = hk − Tkx associated with scenario k. For any x, we have

Q(x, ξk) = max(hk − Tkx)T µk|W T µk ≤ q.

Because Q(x, ξk) is convex, it follows by the subgradient inequality that

Q(x, ξk) ≥ (hk − Tkx)T πk.

By taking expectation, we obtain

Q(x) = Eξ[Q(x, ξ)] ≥K∑

k=1

[pkπ

Tk (hk − Tkx)

],

where pk is the probability associated with scenario k. By θ ≥ Q(x), we obtain the optimal-

ity cut

θ ≥K∑

k=1

[pkπ

Tk (hk − Tkx)

].

The generation of optimality cuts is demonstrated in Figure 2.2. The figure shows how

17


(x)

x1

x2

x3 x

(x)(x)

Figure 2.2: Optimality Cuts

the optimality cuts cut off non-optimal solutions. The points x1, x2, and x3 are the first-

stage solutions at iteration 1, 2, and 3. The optimality cuts φ1(x) and φ2(x) are generated

at points x1 and x2 respectively. We refer the reader to Kall and Wallace [36] for more

detailed description of feasibility cuts and optimality cuts.

L-shaped Algorithm

This description of the L-shaped algorithm is taken from [10] with minor modifications. In

our description of the L-shaped algorithm, we denote s as the index of the feasibility cuts, t

as the index of the optimality cuts, n as the index of the iteration, and k as the index of the

scenarios. The feasibility cut l is written in the form of flx ≥ dl, and the optimality cut l is

written as glx + θ ≥ el, where dl and el are the cut right hand sides.

L-shaped algorithm

Step 0. Set s = t = n = 0.

Step 1. Set n = n + 1. Solve the master problem.

18


min z = cT x + θ,

s.t. Ax = b,

flx ≥ dl, l = 1, ..., s,

glx + θ ≥ el, l = 1, ..., t,

x ≥ 0,

θ ∈ R

(2.5)

If the master problem is infeasible; stop. Otherwise let (xn, θn) be the optimal solution

of (2.5).

Step 2. For k = 1, ...,K solve the feasibility problems.

min w1k = eT v+ + eT v−,

s.t. Wy + Iv+ − Iv− = hk − Tkxn,

y ≥ 0,

v+ ≥ 0,

v− ≥ 0

(2.6)

where eT = (1, ..., 1). If w1k > 0, then generate a feasibility cut by using

fs+1 = (σk)T Tk

and

ds+1 = (σk)T hk,

where σk is the dual multipliers associated with constraints WyIv+−Iv− = hk−Tkx

n.

If for all k, w1k = 0, then go to Step 3.

Step 3. For k = 1, ...,K solve the optimality problems.

19


min w2k = qky,

s.t. Wy = hk − Tkxn,

y ≥ 0

(2.7)

Let πnk be the dual multipliers associated with constraints Wy = hk − Tkx

n. Generate

an optimality cut by using

gt+1 =K∑

k=1

pk(πnk )T Tk (2.8)

and

et+1 =K∑

k=1

pk(πnk )T hk. (2.9)

Let wn = et+1 − gt+1xn. If θn ≥ wn, stop, xn is an optimal solution. Otherwise, set

t = t + 1, and go to Step 1.

2.2.2 Multi-cut L-shaped Method

The multi-cut L-shaped method [10] is an extension of the standard L-shaped method. In

equations (2.8) and (2.9), information is lost due to cut aggregation. In the multi-cut

version of the L-shaped method, an lower bound approximation θk of the recourse function

is defined for each subproblem k, k = 1, ..., K. An optimality cut associated with each

scenario is added to the master problem. Specifically, the inequalities

θk ≥ pkπTk (hk − Tkx) ∀k ∈ 1, .., K,

are added, where pk is the probability associated with scenario k. In addition, the objective

function of the master problem in (2.5) is modified by replacing the term θ by∑K

k=1 θk.

Compared to the original L-shaped method, the multi-cut version of the L-shaped method

should require fewer iterations to solve the problem to optimality. However, because of the

large number of cuts added to the master problem in each iteration, the size of the master

20


problem grows quickly and the master problem becomes more difficult to solve as more

iterations are completed.

Multi-cut L-shaped Algorithm

The description of the multi-cut L-shaped algorithm is also taken from [10]. We use the

same notation as for the L-shaped algorithm, except that there is an additional subscript tk

for the index of optimality cuts associated with scenario k.

Multi-cut L-shaped algorithm

Step 0. Set s = n = 0 and tk = 0 for all k = 1, ..., K.

Step 1. Set n = n + 1. Solve the master problem.

min z = cT x +K∑

k=1

θk,

s.t. Ax = b,

flx ≥ dl, l = 1, ..., s,

gl(k)x + θk ≥ el(k), l(k) = 1, ..., t(k); k = 1, ..., K,

x ≥ 0,

θ ∈ R

(2.10)

If the master problem is infeasible; stop. Otherwise let (xn, θn1 , ..., θn

K) be the optimal

solution of (2.10).

Step 2. For k = 1, ...,K solve the feasibility problems.

min w1k = eT v+ + eT v−,

s.t. Wy + Iv+ − Iv− = hk − Tkxn,

y ≥ 0,

v+ ≥ 0,

v− ≥ 0

(2.11)

21


where eT = (1, ..., 1). If w1k > 0, then generate a feasibility cut by using

fs+1 = (σk)T Tk

and

ds+1 = (σk)T hk,

where σk is the dual multipliers associated with constraints WyIv+−Iv− = hk−Tkx

n.

If for all k, w1k = 0, then go to Step 3.

Step 3. For k = 1, ...,K solve the optimality problems (2.7).

Let πnk be the dual multipliers associated with subproblem k at iteration n. If θn

k ≥pk(πn

k )T (hk − Tkxn) for all k ∈ 1, .., K., stop, xn is an optimal solution. Otherwise,

generate an optimality cut by using

gt(k)+1 = pk(πnk )T Tk

and

et(k)+1 = pk(πnk )T hk.

Set t(k) = t(k) + 1, and go to Step 1.

Remarks

Another variant of the L-shaped method is the L-shaped clustering method [40]. Instead

of aggregating all the cuts as in the L-shaped method or having one cut per each scenario

as in the multi-cut L-shaped method, the L-shaped clustering method partitions the set of

scenarios into C clusters denoted by K1, K2, ...,KC . Each cluster is associated with a lower

bound approximation θj for the partial sum of the recourse function which is associated with

all the subproblems within cluster j. Cuts from the subproblems within the same cluster are

aggregated into a single cut before being added to the master problem. We can think of this

method as a mixture of the L-shaped method and the multi-cut L-shaped method, where if

22


the number of clusters is 1, we get the L-shaped method and if the number of clusters is K,

we get the multi-cut L-shaped method. Using this approach, we can manage the number of

cuts added to the master problem by changing the cluster size.

The description of the L-shaped clustering algorithm is slightly different from the de-

scription of the multi-cut L-shaped algorithm. Mainly, we need to change the index for the

scenario k ∈ 1, .., K to cluster j ∈ 1, .., C. In Step 3 of the description, the generation of

optimality cut coefficients also needs to change to

gt(j)+1 =Kj∑

k=1

pk(πnk )T Tk

and

et(j)+1 =Kj∑

k=1

pk(πnk )T hk.

In Chapter 3, we exploit clustering and cut aggregation in a different way in an attempt

to solve stochastic programs.

2.2.3 Regularized Decomposition Method

The candidate point in the L-shaped method is obtained without regard to the previous

points. This often leads to solutions that are very different from iteration to iteration with-

out providing substantial improvement in the objective value. In addition, the algorithm

does not make use of a good starting solution if one was provided. The regularized decom-

position method [57] overcomes these disadvantages by regularizing the master problem

with a quadratic term added to the objective function. The regularized master problem is

then

minx∈X

ρ

2‖x− wn‖2 + cT x +

K∑

k=1

pkθk, (2.12)

where ρ > 0 is a control parameter, wn is the incumbent solution at iteration n, and θk is

the under approximation of the second-stage problem associated with scenario k. The new

23

2.3. SAMPLE AVERAGE APPROXIMATION

master problem can be solved by using a quadratic programming solver.

This modification allows the algorithm to control the step size. Note that there is a

penalty to the solution x for moving away from the current incumbent solution wn, making

it possible to effectively use a given good starting point. The incumbent solution is updated

if a new solution xn+1 provides significant improvement in the objective value, which can

be measured by comparing the actual objective value f(xn+1) to the predicted value

fn(xn+1) := cT xn+1 +K∑

k=1

pkθn+1k .

For a fixed constant γ ∈ (0, 1), if

f(xn+1) ≤ (1− γ)f(xn) + γfn(xn),

then we set wn+1 = xn+1; otherwise wn+1 = wn. One can also vary the control parameter

ρ by either increasing it if the step is too short or decreasing it when f(xn) > f(wn).

Another method that has the ability to control the step size and utilize a starting solution

is the trust-region method. We provide details of the trust-region method in Chapter 4.

2.3 Sample Average Approximation

The sample average approximation (SAA) method is an effective way to approximately solve

stochastic programs involving a large number of scenarios [61, 42]. The SAA method uses

a sampling technique to obtain a sampled instance and then uses the value of the solution

from the sampled instance to approximate the value of the original instance. Consider the

two-stage stochastic program (2.1). A sample (ξ1, ..., ξN ) of N realizations of the vector ξ

is generated, and problem (2.1) is approximated by solving the following sample average

approximation problem

zN = minx∈X

cT x +

1N

N∑

n=1

Q(x, ξn)

. (2.13)

24


2.3.1 Lower Bound Estimate

Denote the optimal value of formulation (2.1) by z∗. Then it is well known that [45, 35]

E[zN ] ≤ z∗.

This means that zN obtained from (2.13) provides a valid statistical lower bound for the

true optimal value z∗. An estimate of E[zN ], denoted by L, can be computed by solving M

independent SAA problems, and then taking the average of zjN , for j = 1, ..., M .

L =1M

M∑

j=1

zjN .

An approximate 100(1− α)% confidence interval of E[zN ] can be constructed as

[L− k · sL√

M, L +

k · sL√M

],

where k is the value taken from Student’s t-distribution at 100(1 − α/2) percentile with

M − 1 degrees of freedom, and sL is the standard error of L computed by

sL =

1

M − 1

M∑

j=1

(zjN − L)2

1/2

.

2.3.2 Upper Bound Estimate

For any feasible solution x to the stochastic program (2.1), we have f(x) ≥ z∗. Therefore,

evaluating the objective function of (2.1) for any x ∈ X provides an upper bound on z∗.

However, evaluating the objective function f(x) exactly in the case that K is large may be

impossible. Rather, f(x) can be estimated for any x. A near-optimal solution x is required to

obtain a good estimate of the upper bound. For a given near-optimal solution x, the estimate

of the upper bound U can be computed as follows. Generate T independent samples of N

realizations of the vector ξ. For each sample j ∈ 1, ..., T , compute the sample average

estimate of f(x), defined as

25


fN (x) = cT x +1N

N∑

n=1

Q(x, ξn).

Then an estimate of the upper bound is

U =1T

T∑

j=1

f jN

(x).

An approximate 100(1− α)% confidence interval of f(x) can be constructed as

[U − k · sU√

T, U +

k · sU√T

],

where k is the value taken from Student’s t-distribution at 100(1−α/2) percentile with T−1

degrees of freedom, and sU is the standard error of U computed by

sU =

1

T − 1

T∑

j=1

(f jN− U)2

1/2

.

2.3.3 Bias of Sample Average Estimates

The SAA method also introduces bias into the estimate because a subset of samples is drawn

from the set of all possible scenarios. In this context, the bias is the difference between the

true objective value and the expected objective value of the estimate. From the previous

sections, the bias of the lower bound is defined as

z∗ − E[zN ]

or

z∗ − L

by using our estimate of E[zN ]. Similarly, the bias of the upper bound is then

E[fN (x)]− z∗

26

2.4. BOOTSTRAP METHOD

or

U − z∗

by using our estimate of f(x).

Convergence of Optimal Solutions

Shapiro and Homem-de-Mello provided detailed convergence analysis in [61] and [57].

Assuming that the two-stage stochastic linear program has a unique solution x∗ and the

SAA samples are iid, the solution of the SAA problem converges to x∗ as the sample size

increases at an exponential rate. More specifically, let xN be the optimal solution obtained

by solving SAA problem (2.13) of size N . Then the probability of the event xN = x∗tends to 1 exponentially fast as N → ∞. Shapiro also provided a formula to estimate the

sample size required for achieving a given precision [57]. However, the formula given is

considered to be too conservative for practical use. In addition, the practical sample size

required for obtaining a good solution is often problem-dependent.

2.4 Bootstrap Method

The bootstrap method is a nonparametric approach to statistical inference based on resam-

pling from the observations. The concept of the bootstrap was first introduced by Efron

[25] in 1979, and further developed by Efron and Tibshirani [26]. The most common use

of the bootstrap method is to estimate the standard error, confidence interval, or bias for a

statistic.

One of the biggest advantages of bootstrapping is that it can be used to estimate a

statistic without knowing the underlying distribution. Unlike the traditional parametric ap-

proaches, which make statistical inferences by making an assumption about the underlying

distribution, the bootstrap method uses the plug-in principle which allows the estimation of

parameters for a statistic from the sample itself.

The principle of the bootstrap method is to estimate the statistical parameters from the

27

2.4. BOOTSTRAP METHOD

existing samples by resampling repeatedly. A resampled sample is called a bootstrap sample,

which is drawn with replacement by using the same sample size as that of the original. This

means that the bootstrap sample has the same number of elements but may not have exactly

the same elements as the original sample. After obtaining the bootstrap samples, the same

procedure used to calculate the statistic for the original sample is applied to the bootstrap

samples to obtain an estimate of the statistic. Resampling is usually done by using the

Monte Carlo sampling method.

To illustrate the bootstrap method, we present the generic procedure for the bootstrap

method, which is taken from [47] with minor modifications. Consider the case where a

random sample of size n is drawn from an unknown probability distribution F . The basic

steps of the bootstrap procedure are the following:

Step 1. Construct an empirical probability distribution, F (x) from the sample x by placing

a probability of 1/n at each element, x1, x2, .., xn. This gives the nonparametric

maximum likelihood estimate of the population distribution F .

Step 2. From the empirical distribution function F (x), draw a random sample of size n

with replacement.

Stop 3. Calculate the statistic of interest θ from this random sample, yielding θ∗b.

Step 4. Repeat steps 2 and 3 B times, where B is a large number. The practical size of

B depends on the tests to be run on the data. Typically, B should be 50-200 to

estimate the standard error of θ, and at least 1,000 to estimate confidence intervals

around θ(Efron & Tibshirani, 1986, sec. 9).

Step 5. Construct a probability distribution from the B number of θ∗b’s by placing a prob-

ability of 1/B at each point, θ∗1, θ∗2, .., θ∗B. This distribution is the bootstrapped

estimate of the sampling distribution of θ, F ∗(θ∗), which can be used to make in-

ferences about θ.

In Chapter 5, we will use bootstrapping to estimate the bias of the lower bound z∗ −E[zN ] in sample average approximation to stochastic programs.

28

2.5. GRID COMPUTING ENVIRONMENTS AND SOFTWARE TOOLS

2.5 Grid Computing Environments and Software Tools

In this section, we describe the computational environment, as well as the software tools

used for our implementation and experiments. There are two computational grids in the

Center for Optimization Research @ Lehigh (COR@L). The most frequently used one is the

Shark, which made up of roughly 40 processors with Intel-based processors running a Linux

operating system. We used Shark for implementation and for small experiments. Another

recently established grid is Polyps, which is composed of 48 dual-core CPUs. Polyps is also

an Intel/Linux based cluster, and it is used for more intensive computation.

There are also larger grids at Lehigh University. One is a dedicated Beowulf Cluster

with more than 400 core processors. This cluster is comprised of two sub-clusters, named

Inferno and Blaze. Inferno is an Intel Xeon-based cluster and Blaze is a AMD Opteron-based

cluster. Both are running the Linux operating system. Another large cluster is a Windows-

based cluster, which is composed of over 800 machines around the campus. Due to the

intensive use of these machines in the Lehigh community, we only use them for our largest

experiments.

The software packages included in our parallel implementation and experiments are

the Condor job management program, the MW master-worker run-time library, the stochas-

tic programming library SUTIL, and the Asynchronous Trust Region(ATR) algorithm. We

provide a short explanation of this software in the following subsections.

2.5.1 Condor

Condor is a job management system for High-Throughput Computing [24, 63]. It al-

lows jobs to run on a collection of distributed computers by providing mechanisms for

job scheduling and resource management. Users are able to submit jobs and specify the

job requirements to the Condor pool. Condor then manages the jobs by scheduling and

matching them to the machines based on job priority and a given policy.

The computers that make up Condor pools are often controlled by individuals, and

Condor allows the owners to specify their own policies for the running of jobs. By default,

29


the policy is to schedule jobs on a computer only when it is idle. Further, jobs running

on a machine must be immediately suspended or preempted if the owner of the machine

reclaims the machine.

Condor also provides a job migration and checkpointing mechanism by which jobs can

be migrated to other idle processors when the owners reclaim their computers. The check-

pointing mechanism allows jobs to resume execution, which is very useful for jobs running

on a dynamic grid environment.

2.5.2 MW

MW is a C++ library encapsulating a master-worker framework that makes it easier to

implement parallel applications running in a Condor environment. MW is implemented

via three main abstract classes, which are the MWDriver, MWTask, and MWWorker. The

MWDriver class contains the code for the master process that controls and distributes tasks

to the worker processors. The MWTask is the code that carries the data and results of

the tasks. The MWWorker class contains code to execute the tasks. In order to use MW,

one needs to implement only a few functions for each class, and MW will handle all the

other necessary processes automatically. Those processes include communication between

processors, acquiring worker processors, suspending tasks, and resuming tasks.

MW also provides statistics for benchmarking. We often report these results to indicate

the effectiveness of the algorithms we build with MW. For a set W of workers that are

involved in a computation, MW collects the following statistics for each worker j in W :

• uj: Total wall time that worker j was available for computation

• cj: Total CPU time that worker j was executing tasks

• sj: Total wall time that worker j was suspended

• bj: Benchmark factor for worker j

After the computation completes, MW computes a summary of the statistics. Some useful

statistics are described below:

30


• Number of Workers(|W |) is the number of different worker processors involved in

the computation.

• Wall Clock Time(T ) is the wall clock time used for the entire computation.

• Average Number Present Workers is the average number of workers that were avail-

able for computation, which is computed by∑

j∈W uj/T .

• Average Number Active Workers is the average number of workers that were exe-

cuting tasks, which is computed by∑

j∈W cj/T .

• Equivalent Run Time is the total equivalent running time used by all workers, which

is computed by∑

j∈W cjbj . Note that the running times are adjusted by a benchmark

factor bj for each worker j.

• Overall Parallel Performance measures the percentage of utilization of worker pro-

cessors. The overall parallel performance is computed by∑

j∈W cj/∑

j∈W (uj − sj).

Additional details for MW are in the user’s manual [64] and in [31, 39].

2.5.3 SUTIL

SUTIL [16] is a stochastic programming library written in C and C++. SUTIL provides util-

ities for generating and manipulating stochastic programming instances by reading prob-

lem information from SMPS files [11]. Problem instances can be created by using the full

scenario tree or by sampling from the underlying distribution of the random parameters.

Several sampling methods are provided within the SUTIL. Those methods are Monte Carlo

sampling, Latin Hypercube sampling [46], and Antithetic Variates sampling [38]. SUTIL

also provides functionality for passing stochastic information between processors, making

it easier for parallel implementation.

2.5.4 ATR

The Asynchronous Trust Region(ATR) method is a parallel decomposition method for so-

lution of two-stage stochastic programs [40]. The algorithm is implemented with MW

31

2.6. PERFORMANCE PROFILES

and SUTIL. The implementation of the ATR algorithm is based on the L-shaped cluster-

ing method as we discussed in the previous section. In addition to the L-shaped method,

the ATR method is regularized using a trust-region. The trust-region is enforced by using

a linear constraint. The ATR method also includes an asynchronous capability, which al-

lows the algorithm to start a new iteration before the evaluation of the previous iteration

is completed. The asynchronous mechanism is useful when running on the computational

grid environment where the speed of worker processors are not consistent. Running asyn-

chronously allows evaluation of several candidates at the same time and therefore reduces

the idle time of the worker processors.

The asynchronous mechanism is achieved by using a basket. A basket is simply a queue

that hold the points xj for which the function f(xj) will be evaluated. The size of the basket

is defined by user. During the computation, “tasks” will be assigned to worker processors to

evaluate the solutions of the subproblems. When the basket is not fully occupied and a user-

defined number of tasks are completed, the algorithm triggers a signal to generate a new

task. Once a task is completed, the point associated with the task will be removed from the

basket. The amount of synchronicity can be specified via a synchronicity parameter, which

is the fraction of the tasks that needs to be completed before putting a new point into the

basket. If the parameter is set to 1, then the algorithm is running in a synchronous mode.

2.6 Performance Profiles

Performance profiles are often used for comparing the performance of solvers on a given set

of instances [23]. It provides a graphical representation instead of the traditional tabular

forms, making it easier for a reader to understand the relative performance of different

solvers. Each curve in the performance profile plot corresponds to the performance of a

solver. To be more specific, given a set of solvers S on a set of instances P , we define τp,s as

the performance measure of a solver s on a instance p. The relative performance of a solver

32

2.7. APPLICATIONS OF TWO-STAGE STOCHASTIC PROGRAMMING

s with respect to other solvers on a instance p is given by the performance ratio

rp,s =τp,s

minτp,s′ : s′ ∈ S .

The performance profile of the solver s is defined as

ρs(t) =1|P |sizep ∈ P : rp,s ≤ t,

where t ≥ 1. The function ρs(t) is the empirical probability that solver s will have perfor-

mance ratio rp,s within a factor t of the best possible ratio, or in other words, the cumulative

distribution function of the performance ratio.

2.7 Applications of Two-stage Stochastic Programming

In this section, we provide examples of two-stage stochastic linear programs. These ex-

amples are used as test instances in our experiments. The problem set includes many

snip instances from our previous study [33] as well as other instances from literatures.

The instances semi, dcap, and sslp are parts of the Stochastic Integer Programming Test

Problem Library(SIPLIB) [1]. We relax the integrality of these instances as well as modi-

fied the stochastic parameters so that they are all independent, giving the instances more

possible scenarios. The instances electricity, product, biweekly, and weekly are taken from

Topaloglu [65]. The instances 4node, AIRL, assets, chem, phone, and stocfor2 are taken from

Ariyawansa and Felt [4]. The dimension of the instances are given in Table 2.1 and the

short description of the instances are provided as follow.

snip is a stochastic network interdiction problem [33]. The problem can be thought of as

a game between two players. The goal of the first player is to choose which arcs to be

destroyed in order to minimize the expected flow of the second player, while the sec-

ond player maximizes his flow, given that he knows which arcs have been destroyed.

The problem is also subject to a budget constraint, where there is a maximum number

33


of arcs that can be destroyed. The success of an interdiction is stochastic. Instances of

different network sizes are included in our test set.

SSN is a telecommunications network planning problem discussed in Sen, Doverspike, and

Cosares [60]. The goal is to add capacity to an existing network in order to minimize

the expected amount of unserved demand. The problem also has a budget constraint,

and the capacity to be added to point-to-point connections must be determined before

the actual demand is known.

Storm is a cargo flight scheduling problem that appeared in Mulvey and Ruszczynski [48].

The problem is to schedule flights in order to serve uncertain demand at a minimum

cost. This problem is a two-stage stochastic program in which the flight schedules

must be determined in the first-stage and the cargo delivery paths can be determined

in the second-stage after the true demand is known.

LandS is a electricity investment problem that appeared in Louveaux and Smeers [43].

The problem is to determine the optimal capacity investment of four power plants to

meet uncertain demands, subject to a budget constraint and minimum total capacity

constraints. This problem is a two-stage stochastic program in which the capacity of

the power plants must be determined in the first-stage and different operation modes

of the power plants can be operated according to the demands.

gbd is an aircraft allocation problem [42] originally described by Dantzig [18](chapter 28).

The problem is to assign different types of aircrafts to routes in order to minimize the

expected cost and unserved demand. There is a bound on the number of available

aircraft for each type, and the demand is uncertain with known distribution.

20term is a motor freight carrier’s operations problem described in [44]. This problem

is a two-stage stochastic program in which the position of a fleet of vehicles at the

beginning of a day must be determined in the first-stage and the movement of the

fleet on the network can be adjusted in the second-stage. There is a penalty for the

34


unsatisfied point-to-point demands, and the fleet must return to the same starting

position at the end of the day.

dcap is a dynamic capacity acquisition and allocation problem [2]. The problem’s objec-

tive is to determine a capacity acquisition plan in order to minimize the expected cost,

given that the assignment costs and processing requirements are uncertain. The orig-

inal problem is a two-stage integer program with mixed-integer first-stage variables

and pure binary second-stage variables. There are 4 different sizes of this problem

and each has 3 different numbers of scenarios, for a total 12 instances in this problem

class.

sslp is a stochastic server location problems described in [51]. The problem objective is to

locate the servers at some potential locations in order to minimize the total cost and

maximize the expected revenue, subject to the customer demand constraint. In this

problem, the presence of the client is uncertain. The original problem is a two-stage

integer program with pure binary first-stage variables and mixed-binary second-stage

variables. There are total 10 instances in this problem class of different sizes and

number of scenarios.

semi is a capacity planning problem in the semiconductor manufacturing industry with

stochastic demand [27]. The goal is to minimize the expected unserved demand

subject to capacity and budget constraints. The original problem has mixed-integer

first-stage variables and continuous second-stage variables. There are three instances

in this problem class with different number of scenarios.

electricity is a two-stage stochastic electricity planning problem. The first-stage decisions

are to assign the power capacities to each terminal subject to the available resources.

The second-stage decisions are to determine how much power to supply at each de-

mand location, given that the demands are stochastic. There are three instances of

different sizes in this problem class. The mathematical formulation is given in [65].

product is a product distribution problem where products need to be shipped from plants

35


to warehouses before knowing the actual customer demands. There are three in-

stances of different sizes in this problem class. The mathematical formulation is given

in [65].

biweekly is a freight carrier planning problem over a two-week horizon. This problem is to

find the number of vehicles at each terminal and the movement of vehicles between

terminals at each period, subject to the demand of loads that need to be carried

between terminals. The loads for the first week are known, where the loads for the

second week are stochastic. There are three instances of different sizes in this problem

class. The mathematical formulation is given in [65].

weekly is a freight carrier planning problem over a one-week horizon. This problem is

similar to the problem biweekly except that the planning horizon is a one week cycle.

There are three instances of different sizes in this problem class. The problem is

originally from Mak, Morton and Wood [44]. The mathematical formulation is given

in [65].

AIRL is a airlift operations scheduling problem [4], which is belong to the same family as

gbd.

4node is a cargo network scheduling problem [4], which is belong to the same family as

Storm.

assets is an asset management problem [4]. This problem is formulated as a network

problem, where the nodes represent the categories of asset and the arcs represent the

transactions. The objective is to maximize the expected return.

chem is a batch chemical plants design problem [4]. The problem is to decide how many

chemical plants to build in order to serve the stochastic demand. This problem is a LP

relaxation of the original problem.

phone is a telecommunication network planning problem [4]. Given an amount of budget,

the goal is to expand the network in order to minimize the unserved future demand.

36


This instance is very similar to SSN.

PGP is a power generation problem [43]. The objective is to decide the capacities of the

power generations in order to meet the future demand, which is stochastic.

stocfor2 is a two-stage case of a multi-stage forest planning problem [4]. The objective is

to determine which area of the forest should be harvested in order to maximize the

total yield within the overall time horizons. The problem involves uncertainty because

there is some likelihood that the left-standing trees will be destroyed.

The problem dimensions of the instances are given in Table 2.1. The problem dimensions

are listed by the number of rows and columns for the first-stage and second-stage problems.

The total number of scenarios for each instance is also given in the table. In Chapter 5, we

give estimated optimal solution values for each instance.

37


Inst

ance

Scen

ario

sFi

rst-

Stag

eSe

con

d-St

age

Inst

ance

Scen

ario

sFi

rst-

Stag

eSe

con

d-St

age

20te

rm1.

10E+

123,

6312

4,76

4gb

d64

6425

4,17

5,10

4nod

e10

2410

2414

,52

74,1

86ph

one

3276

81,

823

,85

4nod

e16

384

1638

414

,52

74,1

86pr

oduc

tlg

3.27

E+20

475

,150

070

0,14

504n

ode

3276

832

768

14,5

274

,186

prod

uct

md

1.21

E+10

010

,250

250,

1400

4nod

e40

9640

9614

,52

74,1

86pr

oduc

tsm

1.21

E+10

010

,50

220,

250

4nod

e81

9281

9214

,52

74,1

86se

mi2

8.19

E+03

2,61

258

42,9

802

AIR

L125

2,4

6,8

sem

i31.

59E+

062,

612

5842

,980

2A

IRL2

252,

46,

8se

mi4

6.71

E+07

2,61

258

42,9

802

Land

S1.

00E+

062,

47,

12sn

ip10

x10a

3.69

E+19

1,20

060

1,50

2PG

P114

42,

47,

16sn

ip10

x10b

3.69

E+19

1,20

060

1,50

2PG

P36

2,4

7,16

snip

10x1

0c3.

69E+

191,

200

601,

502

asse

tslg

3750

05,

135,

13sn

ip20

x20

1.45

E+76

1,80

024

01,2

002

asse

tssm

100

5,13

5,13

snip

30x3

02.

03E+

177

1,18

0054

01,4

502

biw

eekl

ylg

2.55

E+67

100,

3180

280,

3280

snip

4x9

1.68

E+07

1,67

202,

172

biw

eekl

ym

d1.

24E+

124

50,1

250

300,

1300

snip

7x5

4.19

E+06

1,72

217,

181

biw

eekl

ysm

1.01

E+77

20,5

2014

0,54

0ss

lp10

5010

01.

00E+

100

1,10

60,5

10ch

em2

38,3

940

,41

sslp

1050

1000

1.00

E+15

01,

1060

,510

dcap

233

200

2.62

E+41

6,12

15,2

7ss

lp10

5020

001.

13E+

165

1,10

60,5

10dc

ap23

330

03.

87E+

446,

1215

,27

sslp

1050

508.

88E+

841,

1060

,510

dcap

233

500

3.81

E+48

6,12

15,2

7ss

lp10

5050

08.

88E+

134

1,10

60,5

10dc

ap24

320

01.

68E+

556,

1218

,36

sslp

1545

101.

00E+

451,

1560

,690

dcap

243

300

2.82

E+59

6,12

18,3

6ss

lp15

4515

8.40

E+52

1,15

60,6

90dc

ap24

350

05.

96E+

646,

1218

,36

sslp

1545

52.

84E+

311,

1560

,690

dcap

332

200

2.62

E+41

6,12

12,2

4ss

lp5

2510

01.

00E+

501,

530

,130

dcap

332

300

3.87

E+44

6,12

12,2

4ss

lp5

2550

2.98

E+42

1,5

30,1

30dc

ap33

250

03.

81E+

486,

1212

,24

ssn

1.02

E+70

1,89

175,

706

dcap

342

200

1.68

E+55

6,12

14,3

2st

ocfo

r264

15,1

510

2,96

dcap

342

300

2.82

E+59

6,12

14,3

2st

orm

6.02

E+81

185,

121

528,

1259

dcap

342

500

5.96

E+64

6,12

14,3

2w

eekl

ylg

4.03

E+12

03,

300

800,

7200

elec

tric

ity

lg1.

34E+

102

25,5

020

0,15

00w

eekl

ym

d1.

83E+

123

3,12

044

0,32

40el

ectr

icit

ym

d3.

31E+

475,

5010

0,15

00w

eekl

ysm

8.54

E+11

33,

6032

0,19

20el

ectr

icit

ysm

3.31

E+47

10,2

070

,200

Tabl

e2.

1:Te

stIn

stan

ceD

imen

sion

s

38

Chapter 3

Warm-starting and Dynamic Cut

Aggregation in the L-Shaped Method

The L-shaped decomposition method is a well-known method for solution of large-scale

stochastic programs. For many practical instances, however, the number of scenarios can

be extremely large, and solving the problems exactly using the decomposition method is

still intractable. A more practical approach is to use sampling method encouraged with the

decomposition method [42], which provides an approximation of the objective value to the

original problem. In this chapter, we provide ideas based on sampling and decomposition

method to more accurately solve large-scale stochastic programs.

3.1 Motivation

When solving problems using sample average approximation with different sample sizes,

we often get solutions from small sampled instances that are very “close” to the solutions

of the large sampled instances, perhaps even identical [42]. In practice, we make use of

this information for solution of large sampled instances by providing starting solutions from

small sampled instances.

A good starting solution may speed up the computation time significantly, but there is

still no way to prove that the provided solution is near optimal until we obtain a reasonable

39

3.2. WARM-STARTING METHODS

optimality gap (the difference between the function value f(xj) for a given point xj and

the function value z obtained by (2.5)). In solution of minimization problems using a

decomposition method, an upper bound is provided by evaluating a feasible solution, and

a lower bound is provided by solving the master problem with optimality cuts. A good

starting solution may provide a near-optimal upper bound, but the algorithm could still

take many iterations to raise the lower bound. Although we can use the solution from a

small instance as a starting point to solve a large instance, the cuts from the small instance

may not be valid for the large instance because they are generated by using a different set

of scenarios.

Another difficulty in decomposition method is managing the cuts during the computa-

tion. Although the L-shaped clustering method [41] can vary the aggregation level of the

cuts by using different cluster sizes, the cluster size must be specified before the computa-

tion and it is problem-dependent. A low level of cut aggregation causes the number of cuts

to grow rapidly in the master problem, resulting in a more difficult master problem. A high

level of cut aggregation causes loss of information, resulting in an algorithm that requires

more iterations.

The main discussions in this chapter provide ideas that overcome the above difficulties.

In the following sections, we present warm-starting methods, called the pre-sampling and

scenario-partitioning methods, which make use of the cuts from solutions of small sampled

instances to solve a much larger sampled instance. The scenario partitioning method also

utilizes parallel computing to solve a large number of partitioned problems. In addition, we

provide more efficient cut manipulation methods for the L-shaped decomposition method,

which help solve larger instances faster.

3.2 Warm-starting Methods

3.2.1 Pre-sampling Method

The basic idea of the pre-sampling method is to use information obtained from solution of

a small sampled instance to solve a much larger sampled instance. More specifically, the

40


scenarios used to create a small instance are drawn uniformly without replacement from

the set of scenarios of a large instance. We denote the small instance as the pre-sampled

instance and the large instance as the sampled instance. This method provides two benefits.

First, the solution of the pre-sampled instance could be a better starting point for solving

the large sampled instance than a solution from an arbitrary small instance. Second, the

cuts from the pre-sampled instance can be adjusted to become valid for the large sampled

instance, as we demonstrate below.

First, consider an sampled instance of size |N |:

(SAA(N )) minx

cT x +1|N |

|N |∑

n=1

Q(x, ξn)

s.t. Ax = b

x ≥ 0

(3.1a)

and

Q(x, ξn) = miny

q(ξn)T y

s.t. T (ξn)x + W (ξn)y = h(ξn)

y ≥ 0

(3.1b)

where N is the set of sampled scenarios, and Q(x, ξn) is the value of optimal solution to

the second-stage problem associated with scenario n at first-stage solution x. For a subset

of scenarios η ⊆ N , we obtain a pre-sampled instance SAA(η). We typically choose |η| to

be small enough so that we can solve SAA(η) with the extensive form using standard LP

solver, whereas |N | is typically large and SAA(N ) must be solved by using decomposition

method.

The dual solution π to SAA(η) can be used to construct inequalities for the master

problem when solving SAA(N ) by a decomposition approach.

Theorem 3.1. Let πk be the optimal dual multipliers of the constraints T (ξk)x + W (ξk)y =

41


h(ξk) associated with scenario k obtained by solving SAA(η), the cut

θk ≥ 1|N |π

Tk (hk − Tkx)

is valid for SAA(N ).

Proof. Let Q(x, ξk) be the value of the second-stage problem associated with scenario k for

any x. From Section 2.2.2 or [34], we can construct a valid optimality cut to SAA(η) as

θ′k ≥1|η|Q(x, ξk) ≥ 1

|η|πk(hk − Tkx).

Because scenario k ∈ η is taken from N where |N | is larger than |η|, the contribution of

Q(x, ξk) to the overall objective value in SAA(η) is larger than the contribution of Q(x, ξk)

in SAA(N ). Multiplying both sides of the above inequality by a constant |η||N | , we have

|η||N |θ

′k ≥

1|N |Q(x, ξk) ≥ 1

|N |πk(hk − Tkx).

Substitute |η||N |θ

′k by θk, we have

θk ≥ 1|N |Q(x, ξk) ≥ 1

|N |πTk (hk − Tkx).

Because SAA(N ) has the same second-stage problem Q(x, ξk) as in SAA(η), by θk ≥1|N |Q(x, ξk), the cut θk ≥ 1

|N |πTk (hk − Tkx) is valid for SAA(N ).

3.2.2 Scenario-Partitioning Method

A natural extension of the pre-sampling method is to solve many pre-sampled instances

to obtain more cuts. The scenario-partitioning method partitions the set of scenarios in

N into many subsets and then generates small instances associated with those subsets in

order to solve them separately using the extensive formulation (2.2). We denote the small

instances from partitioning the set of scenarios of SAA(N ) as the partitioned instances, e.g.,

N = η1 ∪ η2 ∪ ... ∪ ηP , and the partitioned instances SAA(ηj) are solved for all j = 1, ..., P .

42

3.3. DYNAMIC CUT MANAGEMENT

The solutions and cuts obtained from the partitioned instances are then used to warm-start

for solution of SAA(N ) using the decomposition method. One advantage of the scenario-

partitioning method is that solution of those partitioned instances is naturally parallelizable,

and the sizes of the partitioned instances can be chosen so that they are small enough to be

solved using a standard LP solver by the extensive form.

A starting solution x0 for solving SAA(N ) can be generated by using a convex com-

bination of solutions from all the partitioned instances. Let x∗j be the optimal solution of

SAA(ηj), ∀j = 1, ..., P , then x0 is computed by

x0 =

P∑

j=1

|ηj |x∗j

|N | .

It is also possible to chose a good solution x∗j from j = 1, .., P by evaluating the function

f(x∗j ), but this approach requires extra computational effort.

In theory, one could obtain |N | cuts from solving all the partitioned instances, one from

each scenario, of the form

θk ≥ 1|N |π

Tk (hk − Tkx) ∀k ∈ N .

In practice however, |N | can be very large, and adding all the cuts obtained from the parti-

tioned instances may not be practical. A more appropriate way is to aggregate cuts before

solving SAA(N ). A simple aggregation method is to aggregate all the cuts from each par-

titioned instance into a single cut to obtain P cuts. The aggregated cut for a partitioned

instance is then

θj ≥ |ηj ||N |

|ηj |∑

k=1

πTk (hk − Tkx) ∀j ∈ 1, ..., P.

3.3 Dynamic Cut Management

When solving stochastic programs with a decomposition method, the master problem con-

tinuously gets cuts from the subproblems and becomes progressively larger and harder to

43


Table 3.1: Average statistics on different numbers of clustersDescription Number of Clusters

50 100 250 500 1000 2500 5000Number of Workers 9.20 8.60 7.90 8.20 7.20 7.20 8.70Wall Clock Time 316.64 311.68 337.01 257.37 268.11 326.95 481.30Average Number Present Workers 3.21 2.96 2.38 2.56 2.70 3.24 4.73Average Number Active Workers 2.83 2.56 1.98 2.03 1.76 1.05 0.53Equivalent Run Time 6257.44 5379.98 4219.51 3639.29 3277.57 2416.56 1885.28Overall Parallel Performance 0.88 0.87 0.85 0.81 0.68 0.34 0.13Number of Iterations 37.00 31.70 25.40 22.50 19.80 14.90 11.80

solve. The idea of dynamic cut management is to manage the cuts within the master prob-

lem adaptively to achieve better overall performance. Cut management can be done by

aggregating cuts or removing unused cuts as necessary. At the beginning of the computa-

tion, because the master problem has a small number of cuts, we add more cuts in order to

get a good lower bound quickly. At later stages of the computation, we reduce the number

of cuts by aggregating them in order to prevent the master problem from growing too large.

We could also consider the possibility of removing some cuts from the previous iterations.

3.3.1 Justification

In the L-shaped clustering algorithm described in Section 2.2.2, the number of clusters has

an important influence on the computation time and it could be problem-dependent. In

general, if the number of clusters is large, many cuts will be added to the master problem

at each iteration. Solving the master problem at each iteration will then be more difficult.

On the other hand, when using a small number of clusters, more cuts are aggregated, and

the algorithm will need more iterations to be terminated.

We conducted experiments to illustrate how the number of clusters can affect the com-

putational performance. The experiments were done with the ATR method as described

in Section 2.5.4 by using 10 available processors on a distributed computational grid. The

instance snip7x5 with a sample size of 5000 was chosen as an example. We varied the num-

bers of clusters from 50 to 5000. In order to reduce the uncertainties of running experiments

in the grid environment, we repeated each experiment 10 times to obtain the average, and

we used the same set of samples for the experiments for each different numbers of clusters.

44


Table 3.1 shows the results of using different numbers of clusters for the different statis-

tics described in Section 2.5.2. As one can see, the best wall clock time was obtained with

500 clusters. As the number of clusters increases, the number of iterations reduces, as does

the parallel performance. In addition, the total work load also decreases. These results

indicate that using a small number of clusters increases the total work load, but benefits

from a higher parallel performance. On the other hand, using a large number of clusters

reduces the total work load, and also reduces the parallel performance.

Note that an optimal number of clusters is difficult to determine. It depends on many

factors, such as the problem being solved, the number of available workers, and the speed

of the processors being used. For example, if we have a very powerful master processor, we

would prefer using a larger number of clusters because the master processor could handle

many cuts yet still take a relatively short time to solve the master problem. On the other

hand, if the master processor is very slow, we would prefer a smaller number of clusters by

running more iterations in order to take advantage of the worker processors.

Changing the cluster size dynamically during the course of the algorithm is one idea

that has been attempted in the early stages of this thesis. The key idea of the dynamic

cluster sizes approach is to vary the number of cuts during the computation. Despite its

difficulty in terms of implementation, the dynamic cluster sizes approach did not seem to

provide promising results. In the successive sections, we provide a different approach to

handle cuts without changing the cluster size.

3.3.2 Cut Management

In order to adjust the number of cuts in the master problem, we need to know when to

aggregate the cuts and when to remove the cuts. A simple idea is to aggregate and remove

more cuts as the parallel performance of the algorithm drops. In our implementation, we

use cut aggregation and cut removal to vary the number of cuts in the master problem at

each iteration. In this section, we introduce a system for measuring the performance of the

algorithm in order to adjust the cut aggregation and cut removal parameters for a good

45


ATR

Cut aggregation /removal level

Algorithm Efficiency

Cut Manager

Performance Measure

Computational time (master/workers) Initial setup

Figure 3.1: Cut management feedback loop diagram

overall performance.

This idea can be illustrated by using a feedback loop diagram as shown in Figure 3.1. At

the beginning of the algorithm, the cut aggregation and removal parameters are initialized

to default values. At each iteration, the computation time for solving the master problem

and subproblems is measured. The algorithm’s efficiency is then calculated, and the cut

manager makes an adjustment to the cut aggregation and removal parameters in order to

determine the number of cuts to be added and removed.

We use the parallel utilization as our performance measure because when using our

parallel decomposition algorithm to solve a stochastic program, the solution time of the

master problem should be relatively small compared to the total solution time of the sub-

problems in order to gain high parallel utilization. We introduce a statistic ek to measure

the utilization of the algorithm at iteration k by

ek =c + Wk/nk

c + Mk + Wk/nk,

where Mk is the time for solution of the master problem at iteration k, Wk is the total

time for solution of subproblems at iteration k, nk is the number of worker processors

available at iteration k, and c is a time constant. The constant c is added to the equation

to maintain a high value of ek if Wk and Mk were relatively smaller than c. In general, the

total solution time of the subproblems Wk is fairly constant at each iteration because the

size and the number of subproblems remain unchanged throughout the algorithm, but the

time for solution of the master problem Mk usually increases as cuts are accumulated over

46


the iterations. However, the time for solution of the master problem is very unpredictable.

Also, the number of workers nk may change from iteration to iteration. Therefore, the value

of ek calculated at each iteration is not very consistent. In order to avoid this, we maintain

an overall utilization Ek and use it as our control variable instead of using ek directly. The

overall utilization Ek is updated at each iteration according to a weight ω ∈ [0, 1]. The

updating formula is given as

Ek = ωEk−1 + (1− ω)ek. (3.2)

The value of Ek is used to adjust the cut aggregation and removal parameters, which we

will discuss in the following sections.

3.3.3 Cut Aggregation

Cut aggregation aims to reduce the total number of cuts in the master problem. Cut aggre-

gation can be performed in many different ways. For example, the cuts may be aggregated

at the workers before sending them back to the master processor. Worker cut aggregation

has been already used in our existing ATR algorithm. In this section, we consider aggregat-

ing cuts at the master processor.

In theory, one could vary the number of cuts as well as the combination of cuts nearly

continuously from iteration to iteration. In our experience, using only a small number of

aggregation levels, including an extreme case in which all the cuts from a same iteration

are aggregated into one cut is better. The cut aggregation level is used to determine how

many cuts need to be aggregated. At each level of aggregation, we use a fixed combination

of cuts in order to have more consistent results. In our implementation, we only aggregate

cuts that come from the same iteration, which makes it easier to remove cuts in the later

stages while keeping a convergent algorithm.

We use two approaches for the cut aggregation. The first approach is to aggregate cuts

from previous iterations that have been already added to the master problem. The main

purpose of this approach is to reduce the number of cuts in the master problem. We refer to

this as the cut clean-up process. In this approach, we search for all the cuts that came from

47


the same iteration and aggregate them into a single cut. Once a cut is aggregated, it will

be removed from the master problem unless it is still active. In the second approach, we

aggregate cuts from the worker processors before adding them to the master problem. This

approach aims to reduce the cut growth rate in the master problem when we feel that there

are too many cuts are added at each iteration. In this approach, we also consider which

level of aggregation should be used based on the performance measure. We refer to this as

multi-level cut aggregation. We discuss in detail of these two approaches as follows.

Cut Clean-up Process

In the cut clean-up process, a flag CleanUp is used to indicate whether or not we should

clean up the cuts. Recall Ek from (3.2) as the overall utilization updated at iteration k, the

flag CleanUp is set according to

CleanUp =

False if Ek ≥ 0.8,

T rue otherwise.

The flag CleanUp can be set or reset many times during the computation. Therefore, we

keep a record of the latest iteration number for which we have already done the clean-up

process. When the flag CleanUp is set, we look to aggregate the cuts that came from the

iteration number beyond the current record up until iteration k − δ, where k is the current

iteration and δ is a constant. We use δ = 5 in our implementation. This means that cuts

from the latest δ iterations remain untouched. The reason is that the newest cuts contain

the most important information, which is essential for finding a good candidate solution.

Suppose at iteration k, the flag CleanUp is set, and the cuts from iteration j < k need

to be cleaned up. Each cut from iteration j is of the form

θi ≥ piπTi (hi − Tix), i = 1, ..., N, (3.3)

where N is the total number of cuts obtained at iteration j. The cut clean-up process

48


aggregates all the cuts from iteration j into a single cut as

N∑

i=1

θi ≥N∑

i=1

[piπTi (hi − Tix)]. (3.4)

Specifically, all inactive cuts (3.3) are removed from the master problem (2.10) and the

aggregated form of the inequality (3.4) is added.

Note that we skip an attempt to clean up cuts for a particular iteration if the cuts have

been already aggregated by the multi-level cut aggregation method that will be discussed

next. This is to prevent over-aggregating cuts, which may result in too much information

loss. However, the remaining cuts may be removed later if they meet the removal criteria,

which will also be discussed in the successive section.

Multi-level Cut Aggregation

In the multi-level cut aggregation, we first define L as the aggregation level, where L can

take integer values 0, 1, 2, or 3. At the beginning of the algorithm, L is set to 0, and the

value of L is updated during computation. At iteration k, Lk is calculated according to the

following criteria:

Lk =

0 , if Lk−1 = 1 and Ek ≥ 0.8,

1 , if Lk−1 = 0 and Ek < 0.6; or Lk−1 = 2 and Ek ≥ 0.6,

2 , if Lk−1 = 1 and Ek < 0.4; or Lk−1 = 3 and Ek ≥ 0.4,

3 , if Lk−1 = 2 and Ek < 0.2.

Recall that our implementation is based on the L-shaped clustering method as discussed

in Section 2.2.2, in which we use a fixed cluster size. Each cluster is associated with a cut

that is returned from a worker. At each aggregation level, a fixed number NL is used to

determine the maximum number of cuts that can be aggregated into a single cut. NL is

chosen to be 1, 10, 100, and N for the aggregation level 0, 1, 2, and 3 respectively, where

N is the total number of clusters. Note that at level 0, none of the cuts are aggregated; and

at level 3, all the cuts are aggregated into a single cut.

49


Because of the asynchronicity, it is possible that the newly obtained cuts are not coming

from the same iteration. In such case, we aggregate cuts separately for each iteration that

the cuts came from. Considering only cuts from a single iteration, we aggregate cuts for

a given aggregation level L as follows. First, we sort the cuts according to the indices of

the clusters. With an associated value of NL, we partition the cuts uniformly into chunks

C1, C2, ..., CdN/NLe according to the sorted indices. Note that the number of cuts in a chunk

can be less than NL if some cuts were missing.

Suppose that Cj is not empty, and the cuts within Cj are of the form

θi ≥ piπTi (hi − Tix), ∀i ∈ Cj ,

then the cut aggregation for chunk j is performed by

∑

i∈Cj

θi ≥∑

i∈Cj

[piπTi (hi − Tix)].

3.3.4 Cut Removal

During the computation of the L-shaped method, cuts are constantly added at each iteration

and accumulated in the master problem. This causes the master problem size to grow and

become harder to solve. In fact, cuts may stay inactive for a long periods of time and they

do not make any contribution to the solution process. The idea of cut removal is to remove

those cuts in order to reduce the master problem size.

In order to remove cuts, we need to keep a record, for each cut in the master problem,

of where the cut came from and how many iterations it has been inactive. For every cut, we

keep a record of its iteration index, its parent solution index, and the number of successive

iterations that the cut remains inactive.

Let rk be the cut-removal age at iteration k, which is the maximum number of successive

iterations that a cut is allowed to stay inactive. A cut may be deleted if all the following

criteria have been met:

1. The cut is not generated at the current iteration.

50

3.4. COMPUTATIONAL EXPERIMENTS

2. The parent solution of the cut is not the current incumbent solution.

3. The cut stays inactive successively for more than rk iterations at the time of iteration k.

Since cuts obtained from the current incumbent solution are never deleted, the algorithm

is still convergent.

We use the overall utilization Ek, as discussed in Section 3.3.2, to determine the cut-

removal age rk. At iteration k, rk is calculated according to

rk = max(EkR, rmin),

where rmin is the minimum allowance of rk, and R is a user defined integer number. We

typically use a value of rmin = 5 and R = 50 or 100 in our experiments. This mechanism

allows varying the length of time that a cut could stay inactive before it is deleted. If the

efficiency is low, then cuts will be removed more often and vise versa.

3.4 Computational Experiments

In this section, we describe the results of two experiments, one for the warm-starting meth-

ods described in Section 3.2 and another for the cut management systems described in

Section 3.3. The results are reported in the separated subsections.

3.4.1 Computational Results for Warm-starting

In this experiment, we tested our pre-sampling method and the scenario-partitioning method

simultaneously. We also tested the use of different solutions as the starting point for the ATR

algorithm. The results are compared to the result of using the ordinary ATR method without

warm-starting.

We used a Monte Carlo(uniform random) sampling method to generate the sampled

instances for our test instances, and then solved each sampled instance with the ATR algo-

rithm. For each sampled instance, we use a partition size of 200, which means that we need

51


to solve 200 partitioned instances for each sampled instance for the warm-starting. The sce-

narios are partitioned evenly so that each partition contains 5% of the total number of sce-

narios. The results of these 200 partitioned instances are used for our scenario-partitioning

method, where the test of the pre-sampling method is carried out by taking only the result

of the first partitioned instance.

The partitioned instances are solved by using SOLVE-DE, which is a program that allows

us to generate and solve the extensive form of a sampling version of a test instance. This

program is implemented in C++ using our SUTIL library and COIN-CLP LP solver. The

program provides a solution and cuts information as the result of warm-starting, which will

be used for the ATR algorithm.

Our results are provided in table form as well as performance profile plots. These results

are obtained from an average of ten trials. We tested our experiment with six different

settings. For each setting, we used the same set of samples to eliminate the sampling error.

In the first setting, we solved the sampled instances by using the ordinary ATR method.

The next three settings were performed by providing a starting point in addition to the

first setting. These three different starting points are marked as HqSol, AvgSol, and 1stSol.

HqSol used a high quality solution which was obtained from solution of a sampled instance

of same sample size with a different set of samples, AvgSol used an average solution from

solutions of 200 partitioned instances, and 1stSol used the solution from the first partitioned

instance. In the fifth setting, we used the scenario-partitioning method, using the average

solution as a starting point and using all the cuts from the partitioned instances. In the last

setting, we use only the solution and cuts from the first partitioned instance as test for our

pre-sampling method.

Table 3.2 shows the average percentage from optimality for the function evaluation with

different starting solutions. The results are summarized in the third, fourth, and last col-

umn respectively. These numbers are the percentage of the difference between the objective

value at the first iteration and the final optimal value for all three different starting solu-

tions. If a starting solution is of high quality, then the difference between the evaluated

value and the final optimal value will be small. In many instances, the quality of solutions

52


Table 3.2: Average percentage from optimality for different starting solutionsInstance N HqSol(%) AvgSol(%) 1stSol(%)20term 10000 0.04 0.00 0.034node 32768 10000 0.00 1.08 2.67AIRL2 20000 0.00 0.02 0.03assets sm 20000 0.00 0.00 0.00biweekly lg 2000 0.02 0.49 1.45electricity lg 2000 0.08 1.11 4.16gbd 20000 0.00 0.06 0.23LandS 20000 0.00 0.00 0.03PGP1 20000 7.70 0.14 0.84phone 20000 0.00 0.00 0.00product sm 2000 0.00 0.11 0.57semi4 1000 0.24 177.83 259.06snip4x9 10000 0.11 1.88 3.33snip7x5 10000 0.14 1.33 2.06ssn 5000 2.50 19.09 76.61stocfor2 20000 0.00 0.00 0.07storm 10000 0.00 0.00 0.00weekly lg 2000 0.05 2.45 7.32weekly md 2000 0.00 5.31 8.21

are very high even those from the first partitioned instances. These results indicate that a

good solution is easy to obtain, even when using a small sample size. However, this is not

always the case, as we can easily see from the instances semi4 and ssn. The results from

the average solutions are slightly better than the results from the first partitioned instance.

In general, a high quality solution provides the best results, but it requires solving a large

sampled instance in order to obtain such solution.

Table 3.3 provides the optimality gap after warm-starting using the scenario-partitioning

method. These results show that our scenario-partitioning method provides a very tight

bound in most instances even before the ATR algorithm is undertaken. As a result, the

computation time and number of iterations are significantly reduced(see Table 3.5 and

Table 3.6).

Table 3.4 provides the computation time for solution of the instances with and without

warm-starting as well as the warm-starting time. The scenario-partitioning method is used

to warm-start. Column three and column four in Table 3.4 are the average wall clock time

53


Table 3.3: Average optimality gaps after warm-startingInstance N UB LB OptGap(%)20term 10000 254370.3 254290.4 0.034node 32768 10000 451.7 446.7 1.11AIRL2 20000 269680.5 269569.4 0.04assets sm 20000 -723.9 -723.9 0.00biweekly lg 2000 -7184.1 -7358.4 2.41electricity lg 2000 -1804.0 -1905.6 5.57gbd 20000 1654.2 1651.3 0.17LandS 20000 225.7 225.6 0.03PGP1 20000 439.6 436.7 0.66phone 20000 36.9 36.9 0.00product sm 2000 -17923.6 -18012.0 0.49semi4 1000 314.5 90.9 197.57snip4x9 10000 10.8 9.8 9.41snip7x5 10000 81.3 77.5 4.74ssn 5000 11.5 1.9 99.50stocfor2 20000 -39772.2 -39806.4 0.09storm 10000 15498030.0 15497399.0 0.00weekly lg 2000 -2442.2 -2658.1 8.62weekly md 2000 -912.5 -1070.1 16.37

for solution of each instance without warm-starting and with warm-starting, respectively.

We can see that the computation time is reduced significantly by performing warm-starting,

in most instances. The last column in this table is the warm-starting time, the average

solution time of the 200 partitioned instances. Note that these results are obtained by

running the experiment on a computational grid with 54 processors. The computation time

required for warm-starting, by solving all 200 partitioned instances in parallel on the same

computational grid, is relatively small compared to the solution time. In most instances,

even if we could perform warm-starting using a single machine, the total time would still

less than solving the sampled instance without warm-starting.

Figure 3.2 shows the performance comparison of six different settings. The performance

profile plots for the wall clock time is given in Figure 3.2 (a) and for the number of iterations

is given in Figure 3.2 (b). Numerical results are also given in Table 3.5 and Table 3.6,

respectively. The results provide very similar outcomes in both the wall clock time and

the number of iterations. That is, the performance using a high quality solution are about

54


Table 3.4: Decomposition time and warm-starting time in secondsInstance N Time w.o.WS Time w. WS WS Time(each)20term 10000 1628.3 403.6 0.834node 32768 10000 273.8 137.9 0.83AIRL2 20000 129.4 84.9 0.02assets sm 20000 96.6 37.3 0.04biweekly lg 2000 2061.8 652.6 1.59electricity lg 2000 729.4 140.9 0.31gbd 20000 188.8 144.3 0.02LandS 20000 138.8 97.0 0.01PGP1 20000 174.1 149.2 0.02phone 20000 149.5 108.4 0.23product sm 2000 497.5 110.1 0.32semi4 1000 1863.3 1088.4 1.79snip4x9 10000 445.4 184.6 0.29snip7x5 10000 267.9 147.7 0.40ssn 5000 606.2 253.1 0.37stocfor2 20000 430.2 94.1 0.36storm 10000 486.7 207.4 3.16weekly lg 2000 1056.3 331.8 0.36weekly md 2000 604.5 307.9 0.18

the same as of using an average solution, and the performance using a solution from the

first partitioned instance with and without using cuts are about the same. The scenario-

partitioning method(Avg+Cut in Figure 3.2) has the best performance and the ordinary

ATR method has the worst performance. These results suggest that there is no significant

advantage in using a high quality solution over an average solution, despite the better

results obtained in Table 3.2. In addition, the pre-sampling method(1st+Cut in Figure

3.2) performs only slightly better than using the solution from the first partitioned instance

without providing cuts. However, the scenario-partitioning method performs very well, as it

stands out of the group. The main reason is because it is able to use all the cut information

from all the partitioned instances, which provides a much tighter lower bound than the

other settings.

55


1 2 3 4 5 6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of wall clock time using partitioning method

ATRHQAvg1stAvg+Cut1st+Cut

(a) Wall clock time

1 1.5 2 2.5 3 3.5 4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of number of iterations using partitioning method

ATRHQAvg1stAvg+Cut1st+Cut

(b) Number of iterations

Figure 3.2: Performance profile of wall clock time and number of iterations for warm-starting

56


Table 3.5: Average wall clock time for warm-startingWall clock time(secs)

Instance N ATR HqSol AvgSol 1stSol AvgSol+Cut 1stSol+Cut20term 10000 1628.3 440.2 353.0 1078.9 403.6 1066.84node 32768 10000 273.8 96.8 102.9 101.7 137.9 103.2AIRL2 20000 129.4 92.6 121.0 153.6 84.9 104.9assets sm 20000 96.6 37.0 39.0 40.3 37.3 40.6biweekly lg 2000 2061.8 6863.8 1112.4 1365.9 652.6 1341.4electricity lg 2000 729.4 216.9 141.5 453.9 140.9 467.6gbd 20000 188.8 218.8 174.2 188.2 144.3 154.6LandS 20000 138.8 98.6 94.0 114.2 97.0 146.2PGP1 20000 174.1 156.5 118.8 137.9 149.2 148.6phone 20000 149.5 77.0 76.1 140.4 108.4 103.7product sm 2000 497.5 132.0 116.3 264.0 110.1 227.0semi4 1000 1863.3 1473.3 1651.5 1654.7 1088.4 1694.3snip4x9 10000 445.4 274.3 216.7 261.0 184.6 289.3snip7x5 10000 267.9 179.2 246.7 197.1 147.7 193.3ssn 5000 606.2 424.8 345.4 336.6 253.1 434.3stocfor2 20000 430.2 90.0 155.8 246.9 94.1 161.8storm 10000 486.7 268.6 342.6 460.7 207.4 321.1weekly lg 2000 1056.3 538.8 532.6 726.8 331.8 750.5weekly md 2000 604.5 453.6 426.9 547.4 307.9 434.5

Table 3.6: Average number of iterations for warm-startingNumber of iterations

Instance N ATR HqSol AvgSol 1stSol AvgSol+Cut 1stSol+Cut20term 10000 279.3 46.8 45.4 146.1 43.0 150.64node 32768 10000 69.6 18.9 20.7 21.2 23.1 21.8AIRL2 20000 31.7 15.0 20.4 20.5 10.3 20.3assets sm 20000 21.0 2.0 2.0 2.0 2.0 2.0biweekly lg 2000 67.8 37.8 37.0 47.8 30.3 44.3electricity lg 2000 81.3 43.3 26.8 60.8 26.0 61.5gbd 20000 45.0 24.8 30.0 29.1 19.4 29.3LandS 20000 44.3 20.1 19.2 29.2 10.9 30.0PGP1 20000 30.8 24.4 21.6 24.6 14.8 25.1phone 20000 9.0 2.0 2.0 2.0 2.0 2.0product sm 2000 65.5 24.3 22.3 40.0 20.3 39.5semi4 1000 187.0 123.6 173.5 167.1 97.6 166.4snip4x9 10000 58.9 57.4 52.6 57.9 32.3 57.1snip7x5 10000 47.8 46.2 44.5 46.9 31.4 47.1ssn 5000 76.9 48.5 46.6 53.1 44.9 54.6stocfor2 20000 100.7 7.6 8.4 16.0 8.4 16.3storm 10000 51.6 24.6 26.3 30.5 16.2 30.7weekly lg 2000 69.2 49.5 41.5 53.0 30.5 53.6weekly md 2000 61.3 47.5 45.5 52.8 31.3 49.8

57


Varying the Size of Partitions

In this experiment, we considered changing the size of the partitioned instances for the

scenario-partitioning method. By increasing the size of the partitioned instances, the cuts

obtained from solving these instances were tighter and the number of partitioned instances

needed to be solved was reduced. However, each partitioned instance becomes harder.

Three instances were tested in this experiment - the ones with the largest optimality gaps

from our previous results, namely, semi4, snip7x5, and ssn. The results are provided in Table

3.7 for each of the instances. The first column in these tables are the size of the partitioned

instances. The average warm-starting time for solution of each partitioned instance is given

in the second column. The solution time and the number of iterations of the ATR method

are given in the third and the fourth columns respectively.

Table 3.7: Average solution time and number of iterations by varying the size of the parti-tioned instances

SAA(ηj) Size WS Time(sec.) Solve Time(sec.) Iteration1 0.27 736.99 80.22 0.55 746.92 76.64 1.28 713.05 76.98 3.25 722.62 72.3

16 8.30 692.32 66.2(a) semi4 with N = 800

SAA(ηj) Size WS Time(sec.) Solve Time(sec.) Iteration25 0.13 195.36 28.350 0.40 205.86 28.2

100 1.28 183.95 29.9200 4.30 171.75 27.9400 12.56 154.55 25.3

(b) snip7x5 with N = 10,000

SAA(ηj) Size WS Time(sec.) Solve Time(sec.) Iteration25 0.38 351.33 47.150 0.93 389.95 44.9

100 2.65 303.89 42.8250 10.01 324.86 41.4500 32.74 281.05 42.7

(c) ssn with N = 5,000

58


From the result of these three instances, we can see that the solution time and number

of iterations of the ATR method only reduced slightly as the size of the partitioned instances

increases. However, despite to the reduction of the number of partitioned instances need to

be solved when increasing the partition size, the warm-starting time increases substantially.

This results suggest that, small partition size is more preferable for scenario partitioning.

3.4.2 Computational Results for Dynamic Cut Management

In this experiment, we tested the cut management systems using 26 instances from our test

set. We used the SAA approach and solved each sampled instance with the ATR algorithm.

As we know that the number of clusters has a large influence on the overall performance,

we set up our experiment with three different numbers of clusters. The number of clusters is

rated as large, medium, and small, with an associated size between 125 to 2500, 25 to 500,

and 5 to 100 respectively. Note that the scale is based on the difficulties of the instances. For

each number of clusters, three different settings were used, which labeled as Non, Clean,

and All. The Non setting was to solve the sampled instances using the ordinary ATR method

without aggregating and removing cuts. In the Clean setting, cuts were aggregated by only

using the cut clean up process as discussed in Section 3.3.3. In the All setting, we used cut

clean up, multi-level cut aggregation, and cut removal in our ATR algorithm. This gives a

total of nine combined settings for the experiment. For each setting of different instances,

10 sampled instances were solved and an average of the results are reported.

The performance profile plots for the large, medium, and small number of clusters are

given in Figure 3.3 to 3.5. In these plots, we compared the performance of the wall clock

time and the total time for solution of the master problem. With a large number of clusters,

Figure 3.3 shows that the non-aggregated version performed poorly compared to the other

settings in both plots. In contrast, the All setting performs slightly better than the Clean

setting. The results of using a medium number of clusters are similar to the results of

using a large number of clusters, as shown in Figure 3.4, although the differences are less

significant for different settings. With a small number of clusters, the performance is more

59


1 2 3 4 5 6 7

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of wall clock time for large clusters

NonCleanAll

2 4 6 8 10 12

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of master problem time for large clusters

NonCleanAll

(a) Wall clock time (b) Master problem time

Figure 3.3: Performance profile for cut management with large number of clusters

1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of wall clock time for medium clusters

NonCleanAll

1 2 3 4 5 6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of master problem time for medium clusters

NonCleanAll


Figure 3.4: Performance profile for cut management with medium number of clusters

evenly matched(see Figure 3.5). In this case, the All setting still performs better in most

instances, but the Clean version performs slightly worse than the Non version. The main

reason is that with a small number of clusters, the master problem can be solved relatively

fast, which makes cut aggregation less necessary. In addition, the overuse of cut aggregation

could result in increasing the number of iterations. In contrast, the Clean setting uses a more

aggressive approach by aggregating all cuts from the same iteration into one cut, where the

All setting is made more gentle by using different levels of cut aggregations.

Note that we do not compare the number of iterations in this experiment because ag-

gregating cuts and removing cuts causes the algorithm to take more iterations. However,

numerical results are given in Table 3.8 - Table 3.10.

60


1 1.2 1.4 1.6 1.8 2

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of wall clock time for small clusters

NonCleanAll

2 4 6 8 10 12 14

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of master problem time for small clusters

NonCleanAll


Figure 3.5: Performance profile for cut management with small number of clusters

61


Tabl

e3.

8:A

vera

gest

atis

tics

oncu

tag

greg

atio

nfo

rla

rge

num

ber

ofcl

uste

rsW

allC

lock

Tim

e(se

cs.)

Mas

ter

Solu

tion

Tim

e(se

cs.)

Nu

mbe

rof

Iter

atio

ns

Inst

ance

NC

lust

ers

Non

Cle

anA

llN

onC

lean

All

Non

Cle

anA

ll20

term

1000

500

2530

.850

0.9

710.

124

61.3

407.

960

9.6

97.2

123.

918

1.0

4nod

e32

768

1000

025

0012

21.8

832.

271

8.1

1044

.847

1.9

417.

944

.748

.668

.6A

IRL1

1000

025

0025

7.4

218.

720

9.3

170.

376

.299

.923

.822

.328

.2as

sets

lg10

000

2500

225.

922

0.7

186.

113

9.4

88.8

76.8

22.2

22.6

22.8

biw

eekl

ysm

5000

1250

432.

742

7.3

246.

230

5.5

248.

112

5.0

19.9

19.8

20.5

chem

1000

025

0032

2.3

225.

822

5.2

227.

110

3.4

83.5

24.4

23.2

24.3

dcap

233

500

1000

025

0028

0.5

227.

420

4.7

98.1

69.8

71.6

16.6

16.3

17.7

dcap

243

200

1000

025

0012

7.4

159.

416

6.1

32.7

31.4

31.5

10.5

10.6

10.7

dcap

332

300

1000

012

5018

6.0

200.

823

0.3

51.0

38.9

42.0

20.8

22.2

22.2

elec

tric

ity

lg20

0050

011

35.6

369.

839

2.0

977.

820

7.4

227.

512

8.6

142.

413

8.5

elec

tric

ity

md

5000

1250

691.

629

6.9

360.

852

9.7

150.

319

7.8

44.5

41.8

44.5

gbd

1000

025

0085

1.0

482.

233

1.9

758.

434

7.6

217.

121

.425

.438

.8La

ndS

1000

025

0037

2.7

256.

119

8.8

258.

013

3.0

102.

720

.422

.330

.7ph

one

1000

025

0010

9.7

72.4

70.4

0.8

0.8

0.8

6.0

6.0

6.0

prod

uct

md

2000

500

2330

.151

5.9

512.

021

78.5

347.

037

7.2

99.2

94.7

102.

4pr

oduc

tsm

2000

500

390.

819

4.2

214.

729

4.1

96.3

94.7

58.1

59.9

57.3

snip

10x1

0c10

0050

031

91.0

844.

654

7.6

3102

.776

9.9

480.

657

.086

.112

5.0

snip

4x9

5000

1250

1790

.465

4.7

429.

817

02.4

550.

931

4.5

31.6

39.6

55.2

snip

7x5

5000

2500

1670

.293

2.6

470.

615

76.4

779.

638

0.9

20.5

23.8

45.7

sslp

1050

2000

2000

1000

1772

.052

5.7

515.

216

49.0

361.

538

4.2

62.9

75.4

98.9

sslp

525

100

5000

1250

1006

.538

2.4

413.

382

4.4

234.

227

3.3

52.7

55.7

70.2

ssn

5000

1250

3270

.714

36.2

1118

.530

19.8

1095

.976

9.5

42.6

51.0

98.2

stoc

for2

5000

1250

1815

.349

0.7

455.

516

67.3

283.

322

7.2

98.4

99.0

97.4

stor

m50

0025

0022

58.0

926.

161

4.0

2072

.063

6.0

385.

025

.729

.944

.4w

eekl

ylg

1000

250

4086

.994

3.3

996.

438

73.7

541.

057

7.4

167.

523

2.9

248.

3w

eekl

ysm

500

125

3574

.083

1.9

762.

034

80.2

655.

854

6.8

310.

042

3.8

464.

8

62


Tabl

e3.

9:A

vera

gest

atis

tics

oncu

tag

greg

atio

nfo

rm

ediu

mnu

mbe

rof

clus

ters

Wal

lClo

ckTi

me(

secs

.)M

aste

rSo

luti

onTi

me(

secs

.)N

um

ber

ofIt

erat

ion

sIn

stan

ceN

Clu

ster

sN

onC

lean

All

Non

Cle

anA

llN

onC

lean

All

20te

rm10

0010

080

1.7

351.

633

9.2

711.

522

0.7

184.

221

0.3

281.

824

0.5

4nod

e32

768

1000

050

031

3.4

356.

037

2.7

132.

210

7.4

116.

357

.768

.667

.9A

IRL1

1000

050

010

0.7

124.

810

0.0

23.0

20.1

21.1

34.3

33.0

33.3

asse

tslg

1000

050

088

.298

.188

.68.

07.

48.

222

.422

.822

.5bi

wee

kly

sm50

0025

013

5.1

125.

814

1.5

28.0

24.5

29.5

21.6

22.7

25.0

chem

1000

050

099

.710

7.2

102.

717

.316

.416

.933

.032

.032

.7dc

ap23

350

010

000

500

129.

619

3.4

162.

99.

59.

28.

620

.020

.218

.8dc

ap24

320

010

000

500

168.

725

8.8

138.

43.

23.

23.

315

.015

.015

.0dc

ap33

230

010

000

250

173.

819

2.8

190.

37.

27.

27.

327

.327

.527

.6el

ectr

icit

ylg

2000

100

267.

021

1.9

224.

155

.148

.733

.710

9.7

103.

396

.2el

ectr

icit

ym

d50

0025

015

3.4

203.

518

9.3

22.1

27.8

26.0

39.1

44.2

41.2

gbd

1000

050

018

8.9

146.

916

9.1

103.

247

.445

.932

.035

.135

.0La

ndS

1000

050

012

3.4

110.

711

6.1

41.4

24.9

24.8

28.7

28.4

28.9

phon

e10

000

500

72.1

71.3

99.1

0.1

0.1

0.1

6.2

6.0

6.2

prod

uct

md

2000

100

280.

232

1.1

298.

113

9.6

79.1

81.9

94.8

90.4

94.6

prod

uct

sm20

0010

014

5.8

177.

021

0.4

11.0

12.2

12.8

46.4

48.6

49.7

snip

10x1

0c10

0010

072

3.2

271.

024

8.9

650.

916

3.9

158.

411

0.4

144.

114

2.8

snip

4x9

5000

250

271.

920

5.5

178.

217

6.6

70.0

69.4

46.4

52.5

52.8

snip

7x5

5000

500

341.

519

7.7

236.

123

1.7

102.

511

3.5

31.5

40.8

41.0

sslp

1050

2000

2000

200

534.

125

9.9

311.

039

0.5

109.

199

.712

7.1

142.

313

2.9

sslp

525

100

5000

250

217.

718

2.0

250.

391

.847

.149

.872

.874

.572

.5ss

n50

0025

017

15.3

725.

169

7.8

1347

.232

2.6

295.

372

.190

.395

.4st

ocfo

r250

0025

028

8.4

294.

624

8.9

133.

085

.266

.411

4.9

113.

410

7.4

stor

m50

0050

054

5.0

436.

033

9.9

259.

110

6.7

102.

637

.741

.040

.8w

eekl

ylg

1000

5077

8.9

641.

155

6.8

428.

718

2.7

135.

123

1.6

237.

924

4.8

wee

kly

sm50

025

736.

548

9.4

344.

659

5.5

275.

898

.747

0.0

471.

651

8.3

63


Tabl

e3.

10:

Ave

rage

stat

isti

cson

cut

aggr

egat

ion

for

smal

lnum

ber

ofcl

uste

rsW

allC

lock

Tim

e(se

cs.)

Mas

ter

Solu

tion

Tim

e(se

cs.)

Nu

mbe

rof

Iter

atio

ns

Inst

ance

NC

lust

ers

Non

Cle

anA

llN

onC

lean

All

Non

Cle

anA

ll20

term

1000

2047

9.0

416.

125

1.0

281.

720

9.7

32.5

459.

546

7.3

344.

74n

ode

3276

810

000

100

249.

926

6.9

231.

017

.519

.314

.876

.378

.474

.6A

IRL1

1000

010

090

.411

8.9

91.0

1.4

1.5

1.4

39.1

39.7

39.0

asse

tslg

1000

010

012

3.1

111.

397

.80.

40.

40.

522

.823

.022

.9bi

wee

kly

sm50

0050

84.6

136.

013

2.1

3.6

4.2

3.7

22.8

25.8

23.8

chem

1000

010

011

6.9

88.1

87.7

1.1

1.0

1.1

34.0

33.1

33.1

dcap

233

500

1000

010

020

2.2

224.

924

9.8

0.5

0.5

0.4

20.3

20.7

20.0

dcap

243

200

1000

010

021

2.5

208.

421

3.0

0.3

0.2

0.3

15.4

15.3

15.3

dcap

332

300

1000

050

178.

617

6.8

177.

90.

50.

50.

531

.331

.231

.7el

ectr

icit

ylg

2000

2017

4.7

173.

515

3.2

3.2

3.1

2.8

77.8

75.1

76.0

elec

tric

ity

md

5000

5016

4.5

206.

320

3.3

2.2

1.9

1.8

44.7

41.9

41.7

gbd

1000

010

013

6.5

135.

918

7.7

6.9

6.9

7.5

41.6

43.1

44.9

Land

S10

000

100

118.

018

8.7

126.

53.

23.

73.

041

.345

.941

.6ph

one

1000

010

087

.811

2.4

91.4

0.0

0.0

0.0

8.0

8.0

8.0

prod

uct

md

2000

2014

5.6

164.

213

5.5

7.0

7.3

6.7

63.4

64.9

61.9

prod

uct

sm20

0020

155.

015

0.3

127.

61.

01.

00.

947

.647

.244

.1sn

ip10

x10c

1000

2026

9.4

275.

022

7.7

153.

712

4.7

64.9

195.

620

8.6

209.

1sn

ip4x

950

0050

214.

118

4.2

190.

719

.117

.517

.064

.962

.364

.3sn

ip7x

550

0010

017

0.8

151.

819

9.7

30.7

27.6

26.7

47.4

46.8

47.1

sslp

1050

2000

2000

4036

8.2

244.

226

0.8

50.3

49.7

14.8

187.

118

9.6

189.

6ss

lp5

2510

050

0050

183.

120

2.7

200.

87.

77.

65.

190

.790

.590

.8ss

n50

0050

1259

.791

2.8

888.

467

0.2

223.

822

0.5

161.

316

1.7

161.

7st

ocfo

r250

0050

237.

224

0.1

225.

97.

57.

24.

410

6.2

104.

310

5.1

stor

m50

0010

034

1.4

371.

433

9.9

29.0

30.4

29.0

49.0

50.2

49.3

wee

kly

lg10

0010

466.

046

8.7

430.

547

.949

.016

.823

5.8

237.

622

4.0

wee

kly

sm50

05

255.

423

9.1

218.

649

.142

.29.

241

5.9

382.

537

1.5

64


We also provide comparison of the wall clock time for all different settings with different

numbers of clusters. The performance profile plots are given in Figure 3.6. As a summary,

the algorithm performs better when using cut aggregation and cut removal methods. In

addition, these results support our justification in Section 3.3.1 that either too small or too

large a number of clusters will result in increasing the total computation time, although

the differences are not significant between the small and medium numbers of clusters. The

reasons for the small differences are that many tested instances are very easy to solve,

and the sample size of the instances may be too small. Therefore, the computation time

is less distinguishable when using small and medium numbers of clusters. Also, the cut

aggregation and cut removal methods have less contribution to the overall performance

compared with that when the number of clusters is large. In addition, the proper number

of clusters is unknown and different for each instance. Therefore, it is not always the case

that a good number of clusters was chosen for the medium cases in our experiments.

1 1.5 2 2.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profile of wall clock time

Non−LargeClean−LargeAll−LargeNon−MediumClean−MediumAll−MediumNon−SmallClean−SmallAll−Small

Figure 3.6: Performance profile of wall clock time for all settings

In Figure 3.7 - 3.9, we showed how the algorithm behaves when using cut aggregation

65


and cut removal methods with different numbers of clusters. The results from instance ssn

are used for the demonstration, as ssn is notoriously one of the most difficult instances. We

plotted the time for solution of the master problem and the number of cuts in the master

problem against the iteration number. As we can see from these plots, the time for solution

of the master problem is closely related to the number of cuts, although there is a lot of

variance in these plots, which is more visible in the non-aggregation setting. The time

for solution of the master problem in the non-aggregation setting also has a much higher

magnitude than the other settings. When employing cut clean up or cut aggregation plus

cut removal, the time for solution of the master problem is significantly reduced. In terms

of the number of cuts, we can see from plots (b) in Figure 3.7 - 3.9 that without aggregating

cuts and removing cuts, the number of cuts increases linearly. When using cut aggregation

and cut removal, the number of cuts is significantly reduced. The average number of cuts

also appeared at different levels using different numbers of clusters. The larger the number

of clusters, the more cuts in the master problem. The pattern of the plots also appears

differently with different numbers of clusters. In the case of large numbers of clusters,

because there are too many cuts added at each iteration, cut aggregation and cut removal

are performed more often, resulting in a smoother plot. In the cases of small and medium

numbers of clusters, cut aggregation and cut removal are performed periodically, resulting

in the plots looking like “sawtooth” pattern. The period of the pattern is shorter in the All

setting than in the Clean setting, which seems that the All setting tried to aggregate and

remove cuts more often than the Clean setting, but in a smaller number. It is also easy to

see that the number of iterations increases as we aggregate and remove cuts.

66


0 20 40 60 80 1000

50

100

150

200

250

300

350

Iterate

Sec

onds

Master problem solving time for ssn

NonCleanAll

10 20 30 40 50 60 70 80 90 100

0.5

1

1.5

2

2.5

3

3.5

4

4.5

x 104

Iterate

Num

ber

of c

uts

Number of cuts during computation for ssn

NonCleanAll

(a) Master problem time (b) Number of cuts

Figure 3.7: Master problem solution time and number of cuts in the master problem duringcomputation with large number of clusters

0 20 40 60 80 1000

10

20

30

40

50

60

Iterate

Sec

onds


NonCleanAll

0 20 40 60 80 1000

2000

4000

6000

8000

10000

12000

14000

16000

18000

Iterate

Num

ber

of c

uts


NonCleanAll


Figure 3.8: Master problem solution time and number of cuts in the master problem duringcomputation with medium number of clusters

0 20 40 60 80 100 120 140 160 1800

5

10

15

Iterate

Sec

onds


NonCleanAll

0 20 40 60 80 100 120 140 160 1800

1000

2000

3000

4000

5000

6000

7000

8000

Iterate

Num

ber

of c

uts


NonCleanAll


Figure 3.9: Master problem solution time and number of cuts in the master problem duringcomputation with small number of clusters

67

3.5. CONCLUSIONS

3.5 Conclusions

In this chapter, we proposed warm-starting methods for solution of stochastic programs us-

ing a decomposition method. The main ideas of the warm-starting methods are to provide a

good starting point as well as optimality cuts to tighten the lower bound. The pre-sampling

method solves a small sampled instance of a given instance to obtain a starting solution

and cuts, where the scenario-partitioning method solves all the partitioned instances and

uses the average solution as a starting solution. Our computational results show that both

methods often provide a good starting solution. However, the performance only improved

significantly when using the scenario-partitioning method. This is simply because all the op-

timality cuts from the partitioned instances must be loaded into the master problem before

beginning the decomposition method.

We also discussed managing cuts in the master problem to improve the performance

of the algorithm. Our method monitors the performance of the algorithm and then makes

adjustment by using cut aggregation and cut removal. The computational results show that

the improvement is most evident when using a large number of clusters. Overall, our cut

management system provides better performance than the ordinary ATR method, and it

also provides more consistent results regardless of the number of clusters.

68

Chapter 4

Nonlinear Programming Methods for

Stochastic Programs

Nonlinear programming algorithms are widely applied to solve continuous optimization

problems, where the objective is a nonlinear function of the decision variables. Even though

many stochastic programs also have an objective that is a nonlinear function of the deci-

sion variables, there is very little literature on using nonlinear programming techniques in

stochastic programming. One reason may be that the objective function f(x) is not differen-

tiable in stochastic programming. Another reason may be that in stochastic programming,

the function evaluation requires significant computation due to the large number of scenar-

ios. However, by breaking down the problem into many subproblems, each subproblem can

be evaluated independently. With the arrival of parallel computing, the function evaluation

of a stochastic program has become relatively less expensive. This is the motivation for

considering nonlinear programming techniques for stochastic programming.

In this chapter, we consider solution of large-scale stochastic program by employing

well-known nonlinear programming algorithmic techniques. One simple technique we ex-

amined in Chapter 3 was to speed up the computational process by providing a starting

point, which can be obtained easily from the solution of a small sampled instance. How-

ever, a starting solution only provides a benefit when it is “regularized”. In other words,

69

4.1. TRUST-REGION METHOD

we must limit the search area to prevent taking a step too far from a given (hopefully

near-optimal) point. One idea to regularize the search is to use a trust-region.

The size of the trust-region radius can have a significant impact on the performance of

the algorithm. If the trust-region is too small, the algorithm will require many steps to reach

an optimal point. On the other hand, if the trust-region is too large, steps may be taken

far away from the optimal point, and the algorithm will take many iterations to achieve

a sufficient reduction in the objective value. Therefore, determining a proper trust-region

radius can be an important issue.

Because the function evaluation becomes relatively cheaper in a computational grid

environment, it is a good idea to evaluate more points in order to obtain a better approx-

imation of the recourse function(by obtaining more cuts). The key issue is how we can

obtain a “good” candidate solution without creating too much additional work for the mas-

ter process. The line search method used in nonlinear programming is well-suited in this

circumstance, as the points along the line can be obtained easily.

In this chapter, we present ideas to improve the trust-region method used for solution of

two-stage stochastic programs using various nonlinear programming techniques. In Section

4.1, we discuss methods of adjusting the trust-region including finding an initial trust-region

radius and trust-region scaling. In Section 4.2, we provide three methods of finding a

candidate solution based on the line search method. Finally, computational results are

given in Section 4.3.

4.1 Trust-region Method

The trust-region method [40] that we discuss here is a variant of the L-shaped method

based on l∞ trust-regions, which is an enhanced L-shaped method obtained by adding the

following trust-region constraint to the current master problem (2.10):

−∆e ≤ x− xI ≤ ∆e, (4.1)

70


where ∆ is the trust-region radius, e = (1, 1, ..., 1)T , and xI is the current incumbent.

The implementation of our trust-region algorithm is an ATR version as mentioned in

Section 2.5.4 in Chapter 2. The ATR algorithm is an iterative algorithm. It starts from a

feasible incumbent solution xI which is either provided by the user or obtained by solving

a relaxed master problem. The function evaluation is performed in order to obtain new cut

information. The new cuts are then added to the current master problem. At a given iter-

ation k, a new candidate solution can be obtained by solving the updated master problem

within the current trust-region. A new candidate solution can be either accepted if its eval-

uated objective value was sufficiently improved or rejected if it was not. If the candidate

solution is accepted, then it becomes the new incumbent solution xI . The trust-region is

also updated around the incumbent solution at every iteration, and the process of finding a

new candidate solution repeats. The algorithm stops when it passes the convergence test.

We refer the reader to [40] for more details.

4.1.1 Classical Trust-region Update Method

The trust-region method defines a model function m(x) around the current search point xI

to approximate the function f(x). A trust-region radius ∆ is enforced to keep the search

within the current region in which we “trust” our model function. In the classical trust-

region method, an initial trust-region radius is provided by users. During the computation,

∆ can be reduced, kept constant, or increased according to the ratio of actual to predicted

reduction at iteration k:

ρk =f(xI)− f(xk)f(xI)−m(xk)

=actual reduction of f

predicted model reduction of f,

where k is the current iteration number and xI is the current incumbent solution.

If ρk is small, which means that the model function is not a good predictor, we should

reduce the trust-region radius. On the other hand, if ρk is large, which means that the

model m(x) is good, then we can “trust” the step prediction and we increase the trust-

region radius.

71


A typical trust-region radius update is to set

∆k+1 =

∆k/β1 if ρk < η1,

∆kβ2 if ρk > η2,

∆k otherwise,

(4.2)

where β1, β2 > 1 and η1 < η2 < 1. Typical values for β1, β2, η1, η2 are β1, β2 = 2, η1 = 1/4,

and η2 = 3/4. We refer readers to Dennis and Schnabel [22], and Conn, Gould, and Toint

[14] for more details on the trust-region radius update.

4.1.2 Selection of an Initial Trust-region Radius

Although the trust-region radius can be adjusted throughout the computation, finding a

good initial trust-region radius is still crucial to good overall performance. Too small a

trust-region leads to small improvement in the objective value, whereas too large a trust-

region requires many iterations to build up the model approximation. Typically, an initial

radius is provided by the user who must have knowledge of the problems to be solved. This

is not always the case when solving a new problem, and it requires trial and error.

There are some automatic trust-region radius initialization methods in the nonlinear

programming literature [22, 14, 59]. A simple strategy is to set the initial trust-region

radius to 1.0 or to use a fraction of the gradient norm from the starting point.

We use a similar approach in our implementation. Our method is based on information

obtained from the subgradient at the starting point. Recall from Chapter 2 that a subgradi-

ent g of the expected recourse functionQ(x) can be derived from the optimal dual solutions

of the subproblems (2.1b) as

g = −K∑

k=1

pkTTk πk, (4.3)

where πk is the optimal dual multiplier from solving (2.1b). For an initial solution x0, we

72


propose to initialize the trust-region radius as

∆0 =1n‖g0‖2, (4.4)

where n is the number of variables in x0, and g0 is the subgradient evaluated at x0. In order

to prevent improper sizes of the radius, the initial trust-region radius is also constrained by

a minimum value ∆0min and a maximum value ∆0

max.

4.1.3 Trust-region Scaling

In our previous discussion of the trust-region method, the trust-region can be seen as a

multi-dimensional cube surrounding the current incumbent, where the length of the side of

the cube is set by the trust-region radius. However, different shaped trust-regions can be

used. In this section, we consider trust-region scaling for changing the shape of the trust-

region. Scaling the trust-region can be useful for problems with different sensitivities for

taking steps in different directions, i.e., we may prefer taking a short step in a more sensitive

direction and vice versa.

As mentioned in the beginning of this section, instead of using a trust-region bound such

as (4.1), the scaled trust-region is bounded by using the following constraint:

−∆v ≤ x− xI ≤ ∆v, (4.5)

where ∆ is the trust-region radius, v is the trust-region scaling vector, and xI is the current

incumbent. This makes changing the size and the shape of the trust-region separable: the

trust-region radius is associated with the size of the trust-region, while the scaling vector is

used to control the shape of the trust-region. The trust-region scale is updated after each

major iteration(after a new incumbent has been updated). In updating the trust-region

scale, the vector v is computed based on a given criterion. After that, it is multiplied by a

scalar in order to re-scale the vector so that the maximum value of all components are equal

or less than 1. Note that the trust-region radius(∆) updating procedure remains unchanged.

73


The scaling vector also has a minimum value vmin that each component can take. So each

component j of v falls between 0 < vmin ≤ vj ≤ 1. Note that the value of vmin must be

strictly positive for the algorithm to converge. A “too small” value of vmin may also affect

the convergence of the trust-region algorithm.

We suggest and test three different scaling methods: scaling by using the previous step

size, by using subgradient information, and by using the BFGS method.

Trust-region Scaling by Using the Previous Step Size

In this trust-region scaling method, we scale the dimension of the trust-region based on the

previous steps. A uniform trust-region is provided at the beginning of the algorithm and

continuously updated after each major iteration. If the new incumbent does not take a full

step in any dimension, the scale of the corresponding dimension will be reduced. The scale

is updated in a weighted manner, which allows us to control the sensitivity of the trust-

region shape to individual steps. Putting more weight on the existing scale factors makes

the trust-region less sensitive to individual changes.

At a given iteration k, let xI ∈ Rn be the current incumbent solution where k ≥ I, and

let xk be the solution of the current iteration k. The current trust-region radius is denoted by

∆, and v ∈ Rn is the trust-region scaling vector. The value of v is initially set to (1, 1, ..., 1).

The current candidate solution xk is obtained by solving the updated master problem with

the following constraint enforced.

−∆v ≤ x− xI ≤ ∆v.

If the candidate solution xk was accepted to become a new incumbent, we compute the

scaling vector v for each element j ∈ 1, ..., n as

vk+1j = ωvk

j + (1− ω)|(xI

j − xkj )|

∆vkj

,

where ω ∈ [0, 1] is the weight for the update and n is the number of elements in v or x.

74


*

−g

Figure 4.1: The scaled trust-region based on the subgradient of the current point

After the update, we check whether multiplying v by a scalar is necessary to renormalize

v so that ‖v‖∞ = 1. In addition, the value of any element that is less than vmin will be set

to vmin.

Trust-region Scaling by using Subgradient Information

At each iteration of the ATR algorithm, the subgradient can be computed by (4.3). The

subgradient suggests a descent direction from the current search point. Another trust-region

scaling idea is to use this subgradient information to scale the trust-region directly.

We consider the steepness of the subgradient for determining the scale of the trust

region. That is, the scale of the trust-region in a given direction is inversely proportional to

the absolute value of the subgradient in that direction. The key idea of this scaling technique

is that, if a direction is more sensitive to the change (it is steeper in that direction), then

it is more risky to take a long step and vise versa. Therefore, the trust-region is scaled by

allowing only small steps in steeper directions and large steps in a flatter directions.

Figure 4.1 is a two-dimensional example to illustrate our idea. A descent direction is

given as the vector −g in the figure. The vector shows a steeper direction on the vertical

axes than the horizontal axes, which means that the vertical direction is more sensitive than

75


the horizontal direction. A smaller scale is applied on the vertical axes while a larger scale

is applied on the horizontal axes. The resulting trust-region is a rectangular shape where

the vertical length is shorter than the horizontal length.

Although the trust-region is centered of the current incumbent, the subgradient infor-

mation is obtained from candidate points around the current incumbent. Therefore, the

subgradient information only provides a rough idea of which direction we should go in,

and we should also take account of the information from the previous iterations.

With this intuition, we are ready to precisely define the scaling. At iteration k, where

the current incumbent solution is xI , and a trust-region radius ∆ is given, we define γj for

each element j ∈ 1, ..., n in v as follows:

γj =

1.0 if |gj | ≤ ‖g‖2vmin,

‖g‖2vmin

|gj | otherwise.

The trust-region scaling vector v is updated for each element j as

vk+1j = ωvk

j + (1− ω)γj ,

where ω ∈ [0, 1] is the weight for the update. Larger ω imply that the scaling is more

sensitive to recent subgradient observations.

Trust-region Scaling by Using the BFGS Method

The BFGS method is a well-known method to approximate the Hessian matrix in continuous

differentiable problems [50]. The Hessian matrix provides information on the curvature of

the objective function. However, in non-differentiable problems, the Hessian matrix is not

defined at certain points. Further, for piecewise-polyhedral function like Q(x), the Hessian

matrix takes value 0 when it is defined. However, there is some “global” curvature to

the function Q(x) made by considering all the linear pieces simultaneously. The idea is

to use the subgradient information to construct an approximation Hessian for the original

problem, and use it to control the shape of the trust-region. Figure 4.2 illustrates our idea.

76


Figure 4.2: BFGS approximation

An approximation of the Hessian matrix B can be constructed by using our modified

BFGS method as described in Algorithm 4.1. The matrix B is initially set to I and contin-

uously updated throughout the iterations. Note that we only perform the BFGS update at

each major iterations, i.e., when a candidate solution is accepted. In Algorithm 4.1, gk is the

subgradient of the recourse function evaluated at iteration k, which is obtained by (4.3).

Algorithm 4.1 Modified BFGS Update Algorithm

Step 0. Initialize x0,B0 ⇐ I.

Step 1. Evaluate g0 by solving the subproblems (2.1b) using LP solver.

Step 2. for k = 0, 1, 2, ... doif termination test is satisfied then

STOPend ifSolve the new master problem (2.10) to obtain xk+1

Evaluate gk+1

if xk+1 is accepted thensk ⇐ xk+1 − xk

yk ⇐ gk+1 − gk

Bk+1 ⇐ Bk + ykyTk

yTk sk

− BksksTk Bk

sTk Bksk

end ifend for

77


Figure 4.3: The scaled trust-region based on BFGS approximation

The idea of the trust-region scaling by using the BFGS method is illustrated via a two-

dimensional contour plot in Figure 4.3. The shape of the contour is determined by the

diagonal of the matrix B. A larger element of the diagonal implies a steeper slope in the

associated dimension. In Figure 4.3, the slope on the vertical axes is steeper than the slope

on the horizontal axes. A larger scale is then applied to v1 instead of v2, where v1 and v2

are the elements of the scaling vector v associated with the horizontal axes and the vertical

axes respectively.

The trust-region scale is computed in inverse proportion to the amount of the main

diagonal of matrix B, and it is constantly updated in a weighted manner after performing

the BFGS update. Let λ be the main diagonal of B, then each element j of the trust-region

scaling vector v is updated according to

vk+1j = ωvk

j + (1− ω)γj ,

where ω ∈ [0, 1] is the weight for the update and γj is calculated by

γj =

1.0 if |λj | < vmin‖λ‖∞,

‖λ‖∞|λj |vmin

otherwise.

78

4.2. PARALLEL SEARCH METHODS WITHIN A TRUST-REGION

4.2 Parallel Search Methods within a Trust-region

One of the biggest concerns for the decomposition method is that the master problem will

become larger and larger as more and more cuts are added during the computation. The

number of cuts added at each iteration can be large, depending on the number of subprob-

lems and how the cuts are aggregated. As a result, solution of the master problem (2.5) for

a new candidate solution can be very time-consuming and can significantly reduce the par-

allel utilization since many workers remain idle waiting for a new candidate solution. The

search methods discussed in this section could potentially solve this problem by evaluating

several candidate solutions in parallel.

As we generate more points to evaluate, we also increase the number of tasks need to

be performed in parallel. This can be handled in our parallel computational environment

by increasing the number of worker processors. However, evaluating more points does not

always lead to faster discovery of a better solution. Therefore, we must empirically find a

balance to limit the maximum number of points that can be searched simultaneously. If we

cannot find a better solution within a specific number of evaluations, we restart the search

process on a different path, which requires solution of an updated master problem.

Another issue we should consider when employing the parallel search methods we pro-

pose is that the more points we evaluate, the more cuts we obtain. This could cause the

master problem to become too large too quickly, so we should aggressively aggregate cuts

or remove inactive cuts using methods as discussed in Chapter 3. We can also select and

add only a portion of cuts so that the master problem will not become too big.

Our parallel search methods are incorporated into the asynchronous mechanism in the

ATR algorithm (see Section 2.5.4 or [40]), which is a variant of the L-shaped method. The

search procedure replaces Step 1 in the L-shaped algorithm (as described in Sections 2.2.1

and 2.2.2) in solving for a new candidate. In the original ATR algorithm, a trigger signal

was used to start a new candidate, and this signal will be used to start a search procedure.

In the following subsections, we discuss three different search methods, which we call

the line search method, the zigzag search method, and the dogleg search method. The main

79


differences of these methods are the search patterns being used.

4.2.1 Line Search

The basic idea for the line search is to seek an improved solution on the line between a

given candidate solution xk,0 and the current incumbent solution xI by evaluating a series

of points along the line. A parameter α ∈ [0, 1] is used to indicate the relative distance from

xI to xk,0. The value α is initially set to 1.0 and reduced by half after each search. The

points generated by the line search method are given as

xk,l = xI + αl(xk,0 − xI), for l = 0, ...,M− 1, (4.6)

where αl = 1/2l and M is the maximum number of points allowed for a given search

direction. This process allows us to solve the master problem (2.10) only once to obtain

the point xk,0 when performing a line search, while the other points along the line can be

generated easily by (4.6). However, xk,0 − xI may not always provide a descent direction

and continuously searching along this direction may produce a point that is too close to

xI . Therefore, after evaluating M points, if an improved solution has not been obtained,

we restart the line search procedure by solving the updated master problem (2.10). This

process repeats until an improved solution is found. Note that the line search procedure is

also restarted after a new incumbent is updated.

4.2.2 Zigzag Search

The zigzag search method considers a search area formed from three points: the previous

incumbent solution xI−1, the current incumbent solution xI , and a candidate solution xk,0.

The candidate solution xk,0 is obtained by solving the master problem. In the zigzag search

method, the search sequence starts from a candidate point xk,0, and follows a zigzag pattern

as shown in Figure 4.4. The sequence of search points are indexed by numbers starting from

0, and the evaluations of search points are performed in an increasing order of the index.

The sequence of search points approaches the current incumbent solution xI as the indices

80


0

1

2

3

4

5

6

xk,0

xI

xI-1

Figure 4.4: Zigzag Search Pattern

of the search points increase. Note that the points indexed by an odd number are obtained

by using the same method as the line search method, in which points are on the line between

the current incumbent solution xI and the candidate solution xk,0. The points indexed by an

even number are located inside the triangular area, in which points are on the line between

the previous search point and the previous incumbent solution xI−1. In a fashion similar

to the line search method, a parameter α ∈ [0, 1] is used to compute the distance between

consecutive search points. The value of α is initially set to 1.0 and reduced by half each

time a search point has an odd index. The search process restarts when a better solution

has been found or when it reaches a maximum number of search points(M).

The points generated by the zigzag search method are given as

xk,l =

xk,0, for l = 0,

xI + αl(xk,0 − xI), for l = 1, 3, 5, ...,

xk,l−1 + αl−1(xk,l−1 − xI−1), for l = 2, 4, 6, ...,

(4.7)

for l = 0, 1, 2, ...,M− 1. The value of αl is computed by

αl =1

2dl/2e , for l = 1, 3, 5, ...,

81


X I-1

X I

X k,0

XC

Figure 4.5: Dogleg Search

and it is updated only when l is odd.

4.2.3 Dogleg Search

The idea of the dogleg search method is similar to the conventional dogleg method used in

unconstrained optimization [50, 15]. That is, the search points are generated along a path

which is composed of two segments. The main difference between dogleg methods are how

the knots of these segments are obtained. In our dogleg method, the dogleg path is the

segments joining the knot xk,0, xc, and xI as shown in Figure 4.5. Points xk,0 and xI are the

candidate solution and the incumbent solution respectively. The point xc is the centroid of

points xk,0, xI , and xI−1, which is calculated as

xc =xI−1 + xI + xk,0

3,

where xI−1 is the previous incumbent solution.

We start the dogleg search from point xk,0, where the value of parameter α is set to

2.0. By reducing the value of α by half at each step, a sequence of search points xkl, for

l = 0, 1, 2, ...,M− 1, can be generated by

82


xkl =

xI + α(xc − xI) for 0 ≤ α ≤ 1,

xc + (α− 1)(xk,0 − xc) for 1 ≤ α ≤ 2,

where M is the maximum number of search points for a given dogleg path. A new dogleg

search is started if the number of search points is equal to M or if a new incumbent is

obtained.


In this section, we provide computational results for our algorithmic enhancements to the

trust-region method for solution of two-stage stochastic linear programs. First, we show

how a different initial trust-region radius could impact the computational performance as

well as comparing these radii to an automatically selected initial radius. Next, we pro-

vide computational results using different trust-region scaling methods. Finally, we give

computational results using different parallel search methods.

4.3.1 Initial Trust-region Radius

We used 31 different instances from our test set in this experiment, and we solved sampled

versions of these instances. The initial trust-region radius was set by using the value ∆0 =

0.5, 2, 10, 50, and 250, as well as the automatic initialized value obtained by (4.4). The

experiment for each value of ∆0 is repeated for ten trials using different samples, and in

each trial, the same samples were used for each different value of radius. The results are

provided in terms of the average of ten trials. For the automatic initial trust-region radius,

we use the method in Section 4.1.2 with ∆0min = 1.0 and ∆0

max = 250. We also allow the

trust-region radius to be varied during the computation.

The results are provided in Table 4.1. The names of the instances are listed in the

first column, and the second column is the associated sample size. The column AutoR

is the average radius determined by the automatic selection method. The second main

column provides the average computation time of using different initial radii, and the third

83


1 1.5 2 2.5 3 3.5 4 4.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of wall clock time using different initial radius

AutoR=0.5R=2.0R=10R=50R=250

1 1.5 2 2.5 3 3.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of number of iterations using different initial radius

AutoR=0.5R=2.0R=10R=50R=250

(a) Wall Clock Time (b) Number of Iterations

Figure 4.6: Performance profile of wall clock time and number of iterations for differentinitial trust-region radius

main column provides the average number of iterations. The same results are provided in

performance profile plots in Figure 4.6 (a) for the average wall clock time and in Figure 4.6

(b) for the average number of iterations.

In comparison to other pre-specified initial radii, our automatic method provides an

average performance. As the results show, a user-provided initial radius is more useable if

the user has the knowledge of the problem being solved. However, it is unrealistic to assume

that the user has the full knowledge of the problems. Therefore, using an automatic selected

initial radius can be a useful option. One reason that our method does not work very well

is that the trust-region radius is constantly adjusted during the computation. Therefore,

an initial trust-region radius may not be extremely important for the overall performance.

Our results also suggest that a smaller initial trust-region radius is more preferable for our

tested instances, and it is likely the best choice for solution of unknown problems.

84


Tabl

e4.

1:A

vera

gew

allc

lock

tim

ean

dnu

mbe

rof

iter

atio

nsus

ing

diff

eren

tin

itia

ltru

st-r

egio

nra

dius

Wal

lClo

ckTi

me(

secs

.)N

um

ber

ofIt

erat

ion

sIn

stan

ceN

Au

toR

Au

toR

=0.

5R

=2

R=

10R

=50

R=

250

Au

toR

=0.

5R

=2

R=

10R

=50

R=

250

20te

rm10

0019

4.9

1005

.833

3.1

332.

844

5.9

260.

012

72.3

403.

315

0.1

148.

417

6.7

110.

743

3.4

4nod

e32

768

5000

250.

020

3.6

147.

729

0.4

415.

353

9.2

547.

349

.035

.059

.286

.287

.993

.0A

IRL2

1000

025

0.0

94.4

104.

491

.497

.388

.390

.822

.329

.725

.424

.620

.921

.3as

sets

lg10

000

1.0

173.

316

6.9

169.

811

7.7

73.9

62.8

31.7

33.1

31.4

23.3

15.8

9.7

biw

eekl

ylg

2000

4.2

233.

229

5.9

224.

835

1.8

463.

355

3.4

18.9

24.2

18.6

23.3

26.5

29.0

biw

eekl

ym

d20

0013

.346

1.1

233.

730

7.2

352.

339

2.1

358.

120

.920

.017

.818

.519

.018

.8bi

wee

kly

sm20

0021

.537

5.3

262.

026

4.7

353.

944

5.5

491.

324

.721

.621

.524

.627

.629

.3ch

em20

000

116.

042

2.2

749.

373

3.5

687.

871

1.9

289.

717

.049

.242

.624

.822

.314

.0dc

ap23

350

010

000

250.

014

9.5

175.

414

9.4

148.

716

5.4

149.

418

.221

.019

.918

.418

.618

.8dc

ap24

330

010

000

250.

014

0.0

172.

213

1.6

138.

412

9.0

132.

015

.720

.315

.615

.615

.815

.5dc

ap33

230

010

000

250.

017

0.7

184.

620

5.4

154.

816

7.8

165.

820

.922

.724

.520

.420

.920

.4dc

ap34

250

010

000

250.

012

9.6

173.

013

1.1

157.

296

.510

1.1

16.8

22.7

16.9

17.0

16.6

17.0

elec

tric

ity

lg20

0013

.335

1.4

298.

028

1.7

338.

340

8.4

375.

013

8.3

99.1

115.

312

8.2

141.

614

1.0

elec

tric

ity

md

5000

14.1

280.

623

0.0

188.

522

0.0

227.

023

4.7

50.7

48.0

39.3

42.7

43.6

44.4

elec

tric

ity

sm10

000

16.1

148.

418

5.9

221.

814

6.0

145.

520

3.4

21.5

31.2

34.8

21.6

21.3

22.7

gbd

5000

30.9

220.

127

3.3

219.

121

1.4

210.

321

7.9

32.6

37.1

32.4

32.2

30.9

31.6

Land

S10

000

7.3

260.

342

4.3

221.

422

6.8

239.

422

8.7

27.2

36.1

24.8

26.1

26.2

25.8

PGP3

1000

017

0.1

362.

327

9.3

260.

825

3.6

394.

533

6.3

32.6

31.5

27.1

27.6

33.4

31.6

phon

e10

000

1.0

56.8

56.4

62.7

99.4

101.

610

1.4

3.0

3.0

5.0

7.0

7.0

7.0

prod

uct

md

1000

26.3

580.

131

1.6

366.

348

0.1

600.

360

0.4

122.

078

.785

.310

4.4

120.

912

5.2

prod

uct

sm20

0048

.167

4.1

404.

533

1.4

713.

772

7.5

700.

694

.767

.460

.397

.899

.598

.7sn

ip4x

920

001.

417

7.9

167.

917

9.6

175.

417

9.5

179.

541

.439

.942

.641

.341

.841

.5sn

ip7x

550

002.

049

5.7

402.

748

4.4

434.

550

7.2

445.

732

.431

.334

.232

.233

.732

.8ss

lp10

5020

0020

0025

0.0

212.

814

0.1

240.

015

3.5

267.

641

0.2

156.

681

.213

4.8

84.1

166.

616

2.2

sslp

1545

1520

0025

0.0

323.

735

6.4

300.

631

6.8

370.

032

4.3

242.

527

4.3

229.

024

7.7

254.

024

4.1

ssn

2000

1.0

754.

378

9.3

711.

562

0.3

1035

.514

99.8

64.4

67.6

62.0

58.3

76.9

94.9

stoc

for2

5000

250.

055

8.3

729.

770

1.6

625.

160

0.6

566.

983

.611

7.0

107.

310

0.4

91.6

84.5

stor

m20

0025

0.0

193.

817

2.7

154.

816

7.0

214.

918

5.7

44.9

45.2

40.3

44.8

46.9

47.8

wee

kly

lg10

0019

.455

7.8

359.

232

2.4

588.

077

7.9

745.

220

9.2

141.

713

0.0

221.

227

5.3

266.

5w

eekl

ym

d20

0042

.461

1.1

220.

231

8.0

515.

167

8.5

797.

514

0.8

69.7

80.5

117.

614

8.2

170.

0w

eekl

ysm

1000

76.4

827.

352

2.1

516.

862

1.8

891.

792

3.0

434.

323

1.4

219.

435

9.6

428.

343

8.0

85


1 1.5 2 2.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile on Wall Clock Time for Trust−region Scaling Methods

UnscaledStep SizeSubgradientBFGS

1 1.2 1.4 1.6 1.8 2

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile on Number of Iterations for Trust−region Scaling Methods

UnscaledStep SizeSubgradientBFGS

(a) Wall Clock Time (b) Number of Iterations

Figure 4.7: Performance profile of wall clock time and number of iterations for trust-regionscaling

4.3.2 Computational Results for Trust-region Scaling

In this experiment, we tested our trust-region scaling methods on 46 instances in our test

suite. The experiment was conducted by using the parameters vmin = 0.1 and ω = 0.75.

The results are reported in terms of the average of ten trials. We compared the results of

using our trust-region scaling methods to the same method without scaling. The perfor-

mance profile plots for the wall clock time and number of iteration are given in Figure 4.7.

Numerical results are also provided in Table 4.2 and Table 4.3.

In Table 4.2 and Table 4.3, the names of the instances are listed in the first column. The

second column N is the sample size of the sampled instances. The statistics for the unscaled

version is titled as unscaled, and the statistics for three different scaling methods are titled

as Step Size, Subgradient, and BFGS. The average wall clock time is given in Table 4.2, and

the average number of iterations is given in Table 4.3.

The results in Figure 4.7 show that, in most instances, our trust-region scaling methods

perform better than the unscaled version. In comparing different scaling methods, the

BFGS method and Step Size method provide more consistent results, with the BFGS method

performs slightly better than the Step Size method in most cases. The Subgradient method

is a slightly more aggressive approach. It provides the best results in some instances and

also provides the worst results in some other instances.

86


Table 4.2: Average wall clock time for different scaling methodsAverage Wall Clock Time(secs.)

Instance N Unscaled Step Size Subgradient BFGS20term 2000 1115.2 800.9 1105.2 983.6AIRL1 5000 76.4 78.4 81.5 71.2AIRL2 5000 66.6 69.4 68.6 66.7biweekly lg 1000 169.6 197.7 561.9 183.9biweekly md 2000 236.9 171.4 411.5 228.4biweekly sm 2000 237.0 213.9 263.8 220.5dcap233 200 10000 166.4 165.1 129.5 141.6dcap233 300 10000 181.1 175.5 128.2 145.6dcap233 500 10000 178.4 185.8 126.0 142.4dcap243 200 10000 133.0 131.4 133.7 136.6dcap243 300 10000 142.8 133.7 138.0 132.4dcap243 500 10000 133.7 132.5 139.3 115.4dcap332 200 10000 208.2 221.7 157.2 301.9dcap332 300 10000 232.3 223.4 157.0 257.3dcap332 500 10000 128.1 126.5 144.8 130.3dcap342 200 10000 165.2 155.4 157.9 133.9dcap342 300 10000 164.0 156.3 162.3 153.7dcap342 500 10000 146.3 143.6 159.2 146.2electricity lg 2000 586.0 564.6 236.7 531.4electricity md 5000 452.4 401.4 343.6 365.5electricity sm 5000 148.6 131.7 260.2 155.9gbd 5000 138.6 149.5 150.7 160.8LandS 5000 86.0 80.7 75.5 75.6phone 10000 61.7 67.8 63.6 64.5product md 1000 387.6 331.0 278.7 278.0product sm 2000 413.3 367.1 273.3 337.4snip10x10a 1000 302.4 273.8 309.8 289.1snip10x10b 1000 407.8 368.4 347.5 336.4snip10x10c 1000 390.7 341.0 374.9 356.0snip4x9 5000 403.9 360.4 334.8 386.0snip7x5 5000 290.1 221.8 208.2 236.0sslp 10 50 100 2000 408.0 398.8 378.9 381.9sslp 10 50 1000 2000 418.7 429.4 410.9 404.6sslp 10 50 2000 2000 462.5 446.7 405.9 397.5sslp 10 50 50 2000 425.2 390.2 371.8 413.7sslp 10 50 500 2000 498.7 402.6 399.3 401.3sslp 15 45 10 2000 826.0 789.6 682.5 695.0sslp 15 45 15 2000 736.0 819.6 821.6 705.4sslp 15 45 5 2000 801.8 694.0 727.1 758.5sslp 5 25 100 5000 308.7 331.4 325.8 318.1sslp 5 25 50 5000 288.3 305.7 340.3 291.6ssn 2000 442.1 453.5 821.2 437.3storm 2000 256.1 282.5 285.9 315.1weekly lg 500 478.8 394.1 242.8 320.3weekly md 500 144.4 133.2 84.4 130.3weekly sm 500 493.5 392.8 345.5 394.6

87


Table 4.3: Average number of iterations for different scaling methodsAverage Number of Interations

Instance N Unscaled Step Size Subgradient BFGS20term 2000 180.9 145.4 178.7 163.4AIRL1 5000 28.0 27.7 28.6 26.4AIRL2 5000 21.8 22.0 22.6 21.4biweekly lg 1000 17.0 18.3 77.6 17.4biweekly md 2000 19.2 15.1 44.4 16.8biweekly sm 2000 22.6 22.3 24.2 21.8dcap233 200 10000 20.6 20.7 15.9 17.2dcap233 300 10000 22.0 22.2 15.2 17.3dcap233 500 10000 21.7 22.3 14.9 17.1dcap243 200 10000 15.0 15.0 14.5 14.7dcap243 300 10000 14.9 14.9 15.0 13.8dcap243 500 10000 14.6 15.0 14.7 12.7dcap332 200 10000 26.1 26.2 19.7 35.7dcap332 300 10000 26.9 26.7 20.3 30.6dcap332 500 10000 15.3 15.3 17.9 14.8dcap342 200 10000 17.5 17.3 17.3 14.9dcap342 300 10000 17.7 17.4 17.7 17.3dcap342 500 10000 15.5 15.5 16.6 16.6electricity lg 2000 125.5 121.7 63.8 119.0electricity md 5000 42.1 38.4 38.9 37.0electricity sm 5000 25.8 22.8 63.1 27.3gbd 5000 33.1 34.5 35.3 36.2LandS 5000 27.7 27.0 23.7 26.3phone 10000 6.1 6.1 6.0 6.0product md 1000 90.3 82.0 73.9 73.4product sm 2000 77.3 70.6 59.6 66.4snip10x10a 1000 94.1 89.8 92.9 90.1snip10x10b 1000 135.3 129.3 122.6 123.3snip10x10c 1000 118.4 113.0 115.7 112.2snip4x9 5000 41.4 39.3 37.7 40.9snip7x5 5000 35.1 29.9 30.5 31.2sslp 10 50 100 2000 144.0 135.3 130.5 131.5sslp 10 50 1000 2000 150.4 142.2 137.7 138.5sslp 10 50 2000 2000 155.3 149.5 127.1 131.4sslp 10 50 50 2000 145.0 127.8 125.0 135.8sslp 10 50 500 2000 189.0 137.3 132.4 134.8sslp 15 45 10 2000 247.8 248.6 224.9 213.8sslp 15 45 15 2000 230.0 258.6 258.2 223.1sslp 15 45 5 2000 244.4 210.5 229.0 223.8sslp 5 25 100 5000 63.0 64.8 61.3 63.0sslp 5 25 50 5000 60.7 62.3 63.8 60.3ssn 2000 58.6 56.4 157.4 57.3storm 2000 38.1 40.8 40.4 42.5weekly lg 500 161.3 140.1 97.6 118.9weekly md 500 93.4 87.3 63.8 86.3weekly sm 500 249.7 230.1 171.2 209.3

88


4.3.3 Computational Results on Parallel Search

In this section, we report the results of using different search methods on our problem test

set. The first purpose of the study is to compare the computational performance of the

three methods. The second purpose is to study how the maximum number of simultaneous

search points affects the computational performance and to discover an appropriate number

of search points for solution of general problems.

The experiments are performed in two parts. The first part of the experiment contains

31 instances from our test set. We performed this part of the experiment in the COR@L

laboratory with a computational grid of 46 Pentium III processors. The second part of the

experiment was aimed at providing insight into the choice of the number of simultaneous

search points. The instances gbd, snip20x20, and ssn were used to illustrate our point.

The results of our experiments are based on the average of ten trials. To minimize the

error from sampling, the same samples were used for all methods in each trial. In the

first part, we ran the experiments by using the three different search methods discussed in

Section 4.2 and by changing the maximum number of search points (M) from 1 to 5. Note

that M = 1 is the regular ATR method, which is used for comparison. We provide four

statistics for these experiments. In particular, we measured the wall clock time for solution

of each instance, the total time required for solution of the master problems, the number of

master problems being solved, and the total number of points being evaluated. Numerical

results are provided in Appendix A.2, and the performance profile plots of these statistics

are given in Figure 4.8 - 4.11.

Figure 4.8 compares the performance of wall clock time using the different methods and

different number of search points. From the figure, we see that there is not much difference

in the three search methods that we use. The Dogleg Search method seems to be the best

among these, where the Line Search and the Zigzag Search methods are about the same.

The Zigzag Search method performs better than the Line Search method when M = 2 and

the results are opposite whenM = 5. In comparison to the regular ATR method, these three

methods perform significantly better. By increasing the number of search points, we could

89


1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of Wall Clock Time with 2 search points

ATRLineSearchZigzagDogleg

1 1.5 2 2.5 3

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



(a) Search points = 2 (b) Search points = 3

1 1.5 2 2.5 3

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



1 1.5 2 2.5 3

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



(c) Search points = 4 (d) Search points = 5

Figure 4.8: Performance profile of wall clock time for search methods and varying numbersof points

potentially reduce the number of master problems that need to be solved, but at the same

time, we may also increase the computation time by evaluating extra points. Therefore, it is

clear that doing the extra search may not always be better, and it is a function of the number

of processors available as well as the relative difficulty of solving the master problem. For

example, if the master problem is relatively easy to solve, then it may be better to solve

the master problem to get a better candidate solution instead of generating one from our

search methods.

Figure 4.9 shows the performance profile plots of the computation time for solution of

the master problem. Clearly, all three search methods reduce the total time for solution of

90


1 1.5 2 2.5 3 3.5 4 4.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of Master Solving Time with 2 search points


1 1.5 2 2.5 3 3.5 4 4.5 5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



1 2 3 4 5 6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




Figure 4.9: Performance profile of total master problems solution time for search methodsand varying numbers of points

the master problems significantly compared to the regular ATR method. This result agrees

with the result in Figure 4.10, which is the performance profile plots of the number of

master problems being solved.

Figure 4.11 shows the performance profile plot of the total number of points evaluated

for the different search methods and for the different values of M. As we expected, the

larger the value of M, the more total points that were evaluated. However, one surprising

result is that the total number of points that need to be evaluated when using each of the

three search methods is less than the number of points used in the regular ATR method.

At M = 5, the evidence is less obvious(see Figure 4.11 (d)). One explanation is that the

91


1 1.5 2 2.5 3 3.5 4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of Number of Master Problems Solved with 2 search points


1 1.5 2 2.5 3 3.5 4 4.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



1 2 3 4 5 6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




Figure 4.10: Performance profile of number of master problems solved for search methodsand varying numbers of points

92


1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance Profile of Number of Points Evaluated with 2 search points


1 1.2 1.4 1.6 1.8 2 2.2 2.4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




1 1.2 1.4 1.6 1.8 2 2.2 2.4

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9



1 1.5 2 2.5

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9




Figure 4.11: Performance profile of number of points evaluated for search methods andvarying numbers of points

candidate solutions generated from our search methods were better than (or as good as)

the candidate solutions obtained from solution of the master problem.

In the second part of the experiment, we study the quality of the points generated by

the different search methods. The experiment is conducted by running the algorithm in a

synchronous mode (see Section 2.5.4), which means that a new point will not be generated

until the previous point evaluation is complete. Doing this allows us to count the number

of search points generated during the computation precisely and to compute the likelihood

of search points being accepted as improving the objective function. A few instances were

selected in this study, namely, gbd, snip20x20, and ssn. We ran ten trials for each search

93

4.4. CONCLUSIONS

method with the same samples and use M = 8. The results are provided in Table 4.4 for

each method in a separate main column. The search points are labeled from 0-7, with point

0 being the point obtained by solving the master problem and point 1 being the first point

generated from our search methods, and so on. The columns Evaluated(%) are the average

percentage of the nth point being evaluated in terms of the total points evaluated in the

entire computation. The columns Accepted(%) are the average percentage of the nth point

being accepted as a new incumbent solution in terms of the total points evaluated. Note that

the summation of the points being accepted is less than 100% (see the last rows in Table

4.4) because not every point is accepted. The results show that, with high probability, most

new incumbent solutions were found within the first two or three points being generated.

This means that generating more search points may not be worth the extra computational

power required for the extra point evaluations, which seems to support our results in Part

I that why the best performance was achieved when M is small. It may also suggest new

mechanisms for selecting points to evaluate.

Of these three instances, gbd is a relatively easy problem compared to the other two, and

it seems there is not much benefit from using our search methods. In addition, the solution

from solving the master problem (point 0 in Table 4.4 (a)) has a very high acceptance

percentage, which also lessens the benefit of using our search methods. The gbd results

are in contrast to the cases of snip20x20 and ssn, because these instances are much more

difficult. In solving snip20x20 and ssn, the computation time was significantly reduced by

using our search methods. The point generated from the Dogleg Search method also has

the highest acceptance rate in total(see Table 4.4 (b) and (c)).

4.4 Conclusions

In this chapter, we discussed how to apply several different nonlinear programming tech-

niques for solution of stochastic programs. In particular, we showed how an initial trust-

region radius can be obtained automatically from the subgradient information at the be-

ginning of the algorithm. We also employed trust-region scaling techniques to improve

94

4.4. CONCLUSIONS

Table 4.4: Statistics on percentage of the number of the nth point being evaluated andbeing accepted for different search methods

Line Search Zigzag Search Dogleg SearchPoint Evaluated(%) Accepted(%) Evaluated(%) Accepted(%) Evaluated(%) Accepted(%)

0 74.2 53.5 74.6 56.5 62.0 48.81 20.7 16.1 18.2 14.2 13.2 6.02 4.6 4.0 3.9 1.3 7.1 1.83 0.6 0.6 2.7 2.1 5.3 1.74 0.0 0.0 0.6 0.6 3.6 0.55 0.0 0.0 0.0 0.0 3.2 0.46 0.0 0.0 0.0 0.0 2.8 0.07 0.0 0.0 0.0 0.0 2.8 0.0

Total 100.0 74.2 100.0 74.6 100.0 59.2

(a) gbd, N = 2000


0 39.0 4.4 40.2 5.3 45.9 5.01 34.7 12.6 34.9 14.7 40.9 28.42 22.0 18.1 20.2 17.4 12.6 12.03 3.9 3.6 2.8 1.1 0.6 0.64 0.3 0.3 1.8 1.7 0.0 0.05 0.0 0.0 0.1 0.1 0.0 0.06 0.0 0.0 0.0 0.0 0.0 0.07 0.0 0.0 0.0 0.0 0.0 0.0

Total 100.0 39.0 100.0 40.2 100.0 45.9

(b) snip20x20, N = 200


0 70.4 44.5 68.3 43.8 74.8 49.61 25.9 25.2 24.4 22.5 25.2 25.22 0.8 0.0 1.9 0.8 0.0 0.03 0.8 0.0 1.1 0.0 0.0 0.04 0.8 0.3 1.1 0.0 0.0 0.05 0.5 0.0 1.1 0.0 0.0 0.06 0.5 0.0 1.1 0.0 0.0 0.07 0.5 0.0 1.1 0.2 0.0 0.0

Total 100.0 69.9 100.0 67.4 100.0 74.8

(c) ssn, N = 1000

95

4.4. CONCLUSIONS

the capability of the existing algorithm. Last but not least, we demonstrated three search

methods that when incorporated into our trust-region method speed up the algorithm con-

siderably.

In conclusion, our experimental results showed that an initial trust-region radius can be

obtained easily from the subgradient information, although the selection of an initial trust-

region radius seems not to have a large impact on performance. Nevertheless, an automatic

mechanism like the one we propose may be useful when little is known about the prob-

lem instance being solved. For trust-region scaling, we obtained slight improvement from

all three of our new scaling methods. Among them, scaling by considering the previous

step size and by using the BFGS update methods provided the most consistent improve-

ment, where the scaling method using subgradient information may have more variable

performance. For the last experiment, we showed that by using our new search methods,

computation time for solution of the master problems is reduced as is the total number

of iterations in many cases, especially for solution of difficult instances. Among our three

search methods, the Dogleg Search method provides the best results. Also, good perfor-

mance can be achieved by using only a small number of additional search points, and the

ATR algorithm can be improved by this simple idea.

96

Chapter 5

Bootstrap Estimate of Bias in Sample

Average Approximation

In this chapter, we describe how a well-known statistical procedure known as bootstrapping

can help to improve the estimate of the objective value in sample average approximation.

Extensive computational results are given to demonstrate the applicability of our approach.

5.1 Motivation

In sample average approximation (SAA), the optimal value of the true problem z∗ is bounded

(in expectation) by the estimates of the lower bound and upper bound (see Section 2.3).

The estimate of the lower bound is obtained by solving a series of sampled instances and

taking the average of the resulting optimal values. An estimate of the upper bound is ob-

tained by approximately evaluating the objective function using a solution from a sampled

instance. We refer the reader to Section 2.3 or Ruszczynski and Shapiro Chapter 6 in [57]

for a more detailed description of estimating the optimality gap.

Recent research has shown that the SAA method can be an effective method for obtain-

ing good solutions to stochastic programs [62, 61, 42]. However, the estimate of the true

objective value in using the SAA method often involves bias in both upper and lower bound

estimates. Computational results [42] have shown that the lower bound bias tends to be

97

5.2. BOOTSTRAP ESTIMATE OF LOWER BOUND BIAS

larger than the upper bound bias, and the amount of bias can be reduced by increasing

the sample size. Unfortunately, solving sampled instances with large sample size requires

significant computation time in many instances. In Chapters 3 and 4, we described a variety

of mechanisms for improving solution times for large instances. In this chapter, we give a

complimentary approach to reduce the bias by bootstrapping. Given that we have a good

estimate of the bias, we can use it to adjust the estimate of the objective value and provide

reasonable results by solving only sampled instances with small sample sizes.

The bootstrap method is widely used in statistics to estimate the standard error and bias

of a statistic from the samples of an unknown distribution. For stochastic programming,

although the distribution of the random parameters is known in most practical cases, it is

difficult or impossible to compute the true objective value exactly due to the complexity

of the problem and the large number of scenarios. Using the bootstrap method requires a

large amount of sample data to obtain a reasonable estimate. This can be impractical in

traditional computing environments due to the amount of time required to solve a large

number of sampled instances. In a computational grid environment, however, this process

can be sped up significantly because solution of the sampled instances necessary to obtain

bootstrap-adjusted estimates can be performed independently and in parallel.

For the above reasons, we propose to estimate the bias using the bootstrap method. In

our work, we mainly consider the bias of the lower bound estimate rather than that of the

upper bound estimate, which is often small in practice.

5.2 Bootstrap Estimate of Lower Bound Bias

Suppose we are using the average of the solutions from the sampled instances to estimate

the true objective value z∗. Then the bias β of the estimate of the lower bound can be

measured by

β = z∗ − E[zN ], (5.1)

98

5.2. BOOTSTRAP ESTIMATE OF LOWER BOUND BIAS

where zN is the optimal value from solving a random sampled instance of size N . The

expected value E[zN ] can be estimated by taking the mean of M independent estimates

(see Section 2.3)

E[zN ] =1M

M∑

i=1

ziN . (5.2)

Recall that an estimate of the optimal value zN corresponding to N random scenarios is

obtained by solving (3.1). To perform bootstrapping, we first define a random sample Ξ =

(ξ1, ξ2, ..., ξN ) that contains N scenarios drawn from the stochastic programing problem’s

probability space. A bootstrap sample

Ξ[ = (ξ[1, ξ

[2, ..., ξ

[N ) (5.3)

is generated by using the same number of scenarios N from Ξ. Each scenario ξ[i ∈ Ξ[,

is drawn (with replacement) from the original samples ξ1, ξ2, ..., ξN with equal probability

1/N . The bootstrapping procedure to estimate the bias of zN from a random sample Ξ is as

follows:

Bootstrapping Procedure

Step 1: Generate B bootstrap samples Ξ[1, Ξ[2, ...,Ξ[B from Ξ, each bootstrap sample con-

sisting of N scenarios generated as in (5.3).

Step 2: Solve the problem SAA(Ξ[j) of (3.1) corresponding to each bootstrap sample Ξ[j

to obtain z[jN for j = 1, ..., B.

Step 3: Estimate the bias β of zN by

βN = zN − 1B

B∑

j=1

z[jN

where zN is the optimal value from solving SAA(Ξ) of (3.1) using the original

sample Ξ.

By repeating the above procedure M times, the expected value of the bias for a sampled

99

5.3. BOOTSTRAP-ADJUSTED ESTIMATE OF THE OPTIMAL VALUE

instance with N scenarios can be computed as

E[βN ] =1M

M∑

i=1

(ziN − 1

B

B∑

j=1

z[ijN ), (5.4)

= E[zN ]− 1MB

M∑

i=1

B∑

j=1

z[ijN .

In general, the number of trials M and the number of bootstrap replications B necessary

to obtain a reasonable estimate of bias is problem-dependent. The suggested values for M

and B are between 50 and 300, and could exceed 1,000 in some cases.

5.3 Bootstrap-adjusted Estimate of the Optimal Value

The SAA method gives a biased estimate of the true objective value z∗, and the estimate

of bias (5.4) is then used to adjust the estimate of the optimal value z∗ for a sampled in-

stance. Using the same notation as described in the Bootstrapping Procedure, the bootstrap-

adjusted estimate zAN of z∗ for a sampled instance of size N is then

zAN = zN + βN ,

and the expected value of the bootstrap-adjusted objective value estimate is computed by

E[zAN ] = 1

M

M∑

i=1

(ziN + βi

N ),

= 1M

M∑

i=1

(2ziN − 1

B

B∑

j=1

z[ijN ),

= 2E[zN ]− 1MB

M∑

i=1

B∑

j=1

z[ijN .

(5.5)

Note that it is possible to use a different value of B in each trial of M because the

estimate of bias is completed in the bootstrapping procedure. Specifically, in a limited

computational budget situation, one may use a value of B < 1 to estimate the objective

value. This means that the bootstrapping procedure is not carried out in all the trials and at

100


most one bootstrap instance SAA(Ξ[) is solved for the individual trial to estimate the bias.

For example, a value of B = 0.5 means that half the SAA trials use bootstrapping, each with

only one bootstrap replication. In such case, the estimate of bias is computed only for those

trials that perform bootstrapping.

We use the mean square error (MSE) to assess the performance of an estimator in (5.5)

for the true objective value z∗. For given values of N , M , and B, the MSE of this estimator

can be computed by

MSE(E[zAN ]) = (z∗ − E[zA

N ])2 +1M

[V ar(zN ) +

V ar(βN )B

+ 2Cov(zN , βN )

]. (5.6)

The true value z∗ is unknown. However, we estimate the value of z∗ to a high accuracy

by evaluating solutions from sampled instances with large sample size. We then use these

estimated value z∗ to compute the MSE of the bootstrap-adjusted estimate. The estimate of

z∗ for all test instances are given in Section 5.4.1.


Several questions arise when using bootstrap to estimate the bias. First, the true optimal

objective value is unknown, how can we assess our bias estimate? Second, although the

bootstrap method does not have a limitation on the statistic to which it can be applied,

we do not know how effective it will be for estimating the objective value of a stochas-

tic program. Computational results will be required to provide the insight. Third, if the

bootstrap method can provide a good estimate of the objective value for a stochastic pro-

gram by solving small sampled instances, are those solutions of high quality? Fourth, if the

computational resources were limited, is the bootstrap an efficient method or should the

sample size be increased? Fifth, how good are the lower and upper bound estimates of the

bootstrap method compared to SAA using the ATR method? Sixth, how many bootstrap

replications B and trials M are needed in order to get a good estimate of z∗ for stochastic

programming? In addition, if the estimate of z∗ was not “good enough” as measured by an

101


estimated gap between the lower and upper bounds, will it be better to rerun the experi-

ments by increasing the sample size N or stay with the same sample size and solve more

bootstrap instances? As we know, this could be problem-dependent. However, we hope that

our computational experiments will provide reasonable guidelines for practitioners.

To answer these questions, we constructed experiments to test the bootstrap method

on our instances. We solved a series of deterministic equivalent problems of the sampled

instances using different numbers of scenarios, namely N = 5, 10, 50, 100, 200, 400, and

800. For each number of scenarios N , we ran 200 trials (M = 200). For each trial, we ran

200 bootstrap replications(B = 200). The results were used to compute an estimate of bias

according to equation (5.4). Note that we use N = 5, 10, 15, and 20 for the instances in

Table 5.1 (b) instead due to the difficulty of solving these instances with large sample size.

Although the computation time for solution of each sampled instance is small, the ex-

periment was a time consuming task mainly because of the huge number of instances that

needed to be solved. For example, in order to estimate the bias of a instance with a sample

of 50 scenarios, we solved 40,200 deterministic equivalent problems. We provide the com-

putation time for solution of one deterministic equivalent problem of our test instances on

an Intel Pentium III 1.13 in Table 5.1. The table is also divided into two sub-tables due to

the different sample sizes used.

5.4.1 Estimating the True Optimal Solution Values

In order to compare the results with the true bias, the true optimal value for each instance

was required. For this purpose, we ran another experiment by solving all the test instances

using the decomposition method. We start by using a reasonably large sample size, and

increase it if the estimated optimality gap from the SAA method was still above 1%. We

used the more conservative approach mentioned in [44] to compute the estimate of the

optimality gap by taking the difference of the mean between the upper bound and lower

bound estimates plus the half lengths of the confidence intervals from both upper and lower

bounds. The estimated optimality gap was computed by using the 95% confidence bounds

102


Table 5.1: Average running times(in seconds) for bootstrap by solving the deterministicequivalent problems

Instance 5 10 50 100 200 400 80020term 0.192 0.472 4.342 16.602 74.522 - -4node 1024 0.072 0.232 4.002 12.972 48.042 179.872 -4node 16384 0.082 0.262 3.732 13.762 66.812 200.452 -4node 32768 0.092 0.252 3.992 15.082 54.092 182.572 -4node 4096 0.092 0.292 4.002 14.792 56.082 181.652 -4node 8192 0.092 0.242 4.572 14.902 50.302 217.382 -AIRL1 0.002 0.002 0.022 0.062 0.202 0.662 3.052AIRL2 0.002 0.002 0.022 0.052 0.192 0.512 1.702assets lg 0.002 0.002 0.042 0.122 0.373 2.152 5.152assets sm 0.002 0.002 0.042 0.132 0.462 1.712 6.312biweekly lg 1.012 2.032 14.252 47.962 206.212 - -biweekly md 0.322 0.712 4.322 11.942 47.882 - -biweekly sm 0.102 0.272 2.742 11.102 51.682 - -chem 0.012 0.022 0.182 0.622 2.272 13.922 46.132dcap233 200 0.002 0.002 0.072 0.182 0.552 1.592 6.172dcap233 300 0.002 0.012 0.072 0.222 0.512 2.102 9.302dcap233 500 0.002 0.012 0.052 0.202 0.592 2.062 8.662dcap243 200 0.012 0.012 0.082 0.252 0.802 2.112 9.802dcap243 300 0.012 0.012 0.092 0.242 0.682 2.212 11.472dcap243 500 0.002 0.012 0.092 0.242 0.682 2.902 13.882dcap332 200 0.002 0.002 0.062 0.172 0.682 3.482 11.402dcap332 300 0.002 0.002 0.062 0.162 0.572 2.682 14.642dcap332 500 0.012 0.012 0.062 0.202 0.702 2.382 12.522dcap342 200 0.002 0.012 0.072 0.322 1.032 3.242 19.072dcap342 300 0.012 0.012 0.082 0.262 1.012 4.152 13.922dcap342 500 0.002 0.012 0.092 0.282 0.792 3.302 12.272electricity lg 0.102 0.322 1.312 3.052 7.832 24.372 -electricity md 0.132 0.262 2.152 5.772 20.372 70.742 -electricity sm 0.002 0.022 0.142 0.362 0.812 1.892 4.712gbd 0.002 0.002 0.022 0.062 0.202 0.832 -LandS 0.002 0.002 0.032 0.072 0.152 0.482 -PGP1 0.002 0.002 0.022 0.052 0.152 0.652 2.562PGP3 0.002 0.002 0.022 0.052 0.162 0.672 3.232phone 0.012 0.042 0.292 0.912 2.942 11.622 43.262product lg 0.532 1.162 15.232 65.982 - - -product md 0.242 0.652 5.962 22.002 88.852 - -product sm 0.042 0.082 0.782 2.392 8.502 34.202 -snip10x10a 0.202 0.622 13.612 43.812 - - -snip10x10b 0.202 0.552 10.982 38.992 - - -snip10x10c 0.192 0.602 10.042 47.962 - - -snip4x9 0.042 0.102 1.222 4.502 20.872 52.092 -snip7x5 0.052 0.152 2.222 8.922 28.582 97.102 -sslp 10 50 100 0.132 0.352 3.572 10.902 35.942 135.432 -sslp 10 50 1000 0.142 0.252 3.112 10.882 36.592 126.332 -sslp 10 50 2000 0.112 0.282 3.162 10.172 33.072 128.942 -sslp 10 50 50 0.122 0.382 3.312 9.932 34.302 129.462 -sslp 10 50 500 0.122 0.292 3.492 9.742 33.012 127.452 -sslp 15 45 10 0.212 0.562 5.392 15.842 50.062 187.012 -sslp 15 45 15 0.152 0.392 5.202 15.362 49.772 186.902 -sslp 15 45 5 0.242 0.592 5.422 15.712 51.002 192.002 -sslp 5 25 100 0.012 0.052 0.342 1.022 3.832 12.642 56.482sslp 5 25 50 0.012 0.032 0.342 1.062 3.292 12.962 54.322ssn 0.152 0.522 5.832 21.782 45.392 159.512 -stocfor2 0.032 0.062 0.582 1.662 5.752 21.552 103.922storm 0.662 1.932 16.152 52.992 - - -weekly lg 1.372 3.012 29.462 85.732 - - -weekly md 0.592 1.512 12.012 46.072 - - -weekly sm 0.342 0.902 13.862 63.202 - - -

(a) Problem set A

103


Table 5.1: Average running times(in seconds) for bootstrap by solving the deterministicequivalent problems

Instance 5 10 15 20semi2 8.99 23.55 42.11 -semi3 8.56 22.57 40.20 -semi4 8.50 22.48 40.20 -snip20x20 3.22 20.10 43.27 66.47snip30x30 19.71 107.09 - -

(b) Problem set B

of the lower bound and upper bound estimate. Ten sampled instances of sample size N

were solved to estimate the lower bound. For each solution obtained from the sampled

instances, a sample size of 10 × N was used for evaluation of the upper bound. The ’true’

optimal values used for computing the bias were obtained from using the result with the

best optimality gap. For ease of comparison, we also computed the estimated optimality

gap in terms of percentage to the estimated mean.

Table 5.2 provides the estimated upper bound, lower bound, and optimality gap, along

with a 95% confidence interval. The last column shows the estimated optimality gap as a

percentage of the objective value. The best gaps so far for all instances are within 1.0%,

except for snip30x30, snip4x9, and ssn, which have the gaps of 1.13%, 1.24%, and 1.17%

respectively. The estimated “true” optimal values are given in tables in Appendix A.3. The

contribution of these results is to provide researchers a variety of test instances for stochastic

programming. All the instances in SMPS format and the results will be made available on

COR@L web site.

104


Tabl

e5.

2:U

pper

boun

d,lo

wer

boun

d,an

dop

tim

alit

yga

pw

ith

95%

confi

denc

ein

terv

alIn

stan

ceN

UB

Mea

nU

Bh.

l.LB

Mea

nLB

h.l.

95%

CI

Gap

Ga

p(%

)In

stan

ceN

UB

Mea

nU

Bh.

l.LB

Mea

nLB

h.l.

95%

CI

Gap

Ga

p(%

)20

term

2000

2541

52.0

411

2.72

2543

43.1

272

.39

185.

120.

07sn

ip20

x20

2000

182.

410.

7418

2.19

0.62

1.57

0.86

4nod

e10

2450

0043

4.10

0.01

434.

100.

030.

040.

01sn

ip20

x20

5000

182.

780.

4318

1.98

0.41

1.64

0.90

4nod

e16

384

5000

446.

830.

0144

6.87

0.03

0.04

0.01

snip

30x3

010

0022

7.73

0.87

226.

050.

983.

541.

564n

ode

3276

850

0044

6.83

0.01

446.

860.

030.

040.

01sn

ip30

x30

2000

227.

750.

9022

6.70

0.60

2.55

1.13

4nod

e40

9650

0044

6.83

0.01

446.

860.

020.

040.

01sn

ip4x

950

0010

.78

0.14

10.6

20.

110.

413.

884n

ode

8192

5000

446.

830.

0144

6.85

0.02

0.04

0.01

snip

4x9

1000

010

.69

0.06

10.6

40.

060.

181.

68A

IRL1

5000

2490

69.6

921

.51

2491

06.3

010

2.01

123.

510.

05sn

ip4x

920

000

10.6

60.

0810

.61

0.05

0.18

1.68

AIR

L250

0026

9583

.17

42.0

926

9661

.53

142.

4718

4.56

0.07

snip

4x9

5000

010

.67

0.04

10.6

50.

070.

131.

24as

sets

lg50

00-6

95.1

90.

38-6

95.6

10.

451.

250.

18sn

ip7x

550

0080

.60

0.18

79.8

30.

401.

351.

69as

sets

sm50

00-7

24.6

50.

04-7

23.9

70.

470.

520.

07sn

ip7x

510

000

80.3

90.

1679

.87

0.29

0.96

1.21

biw

eekl

ylg

2000

-720

7.50

11.0

9-7

212.

6215

.69

31.9

10.

44sn

ip7x

520

000

80.2

90.

1280

.23

0.21

0.39

0.48

biw

eekl

ym

d20

00-1

466.

542.

94-1

470.

504.

5011

.39

0.78

sslp

1050

100

2000

-398

.78

0.67

-399

.19

1.42

2.50

0.63

biw

eekl

ysm

2000

-849

1.78

15.1

5-8

493.

7714

.64

31.7

80.

37ss

lp10

5010

050

00-3

98.9

00.

64-3

98.7

10.

421.

060.

26ch

em50

00-1

2988

.13

15.8

3-1

3019

.41

13.6

760

.78

0.47

sslp

1050

1000

2000

-395

.65

0.49

-395

.92

1.35

2.11

0.53

dcap

233

200

1000

064

1.98

0.72

641.

521.

943.

110.

49ss

lp10

5010

0050

00-3

95.7

80.

59-3

95.2

40.

651.

250.

32dc

ap23

330

010

000

555.

300.

6255

3.03

1.49

4.38

0.79

sslp

1050

2000

2000

-390

.78

0.60

-390

.69

0.73

1.33

0.34

dcap

233

500

1000

058

6.11

1.39

587.

881.

592.

970.

51ss

lp10

5020

0050

00-3

90.1

80.

54-3

90.8

20.

571.

760.

45dc

ap24

320

010

000

1168

.23

0.61

1168

.72

1.09

1.69

0.15

sslp

1050

5020

00-4

08.6

00.

74-4

08.8

70.

821.

830.

45dc

ap24

330

010

000

1272

.61

1.40

1274

.53

1.74

3.14

0.25

sslp

1050

5050

00-4

07.5

50.

53-4

07.7

40.

270.

980.

24dc

ap24

350

010

000

1048

.21

0.68

1048

.31

1.00

1.68

0.16

sslp

1050

500

2000

-392

.98

0.74

-392

.84

0.92

1.66

0.42

dcap

332

200

1000

016

2.49

0.11

162.

660.

220.

330.

20ss

lp10

5050

050

00-3

92.7

80.

54-3

93.6

00.

672.

030.

52dc

ap33

230

010

000

178.

260.

2017

8.30

0.26

0.47

0.26

sslp

1545

1020

00-2

73.4

20.

48-2

73.6

00.

751.

400.

51dc

ap33

250

010

000

130.

060.

1113

0.49

0.19

0.30

0.23

sslp

1545

1050

00-2

73.5

40.

30-2

73.1

90.

360.

660.

24dc

ap34

220

010

000

331.

811.

2933

3.49

1.74

3.03

0.91

sslp

1545

1520

00-2

70.5

80.

52-2

70.2

90.

931.

440.

53dc

ap34

230

010

000

372.

790.

9937

2.88

0.77

1.76

0.47

sslp

1545

1550

00-2

70.4

30.

39-2

70.2

90.

350.

740.

27dc

ap34

250

010

000

354.

840.

5535

4.43

0.86

1.82

0.51

sslp

1545

520

00-2

66.3

80.

36-2

66.3

20.

560.

920.

35el

ectr

icit

ylg

2000

-181

5.89

3.29

-182

0.03

5.92

13.3

50.

74ss

lp15

455

5000

-266

.42

0.28

-266

.54

0.27

0.67

0.25

elec

tric

ity

md

2000

-586

0.81

5.04

-587

8.15

11.8

934

.27

0.58

sslp

525

100

2000

-169

.08

0.49

-169

.70

0.57

1.68

0.99

elec

tric

ity

sm20

00-1

437.

760.

32-1

437.

176.

857.

170.

50ss

lp5

2510

050

00-1

69.6

10.

18-1

69.4

10.

320.

500.

30gb

d50

0016

45.4

32.

7616

50.6

48.

0210

.79

0.66

sslp

525

5020

00-1

63.3

70.

43-1

63.4

40.

511.

010.

62La

ndS

1000

022

5.44

0.14

225.

650.

400.

530.

24ss

lp5

2550

5000

-163

.62

0.19

-163

.89

0.37

0.84

0.51

PGP1

5000

438.

900.

0343

8.64

1.02

1.30

0.30

ssn

2000

10.0

40.

259.

620.

401.

0711

.11

PGP3

5000

499.

901.

0649

4.28

3.67

10.3

62.

10ss

n10

000

9.97

0.15

9.89

0.16

0.39

3.93

PGP3

1000

049

8.44

0.88

496.

581.

844.

590.

92ss

n50

000

9.89

0.06

9.88

0.05

0.12

1.17

PGP3

2000

049

7.94

0.48

496.

801.

783.

400.

68st

ocfo

r250

00-3

9787

.07

28.7

5-3

9784

.05

30.7

759

.52

0.15

phon

e50

0036

.88

0.00

36.8

90.

010.

020.

05st

orm

2000

1549

5278

.08

3947

.07

1549

9283

.48

7091

.43

1103

8.50

0.07

prod

uct

lg20

00-2

9874

.17

31.7

0-2

9861

.98

25.8

957

.59

0.19

wee

kly

lg20

00-2

501.

3911

.71

-251

2.60

12.7

735

.69

1.43

prod

uct

md

2000

-282

20.9

99.

68-2

8235

.25

25.3

749

.32

0.17

wee

kly

lg50

00-2

503.

109.

59-2

506.

969.

1022

.56

0.90

prod

uct

sm20

00-1

7921

.18

22.6

3-1

7956

.56

29.7

287

.74

0.49

wee

kly

lg10

000

-250

0.20

8.04

-250

9.27

8.27

25.3

71.

01se

mi2

1000

113.

660.

0011

3.66

0.00

0.00

0.00

wee

kly

md

2000

-963

.39

9.15

-964

.86

5.63

16.2

51.

69se

mi3

1000

113.

700.

0611

3.64

0.03

0.15

0.13

wee

kly

md

5000

-967

.73

4.31

-964

.32

7.44

11.7

51.

22se

mi4

1000

113.

110.

2311

3.20

0.16

0.38

0.34

wee

kly

md

1000

0-9

62.7

53.

22-9

64.0

14.

919.

390.

98sn

ip10

x10a

5000

88.3

70.

2088

.00

0.20

0.77

0.87

wee

kly

sm20

00-3

613.

9113

.13

-362

3.42

19.4

742

.12

1.17

snip

10x1

0a10

000

88.2

10.

1588

.19

0.22

0.39

0.45

wee

kly

sm50

00-3

610.

197.

40-3

614.

6511

.20

23.0

60.

64sn

ip10

x10b

5000

70.0

30.

2069

.78

0.31

0.76

1.09

wee

kly

sm10

000

-361

1.42

11.5

8-3

618.

416.

9425

.51

0.71

snip

10x1

0b10

000

70.0

60.

1069

.98

0.12

0.29

0.41

snip

10x1

0c50

0076

.87

0.10

76.6

70.

370.

680.

88sn

ip10

x10c

1000

076

.78

0.21

76.6

80.

260.

580.

75

105


Table 5.3: Count of the instances that the bias is within a specified value for bootstrapestimate

N # Instances Bias(< 5%) Bias(< 5%) Error(< 5%) Bias(< 1%) Bias(< 1%) Error(< 1%)5 63 42 51 51 12 26 28

10 63 48 55 56 28 39 4850 58 56 57 57 44 48 54

100 58 56 57 57 52 54 54200 50 49 49 50 46 48 48400 39 39 39 39 38 38 39800 19 19 19 19 19 19 19

5.4.2 Bias and Bias Estimate

The results in this subsection are obtained from the experiment mentioned at the beginning

of Section 5.4. The bias and the estimate of bias are computed according to (5.1) and (5.4).

The results are given in Appendix A.15 and the summary of these results are provided in

Table 5.3. In Table 5.3, there are three main columns. The first main column provides

the sample size and the number of instances tested for the corresponding sample size. The

second and the third main columns are the number of instances that pass the 5% and 1%

criteria for the amount of bias, the estimate of bias, and the estimate error respectively. The

bias and the estimate of bias are obtained by (5.1) and (5.4), where the estimate error is the

difference between these two quantities. The estimate error also indicates the difference

between the ’true’ objective value and the estimated objective value after correcting the

bias. As showed in Table A.15, a positive amount indicates an underestimate of the true

value, where a negative amount indicates an overestimate of the true value.

As we can see from Table 5.3, the biases in many tested instances are very small even

when using small sample sizes. For example, with a sample size of 50, more than 96% of

the instances have a true bias within 5%, and 75% of the instances have a bias of less than

1% (see the first sub-columns). If we correct the bias (see the last sub-columns), then the

number of instances with an estimation error within 5% and 1% go up to 98% and 93%

respectively. In some cases such as the instances phone and assets lg, the amount of bias is

small enough that it might be considered as sampling error. In general, the bias tends to be

underestimated, and the difference between the ’true’ bias and the estimated bias decreases

as the sample size increases. The magnitude of the amount of underestimate can be as large

106


as 10-25% in some cases depending on the sample size we used.

We illustrated the relationship of the bias and the estimate of bias via a scatter plot in

Figure 5.1. It is easy to see that the amount of bias has a positive correlation with the esti-

mate of bias. (The correlation coefficient is 0.5282.) The result is more evident when the

amount of bias is large. The plot also suggests that the amount of bias is mostly underesti-

mated. In Figure 5.2, we provide a few plots of the amount of bias and the estimate of bias

obtained by varying the sample size. These examples provide evidence that the amount of

bias was reduced as the sample size increases, and the amount of bias is underestimated

using the bootstrap method. When the sample size is large enough, as for example with

gbd in Figure 5.2 (b), the amount of bias could be small enough that the bootstrap method

cannot provide an accurate estimate. The example of ssn in Figure 5.2 (c) provides evidence

that the bias can be substantially underestimated when the sample size is small. This is due

to the nature of this particular instance. In particular, the objective values of the sampled

instances are mostly 0 when the sample size is very small. In summary, the bias estimate

may suggest when bias is present, but when the estimate is small, this does not mean that

the bias does not exist.

Observations

In our experiments, we have further investigated our results by using a CDF plot of the

objective value for both SAA and bootstrap instances. We make a few observations here.

First, there are instances for which the biases are overestimated (observe the negative values

in the last column of Table A.15), but the amount by which z∗ is overestimated is very small,

and this is more likely to happen when the sample size is large. For example, the amount of

overestimate is less than 0.2% for dcap243 500 and less than 0.4% for semi2. Because this

phenomenon does not arise in the other instances within the same class, this is likely caused

by sampling error. Second, the CDF plots are generally smooth due to the large number of

replications. However, in the chem instance, as well as some of the snip instances with small

sample sizes, the CDF plots appear stair-like. This happens because the objective value of

107


−5 0 5 10 15 20 25 30−20

0

20

40

60

80

100

Bias Estimate(%)

Bia

s(%

)

Figure 5.1: Scatter plot of the bias and bias estimate in percentage of the objective value

these instances can only take discrete values. This is more visible when the sample size

is small. Figure 5.3 shows the CDF plots of the chem instance with sample size of 50 and

400. Third, for the instance semi2, it seems that this instance has a fixed upper bound, i.e.,

we never obtain an objective value greater than 113.658 from our bootstrap experiment

results. As the sample size increases, an increasing number of results will have the value

113.658. We believed that this value is the optimal value for this instance. We also provide

CDF plots of this instance in Figure 5.4, as they appear different from the CDF plots of other

instances.

5.4.3 Upper Bound Bias

In the previous experiment, we provide results of the bootstrap method that show how to

obtain a reasonable estimate of the objective value in most instances, even when using only

small sample sizes. In practice, however, one may need a good solution of the problem in

addition to a good estimate of the objective value. This gives rise to another question: Can

108


101

102

0

2

4

6

8

10

Sample size

Bia

s(%

)

Bias and bias estimate with different sample size for electricity_lg

BiasBias estimate

101

102

103

0

1

2

3

4

5

6

Sample size

Bia

s(%

)

Bias and bias estimate with different sample size for gbd

BiasBias estimate

(a) electricity lg (b) gbd

101

102

103

0

20

40

60

80

100

120

Sample size

Bia

s(%

)

Bias and bias estimate with different sample size for ssn

BiasBias estimate

101

102

103

0

1

2

3

4

5

6

7

Sample size

Bia

s(%

)

Bias and bias estimate with different sample size for dcap332_300

BiasBias estimate

(c) ssn (d) dcap332 300

Figure 5.2: Stemplot of the bias and bias estimate in percentage

−1.5 −1.45 −1.4 −1.35 −1.3 −1.25 −1.2

x 104

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

z

F(z

)

CDF plots for chem with sample size 50

SAA samplesBootStrap samples

−1.37 −1.36 −1.35 −1.34 −1.33 −1.32 −1.31 −1.3 −1.29 −1.28 −1.27

x 104

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

z

F(z

)

CDF plots for chem with sample size 400


(a) N = 50 (b) N = 4O0

Figure 5.3: CDF plots for chem

109


75 80 85 90 95 100 105 110 1150

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

z

F(z

)

CDF plots for semi2 with sample size 5


98 100 102 104 106 108 110 112 1140

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

z

F(z

)

CDF plots for semi2 with sample size 15


(a) N = 5 (b) N = 15

Figure 5.4: CDF plots for semi2

Table 5.4: Head count of the instances that the bias is within a specified value for small NN # Instances LB Bias(< 5%) UB Bias(< 5%) LB Bias(< 1%) UB Bias(< 1%)5 63 37 36 13 10

10 63 49 43 20 2820 59 54 52 28 3350 58 56 55 39 43

we use the solutions obtained from our bootstrap procedure to get a good estimate of an

upper bound on the objective function? We conducted another experiment to provide a

better understanding of the answer to this question.

In this experiment, we solved 10 sampled instances for each sample size of 5, 10, 20,

and 50 to obtain a set of solutions. We used these solutions to assess quality. The reason

we use only small sample sizes is because using larger sample sizes is less practical, and

our goal is to find out how often solutions to instances with small sample size are of high

quality. However, when testing for the quality of solutions, we use a much larger sample

size of 100,100 to ensure that the evaluation of the objective values for candidate solutions

are accurate. We computed the 95% confidence interval of the lower bound and upper

bound estimates as described in Section 2.3. In addition, we computed the lower bound

bias and upper bound bias as a percentage of the true objective value. Numerical results

are provided in Table A.14 in Appendix A.3. We also provided the summary of Table A.14

in Table 5.4.

110


Table 5.4 shows the number of instances that have the lower or upper bound bias less

than 5% and 1%. As we can see, the results are not very good until the sample size reaches

50. However, this is problem-dependent. In many (more than 10) instances, both the lower

bound and upper bound biases are very small even with a sample size of 5. On the other

hand, many instances also have a large amount of bias either for the lower or upper bound

or both. For the instance ssn, for example, the biases are still very large for sample size

50, with 50% bias for the lower bound and 33% for the upper bound. Another example

is the instances in the 4node family, which have very large amount of upper bound biases,

while the lower bound biases are very small. This implies that solutions obtained from

instances with small sample size are less useful in practice. For these instances, increasing

the sample size is clearly a better option, with the cost being the solution of much more

difficult instances. Although we cannot always obtain a good solution by solving instances

with small sample size, it is not difficult to identify that the solution obtained is of low

quality simply by computing the estimated optimality gap.

5.4.4 Bootstrap Method with Limited Computational Budget

In this section, we further investigate our results from the bootstrap experiments in order to

answer the following question: If the computational budget was limited, then what choices

of trials (M) and replications (B) yield the best estimate? In our context, because the com-

putation time for solution of a sampled instance or a bootstrap instance is about the same

for a particular instance, the computational budget constraint we used is based on solving a

fixed number of instances (sampled instances plus bootstrap instances). We provide results

obtained by varying M and B, while keeping the total number of instances constant. Our

results are based on solving 200 instances. We used values of B = 0, 1, 2, 3, 9, 19, 39, 99 and

a truncated value of M from 200. For example, when B is 1, the value of M is 100 (100

sampled instances plus 100 bootstrap instances), while when B is 3, M is 50 (50 sampled

instances plus 50 × 3 bootstrap instances). The results are reported by using mean square

error (MSE) as our measurement criterion for all the test instances.

111


Table 5.5: The value of B for minimum MSE at different sample sizesN N

Instance 5 10 50 100 200 400 800 Instance 5 10 50 100 200 400 80020term 1 0 0 0 0 - - PGP3 19 9 1 2 1 1 14node 1024 2 2 0 0 0 0 - phone 0 0 0 0 0 0 -4node 16384 39 1 0 0 0 0 - product lg 99 1 0 0 - - -4node 32768 1 1 0 0 0 - - product md 1 1 0 0 0 - -4node 4096 1 99 0 0 0 - - product sm 1 0 0 0 0 0 -4node 8192 1 1 1 0 0 - - semi2 1 0 - - - - -AIRL1 1 1 0 0 0 0 0 semi3 19 99 - - - - -AIRL2 1 0 0 0 0 0 0 semi4 39 19 - - - - -assets lg 0 0 0 0 0 0 0 snip10x10a 9 1 1 1 - - -assets sm 1 0 0 0 0 0 0 snip10x10b 2 1 1 0 - - -biweekly lg 99 1 1 1 1 - - snip10x10c 9 3 1 1 - - -biweekly md 1 1 39 1 1 - - snip20x20 99 2 - - - - -biweekly sm 1 1 1 0 0 - - snip30x30 3 2 - - - - -chem 3 2 99 19 1 1 - snip4x9 2 2 1 1 1 1 -dcap233 200 0 0 0 0 0 0 0 snip7x5 2 1 2 1 2 2 -dcap233 300 99 0 0 0 0 0 0 sslp 10 50 100 0 0 0 0 0 0 -dcap233 500 0 0 0 0 0 0 0 sslp 10 50 1000 1 0 0 0 0 0 -dcap243 200 0 0 0 0 0 0 0 sslp 10 50 2000 0 0 0 0 0 0 -dcap243 300 0 0 0 0 0 0 0 sslp 10 50 50 2 1 0 0 0 2 -dcap243 500 0 0 0 0 0 0 1 sslp 10 50 500 1 0 0 0 0 0 -dcap332 200 99 1 1 2 1 0 0 sslp 15 45 10 2 0 0 0 0 0 -dcap332 300 39 3 1 1 2 1 1 sslp 15 45 15 2 0 0 0 0 0 -dcap332 500 1 1 2 0 0 0 0 sslp 15 45 5 1 0 0 0 0 0 -dcap342 200 1 0 0 0 0 0 0 sslp 5 25 100 0 0 0 0 0 0 0dcap342 300 0 0 0 0 0 0 0 sslp 5 25 50 1 1 0 0 0 0 -dcap342 500 0 0 0 0 0 0 0 ssn - 1 1 9 19 - -electricity lg 1 3 2 1 1 99 - stocfor2 1 3 0 0 0 0 -electricity md 1 1 0 1 0 0 - storm 1 0 0 0 - - -electricity sm 1 1 1 0 0 0 1 weekly lg 1 2 1 0 - - -gbd 99 0 0 0 0 - - weekly md 2 1 2 1 - - -LandS 0 0 0 0 0 - - weekly sm 2 1 1 1 - - -PGP1 0 3 0 3 0 0 -

The mean square errors are computed by (5.6) using the best-known objective value

for each instance. Numerical results are provided in Table A.16 in Appendix A.3. We also

provide a summary of Table A.16 in Table 5.5 by providing the value of B with the minimum

MSE for different sample sizes.

We make two observations from the results in Table 5.5. First, for a fixed sample size

N , there is an optimal value of B for minimizing the MSE. Second, as the sample size N

increases, the optimal value of B tends to decrease. In general, the optimal value of B is

often small except for instances with very small sample sizes. In most cases, the optimal

value of B is either 0 or 1 (where B = 0 is the case when not using the bootstrap method).

This can be explained as follow. Most of our test instances have a significant amount of bias

when the sample size is small, but the bias reduces rapidly as the sample size increases.

Therefore, when the bias is large, more bootstrap replications will be needed in order to

obtain an accurate of the bias estimate. When the bias is small, very few bootstrap repli-

cations will be needed for bootstrapping, and in some cases, it is better to not performing

bootstrapping at all.

112


5.4.5 Comparison of Bootstrap Method and ATR Method

In this section, we attempt to answer the question: How can the computational grid best

be used to get a high-quality solution (and bound) for a stochastic program? To do so,

we compared the bootstrap method to the SAA method (using ATR to solve the sampled

instances) in terms of computation time and accuracy of the estimate. The results were

taken from our previous experiments in this and other chapters. Note that we only provide

comparison for some instances where the results were obtained from the same computa-

tional grid, so that we could make direct comparison easily. For the computation time of

the ATR method, we used the results from the ATR version without applying any of the so-

phisticated methods we described in Chapters 3 and 4. For ease of recognition, we defined

a resource factor R, which is the fraction of computation time used in the bootstrap method

as compared to the time used in the ATR method. The computation time of the bootstrap

method was estimated based on the same number of processors used in the ATR method.

We also provide a 95% confidence interval of the estimated objective value using a few

different combinations of bootstrap sample sizes (N), number of trials (M), and number of

replications (B). In particular, we used values of 50, 100, 200, and 400 for N , and values

of 10 and 200 for both M and B. The results were then compared to the 95% confidence

interval of the estimated objective value computed using the ATR method. These statistics

are given in Table 5.6. The sample size of the instances solved with the ATR method is given

in the second column, and the value of N , M , and B for the bootstrap method are given in

the third column. The fifth and the sixth columns are the mean and the half length of the

95% confidence interval of the estimated objective value from the ATR method, while the

seventh and eighth columns are the mean and half length of the 95% confidence interval of

the estimated objective value (bias-adjusted) from the bootstrap method. The last column

is the resource factor R.

For each instance in Table 5.6, the value of the resource factor R is sorted in an in-

creasing order. For each sample size N , the values of M and B are altered between 10

and 200. This gives almost the same value of R for each pair of alternatives, although the

113


combination of M = 200 and B = 10 has a slightly larger value of R due to the additional

190 sampled instances needed to be solved. From the results in Table 5.6, we can see that

using the combination of M = 200 and B = 10 provides better result than the combination

of M = 10 and B = 200 for all values of N . More specifically, the combination of M = 200

and B = 10 gives a much smaller confidence band and a mean that is closer to the true

objective value in the fourth column. For larger values of N , i.e., N = 200 and 400, using

the combination of M = 200 and B = 10 also provides an estimate that is comparable or

sometimes better than the estimate from the ATR method, which is obtained by using a

much larger sample size. However, for the instance ssn (which has a large amount of bias

and a significant underestimate of the bias even for a sample size of 200), the true objec-

tive value falls outside the confidence band by a significant amount for the combination of

M = 200 and B = 10. On the other hand, the combination of M = 10 and B = 200 gives

a more conservative result. It is worth mentioning that solving this instance would even

be difficult using the ATR method. We can see from Table 5.6 that the true objective value

almost falls outside the confidence band with a sample size of 2000, and from Table 5.2

that the estimated gap is as high as 1.17%, even with a sample size of 50,000.

In terms of computation time for the configurations used in Table 5.6, the bootstrap

method appears to be faster than the ATR method in most cases, but the amount is less

than one order of magnitude for a similar level of accuracy. In instances 4node 32768 with

N = 200, snip7x5 with N = 400, and storm with N = 100, however, the computation

times for the bootstrap method were longer than the ATR method, but the results of the

estimate seem to be slightly better. In conclusion, it seems that in general, the bootstrap

method provides a fast alternative to SAA for achieving a high-quality solution and bounds

on optimal solution value.

114


Table 5.6: Comparison of lower bound estimate between the bootstrap method and the ATR

method

Instance N (ATR) NxMxB ’True’ Obj ATR Obj ATR h.l. BS Obj BS h.l. R

20term 2000 50x10x200 254277.20 254343.12 72.39 254181.24 904.47 0.02

20term 2000 50x200x10 254277.20 254343.12 72.39 254374.84 208.88 0.02

20term 2000 100x10x200 254277.20 254343.12 72.39 254676.99 405.76 0.08

20term 2000 100x200x10 254277.20 254343.12 72.39 254241.75 147.40 0.09

20term 2000 200x10x200 254277.20 254343.12 72.39 254428.22 439.63 0.37

20term 2000 200x200x10 254277.20 254343.12 72.39 254377.65 102.11 0.41

4node 32768 5000 50x10x200 446.85 446.86 0.03 447.07 0.72 0.08

4node 32768 5000 50x200x10 446.85 446.86 0.03 447.09 0.10 0.09

4node 32768 5000 100x10x200 446.85 446.86 0.03 446.77 0.19 0.32

4node 32768 5000 100x200x10 446.85 446.86 0.03 446.93 0.05 0.35

4node 32768 5000 200x10x200 446.85 446.86 0.03 446.86 0.09 1.16

4node 32768 5000 200x200x10 446.85 446.86 0.03 446.86 0.02 1.26

biweekly md 2000 50x10x200 -1468.56 -1470.50 4.50 -1489.12 42.85 0.02

biweekly md 2000 50x200x10 -1468.56 -1470.50 4.50 -1480.13 7.01 0.02

biweekly md 2000 100x10x200 -1468.56 -1470.50 4.50 -1468.91 21.32 0.06

biweekly md 2000 100x200x10 -1468.56 -1470.50 4.50 -1473.55 4.49 0.06

biweekly md 2000 200x10x200 -1468.56 -1470.50 4.50 -1481.14 21.11 0.23

biweekly md 2000 200x200x10 -1468.56 -1470.50 4.50 -1470.04 3.38 0.25

biweekly sm 2000 50x10x200 -8498.19 -8493.77 14.64 -8491.73 170.16 0.03

biweekly sm 2000 50x200x10 -8498.19 -8493.77 14.64 -8497.23 30.80 0.03

biweekly sm 2000 100x10x200 -8498.19 -8493.77 14.64 -8524.99 119.04 0.13

biweekly sm 2000 100x200x10 -8498.19 -8493.77 14.64 -8498.76 22.29 0.14

biweekly sm 2000 200x10x200 -8498.19 -8493.77 14.64 -8460.05 83.80 0.60

biweekly sm 2000 200x200x10 -8498.19 -8493.77 14.64 -8497.25 16.12 0.66

electricity lg 2000 50x10x200 -1818.75 -1820.03 5.92 -1791.92 41.04 0.01








product md 2000 50x10x200 -28226.79 -28235.25 25.37 -28212.76 246.66 0.02

product md 2000 50x200x10 -28226.79 -28235.25 25.37 -28243.19 48.59 0.02

product md 2000 100x10x200 -28226.79 -28235.25 25.37 -28189.56 157.81 0.06

product md 2000 100x200x10 -28226.79 -28235.25 25.37 -28230.42 34.89 0.07

product md 2000 200x10x200 -28226.79 -28235.25 25.37 -28204.67 83.79 0.26

product md 2000 200x200x10 -28226.79 -28235.25 25.37 -28244.47 23.79 0.29

product sm 2000 50x10x200 -17940.84 -17956.56 29.72 -17854.61 169.28 0.01

product sm 2000 50x200x10 -17940.84 -17956.56 29.72 -17927.36 36.29 0.01

product sm 2000 100x10x200 -17940.84 -17956.56 29.72 -17950.90 111.81 0.04

product sm 2000 100x200x10 -17940.84 -17956.56 29.72 -17972.92 23.87 0.04

product sm 2000 200x10x200 -17940.84 -17956.56 29.72 -17951.82 79.13 0.13

product sm 2000 200x200x10 -17940.84 -17956.56 29.72 -17936.91 20.31 0.14

product sm 2000 400x10x200 -17940.84 -17956.56 29.72 -17960.82 66.62 0.53

product sm 2000 400x200x10 -17940.84 -17956.56 29.72 -17941.25 12.93 0.58

snip4x9 5000 50x10x200 10.65 10.62 0.11 10.84 2.10 0.02

snip4x9 5000 50x200x10 10.65 10.62 0.11 10.48 0.34 0.02

snip4x9 5000 100x10x200 10.65 10.62 0.11 10.13 0.62 0.06

Continued on next page

115


Table 5.6 – continued from previous page

Instance N (ATR) NxMxB ’True’ Obj ATR Obj ATR h.l. BS Obj BS h.l. R

snip4x9 5000 100x200x10 10.65 10.62 0.11 10.39 0.21 0.07

snip4x9 5000 200x10x200 10.65 10.62 0.11 11.24 0.63 0.30

snip4x9 5000 200x200x10 10.65 10.62 0.11 10.47 0.16 0.33

snip4x9 5000 400x10x200 10.65 10.62 0.11 10.43 0.70 0.74

snip4x9 5000 400x200x10 10.65 10.62 0.11 10.59 0.12 0.81

snip7x5 5000 50x10x200 80.23 79.83 0.40 80.34 3.70 0.05

snip7x5 5000 50x200x10 80.23 79.83 0.40 78.39 0.88 0.05

snip7x5 5000 100x10x200 80.23 79.83 0.40 77.66 3.78 0.18

snip7x5 5000 100x200x10 80.23 79.83 0.40 79.35 0.61 0.20

snip7x5 5000 200x10x200 80.23 79.83 0.40 79.01 3.12 0.59

snip7x5 5000 200x200x10 80.23 79.83 0.40 79.69 0.43 0.65

snip7x5 5000 400x10x200 80.23 79.83 0.40 79.87 1.27 2.01

snip7x5 5000 400x200x10 80.23 79.83 0.40 80.15 0.31 2.19

sslp 10 50 2000 5000 50x10x200 -390.19 -390.82 0.57 -392.83 8.01 0.02

sslp 10 50 2000 5000 50x200x10 -390.19 -390.82 0.57 -389.34 1.38 0.02

sslp 10 50 2000 5000 100x10x200 -390.19 -390.82 0.57 -390.94 4.82 0.06

sslp 10 50 2000 5000 100x200x10 -390.19 -390.82 0.57 -390.22 0.93 0.06

sslp 10 50 2000 5000 200x10x200 -390.19 -390.82 0.57 -391.94 3.84 0.18

sslp 10 50 2000 5000 200x200x10 -390.19 -390.82 0.57 -390.27 0.61 0.20

sslp 10 50 2000 5000 400x10x200 -390.19 -390.82 0.57 -389.25 2.50 0.71

sslp 10 50 2000 5000 400x200x10 -390.19 -390.82 0.57 -390.54 0.44 0.78

ssn 2000 50x10x200 9.93 9.62 0.40 6.04 1.35 0.02

ssn 2000 50x200x10 9.93 9.62 0.40 7.34 0.42 0.02

ssn 2000 100x10x200 9.93 9.62 0.40 9.08 1.11 0.07

ssn 2000 100x200x10 9.93 9.62 0.40 8.61 0.27 0.08

ssn 2000 200x10x200 9.93 9.62 0.40 10.08 0.47 0.15

ssn 2000 200x200x10 9.93 9.62 0.40 9.42 0.18 0.16

stocfor2 5000 50x10x200 -39774.59 -39784.05 30.77 -39827.09 198.49 0.01

stocfor2 5000 50x200x10 -39774.59 -39784.05 30.77 -39769.14 48.92 0.01

stocfor2 5000 100x10x200 -39774.59 -39784.05 30.77 -39886.25 216.86 0.02

stocfor2 5000 100x200x10 -39774.59 -39784.05 30.77 -39753.97 42.58 0.02

stocfor2 5000 200x10x200 -39774.59 -39784.05 30.77 -39845.43 167.09 0.08

stocfor2 5000 200x200x10 -39774.59 -39784.05 30.77 -39775.28 28.14 0.08

stocfor2 5000 400x10x200 -39774.59 -39784.05 30.77 -39765.45 67.77 0.29

stocfor2 5000 400x200x10 -39774.59 -39784.05 30.77 -39772.60 21.28 0.32

storm 2000 50x10x200 15496361.11 15499283.48 7091.43 15474418.30 37605.77 0.36

storm 2000 50x200x10 15496361.11 15499283.48 7091.43 15492741.95 6163.44 0.39

storm 2000 100x10x200 15496361.11 15499283.48 7091.43 15493383.45 28035.76 1.17

storm 2000 100x200x10 15496361.11 15499283.48 7091.43 15499537.80 4656.73 1.28

weekly lg 2000 50x10x200 -2502.56 -2512.60 12.77 -2447.13 109.84 0.17

weekly lg 2000 50x200x10 -2502.56 -2512.60 12.77 -2497.74 17.46 0.19

weekly lg 2000 100x10x200 -2502.56 -2512.60 12.77 -2518.39 43.31 0.51

weekly lg 2000 100x200x10 -2502.56 -2512.60 12.77 -2497.13 12.31 0.56

weekly md 5000 50x10x200 -964.23 -964.32 7.44 -953.53 47.23 0.11

weekly md 5000 50x200x10 -964.23 -964.32 7.44 -965.63 10.68 0.12

weekly md 5000 100x10x200 -964.23 -964.32 7.44 -941.00 37.56 0.43

weekly md 5000 100x200x10 -964.23 -964.32 7.44 -965.64 7.45 0.48

weekly sm 2000 50x10x200 -3617.91 -3623.42 19.47 -3622.43 111.06 0.06

weekly sm 2000 50x200x10 -3617.91 -3623.42 19.47 -3622.44 23.53 0.07

weekly sm 2000 100x10x200 -3617.91 -3623.42 19.47 -3579.40 90.39 0.27

weekly sm 2000 100x200x10 -3617.91 -3623.42 19.47 -3628.88 18.59 0.30

116


5.4.6 Choosing the value of M and B

In this section, we provide guidelines for choosing appropriate values of M and B for boot-

strapping in order to estimate the bias using an SAA approach to solve stochastic programs.

In Sections 5.4.4 and 5.4.5, our results suggested that only a small value of B was needed

for a good estimate of bias. However, we only used a small number of results for the

comparisons in both Section 5.4.4 and Section 5.4.5 due to the computational budget con-

straint, and this may not be sufficient to draw a conclusion. In order to provide a better

understanding of how to choose a suitable value for M and B, we plotted the SAA sample

mean and the bootstrap sample mean of the objective value by varying the values of M and

B from 1 to 200, while fixing the other variable to 200.

Due to the large number of instances and the similarity of results, we only provide a

few examples of the plots in Figure 5.5. These instances were chosen to be instances with

a noticeable amount of bias. There are two plots for each instance in Figure 5.5. The plots

on the left were obtained by varying the value of B while fixing M to 200, and the plots on

the right were obtained by varying the value of M while fixing B to 200. In each plot, the

solid line is the true objective value, given for reference. The dash-dot line is the mean for

the SAA samples, and the dash line is the mean for the bootstrap samples. The true bias can

be identified as the distance between the true objective value and the SAA sample mean,

while the estimated bias is the distance between the SAA sample mean and the bootstrap

sample mean. To obtain the bootstrap-adjusted estimate of the objective value, one could

simply add the distance of the estimated bias on top of the SAA sample mean.

Our results suggest that larger values of M than B are required to obtain a reasonable

(more consistent) result, although the specific values are problem-dependent. In general,

a value of B between 10 and 30 is sufficient to accurately estimate the bootstrap sample

mean. Further increasing B seems not to improve the estimate of bias, as the bootstrap

sample mean remains fairly constant as B increases. This also agrees with our results in

117

5.5. CONCLUSION

Sections 5.4.4 and 5.4.5. On the other hand, the value of M needs to be large enough in

order to obtain a good estimate. From the results of our experiments, we conclude that the

value of M should be at least 50 to obtain a reasonable estimate, and a value larger than

100 is generally recommended.

These results also suggest that, when the value of M and B is “large enough”, the

estimated bias converges to a certain value for a fixed sample size N . If the estimate was

not “good enough”, as measured by an estimated gap between the lower and upper bounds,

a larger sample size N would be needed to close this gap.

5.5 Conclusion

In this chapter, we discussed experiments using the bootstrap method to estimate the bias in

the SAA method for solution of stochastic programs. Our results showed that the bootstrap

method can be an efficient method for estimating the bias, especially for sampled instances

with small sample sizes.

Most of our tested instances have bias, and the amount of bias decreases rapidly as

the sample size(N) of the sampled instance increases. The bootstrap method has proved

to be an effective method in this circumstance, as it requires solving instances of small

sample size. In addition, the bootstrap method also effectively utilizes the power of the

computational grid, and is easier to implement than the decomposition method.

The bias is generally underestimated, and the difference between the ’true’ bias and the

estimated bias decreases as the SAA sample size increases. In most instances, the samples

size required to obtain a “good” estimate of bias is small. By adjusting the bias, the boot-

strap method is capable of obtaining a reasonable estimate of the objective value, which is

comparable to the results of using the decomposition method to solve much larger sampled

instances.

Choosing the number of trials (M) and the number of replications (B) is also a key

factor in using the bootstrap method efficiently. Our experiments show that in order to

obtain an accurate estimate, a larger number of trials is required compared to the number

118

5.5. CONCLUSION

0 50 100 150 200−1860

−1855

−1850

−1845

−1840

−1835

−1830

−1825

−1820

−1815

number of bootstrap replications(B)

z

Objective value with different bootstrap replication for electricity_lg with sample size 50

z* SAA samples BootStrap samples

0 50 100 150 200−1870

−1860

−1850

−1840

−1830

−1820

−1810

−1800

−1790

number of trials(M)

z

Objective value with different trials for electricity_lg with sample size 50


(a1) electricity lg, N = 50, Vary B (a2) electricity lg, N = 50, Vary M

0 50 100 150 20094

96

98

100

102

104

106

108

110

112

114

number of bootstrap replications

z

Objective value with different number of bootstrap replication for semi4 with sample size 10


0 50 100 150 20090

95

100

105

110

115

number of bootstrap trials

z

Objective value with different number of bootstrap trial for semi4 with sample size 10


(b1) semi4, N = 10, Vary B (b2) semi4, N = 10, Vary M

0 50 100 150 200110

120

130

140

150

160

170

180

190


z

Objective value with different number of bootstrap replication for snip20x20 with sample size 5


0 50 100 150 200110

120

130

140

150

160

170

180

190


z

Objective value with different number of bootstrap trial for snip20x20 with sample size 5


(c1) snip20x20, N = 5, Vary B (c2) snip20x20, N = 5, Vary M

0 50 100 150 2007

7.5

8

8.5

9

9.5

10


z

Objective value with different number of bootstrap replication for ssn with sample size 200


0 50 100 150 2006.5

7

7.5

8

8.5

9

9.5

10


z

Objective value with different number of bootstrap trial for ssn with sample size 200


(d1) ssn, N = 200, Vary B (d2) ssn, N = 200, Vary M

Figure 5.5: SAA sample mean and bootstrap sample mean plots by varying B and M

119

5.5. CONCLUSION

of replications. This suggests that a value of B between 10 and 30 is sufficient, while a

value of M larger than 100 is generally recommended.

There are some limitations in using the bootstrap method. First, if the bias is large and

the estimate is not accurate enough, the sample size of the sampled instances needs to be

increased. This may significantly increase the total computation time, which makes the

bootstrap method less practical. Second, the maximum size of the sampled instances that

can be solved using the deterministic equivalent form is limited, especially for instances

with a very large second-stage problem. In our experiments, however, most instances are

able to obtain a sufficiently accurate estimate. If the estimate is not accurate enough for

a limited sample size, then other methods may be considered. Although it is possible to

solve the bootstrap instances using decomposition method, it may not be computationally

practical.

120

Chapter 6

Case Study: Solving Large Stochastic

Programs

In this study, we provide a comparison of different methods for solution of large-scale

stochastic programs. In particular, we use the methods or the combination of methods

discussed in Chapters 3 and 4 to solve 25 selected stochastic programming instances. The

experiments were conducted using six different settings, as described below.

Non: In this setting, we used the regular ATR method without any additional tech-

niques. This setting is provided as a reference for baseline comparison.

MngCuts: In this setting, the cut management methods discussed in Chapter 3 were

used, including the cut clean up method, the multi-level cut aggregation method, and

the cut removal method.

Dogleg: In this setting, the Dogleg search method discussed in Chapter 4 was used as

a representative of the parallel search methods. The number of search points was set

to 3 as it provides the best performance from our previous results.

Warm: In this setting, the scenario-partitioning method discussed in Chapter 3 was

used as a warm-starting method for the ATR method.

121

All: In this setting, we used a combination of the three previous settings above. (Mng-

Cuts, Dogleg, and Warm). setting.

All noSol: This is the same setting as the All setting, but without providing a starting

solution.

We ran two experiments on two different computational grids with the same set of

instances, but with different sample sizes. The first experiment was run by solving smaller

sampled instances on the grid polyps in COR@L Labartory, which is composed of 46 Intel

Pentium III 1.13GHz machines. The second experiment was performed by solving larger

sampled instances on a larger grid blaze, which is composed of 340 64 bit x86 64 CPU

cores (we only used 72 of them in this experiment). Comparing these two different grids,

polyps has fewer and slower processors, but it could provide more consistent results because

there is less sharing of resources. On the other hand, blaze has more powerful processors,

but the resources are not consistent due to heavy usage.

As pointed out in our previous results in Chapter 3, the number of clusters is a key factor

in the performance of the algorithm. The proper number of clusters is likely unknown to

the user because it depends on the problem being solved, as well as the computer being

used, especially for the master processor. A more powerful master processor will be able to

effectively handle a larger number of clusters for the same problem when compared to a

slower processor. In our experiments, we used a number of clusters of 100 to 200 for the

instances solved on polyps and of 200 to 500 for those on blaze.

The parameters used in our experiments were kept uniform for all six settings to the

degree this was possible. The only parameters changed were those related to each of the

different settings. The same cluster size was used and the same set of samples were gen-

erated for all settings in each instance. We did not use any scaling method in these exper-

iments because the difference is minor. The trust-region radius was set manually, and the

maximum running time for any setting was set to 6 hours or 21600 seconds.

122

6.1. COMPUTATIONAL RESULTS

6.1 Computational Results

We provided our results in the form of both performance profile plots and tables. The per-

formance profile plots are given for the wall clock time in Figure 6.1, for the total master

problem solution time in Figure 6.2, and for the number of iterations in Figure 6.3. The

results from the computational grids polyps and blaze are plotted side by side in these fig-

ures. Numerical results of these statistics are also given in Table 6.1 to 6.3 respectively. In

addition, the average number of worker processors during the computation is given in Table

6.4. In these tables, the results from polyps are given in sub-tables (a) and the results from

blaze are given in sub-tables (b).

2 4 6 8 10 12 14

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of wall clock time for large scale problems

AllAll_noSolWarmDoglegMngCutsNon

2 4 6 8 10 12 14

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of wall clock time for large scale problems


(a) On grid polyps (b) On grid blaze

Figure 6.1: Performance profile plots of wall clock time for large-scale instances

In terms of total computation time and master problem solution time, Figures 6.1 and

6.2 provide similar results for both the experiments run on polyps and blaze. The results

show that using any additional method could provide better performance than the ordi-

nary ATR method. It is clear that the Warm setting and the All setting provided the best

performance in these experiments. The All setting was only slightly better than the Warm

setting despite the additional two methods being used in combination. In the other settings,

the performances of the Dogleg and the MngCuts settings were about the same, where the

All noSol setting performed slightly better than these two settings.

A good starting point can reduce the computation time significantly, as we can see from

123

6.1. COMPUTATIONAL RESULTS

100 200 300 400 500 600 700 800 900 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of master problem time for large scale problems


100 200 300 400 500 600 700 800 900 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of master problem time for large scale problems



Figure 6.2: Performance profile plots of master problem solution time for large-scale in-stances

10 20 30 40 50 60

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of number of iterations for large scale problems


10 20 30 40 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Performance profiles of number of iterations for large scale problems



Figure 6.3: Performance profile plots of number of iterations for large-scale instances

124

6.2. CONCLUSION

the direct comparison of the performance profile plots between the All noSol and the All

settings, in which the former setting does not utilize a starting point. However, a bad

starting point can also increase the computation time significantly. For example, in the case

of instance snip20x20 (see Table 6.1), the All setting and Warm setting were the only two

settings that provided with a starting point. Both settings performed poorly compared to

the others, especially when compared to the cases of All noSol and Dogleg settings.

In terms of number of iterations, it is not necessary that the method with the fewest

iterations will be the fastest method because the time spent in each iteration could be

different. For example, for the instance snip20x20 in Table 6.3 (b), the MngCuts setting has

the largest number of iterations, but it is the third fastest method among the six settings.

The reason is because the total time for solution of this instance depends heavily on the

master problem solution time. On the other hand, the instance semi4 has a very large

second-stage problem, where the master problem is relatively easy to solve. As a result,

the total computation time is more likely to depend on the number of iterations, as we can

see in Tables 6.1 and 6.3. In general, providing a starting point can reduce the number

of iterations significantly, while using the cut management method tends to increase the

number of iterations due to the cut aggregation.

6.2 Conclusion

In this study, we tested the methods implemented in Chapters 3 and 4 for solution of large-

scale stochastic programs. The warm-starting method is used to obtain cuts and a starting

point for use with a decomposition method in order to reduce the total number of iterations.

The cut management methods and parallel search methods focus mainly on reduction of

master problem solution time, which is often the bottleneck in solving large-scale instances.

Our results show that using a combination of warm-starting and other methods gives

the best performance. However, it is not always the case that using all the methods will

be the best approach. In general, the best method is instance-specific. The warm-starting

method should be used for solution of difficult instances. In addition, if the instance has

125

6.2. CONCLUSION

a relatively easy master problem, then using cut aggregation may not be a good approach

because it could potentially increase the total number of iterrations. On the other hand, if

the master problem is difficult, then the parallel search methods and the cut management

methods are the better choices.

126

6.2. CONCLUSION

Table 6.1: Total computation time for solution of large-scale instances

Total time(secs.)Instance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 921.1 21868.5 751.7 3473.9 3947.1 4775.04node 32768 50000 616.2 3688.9 616.2 2041.2 1966.3 2128.1AIRL2 100000 207.2 452.9 191.6 612.9 565.5 767.1assets lg 100000 109.9 596.8 126.7 729.0 706.6 704.9biweekly lg 2000 535.4 1913.0 428.2 5471.6 2678.1 24190.3chem 100000 287.9 731.9 277.6 569.8 490.5 490.2dcap233 300 100000 509.1 279.3 277.7 3462.2 3042.0 3578.3dcap342 500 100000 340.6 804.1 344.9 4880.8 3812.1 3826.6electricity lg 10000 489.1 1784.7 488.1 1147.8 2211.6 2488.1gbd 100000 828.6 1928.6 788.3 1886.6 2249.4 1807.9LandS 100000 118.9 728.0 149.3 821.4 990.1 922.1PGP3 100000 250.5 427.6 241.9 396.0 437.8 436.7phone 200000 982.8 5124.5 759.4 1690.7 1467.4 1649.8product lg 5000 1880.3 2589.4 3300.5 24880.4 5449.3 24145.3semi4 1000 9754.6 18836.6 7780.9 12537.1 16591.1 16897.8snip10x10a 5000 458.6 730.8 1009.3 540.2 617.6 1152.0snip10x10b 5000 1255.7 584.3 1217.9 474.9 787.1 1698.5snip10x10c 5000 566.6 1430.5 1132.6 583.9 744.5 1665.0snip20x20 2000 19062.6 2904.5 21922.8 6586.1 21666.9 21997.4snip4x9 20000 378.9 542.3 396.0 894.0 671.2 1077.3snip7x5 20000 430.3 559.1 467.9 483.6 582.5 547.0ssn 20000 3124.7 10811.0 3863.4 7153.5 6426.4 10012.3stocfor2 20000 82.2 593.8 81.9 597.0 841.1 838.0storm 20000 710.2 2649.2 759.1 2366.3 2232.8 2282.1weekly lg 20000 6658.4 8017.2 4253.2 5672.3 8776.7 10808.9

(a) On grid polyps

Total time(secs.)Instance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 656.6 1918.6 450.3 11209.4 1471.6 2096.74node 32768 100000 323.9 2984.6 529.4 2723.2 3750.4 3335.9AIRL2 500000 612.2 4588.0 349.3 4696.4 4040.8 4206.6assets lg 500000 678.9 5149.8 515.4 6310.0 5661.3 5506.1biweekly lg 10000 1266.9 910.6 1541.7 1839.9 1325.1 2185.7chem 200000 1038.1 687.1 813.3 2147.8 1131.1 1279.6dcap233 300 200000 979.6 816.3 540.6 5153.7 4464.8 4421.3dcap342 500 200000 627.1 494.2 497.7 5314.8 5849.5 4657.3electricity lg 20000 811.8 2848.8 1162.6 2000.9 3235.1 2860.3gbd 100000 775.0 1653.1 452.6 1003.5 764.4 1267.6LandS 100000 100.4 941.0 91.9 491.7 439.5 1158.3PGP3 200000 843.5 2218.2 1680.4 2608.7 1791.6 1849.4phone 500000 2197.6 4963.1 1509.8 2888.9 3174.9 2950.2product lg 10000 1395.5 2292.8 2146.8 3626.7 4165.0 5742.6semi4 1000 1672.9 2171.8 2084.8 2030.3 2932.3 2520.2snip10x10a 20000 1777.4 1707.0 2570.3 4409.8 1313.6 1531.3snip10x10b 20000 2529.9 1775.2 2930.9 1505.6 2267.1 2818.5snip10x10c 20000 2008.4 2398.8 2529.9 1404.9 2470.5 2919.8snip20x20 2000 6102.8 737.2 20661.6 1929.7 5539.2 15045.2snip4x9 20000 496.0 643.3 524.3 849.8 328.3 474.9snip7x5 50000 1002.2 1787.8 857.5 1148.6 1076.2 1629.2ssn 20000 1040.6 3521.5 1504.0 2471.0 2395.2 2388.9stocfor2 50000 110.0 884.5 396.7 732.6 1129.0 837.1storm 20000 446.7 1130.2 471.9 902.6 1398.4 1617.0weekly lg 10000 1093.2 879.0 1153.3 843.6 916.0 1456.0

(b) On grid blaze

127

6.2. CONCLUSION

Table 6.2: Master problem solution time for large-scale instances

Master Problem Time(secs.)Instance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 17.580 1552.000 11.680 749.800 420.900 1503.0004node 32768 50000 0.024 52.420 0.020 19.330 35.370 51.920AIRL2 100000 0.024 0.496 0.040 2.480 2.476 2.652assets lg 100000 0.044 0.272 0.044 0.984 2.120 2.080biweekly lg 2000 256.500 1338.000 286.700 640.600 2035.000 1355.000chem 100000 0.216 2.564 0.244 4.920 5.336 5.608dcap233 300 100000 0.008 0.004 0.008 3.336 3.524 3.664dcap342 500 100000 0.004 0.020 0.008 2.636 2.328 2.528electricity lg 10000 10.710 151.400 20.710 119.500 433.100 914.800gbd 100000 1.176 7.520 1.968 28.510 52.700 36.910LandS 100000 0.088 3.116 0.072 9.697 17.090 14.630PGP3 100000 0.224 5.712 0.42 3.212 9.629 8.009phone 200000 0.016 0.192 0.012 0.032 0.036 0.044product lg 5000 308.100 542.900 2090.000 3609.000 1689.000 5061.000semi4 1000 38.490 162.600 68.700 141.800 187.000 668.100snip10x10a 5000 60.430 69.520 626.600 249.200 244.900 864.500snip10x10b 5000 563.800 73.360 781.900 153.200 286.800 1322.000snip10x10c 5000 110.900 494.000 732.400 265.800 295.200 1322.000snip20x20 2000 14090.000 901.200 20480.000 5843.000 14610.000 21380.000snip4x9 20000 26.170 31.510 38.840 117.800 117.500 295.300snip7x5 20000 36.730 57.720 57.910 40.230 91.260 122.100ssn 20000 376.800 1417.000 1459.000 3023.000 1102.000 5615.000stocfor2 20000 0.036 8.949 0.032 12.690 15.470 25.080storm 20000 1.088 43.490 0.708 79.530 128.800 125.300weekly lg 20000 229.400 283.500 183.700 335.500 1191.000 2265.000

(a) On grid polyps

Master Problem Time(secs.)Instance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 8.891 155.100 7.074 285.400 155.900 527.3004node 32768 100000 0.011 17.9 0.0 22.4 32.3 51.5AIRL2 500000 0.003 0.375 0.003 1.022 1.440 1.427assets lg 500000 0.020 0.278 0.021 0.492 1.096 1.064biweekly lg 10000 306.800 270.700 256.500 692.400 342.600 1216.000chem 200000 0.312 1.342 0.350 1.966 3.503 3.110dcap233 300 200000 0.003 0.004 0.003 1.869 2.074 2.076dcap342 500 200000 0.003 0.002 0.002 1.107 1.778 1.630electricity lg 20000 7.765 52.380 11.170 48.190 223.000 506.200gbd 100000 0.205 1.552 0.320 4.107 6.444 6.962LandS 100000 0.029 0.721 0.031 1.252 2.226 3.649PGP3 200000 0.367 2.570 1.717 3.130 5.082 4.119phone 500000 0.007 0.023 0.007 0.017 0.019 0.018product lg 10000 152.200 338.700 790.500 1631.000 910.000 3838.000semi4 1000 11.370 21.440 13.850 22.470 57.110 105.100snip10x10a 20000 179.800 44.680 545.000 389.000 216.100 625.300snip10x10b 20000 293.200 22.860 591.000 127.300 366.200 1309.000snip10x10c 20000 212.000 221.700 478.200 178.300 417.400 1177.000snip20x20 2000 5054.000 234.100 19890.000 1074.000 4243.000 14680.000snip4x9 20000 5.055 8.162 6.100 11.690 25.490 26.650snip7x5 50000 8.495 12.220 9.041 5.944 25.110 33.570ssn 20000 88.830 337.400 158.500 403.300 289.500 871.200stocfor2 50000 0.037 5.927 0.033 8.340 8.736 10.990storm 20000 0.184 4.436 0.256 9.412 16.150 18.890weekly lg 10000 17.820 30.020 38.980 41.010 135.800 180.700

(b) On grid blaze

128

6.2. CONCLUSION

Table 6.3: Number of iterations for solution of large-scale instances

Number of IterationsInstance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 38 1228 28 181 216 2374node 32768 50000 2 126 2 69 75 81AIRL2 100000 8 23 7 33 30 31assets lg 100000 2 22 2 28 27 27biweekly lg 2000 53 165 36 71 201 79chem 100000 11 37 10 49 41 41dcap233 300 100000 2 2 2 27 24 26dcap342 500 100000 2 4 2 28 23 23electricity lg 10000 32 134 32 80 164 156gbd 100000 20 48 19 47 56 45LandS 100000 2 28 2 32 39 36PGP3 100000 17 36 16 33 37 33phone 200000 2 13 2 5 5 5product lg 5000 103 165 96 160 361 152semi4 1000 138 292 109 189 269 263snip10x10a 5000 116 138 137 91 152 114snip10x10b 5000 241 126 170 102 215 166snip10x10c 5000 142 258 147 107 188 146snip20x20 2000 789 318 194 128 1412 113snip4x9 20000 30 50 32 59 62 63snip7x5 20000 38 50 39 47 56 47ssn 20000 74 245 68 114 157 123stocfor2 20000 2 100 2 104 106 106storm 20000 8 51 7 48 49 50weekly lg 20000 65 80 42 63 92 102

(a) On grid polyps

Number of IterationsInstance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 38 264 28 171 207 2034node 32768 100000 2 79 2 87 111 108AIRL2 500000 2 32 2 37 32 32assets lg 500000 2 26 2 34 30 30biweekly lg 10000 60 98 46 106 151 99chem 200000 17 40 15 52 43 43dcap233 300 200000 2 2 2 27 24 24dcap342 500 200000 2 2 2 25 23 22electricity lg 20000 33 121 34 77 172 166gbd 100000 17 53 15 47 52 47LandS 100000 2 34 2 35 29 44PGP3 200000 17 36 20 41 32 34phone 500000 2 7 2 5 5 5product lg 10000 63 154 80 149 275 172semi4 1000 105 188 103 142 203 216snip10x10a 20000 195 166 186 160 120 137snip10x10b 20000 272 137 235 134 234 234snip10x10c 20000 200 238 172 136 224 189snip20x20 2000 627 231 303 135 1100 238snip4x9 20000 40 54 35 57 55 56snip7x5 50000 51 77 42 44 62 69ssn 20000 65 238 62 119 131 134stocfor2 50000 2 102 2 102 103 105storm 20000 8 45 8 42 49 51weekly lg 10000 44 68 46 66 85 86

(b) On grid blaze

129

6.2. CONCLUSION

Table 6.4: Number of average workers for solution of large-scale instances

Number of Average WorkersInstance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 38.4 44.3 36.4 43.7 41.2 43.84node 32768 50000 10.9 43.6 11.0 37.7 42.7 42.9AIRL2 100000 22.7 34.9 20.7 37.6 36.9 28.1assets lg 100000 12.2 37.0 10.4 38.6 38.5 38.5biweekly lg 2000 35.7 42.5 33.6 44.1 43.2 44.8chem 100000 21.5 24.6 20.3 37.0 35.6 35.7dcap233 300 100000 14.9 27.6 27.5 43.3 43.5 40.3dcap342 500 100000 30.5 23.5 30.1 42.3 43.8 43.8electricity lg 10000 35.4 42.4 35.5 40.0 42.9 43.1gbd 100000 39.4 42.6 39.1 42.6 43.0 42.4LandS 100000 10.4 38.7 7.7 39.5 40.4 40.0PGP3 100000 26.5 34.3 25.7 33.6 34.5 30.6phone 200000 26.9 44.1 37.3 36.3 41.9 37.2product lg 5000 35.8 42.2 40.7 44.8 44.1 44.8semi4 1000 44.3 44.7 44.2 44.5 44.7 44.1snip10x10a 5000 33.5 23.7 40.2 36.4 37.6 41.0snip10x10b 5000 41.3 28.1 41.2 35.2 39.1 42.3snip10x10c 5000 36.5 34.4 40.8 37.1 38.8 42.2snip20x20 2000 44.1 42.2 44.2 44.3 44.8 44.8snip4x9 20000 32.1 36.3 32.9 28.4 38.2 31.3snip7x5 20000 35.7 36.3 34.3 35.5 37.2 36.5ssn 20000 42.4 44.2 43.7 43.6 44.3 43.5stocfor2 20000 3.4 37.1 3.4 37.3 27.0 27.3storm 20000 32.8 43.0 23.9 39.5 42.9 42.9weekly lg 20000 42.2 42.4 43.9 44.3 44.4 43.6

(a) On grid polyps

Number of Average WorkersInstance N All All nosol Warm Dogleg3 ManageCut Non20term 20000 22.4 55.1 23.9 5.9 56.2 49.34node 32768 100000 12.6 57.7 5.9 68.1 62.9 68.8AIRL2 500000 21.8 59.8 37.0 67.6 67.5 64.7assets lg 500000 27.6 63.6 39.1 68.5 66.3 67.7biweekly lg 10000 18.1 39.9 9.3 29.4 46.6 45.8chem 200000 13.8 52.3 15.1 20.3 31.5 28.2dcap233 300 200000 15.5 18.1 24.8 58.5 60.0 60.4dcap342 500 200000 29.0 37.2 37.3 69.7 58.0 69.4electricity lg 20000 42.8 47.3 34.1 41.4 57.0 65.1gbd 100000 17.4 27.4 26.4 39.8 57.9 32.0LandS 100000 6.3 19.3 6.8 37.7 36.4 20.6PGP3 200000 15.2 12.8 9.0 12.5 14.2 15.4phone 500000 40.9 69.7 57.6 68.3 62.0 67.8product lg 10000 48.6 66.9 52.6 64.6 66.4 69.2semi4 1000 40.1 48.1 31.4 41.6 38.4 49.6snip10x10a 20000 65.6 52.6 49.7 20.6 59.4 65.9snip10x10b 20000 65.2 41.8 52.8 49.8 58.0 68.2snip10x10c 20000 59.3 58.2 45.7 55.2 52.3 55.6snip20x20 2000 69.7 55.0 70.5 33.5 70.0 71.0snip4x9 20000 15.3 17.3 12.9 13.0 32.9 23.6snip7x5 50000 53.1 43.0 49.5 36.5 55.3 40.8ssn 20000 54.2 63.0 37.1 48.3 48.0 64.4stocfor2 50000 5.1 48.3 1.2 57.2 37.6 52.5storm 20000 16.7 41.1 15.8 45.8 34.1 30.5weekly lg 10000 22.2 51.2 23.1 43.2 53.3 36.4

(b) On grid blaze

130

Chapter 7

Conclusion

7.1 Contributions

In this thesis, we focused on developing and improving computational techniques for so-

lution of large-scale stochastic programs by utilizing computational grids. We studied and

tested our techniques on a large collection of stochastic programming instances.

The techniques discussed in Chapters 3 and 4 are based on a decomposition approach.

In Chapter 3, we proposed two warm-starting methods that could be used prior to solving

the instance with the L-shaped decomposition method. The first method is the pre-sampling

method, which solves one small sampled instance of the original instance to obtain a start-

ing solution and cuts. The second method is the scenario-partitioning method, in which the

set of scenarios is partitioned into subsets to form small sampled instances, and each small

instance is solved in parallel to obtain cuts and solutions. A starting solution can be obtained

by using a linear combination of solutions from these small instances. From our results, the

total computation time can be reduced significantly by using these approaches, especially

the scenario-partitioning method. The warm starting time is relatively small compared to

the total computation time.

One difficulty in solving large-scale stochastic programs with the decomposition method

is that the master problem becomes larger as cuts accumulate during the computation. We

suggested a cut management mechanism to handle this difficulty in order to maintain a

131

7.1. CONTRIBUTIONS

proper size of the master problem. We developed three different methods to reduce the

number of cuts in the master problem. We monitor the behavior of the algorithm in or-

der to determine when each method should be used. The cut clean-up method aggregates

cuts from the previous iterations. The multi-level cut aggregation method aggregates cuts

obtained from the current iteration before adding to the master problem. The cut removal

method removes cuts if a cut remains inactive for a long period of time. Our results showed

that by using these methods, a significant performance improvement can be achieved, es-

pecially when solving difficult instances and using a decomposition approach with many

clusters.

In Chapter 4, we used nonlinear programming techniques to improve a sophisticated

asynchronous trust-region based method for solution of two-stage stochastic linear pro-

grams. First, we showed how an initial trust-region radius can be obtained by using the sub-

gradient information of the recourse function. As the trust-region radius is also constantly

adjusted during the computation in our algorithm, we found that the initial trust-region

radius is not a very important factor in the overall performance. Second, we employed

three scaling techniques to adjust the trust-region: using the previous step size, using the

subgradient information, and using the BFGS method. The idea of scaling the trust-region

is to adjust the shape of the trust-region in order to better handle problems with variables

having different scales. By using these scaling techniques, we demonstrated some slight

improvement over the original algorithm. Third, we used three different search methods

to perform parallel search. These methods are called line search, zigzag search, and dogleg

search according to the pattern of the search paths. Only one master problem needs to be

solved to obtain a feasible path, and points are generated along this feasible path to reduce

the time for finding a candidate solution. The maximum number of points that can be gen-

erated for a given path is limited by a parameterM. Our computational results showed that

the performance of the original algorithm was significantly improved when using a small

value of M.

Sample average approximation (SAA) method is an efficient method for solving large

stochastic programs approximately. However, this method also introduces bias into the

132

7.2. FUTURE RESEARCH

estimate of the optimal solution value. In Chapter 5, we introduced the bootstrap method

as a statistical tool to estimate the bias in using the SAA method. The bootstrap method

repeatedly solves a number of bootstrap instances that are formed by drawing samples (with

replacement) from the sampled instance. The bias is then estimated from the difference

of the objective values between the sampled instance and the bootstrap instances. The

estimated bias is then used to adjust the objective value in order to provide a better estimate

of the optimal value. In our studies, the estimate of bias is generally an underestimate of

the true bias. The difference of the bias and the estimated bias can be reduced by increasing

the sample size of the sampled instances. Our results showed that, the bootstrap method

is capable of producing results that are comparable to the results obtained when using the

ATR method, and it is a very effective method in terms of computation for approximately

solving stochastic programs.

The contributions made in this thesis have led to improvements in the solution tech-

niques for solution of large-scale stochastic programs, making stochastic programming a

better practical tool for solving real-life application problems. We also demonstrated dif-

ferent ways of utilizing the power of parallel computing to solve stochastic programs. In

addition, the collection of stochastic programming instances used in this thesis and the re-

sults obtained from our study is also a contribution to many researchers for future studies

in stochastic programming.

7.2 Future Research

7.2.1 Inexact Master Solution

One difficulty in the decomposition method is that the master problem becomes larger as

cuts are accumulated during the computation, resulting in a more difficult master problem

over time. In fact, it is not necessary to solve the master problem to optimality at each itera-

tion in order to provide a “good” candidate solution. The idea of the inexact master solution

method is to provide a feasible candidate solution without solving the master problem opti-

mally. However, there are still open questions remain for this approach such as determining

133


good stopping criteria, determining whether the inexact solutions are of high quality, and

determining how this affects the convergence rate of the algorithm.

7.2.2 Updating the Trust-region Radius

Another important area that we feel deserves additional research is the trust-region updat-

ing for stochastic programs. In general, the size of the trust-region radius is determined

by how accurately the model function predicts the actual function, and different rules can

be used to determine how the radius should be increased or reduced. Different updating

procedures influence the overall performance of the algorithm. We may consider other up-

dating rules besides the “traditional” rules in future work. For example, the previous step

size may be considered when determining the amount of radius increase or decrease. In

addition, the trust-region updating procedure relies on a set of parameters that may behave

differently depending on the problem type. A large number of instances should be tested to

determine the appropriate values for these parameters.

7.2.3 m Out of n Bootstrap

The bootstrap method requires the solution of many bootstrap instances to obtain an accu-

rate estimate. Therefore, the size of the instance has a large influence on how effective the

bootstrap method is in solving practical problems. On the one hand, increasing the sample

size provides a more accurate result, while on the other hand, it also significantly increases

the total computation time. In some instances, especially for instances with a large second-

stage, the bias can be still large and far underestimated even when solving with the largest

sample size that can be solved in an extensive form. This leads to a new idea of applying

the m out of n bootstrap method [8] (which is also known as the subsampling bootstrap

method [52]) to estimate the bias. The m out of n bootstrap re-samples a small sample size

of m samples from a larger sample size of n samples to obtain an estimate. This method pro-

vides more consistent results than the classical bootstrap method for ill-behaved problems

[52], and it also requires solving smaller sampled instances, which could be a promising

134


approach for estimating bias in stochastic programming.

135

Appendix A

Tables

A.1 Statistics on Test Instances

Table A.1 provides the estimate of the first-stage cost and the estimate of the objective value

for all test instances. These results are obtained by solving 10 SAA problems of size N to

obtain a set of solutions, and then evaluate each solution by using a sample size of 10N .

The results in Table A.1 are computed based on the average values of 10 trials. The name

of the instances, the sample size, the average objective value, and the average first-stage

cost are listed in each column accordingly.

136

A.1. STATISTICS ON TEST INSTANCES

Table A.1: Average of the objective value and the first-stage costInstance 10N z fsc20term 20000 254340.164 53355.0004node 1024 50000 434.107 355.4134node 16384 50000 446.843 368.1564node 32768 50000 446.853 368.1564node 4096 50000 446.850 368.1564node 8192 50000 446.849 368.156AIRL1 50000 249084.680 256738.709AIRL2 50000 269608.853 267298.765assets lg 50000 -695.547 0.000assets sm 50000 -724.086 0.000biweekly lg 20000 -7206.728 -10418.882biweekly md 20000 -1468.368 470.276biweekly sm 20000 -8496.105 539.964chem 50000 -12998.118 11270.833dcap233 200 100000 641.075 129.305dcap233 300 100000 553.997 122.885dcap233 500 100000 586.877 87.782dcap243 200 100000 1168.987 157.594dcap243 300 100000 1273.237 148.786dcap243 500 100000 1048.635 150.554dcap332 200 100000 162.746 110.512dcap332 300 100000 178.263 127.917dcap332 500 100000 130.410 85.630dcap342 200 100000 332.556 119.008dcap342 300 100000 373.143 122.121dcap342 500 100000 355.117 135.154electricity lg 20000 -1818.623 2978.527electricity md 20000 -5865.937 6428.654electricity sm 20000 -1440.155 1881.335gbd 50000 1651.939 882.789LandS 100000 225.487 97.609PGP1 50000 438.402 157.500PGP3 200000 496.283 197.002phone 50000 36.895 0.000product lg 20000 -29863.039 1415.365product md 20000 -28220.557 7316.124product sm 20000 -17938.671 5275.445semi2 10000 113.658 113.658semi3 10000 113.699 113.454semi4 10000 113.016 111.451snip10x10a 100000 88.174 0.000snip10x10b 100000 70.019 0.000snip10x10c 100000 76.769 0.000snip20x20 50000 182.297 0.000snip30x30 20000 227.522 0.000snip4x9 50000 10.707 0.000snip7x5 50000 80.270 0.000sslp 10 50 100 50000 -398.838 159.235sslp 10 50 1000 50000 -395.419 156.233sslp 10 50 2000 50000 -390.240 154.731sslp 10 50 50 50000 -407.910 160.268sslp 10 50 500 50000 -393.056 155.750sslp 15 45 10 50000 -273.419 200.403sslp 15 45 15 50000 -270.440 198.801sslp 15 45 5 50000 -266.401 199.345sslp 5 25 100 50000 -169.448 78.919sslp 5 25 50 50000 -163.631 76.261ssn 20000 9.981 0.000stocfor2 50000 -39774.880 -5203.233storm 20000 15497756.110 5827229.911weekly lg 100000 -2502.444 11722.000weekly md 100000 -963.891 7000.000weekly sm 100000 -3617.663 18500.005

137

A.2. STATISTICS FOR SEARCH METHODS

A.2 Statistics for Search Methods

We refer the results in Table A.2 - A.13 to Section 4.3.3 in Chapter 4. The first and the second

columns in these tables are the name of the instances and the sample size respectively. The

rest of the columns are the statistics for using different number of search points from 1 to 5

respectively. Note that the column ATR is where the number of search point equal to 1.

Table A.2: Average wall clock time for line searchAverage Wall Clock Time for Line Search(secs.)

Instance N ATR 2 points 3 points 4 points 5 points20term 2000 869.0 535.8 546.8 578.4 580.14node 32768 5000 204.5 262.1 187.5 237.0 187.9AIRL2 20000 187.8 155.6 138.0 188.3 214.3assets lg 20000 107.3 135.1 142.8 182.4 193.2biweekly md 2000 908.8 531.1 471.9 376.4 535.1biweekly sm 2000 375.1 284.6 252.9 313.8 302.7chem 20000 177.8 226.9 144.7 143.9 204.7dcap233 500 20000 176.7 175.1 192.3 237.8 237.9dcap243 500 20000 182.2 173.8 174.0 211.1 174.1dcap332 300 20000 187.0 199.5 192.7 202.2 203.2dcap342 500 20000 196.2 201.3 197.5 197.7 254.0electricity lg 2000 572.6 326.8 301.8 304.6 305.5electricity md 5000 412.2 337.5 328.9 357.3 339.4gbd 10000 103.5 191.4 105.9 114.1 185.6LandS 20000 208.3 121.4 241.2 201.0 123.6PGP1 10000 75.8 71.1 76.1 77.5 80.4PGP3 50000 180.9 231.0 233.0 169.6 261.3phone 20000 112.6 113.7 188.9 140.4 140.7product lg 1000 2706.7 1126.2 929.9 941.8 894.8product md 2000 1489.1 714.3 673.0 642.5 752.1product sm 2000 280.7 229.7 216.1 218.8 234.5snip4x9 4 5000 306.3 299.0 305.9 298.9 309.7snip7x5 4 5000 211.8 223.7 208.6 172.5 207.6sslp 10 50 2000 5000 788.2 431.7 449.5 434.6 432.8sslp 5 25 100 10000 302.8 297.7 223.8 289.5 211.4ssn 2000 1328.1 740.4 678.3 707.9 653.9stocfor2 5000 325.5 310.3 318.4 360.2 310.9storm 2000 197.8 185.2 307.7 182.5 181.6weekly lg 2000 735.6 560.2 598.0 460.7 472.4weekly md 5000 463.0 509.9 466.6 484.7 414.6weekly sm 2000 1016.8 663.3 679.6 547.4 600.4

138


Table A.3: Average wall clock time for zigzag searchAverage Wall Clock Time for Zigzag Search(secs.)


139


Table A.4: Average wall clock time for dogleg searchAverage Wall Clock Time for Dogleg Search(secs.)


140


Table A.5: Average master problems solving time for line searchAverage Master Problems Solving Time for Line Search(secs.)


141


Table A.6: Average master problems solving time for zigzag searchAverage Master Problems Solving Time for Zigzag Search(secs.)


142


Table A.7: Average master problems solving time for dogleg searchAverage Master Problems Solving Time for Dogleg Search(secs.)


143


Table A.8: Average number of master problems solved for line searchAverage Number of Master Problems Solved for Line Search


144


Table A.9: Average number of master problems solved for zigzag searchAverage Number of Master Problems Solved for Zigzag Search


145


Table A.10: Average number of master problems solved for dogleg searchAverage Number of Master Problems Solved for Dogleg Search


146


Table A.11: Average number of points evaluated for line searchAverage Number of Points Evaluated for Line Search


147


Table A.12: Average number of points evaluated for zigzag searchAverage Number of Points Evaluated for Zigzag Search


148


Table A.13: Average number of points evaluated for dogleg searchAverage Number of Points Evaluated for Dogleg Search


149

A.3. STATISTICS FOR BOOTSTRAP METHOD

A.3 Statistics for Bootstrap Method

We refer the results in Table A.14 to Section 5.4.3.

Table A.14: Lower bound and upper bound biases with 95% confidence interval for prob-

lems of small sample sizes

Instance N z∗ 95% CI LB Band 95% CI UB Band LB Bias(%) UB Bias(%)

20term 5 254313.00 254970.72±2739.17 256519.14±1194.13 -0.26 0.87

20term 10 254313.00 254070.81±2053.67 255220.42±909.26 0.10 0.36

20term 20 254313.00 254384.66±2365.49 254790.45±485.96 -0.03 0.19

20term 50 254313.00 254340.15±717.63 254303.18±62.51 -0.01 0.00

4node 1024 5 434.11 431.40±4.32 734.77±301.97 0.62 69.26

4node 1024 10 434.11 433.32±1.34 516.26±121.46 0.18 18.92

4node 1024 20 434.11 434.14±0.25 438.93±10.95 -0.01 1.11

4node 1024 50 434.11 434.19±0.16 434.10±0.01 -0.02 0.00

4node 16384 5 446.85 438.71±2.96 948.27±185.31 1.82 112.21

4node 16384 10 446.85 442.53±2.54 687.46±119.30 0.97 53.85

4node 16384 20 446.85 445.34±1.58 521.48±38.75 0.34 16.70

4node 16384 50 446.85 446.43±0.92 460.64±18.87 0.09 3.09

4node 32768 5 446.85 437.94±3.48 902.44±201.88 1.99 101.95

4node 32768 10 446.85 440.47±4.57 769.89±175.70 1.43 72.29

4node 32768 20 446.85 444.64±1.64 538.76±39.33 0.50 20.57

4node 32768 50 446.85 446.87±0.21 457.84±14.22 0.00 2.46

4node 4096 5 446.85 439.17±4.44 873.95±258.48 1.72 95.58

4node 4096 10 446.85 442.20±1.82 668.70±108.40 1.04 49.65

4node 4096 20 446.85 444.71±1.90 548.82±74.22 0.48 22.82

4node 4096 50 446.85 446.88±0.43 469.99±22.73 -0.01 5.18

4node 8192 5 446.85 440.37±4.77 984.80±295.19 1.45 120.39

4node 8192 10 446.85 440.81±1.77 726.12±119.89 1.35 62.50

4node 8192 20 446.85 444.27±1.62 584.89±69.80 0.58 30.89

4node 8192 50 446.85 447.06±0.25 451.96±5.34 -0.05 1.14

AIRL1 5 249080.44 248219.67±2851.45 250580.03±1192.60 0.35 0.60

AIRL1 10 249080.44 249883.97±2387.95 249763.55±750.79 -0.32 0.27

AIRL1 20 249080.44 249058.90±1072.66 249323.93±236.22 0.01 0.10

AIRL1 50 249080.44 249433.80±998.88 249156.99±80.59 -0.14 0.03

AIRL2 5 269618.86 268434.43±4565.65 270948.42±1355.74 0.44 0.49

AIRL2 10 269618.86 268131.31±2129.37 269710.65±198.68 0.55 0.03

AIRL2 20 269618.86 268835.45±1941.84 269773.98±202.98 0.29 0.06


150


Table A.14 – continued from previous page


AIRL2 50 269618.86 269538.25±1307.87 269672.77±125.39 0.03 0.02

assets lg 5 -695.66 -693.76±21.43 -695.16±0.02 -0.27 0.07

assets lg 10 -695.66 -694.38±12.11 -695.19±0.03 -0.18 0.07

assets lg 20 -695.66 -700.07±7.92 -695.19±0.03 0.63 0.07

assets lg 50 -695.66 -697.78±6.40 -695.20±0.02 0.31 0.07

assets sm 5 -724.01 -744.19±22.03 -719.84±2.61 2.79 0.58

assets sm 10 -724.01 -712.49±12.78 -723.41±1.80 -1.59 0.08

assets sm 20 -724.01 -726.28±10.82 -723.36±1.96 0.31 0.09

assets sm 50 -724.01 -722.58±4.92 -724.18±0.02 -0.20 -0.02

biweekly lg 5 -7206.16 -7592.22±192.32 -6861.67±111.70 5.36 4.78

biweekly lg 10 -7206.16 -7429.79±136.67 -7011.85±50.04 3.10 2.70

biweekly lg 20 -7206.16 -7289.53±111.85 -7118.31±31.58 1.16 1.22

biweekly lg 50 -7206.16 -7299.93±62.28 -7176.82±13.73 1.30 0.41

biweekly md 5 -1468.56 -1753.63±129.36 -1245.01±122.01 19.41 15.22

biweekly md 10 -1468.56 -1539.53±57.13 -1391.64±32.21 4.83 5.24

biweekly md 20 -1468.56 -1480.48±48.20 -1430.13±27.29 0.81 2.62

biweekly md 50 -1468.56 -1516.62±44.44 -1441.81±20.31 3.27 1.82

biweekly sm 5 -8498.19 -8620.56±341.90 -8165.55±96.87 1.44 3.91

biweekly sm 10 -8498.19 -8573.41±245.71 -8314.78±74.06 0.89 2.16

biweekly sm 20 -8498.19 -8569.86±216.99 -8360.12±47.21 0.84 1.62

biweekly sm 50 -8498.19 -8538.81±200.59 -8429.12±27.66 0.48 0.81

chem 5 -12999.04 -13245.17±454.12 -12995.12±12.40 1.89 0.03

chem 10 -12999.04 -13248.17±350.80 -12982.16±15.48 1.92 0.13

chem 20 -12999.04 -13093.17±280.06 -12990.81±13.94 0.72 0.06

chem 50 -12999.04 -13130.57±132.30 -12977.49±15.67 1.01 0.17

dcap233 200 5 641.12 581.68±60.41 652.53±7.94 9.27 1.78

dcap233 200 10 641.12 664.08±40.75 649.82±4.90 -3.58 1.36

dcap233 200 20 641.12 651.23±51.55 644.23±4.29 -1.58 0.49

dcap233 200 50 641.12 648.54±31.42 642.43±1.33 -1.16 0.20

dcap233 300 5 553.86 502.24±45.38 564.36±6.14 9.32 1.90

dcap233 300 10 553.86 537.46±34.50 566.22±5.05 2.96 2.23

dcap233 300 20 553.86 550.40±34.92 560.66±6.13 0.63 1.23

dcap233 300 50 553.86 548.93±20.66 555.69±1.41 0.89 0.33

dcap233 500 5 587.24 565.13±60.09 607.32±28.18 3.77 3.42

dcap233 500 10 587.24 571.68±42.67 590.81±2.38 2.65 0.61

dcap233 500 20 587.24 606.34±23.04 587.36±2.34 -3.25 0.02

dcap233 500 50 587.24 590.79±22.67 588.64±1.72 -0.60 0.24

dcap243 200 5 1168.94 1165.49±58.54 1187.45±30.01 0.30 1.58


151




dcap243 200 10 1168.94 1142.38±24.27 1171.95±3.83 2.27 0.26

dcap243 200 20 1168.94 1202.71±34.10 1168.00±1.39 -2.89 -0.08

dcap243 200 50 1168.94 1160.73±25.15 1168.29±1.60 0.70 -0.06

dcap243 300 5 1273.68 1250.50±59.70 1294.34±21.21 1.82 1.62

dcap243 300 10 1273.68 1209.19±55.51 1281.06±10.69 5.06 0.58

dcap243 300 20 1273.68 1245.95±38.97 1273.66±0.75 2.18 0.00

dcap243 300 50 1273.68 1289.79±41.45 1273.17±2.08 -1.27 -0.04

dcap243 500 5 1048.61 1051.72±56.66 1059.76±9.38 -0.30 1.06

dcap243 500 10 1048.61 1079.19±37.56 1052.75±2.35 -2.92 0.39

dcap243 500 20 1048.61 1043.62±35.31 1051.50±1.93 0.48 0.28

dcap243 500 50 1048.61 1052.39±20.41 1049.60±1.74 -0.36 0.09

dcap332 200 5 162.75 146.01±5.00 188.63±19.08 10.28 15.90

dcap332 200 10 162.75 160.15±9.22 168.85±4.27 1.59 3.75

dcap332 200 20 162.75 159.89±5.36 165.87±1.78 1.75 1.92

dcap332 200 50 162.75 159.93±2.29 163.44±0.58 1.73 0.43

dcap332 300 5 178.28 164.53±10.18 201.63±22.90 7.71 13.10

dcap332 300 10 178.28 167.68±4.82 184.19±3.07 5.95 3.31

dcap332 300 20 178.28 169.05±3.27 181.43±1.76 5.17 1.77

dcap332 300 50 178.28 177.93±3.48 179.67±1.30 0.19 0.78

dcap332 500 5 130.36 116.28±4.93 163.83±20.74 10.80 25.68

dcap332 500 10 130.36 132.98±15.82 133.15±3.23 -2.02 2.14

dcap332 500 20 130.36 126.49±4.68 133.02±3.46 2.97 2.04

dcap332 500 50 130.36 133.27±4.71 130.39±0.23 -2.24 0.03

dcap342 200 5 332.90 365.90±50.66 343.92±8.48 -9.91 3.31

dcap342 200 10 332.90 327.96±21.91 335.04±3.03 1.48 0.64

dcap342 200 20 332.90 347.40±17.45 335.68±2.40 -4.36 0.84

dcap342 200 50 332.90 319.28±16.17 333.93±1.21 4.09 0.31

dcap342 300 5 373.06 313.81±68.22 385.73±6.84 15.88 3.40

dcap342 300 10 373.06 345.20±49.16 376.02±3.46 7.47 0.79

dcap342 300 20 373.06 361.94±31.01 373.82±1.57 2.98 0.20

dcap342 300 50 373.06 378.23±14.74 373.96±1.10 -1.39 0.24

dcap342 500 5 355.23 363.63±54.60 358.84±3.60 -2.36 1.01

dcap342 500 10 355.23 352.55±28.79 362.73±10.17 0.76 2.11

dcap342 500 20 355.23 344.06±37.07 355.93±1.04 3.15 0.20

dcap342 500 50 355.23 347.32±14.31 355.13±0.84 2.23 -0.03

electricity lg 5 -1818.75 -1959.51±221.06 -1668.99±22.78 7.74 8.23

electricity lg 10 -1818.75 -1892.21±68.84 -1751.42±20.52 4.04 3.70

electricity lg 20 -1818.75 -1836.64±55.10 -1776.97±16.87 0.98 2.30


152




electricity lg 50 -1818.75 -1848.18±49.47 -1803.08±6.78 1.62 0.86

electricity md 5 -5866.93 -6228.82±233.47 -5663.70±51.76 6.17 3.46

electricity md 10 -5866.93 -5957.30±106.10 -5780.80±37.85 1.54 1.47

electricity md 20 -5866.93 -6047.98±150.05 -5817.39±7.24 3.09 0.84

electricity md 50 -5866.93 -5854.90±79.75 -5842.85±8.34 -0.21 0.41

electricity sm 5 -1440.27 -1548.09±74.99 -1347.30±44.82 7.49 6.45

electricity sm 10 -1440.27 -1445.17±67.78 -1397.78±13.62 0.34 2.95

electricity sm 20 -1440.27 -1499.65±52.75 -1410.82±7.09 4.12 2.04

electricity sm 50 -1440.27 -1433.91±46.43 -1433.92±5.58 -0.44 0.44

gbd 5 1655.62 1574.22±238.10 1736.15±116.89 4.92 4.86

gbd 10 1655.62 1630.23±72.36 1684.73±12.45 1.53 1.76

gbd 20 1655.62 1629.81±115.49 1667.03±6.80 1.56 0.69

gbd 50 1655.62 1677.31±52.12 1660.69±3.64 -1.31 0.31

LandS 5 225.49 215.83±18.00 226.68±0.74 4.28 0.53

LandS 10 225.49 235.27±12.45 226.11±0.48 -4.34 0.28

LandS 20 225.49 221.53±8.47 225.67±0.28 1.76 0.08

LandS 50 225.49 223.29±6.36 225.63±0.24 0.97 0.06

PGP1 5 438.27 424.57±21.51 446.24±1.89 3.13 1.82

PGP1 10 438.27 438.03±13.37 447.15±0.96 0.06 2.03

PGP1 20 438.27 426.63±13.75 445.06±2.45 2.66 1.55

PGP1 50 438.27 439.09±5.09 440.95±2.43 -0.19 0.61

PGP3 5 496.23 443.50±33.53 737.68±88.70 10.63 48.66

PGP3 10 496.23 485.68±40.76 619.35±110.34 2.12 24.81

PGP3 20 496.23 478.13±16.28 528.04±67.11 3.65 6.41

PGP3 50 496.23 468.27±23.93 558.66±89.82 5.63 12.58

phone 5 36.89 37.12±0.47 36.89±0.03 -0.63 -0.01

phone 10 36.89 36.83±0.30 36.88±0.00 0.17 -0.05

phone 20 36.89 36.77±0.25 36.88±0.00 0.33 -0.03

phone 50 36.89 36.96±0.16 36.88±0.00 -0.19 -0.04

product lg 5 -29862.75 -29810.47±528.92 -29383.38±77.40 -0.18 1.61

product lg 10 -29862.75 -29904.24±502.04 -29660.94±35.91 0.14 0.68

product lg 20 -29862.75 -29869.78±264.56 -29746.50±23.44 0.02 0.39

product lg 50 -29862.75 -29948.16±103.37 -29818.79±14.28 0.29 0.15

product md 5 -28226.79 -28258.95±745.11 -28003.32±34.37 0.11 0.79

product md 10 -28226.79 -28120.05±698.07 -28067.84±41.24 -0.38 0.56

product md 20 -28226.79 -28348.77±398.56 -28159.24±12.77 0.43 0.24

product md 50 -28226.79 -28249.69±257.51 -28190.01±14.12 0.08 0.13

product sm 5 -17940.84 -17977.73±627.95 -17652.98±80.60 0.21 1.60


153




product sm 10 -17940.84 -17864.63±331.24 -17814.83±42.90 -0.42 0.70

product sm 20 -17940.84 -18175.97±336.75 -17888.10±19.99 1.31 0.29

product sm 50 -17940.84 -17997.57±200.82 -17908.75±10.68 0.32 0.18

semi2 5 113.66 111.98±1.24 221.43±47.48 1.48 94.82

semi2 10 113.66 113.35±0.40 131.88±21.36 0.27 16.03

semi2 15 113.66 113.55±0.17 120.81±13.42 0.10 6.29

semi3 5 113.70 99.26±4.19 342.60±92.97 12.69 201.33

semi3 10 113.70 106.07±3.75 256.01±46.98 6.71 125.17

semi3 15 113.70 108.59±3.40 190.97±59.59 4.49 67.97

semi4 5 113.12 89.71±5.83 497.10±122.06 20.70 339.43

semi4 10 113.12 98.71±5.63 253.00±44.80 12.74 123.65

semi4 15 113.12 104.41±4.60 163.02±35.62 7.70 44.10

snip10x10a 5 88.17 78.37±8.80 98.48±7.53 11.12 11.69

snip10x10a 10 88.17 82.95±5.10 92.86±3.46 5.92 5.32

snip10x10a 20 88.17 84.33±4.03 89.85±0.99 4.36 1.90

snip10x10a 50 88.17 88.19±2.47 89.48±1.17 -0.02 1.49

snip10x10b 5 70.03 66.34±6.68 79.79±5.15 5.26 13.95

snip10x10b 10 70.03 67.61±5.92 76.19±1.67 3.45 8.80

snip10x10b 20 70.03 66.98±3.56 73.50±1.41 4.36 4.97

snip10x10b 50 70.03 68.82±1.79 71.16±0.58 1.73 1.63

snip10x10c 5 76.76 68.04±8.17 85.26±3.36 11.36 11.08

snip10x10c 10 76.76 70.27±4.22 81.89±1.69 8.45 6.68

snip10x10c 20 76.76 74.47±3.63 79.70±1.21 2.97 3.83

snip10x10c 50 76.76 75.68±3.01 77.60±0.32 1.41 1.11

snip20x20 5 182.44 143.64±16.35 205.12±4.46 21.27 12.43

snip20x20 10 182.44 151.90±5.58 200.12±5.35 16.74 9.69

snip20x20 15 182.44 163.58±9.10 192.55±2.52 10.34 5.54

snip20x20 20 182.44 162.11±5.38 191.28±3.92 11.15 4.84

snip30x30 5 227.43 180.33±10.65 253.61±6.38 20.71 11.51

snip30x30 10 227.43 198.44±8.13 249.45±8.38 12.74 9.68

snip4x9 5 10.65 6.40±3.74 13.32±2.50 39.92 25.06

snip4x9 10 10.65 5.90±2.34 12.88±2.35 44.61 20.95

snip4x9 20 10.65 9.60±2.06 11.64±0.94 9.88 9.24

snip4x9 50 10.65 10.67±1.55 10.90±0.12 -0.16 2.28

snip7x5 5 80.23 66.50±8.96 84.54±2.59 17.11 5.38

snip7x5 10 80.23 61.35±4.68 81.96±1.04 23.53 2.16

snip7x5 20 80.23 77.56±5.53 82.31±1.12 3.32 2.60


154




snip7x5 50 80.23 77.54±3.91 82.04±1.06 3.35 2.26

sslp 10 50 100 5 -398.90 -405.43±12.88 -393.22±3.08 1.64 1.42

sslp 10 50 100 10 -398.90 -407.00±19.86 -396.03±2.22 2.03 0.72

sslp 10 50 100 20 -398.90 -395.15±13.44 -396.44±2.67 -0.94 0.62

sslp 10 50 100 50 -398.90 -399.58±9.38 -398.68±0.38 0.17 0.05

sslp 10 50 1000 5 -395.36 -393.43±18.76 -390.94±3.30 -0.49 1.12

sslp 10 50 1000 10 -395.36 -390.37±9.91 -393.14±1.62 -1.26 0.56

sslp 10 50 1000 20 -395.36 -394.92±13.97 -393.68±1.13 -0.11 0.42

sslp 10 50 1000 50 -395.36 -396.87±6.49 -394.43±0.93 0.38 0.24

sslp 10 50 2000 5 -390.19 -394.47±26.13 -384.72±7.38 1.10 1.40

sslp 10 50 2000 10 -390.19 -399.70±10.39 -388.09±2.74 2.44 0.54

sslp 10 50 2000 20 -390.19 -387.41±9.95 -388.98±1.25 -0.71 0.31

sslp 10 50 2000 50 -390.19 -389.55±5.79 -389.23±1.07 -0.16 0.25

sslp 10 50 50 5 -407.93 -414.17±18.85 -403.71±3.12 1.53 1.04

sslp 10 50 50 10 -407.93 -403.49±9.39 -405.29±1.85 -1.09 0.65

sslp 10 50 50 20 -407.93 -407.15±7.67 -406.52±1.78 -0.19 0.35

sslp 10 50 50 50 -407.93 -410.49±5.50 -407.59±0.92 0.63 0.08

sslp 10 50 500 5 -393.09 -387.18±22.29 -386.67±4.79 -1.50 1.63

sslp 10 50 500 10 -393.09 -390.93±12.42 -391.08±1.23 -0.55 0.51

sslp 10 50 500 20 -393.09 -399.28±8.14 -391.70±1.48 1.57 0.36

sslp 10 50 500 50 -393.09 -394.76±6.45 -392.26±0.44 0.43 0.21

sslp 15 45 10 5 -273.39 -265.84±7.81 -268.05±3.47 -2.76 1.95

sslp 15 45 10 10 -273.39 -275.38±9.84 -270.99±2.45 0.73 0.88

sslp 15 45 10 20 -273.39 -271.46±7.26 -272.95±0.72 -0.71 0.16

sslp 15 45 10 50 -273.39 -273.01±3.80 -272.56±0.44 -0.14 0.30

sslp 15 45 15 5 -270.40 -273.42±22.76 -265.90±3.41 1.12 1.66

sslp 15 45 15 10 -270.40 -272.03±10.71 -267.79±1.50 0.60 0.97

sslp 15 45 15 20 -270.40 -269.77±7.66 -269.47±0.50 -0.24 0.34

sslp 15 45 15 50 -270.40 -273.10±4.08 -269.49±0.65 1.00 0.34

sslp 15 45 5 5 -266.41 -277.90±13.41 -261.90±2.87 4.31 1.70

sslp 15 45 5 10 -266.41 -264.74±12.50 -265.06±1.29 -0.63 0.51

sslp 15 45 5 20 -266.41 -266.04±8.73 -265.85±0.72 -0.14 0.21

sslp 15 45 5 50 -266.41 -267.20±3.26 -265.72±0.46 0.30 0.26

sslp 5 25 100 5 -169.40 -171.32±10.75 -166.96±1.24 1.13 1.44

sslp 5 25 100 10 -169.40 -161.59±7.24 -168.27±1.08 -4.61 0.67

sslp 5 25 100 20 -169.40 -171.76±7.96 -168.42±0.63 1.39 0.58

sslp 5 25 100 50 -169.40 -168.31±5.42 -169.06±0.44 -0.65 0.20

sslp 5 25 50 5 -163.63 -158.89±12.57 -157.59±4.42 -2.90 3.69


155




sslp 5 25 50 10 -163.63 -166.83±8.49 -162.57±0.37 1.96 0.65

sslp 5 25 50 20 -163.63 -166.42±6.73 -163.01±0.34 1.70 0.38

sslp 5 25 50 50 -163.63 -165.11±2.66 -163.26±0.34 0.90 0.22

ssn 5 9.93 0.00±0.00 38.74±4.37 100.00 290.25

ssn 10 9.93 0.00±0.00 27.41±3.25 100.00 176.13

ssn 20 9.93 0.93±1.56 18.63±3.29 90.67 87.70

ssn 50 9.93 4.96±1.04 13.22±0.57 50.08 33.11

stocfor2 5 -39774.59 -40848.30±620.78 -37086.74±2350.84 2.70 6.76

stocfor2 10 -39774.59 -40261.13±482.95 -38763.63±782.86 1.22 2.54

stocfor2 20 -39774.59 -40079.79±442.38 -39394.87±316.04 0.77 0.95

stocfor2 50 -39774.59 -39819.13±203.92 -39735.32±36.18 0.11 0.10

storm 5 15498730.70 15457847.02±107649.37 15501337.03±2814.48 0.26 0.02

storm 10 15498730.70 15508664.26±79098.82 15496673.42±2901.34 -0.06 -0.01

storm 20 15498730.70 15494505.90±54792.91 15498748.94±1945.80 0.03 0.00

storm 50 15498730.70 15516932.48±37107.32 15498333.68±2783.48 -0.12 0.00

weekly lg 5 -2502.56 -2884.42±225.54 -2089.74±81.47 15.26 16.50

weekly lg 10 -2502.56 -2587.86±111.12 -2368.71±49.74 3.41 5.35

weekly lg 20 -2502.56 -2598.59±125.25 -2410.89±44.42 3.84 3.66

weekly lg 50 -2502.56 -2507.67±108.14 -2468.86±16.37 0.20 1.35

weekly md 5 -964.23 -1073.41±136.45 -780.20±72.84 11.32 19.09

weekly md 10 -964.23 -1088.73±73.41 -894.62±23.28 12.91 7.22

weekly md 20 -964.23 -1038.03±96.91 -927.84±20.87 7.65 3.77

weekly md 50 -964.23 -987.19±49.08 -933.24±12.28 2.38 3.21

weekly sm 5 -3617.91 -4228.59±449.20 -3167.08±130.28 16.88 12.46

weekly sm 10 -3617.91 -4016.97±325.04 -3416.04±64.69 11.03 5.58

weekly sm 20 -3617.91 -3740.93±129.16 -3496.30±67.49 3.40 3.36

weekly sm 50 -3617.91 -3720.76±107.51 -3573.63±24.35 2.84 1.22

We refer the results in Table A.15 to Section 5.4.2. In the table, the column z∗ is the best

estimated objective value for each problem instance. The column SAA mean is obtained by

(5.2), and the column BS Mean is the mean of the bootstrap samples. The sixth and the

seventh columns are the bias and the estimate of bias as a percentage of the true objective

value z∗, which are obtained by (5.1) and (5.4) respectively. The last column is the error of

the estimate, which is the difference between the bias and the estimate of bias.

156


Table A.15: Bootstrap estimate of bias for all tested instances

Instance N z∗ SAA Mean BS Mean Bias(%) Bias(%) Error(%)

20term 5 254313.000 252647.595 250984.581 0.65 0.65 0.00

20term 10 254313.000 254197.315 253528.269 0.05 0.26 -0.22

20term 50 254313.000 254286.550 254225.944 0.01 0.02 -0.01

20term 100 254313.000 254213.210 254181.089 0.04 0.01 0.03

20term 200 254313.000 254366.795 254349.799 -0.02 0.01 -0.03

4node 1024 5 434.108 430.235 426.958 0.89 0.75 0.14

4node 1024 10 434.108 433.059 431.437 0.24 0.37 -0.13

4node 1024 50 434.108 434.130 434.121 0.00 0.00 -0.01

4node 1024 100 434.108 434.120 434.121 0.00 0.00 0.00

4node 1024 200 434.108 434.108 434.107 0.00 0.00 0.00

4node 1024 400 434.108 434.103 434.104 0.00 0.00 0.00

4node 16384 5 446.848 436.422 431.958 2.33 1.00 1.33

4node 16384 10 446.848 442.204 438.971 1.04 0.72 0.32

4node 16384 50 446.848 446.753 446.378 0.02 0.08 -0.06

4node 16384 100 446.848 446.872 446.825 -0.01 0.01 -0.02

4node 16384 200 446.848 446.850 446.849 0.00 0.00 0.00

4node 16384 400 446.848 446.860 446.859 0.00 0.00 0.00

4node 32768 5 446.853 437.498 432.902 2.09 1.03 1.06

4node 32768 10 446.853 442.240 439.148 1.03 0.69 0.34

4node 32768 50 446.853 446.704 446.278 0.03 0.10 -0.06

4node 32768 100 446.853 446.856 446.793 0.00 0.01 -0.01

4node 32768 200 446.853 446.855 446.854 0.00 0.00 0.00

4node 4096 5 446.849 436.870 432.190 2.23 1.05 1.19

4node 4096 10 446.849 442.303 439.298 1.02 0.67 0.34

4node 4096 50 446.849 446.738 446.330 0.02 0.09 -0.07

4node 4096 100 446.849 446.847 446.800 0.00 0.01 -0.01

4node 4096 200 446.849 446.846 446.846 0.00 0.00 0.00

4node 8192 5 446.852 436.657 432.103 2.28 1.02 1.26

4node 8192 10 446.852 441.976 438.924 1.09 0.68 0.41

4node 8192 50 446.852 446.660 446.262 0.04 0.09 -0.05

4node 8192 100 446.852 446.875 446.833 -0.01 0.01 -0.01

4node 8192 200 446.852 446.851 446.851 0.00 0.00 0.00

AIRL1 5 249080.439 247941.265 246805.764 0.46 0.46 0.00

AIRL1 10 249080.439 248516.960 247856.095 0.23 0.27 -0.04

AIRL1 50 249080.439 248888.565 248710.204 0.08 0.07 0.01

AIRL1 100 249080.439 249015.140 248920.936 0.03 0.04 -0.01


157




AIRL1 200 249080.439 249071.320 249024.649 0.00 0.02 -0.02

AIRL1 400 249080.439 249070.735 249044.528 0.00 0.01 -0.01

AIRL1 800 249080.439 249083.385 249070.902 0.00 0.01 -0.01

AIRL2 5 269618.861 268614.810 267170.556 0.37 0.54 -0.16

AIRL2 10 269618.861 269427.040 268563.290 0.07 0.32 -0.25

AIRL2 50 269618.861 269423.860 269285.595 0.07 0.05 0.02

AIRL2 100 269618.861 269632.675 269563.658 -0.01 0.03 -0.03

AIRL2 200 269618.861 269686.465 269646.245 -0.03 0.01 -0.04

AIRL2 400 269618.861 269609.780 269597.332 0.00 0.00 0.00

AIRL2 800 269618.861 269648.780 269649.198 -0.01 0.00 -0.01

assets lg 5 -695.658 -695.564 -695.460 -0.01 -0.01 0.00

assets lg 10 -695.658 -696.199 -696.165 0.08 0.00 0.08

assets lg 50 -695.658 -694.975 -694.941 -0.10 0.00 -0.09

assets lg 100 -695.658 -696.187 -696.176 0.08 0.00 0.08

assets lg 200 -695.658 -696.605 -696.599 0.14 0.00 0.14

assets lg 400 -695.658 -695.813 -695.842 0.02 0.00 0.02

assets lg 800 -695.658 -695.770 -695.772 0.02 0.00 0.02

assets sm 5 -724.009 -728.743 -730.686 0.65 0.27 0.39

assets sm 10 -724.009 -723.158 -724.122 -0.12 0.13 -0.25

assets sm 50 -724.009 -724.072 -724.164 0.01 0.01 0.00

assets sm 100 -724.009 -724.307 -724.289 0.04 0.00 0.04

assets sm 200 -724.009 -723.875 -723.840 -0.02 0.00 -0.01

assets sm 400 -724.009 -724.066 -724.075 0.01 0.00 0.01

assets sm 800 -724.009 -723.891 -723.888 -0.02 0.00 -0.02

biweekly lg 5 -7206.163 -7489.418 -7697.862 3.93 2.89 1.04

biweekly lg 10 -7206.163 -7327.907 -7449.703 1.69 1.69 0.00

biweekly lg 50 -7206.163 -7241.241 -7267.937 0.49 0.37 0.12

biweekly lg 100 -7206.163 -7218.170 -7231.496 0.17 0.18 -0.02

biweekly lg 200 -7206.163 -7219.486 -7225.379 0.18 0.08 0.10

biweekly md 5 -1468.558 -1686.539 -1852.193 14.84 11.28 3.56

biweekly md 10 -1468.558 -1570.495 -1662.257 6.94 6.25 0.69

biweekly md 50 -1468.558 -1500.941 -1521.891 2.21 1.43 0.78

biweekly md 100 -1468.558 -1485.113 -1496.290 1.13 0.76 0.37

biweekly md 200 -1468.558 -1474.920 -1480.609 0.43 0.39 0.05

biweekly sm 5 -8498.188 -8937.726 -9266.056 5.17 3.86 1.31

biweekly sm 10 -8498.188 -8695.570 -8885.955 2.32 2.24 0.08

biweekly sm 50 -8498.188 -8539.947 -8581.588 0.49 0.49 0.00

biweekly sm 100 -8498.188 -8519.370 -8540.307 0.25 0.25 0.00


158




biweekly sm 200 -8498.188 -8506.338 -8516.680 0.10 0.12 -0.03

chem 5 -12999.038 -13330.450 -13419.162 2.55 0.68 1.87

chem 10 -12999.038 -13182.950 -13259.321 1.41 0.59 0.83

chem 50 -12999.038 -13099.820 -13132.528 0.78 0.25 0.52

chem 100 -12999.038 -13054.955 -13079.452 0.43 0.19 0.24

chem 200 -12999.038 -13030.675 -13047.040 0.24 0.13 0.12

chem 400 -12999.038 -13027.736 -13038.845 0.22 0.09 0.14

dcap233 200 5 641.123 639.342 635.441 0.28 0.61 -0.33

dcap233 200 10 641.123 641.094 636.812 0.00 0.67 -0.66

dcap233 200 50 641.123 637.755 636.746 0.53 0.16 0.37

dcap233 200 100 641.123 638.340 637.990 0.43 0.05 0.38

dcap233 200 200 641.123 639.559 639.310 0.24 0.04 0.21

dcap233 200 400 641.123 640.777 640.635 0.05 0.02 0.03

dcap233 200 800 641.123 639.972 639.878 0.18 0.01 0.16

dcap233 300 5 553.858 543.369 540.166 1.89 0.58 1.32

dcap233 300 10 553.858 552.891 549.592 0.17 0.60 -0.42

dcap233 300 50 553.858 554.205 552.984 -0.06 0.22 -0.28

dcap233 300 100 553.858 553.946 553.380 -0.02 0.10 -0.12

dcap233 300 200 553.858 553.113 552.935 0.13 0.03 0.10

dcap233 300 400 553.858 553.115 552.982 0.13 0.02 0.11

dcap233 300 800 553.858 552.980 552.951 0.16 0.01 0.15

dcap233 500 5 587.237 581.256 578.434 1.02 0.48 0.54

dcap233 500 10 587.237 581.608 578.535 0.96 0.52 0.44

dcap233 500 50 587.237 584.155 583.002 0.52 0.20 0.33

dcap233 500 100 587.237 586.349 585.725 0.15 0.11 0.04

dcap233 500 200 587.237 585.061 584.637 0.37 0.07 0.30

dcap233 500 400 587.237 588.423 588.215 -0.20 0.04 -0.24

dcap233 500 800 587.237 586.808 586.771 0.07 0.01 0.07

dcap243 200 5 1168.942 1155.309 1151.928 1.17 0.29 0.88

dcap243 200 10 1168.942 1164.183 1161.385 0.41 0.24 0.17

dcap243 200 50 1168.942 1171.831 1170.910 -0.25 0.08 -0.33

dcap243 200 100 1168.942 1169.212 1168.680 -0.02 0.05 -0.07

dcap243 200 200 1168.942 1168.822 1168.616 0.01 0.02 -0.01

dcap243 200 400 1168.942 1168.764 1168.591 0.02 0.01 0.00

dcap243 200 800 1168.942 1169.288 1169.245 -0.03 0.00 -0.03

dcap243 300 5 1273.676 1254.504 1249.968 1.51 0.36 1.15

dcap243 300 10 1273.676 1271.203 1267.833 0.19 0.26 -0.07

dcap243 300 50 1273.676 1267.824 1266.991 0.46 0.07 0.39


159




dcap243 300 100 1273.676 1274.550 1274.046 -0.07 0.04 -0.11

dcap243 300 200 1273.676 1272.128 1271.902 0.12 0.02 0.10

dcap243 300 400 1273.676 1274.722 1274.702 -0.08 0.00 -0.08

dcap243 300 800 1273.676 1275.281 1275.214 -0.13 0.01 -0.13

dcap243 500 5 1048.613 1048.795 1046.833 -0.02 0.19 -0.20

dcap243 500 10 1048.613 1047.792 1045.619 0.08 0.21 -0.13

dcap243 500 50 1048.613 1048.243 1047.357 0.04 0.08 -0.05

dcap243 500 100 1048.613 1050.005 1049.563 -0.13 0.04 -0.17

dcap243 500 200 1048.613 1048.422 1048.151 0.02 0.03 -0.01

dcap243 500 400 1048.613 1049.656 1049.584 -0.10 0.01 -0.11

dcap243 500 800 1048.613 1050.156 1050.165 -0.15 0.00 -0.15

dcap332 200 5 162.749 148.589 145.452 8.70 1.93 6.77

dcap332 200 10 162.749 156.255 153.086 3.99 1.95 2.04

dcap332 200 50 162.749 161.352 159.769 0.86 0.97 -0.11

dcap332 200 100 162.749 161.571 160.785 0.72 0.48 0.24

dcap332 200 200 162.749 161.946 161.521 0.49 0.26 0.23

dcap332 200 400 162.749 162.472 162.259 0.17 0.13 0.04

dcap332 200 800 162.749 162.665 162.560 0.05 0.06 -0.01

dcap332 300 5 178.280 167.358 163.773 6.13 2.01 4.11

dcap332 300 10 178.280 171.533 168.487 3.78 1.71 2.08

dcap332 300 50 178.280 176.546 175.304 0.97 0.70 0.28

dcap332 300 100 178.280 177.635 177.014 0.36 0.35 0.01

dcap332 300 200 178.280 177.711 177.360 0.32 0.20 0.12

dcap332 300 400 178.280 177.999 177.811 0.16 0.11 0.05

dcap332 300 800 178.280 178.034 177.931 0.14 0.06 0.08

dcap332 500 5 130.356 124.571 122.681 4.44 1.45 2.99

dcap332 500 10 130.356 128.011 126.314 1.80 1.30 0.50

dcap332 500 50 130.356 128.751 128.205 1.23 0.42 0.81

dcap332 500 100 130.356 129.932 129.673 0.33 0.20 0.13

dcap332 500 200 130.356 130.132 129.979 0.17 0.12 0.05

dcap332 500 400 130.356 130.446 130.360 -0.07 0.07 -0.14

dcap332 500 800 130.356 130.294 130.250 0.05 0.03 0.01

dcap342 200 5 332.904 320.578 317.674 3.70 0.87 2.83

dcap342 200 10 332.904 328.569 325.715 1.30 0.86 0.44

dcap342 200 50 332.904 333.653 332.360 -0.23 0.39 -0.61

dcap342 200 100 332.904 330.751 330.049 0.65 0.21 0.44

dcap342 200 200 332.904 332.824 332.430 0.02 0.12 -0.09

dcap342 200 400 332.904 332.853 332.671 0.02 0.05 -0.04


160




dcap342 200 800 332.904 333.004 332.899 -0.03 0.03 -0.06

dcap342 300 5 373.059 356.942 355.873 4.32 0.29 4.03

dcap342 300 10 373.059 367.827 365.950 1.40 0.50 0.90

dcap342 300 50 373.059 375.657 374.574 -0.70 0.29 -0.99

dcap342 300 100 373.059 373.087 372.647 -0.01 0.12 -0.13

dcap342 300 200 373.059 372.142 371.928 0.25 0.06 0.19

dcap342 300 400 373.059 371.126 371.004 0.52 0.03 0.49

dcap342 300 800 373.059 373.230 373.170 -0.05 0.02 -0.06

dcap342 500 5 355.234 357.804 355.214 -0.72 0.73 -1.45

dcap342 500 10 355.234 350.964 348.582 1.20 0.67 0.53

dcap342 500 50 355.234 351.837 351.124 0.96 0.20 0.76

dcap342 500 100 355.234 352.633 352.300 0.73 0.09 0.64

dcap342 500 200 355.234 354.450 354.277 0.22 0.05 0.17

dcap342 500 400 355.234 354.307 354.287 0.26 0.01 0.25

dcap342 500 800 355.234 354.198 354.179 0.29 0.01 0.29

electric lg 5 -1818.749 -1984.071 -2130.948 9.09 8.08 1.01

electric lg 10 -1818.749 -1915.661 -1995.596 5.33 4.40 0.93

electric lg 50 -1818.749 -1840.317 -1857.479 1.19 0.94 0.24

electric lg 100 -1818.749 -1832.232 -1841.523 0.74 0.51 0.23

electric lg 200 -1818.749 -1827.793 -1832.536 0.50 0.26 0.24

electric lg 400 -1818.749 -1822.768 -1825.513 0.22 0.15 0.07

electric md 5 -5866.932 -6021.294 -6176.698 2.63 2.65 -0.02

electric md 10 -5866.932 -5929.682 -6013.588 1.07 1.43 -0.36

electric md 50 -5866.932 -5885.359 -5901.494 0.31 0.28 0.04

electric md 100 -5866.932 -5883.972 -5893.271 0.29 0.16 0.13

electric md 200 -5866.932 -5871.112 -5875.654 0.07 0.08 -0.01

electric md 400 -5866.932 -5871.094 -5873.414 0.07 0.04 0.03

electricity sm 5 -1440.269 -1521.629 -1612.517 5.65 6.31 -0.66

electricity sm 10 -1440.269 -1493.937 -1542.405 3.73 3.37 0.36

electricity sm 50 -1440.269 -1449.549 -1459.592 0.64 0.70 -0.05

electricity sm 100 -1440.269 -1444.862 -1449.748 0.32 0.34 -0.02

electricity sm 200 -1440.269 -1443.169 -1445.885 0.20 0.19 0.01

electricity sm 400 -1440.269 -1441.825 -1443.118 0.11 0.09 0.02

electricity sm 800 -1440.269 -1438.555 -1439.191 -0.12 0.04 -0.16

gbd 5 1655.620 1572.631 1520.792 5.01 3.13 1.88

gbd 10 1655.620 1632.491 1604.976 1.40 1.66 -0.26

gbd 50 1655.620 1666.525 1661.437 -0.66 0.31 -0.97

gbd 100 1655.620 1639.969 1637.317 0.95 0.16 0.79


161




gbd 200 1655.620 1660.543 1659.480 -0.30 0.06 -0.36

gbd 500 1655.620 1654.814 1654.341 0.05 0.03 0.02

LandS 5 225.487 220.593 219.772 2.17 0.36 1.81

LandS 10 225.487 226.632 226.149 -0.51 0.21 -0.72

LandS 50 225.487 225.843 225.782 -0.16 0.03 -0.19

LandS 100 225.487 225.375 225.331 0.05 0.02 0.03

LandS 200 225.487 226.157 226.109 -0.30 0.02 -0.32

LandS 500 225.487 225.434 225.413 0.02 0.01 0.01

PGP1 5 438.273 431.168 430.570 1.62 0.14 1.48

PGP1 10 438.273 430.929 430.336 1.68 0.14 1.54

PGP1 50 438.273 433.326 432.124 1.13 0.27 0.85

PGP1 100 438.273 436.200 434.970 0.47 0.28 0.19

PGP1 200 438.273 438.474 437.330 -0.05 0.26 -0.31

PGP1 400 438.273 439.185 438.600 -0.21 0.13 -0.34

PGP3 5 496.227 448.984 445.089 9.52 0.78 8.74

PGP3 10 496.227 453.691 447.974 8.57 1.15 7.42

PGP3 50 496.227 482.223 475.707 2.82 1.31 1.51

PGP3 100 496.227 484.256 480.161 2.41 0.83 1.59

PGP3 200 496.227 491.140 488.282 1.03 0.58 0.45

PGP3 400 496.227 493.753 491.733 0.50 0.41 0.09

PGP3 800 496.227 494.188 492.738 0.41 0.29 0.12

phone 5 36.893 36.904 36.906 -0.03 -0.01 -0.03

phone 10 36.893 36.880 36.885 0.04 -0.01 0.05

phone 50 36.893 36.892 36.891 0.00 0.00 0.00

phone 100 36.893 36.895 36.896 -0.01 0.00 0.00

phone 200 36.893 36.897 36.898 -0.01 0.00 -0.01

phone 400 36.893 36.907 36.907 -0.04 0.00 -0.04

product lg 5 -29862.749 -30248.057 -30607.375 1.29 1.20 0.09

product lg 10 -29862.749 -30059.120 -30253.000 0.66 0.65 0.01

product lg 50 -29862.749 -29931.823 -29974.404 0.23 0.14 0.09

product lg 100 -29862.749 -29912.411 -29931.638 0.17 0.06 0.10

product md 5 -28226.791 -28493.250 -28726.486 0.94 0.83 0.12

product md 10 -28226.791 -28407.064 -28523.589 0.64 0.41 0.23

product md 50 -28226.791 -28270.813 -28299.361 0.16 0.10 0.05

product md 100 -28226.791 -28239.070 -28251.036 0.04 0.04 0.00

product md 200 -28226.791 -28250.862 -28258.211 0.09 0.03 0.06

product sm 5 -17940.840 -18159.902 -18363.863 1.22 1.14 0.08

product sm 10 -17940.840 -18032.786 -18139.318 0.51 0.59 -0.08


162




product sm 50 -17940.840 -17953.253 -17978.976 0.07 0.14 -0.07

product sm 100 -17940.840 -17978.082 -17989.629 0.21 0.06 0.14

product sm 200 -17940.840 -17943.844 -17949.276 0.02 0.03 -0.01

product sm 400 -17940.840 -17944.948 -17946.735 0.02 0.01 0.01

semi2 5 113.658 111.651 109.333 1.77 2.04 -0.27

semi2 10 113.658 113.251 112.395 0.36 0.75 -0.39

semi2 15 113.658 113.527 113.162 0.12 0.32 -0.21

semi3 5 113.696 97.780 92.035 14.00 5.05 8.95

semi3 10 113.696 104.991 100.949 7.66 3.56 4.10

semi3 15 113.696 108.411 105.070 4.65 2.94 1.71

semi4 5 113.125 91.603 85.597 19.02 5.31 13.72

semi4 10 113.125 99.324 94.488 12.20 4.27 7.92

semi4 15 113.125 102.749 98.546 9.17 3.72 5.46

snip10x10a 5 88.168 79.728 75.624 9.57 4.65 4.92

snip10x10a 10 88.168 84.915 81.851 3.69 3.48 0.21

snip10x10a 50 88.168 87.185 86.343 1.12 0.96 0.16

snip10x10a 100 88.168 87.494 87.049 0.77 0.50 0.26

snip10x10b 5 70.025 64.445 64.056 7.97 0.56 7.41

snip10x10b 10 70.025 66.356 65.189 5.24 1.67 3.57

snip10x10b 50 70.025 69.106 68.289 1.31 1.17 0.15

snip10x10b 100 70.025 69.763 69.298 0.37 0.66 -0.29

snip10x10c 5 76.756 68.961 66.251 10.16 3.53 6.62

snip10x10c 10 76.756 71.733 68.977 6.54 3.59 2.95

snip10x10c 50 76.756 75.870 75.034 1.15 1.09 0.06

snip10x10c 100 76.756 76.117 75.666 0.83 0.59 0.25

snip20x20 5 182.445 138.879 119.623 23.88 10.55 13.33

snip20x20 10 182.445 157.963 143.069 13.42 8.16 5.25

snip20x20 15 182.445 164.043 152.077 10.09 6.56 3.53

snip20x20 20 182.445 167.788 157.740 8.03 5.51 2.53

snip30x30 5 227.427 176.780 153.298 22.27 10.33 11.94

snip30x30 10 227.427 198.090 180.724 12.90 7.64 5.26

snip4x9 5 10.653 7.775 7.080 27.01 6.53 20.49

snip4x9 10 10.653 8.417 7.611 20.98 7.57 13.41

snip4x9 50 10.653 10.018 9.540 5.95 4.49 1.47

snip4x9 100 10.653 10.044 9.732 5.72 2.93 2.79

snip4x9 200 10.653 10.305 10.099 3.27 1.93 1.34

snip4x9 400 10.653 10.453 10.323 1.88 1.22 0.66

snip7x5 5 80.225 64.797 59.376 19.23 6.76 12.47


163




snip7x5 10 80.225 71.251 67.133 11.19 5.13 6.05

snip7x5 50 80.225 76.391 74.584 4.78 2.25 2.53

snip7x5 100 80.225 78.117 76.900 2.63 1.52 1.11

snip7x5 200 80.225 78.964 78.177 1.57 0.98 0.59

snip7x5 400 80.225 79.609 79.080 0.77 0.66 0.11

sslp 10 50 100 5 -398.900 -410.296 -415.001 2.86 1.18 1.68

sslp 10 50 100 10 -398.900 -401.412 -404.021 0.63 0.65 -0.02

sslp 10 50 100 50 -398.900 -399.049 -399.572 0.04 0.13 -0.09

sslp 10 50 100 100 -398.900 -398.701 -398.949 -0.05 0.06 -0.11

sslp 10 50 100 200 -398.900 -398.815 -398.979 -0.02 0.04 -0.06

sslp 10 50 100 400 -398.900 -399.134 -399.198 0.06 0.02 0.04

sslp 10 50 1000 5 -395.360 -401.203 -405.836 1.48 1.17 0.31

sslp 10 50 1000 10 -395.360 -398.040 -400.754 0.68 0.69 -0.01

sslp 10 50 1000 50 -395.360 -395.599 -396.134 0.06 0.14 -0.07

sslp 10 50 1000 100 -395.360 -396.020 -396.267 0.17 0.06 0.10

sslp 10 50 1000 200 -395.360 -395.287 -395.445 -0.02 0.04 -0.06

sslp 10 50 1000 400 -395.360 -395.525 -395.585 0.04 0.02 0.03

sslp 10 50 2000 5 -390.194 -394.213 -399.418 1.03 1.33 -0.30

sslp 10 50 2000 10 -390.194 -392.980 -395.881 0.71 0.74 -0.03

sslp 10 50 2000 50 -390.194 -390.150 -390.583 -0.01 0.11 -0.12

sslp 10 50 2000 100 -390.194 -390.443 -390.702 0.06 0.07 0.00

sslp 10 50 2000 200 -390.194 -390.251 -390.361 0.01 0.03 -0.01

sslp 10 50 2000 400 -390.194 -390.537 -390.577 0.09 0.01 0.08

sslp 10 50 50 5 -407.933 -413.927 -418.965 1.47 1.23 0.23

sslp 10 50 50 10 -407.933 -411.440 -413.879 0.86 0.60 0.26

sslp 10 50 50 50 -407.933 -408.420 -408.935 0.12 0.13 -0.01

sslp 10 50 50 100 -407.933 -408.780 -409.037 0.21 0.06 0.14

sslp 10 50 50 200 -407.933 -407.889 -408.027 -0.01 0.03 -0.04

sslp 10 50 50 400 -407.933 -408.684 -408.757 0.18 0.02 0.17

sslp 10 50 500 5 -393.093 -400.187 -405.573 1.80 1.37 0.43

sslp 10 50 500 10 -393.093 -395.044 -397.677 0.50 0.67 -0.17

sslp 10 50 500 50 -393.093 -393.307 -393.832 0.05 0.13 -0.08

sslp 10 50 500 100 -393.093 -393.800 -394.077 0.18 0.07 0.11

sslp 10 50 500 200 -393.093 -392.944 -393.125 -0.04 0.05 -0.08

sslp 10 50 500 400 -393.093 -393.216 -393.286 0.03 0.02 0.01

sslp 15 45 10 5 -273.389 -278.016 -282.249 1.69 1.55 0.14

sslp 15 45 10 10 -273.389 -275.616 -277.757 0.81 0.78 0.03

sslp 15 45 10 50 -273.389 -274.108 -274.638 0.26 0.19 0.07


164




sslp 15 45 10 100 -273.389 -273.418 -273.646 0.01 0.08 -0.07

sslp 15 45 10 200 -273.389 -273.593 -273.718 0.07 0.05 0.03

sslp 15 45 10 400 -273.389 -273.133 -273.188 -0.09 0.02 -0.11

sslp 15 45 15 5 -270.401 -277.200 -281.626 2.51 1.64 0.88

sslp 15 45 15 10 -270.401 -272.106 -274.428 0.63 0.86 -0.23

sslp 15 45 15 50 -270.401 -270.575 -271.027 0.06 0.17 -0.10

sslp 15 45 15 100 -270.401 -270.580 -270.829 0.07 0.09 -0.03

sslp 15 45 15 200 -270.401 -269.883 -269.971 -0.19 0.03 -0.22

sslp 15 45 15 400 -270.401 -270.138 -270.199 -0.10 0.02 -0.12

sslp 15 45 5 5 -266.414 -271.031 -274.988 1.73 1.49 0.25

sslp 15 45 5 10 -266.414 -268.534 -270.674 0.80 0.80 -0.01

sslp 15 45 5 50 -266.414 -266.786 -267.203 0.14 0.16 -0.02

sslp 15 45 5 100 -266.414 -266.664 -266.953 0.09 0.11 -0.01

sslp 15 45 5 200 -266.414 -266.287 -266.403 -0.05 0.04 -0.09

sslp 15 45 5 400 -266.414 -266.683 -266.752 0.10 0.03 0.08

sslp 5 25 100 5 -169.402 -171.994 -175.067 1.53 1.81 -0.28

sslp 5 25 100 10 -169.402 -171.165 -172.866 1.04 1.00 0.04

sslp 5 25 100 50 -169.402 -169.782 -170.084 0.22 0.18 0.05

sslp 5 25 100 100 -169.402 -169.753 -169.953 0.21 0.12 0.09

sslp 5 25 100 200 -169.402 -169.425 -169.491 0.01 0.04 -0.03

sslp 5 25 100 400 -169.402 -169.537 -169.571 0.08 0.02 0.06

sslp 5 25 100 800 -169.402 -169.526 -169.544 0.07 0.01 0.06

sslp 5 25 50 5 -163.629 -167.537 -170.901 2.39 2.06 0.33

sslp 5 25 50 10 -163.629 -165.945 -167.496 1.42 0.95 0.47

sslp 5 25 50 50 -163.629 -163.887 -164.229 0.16 0.21 -0.05

sslp 5 25 50 100 -163.629 -164.190 -164.368 0.34 0.11 0.23

sslp 5 25 50 200 -163.629 -163.439 -163.510 -0.12 0.04 -0.16

sslp 5 25 50 400 -163.629 -163.833 -163.878 0.12 0.03 0.10

ssn 5 9.928 0.000 0.000 100.00 0.00 100.00

ssn 10 9.928 0.029 0.001 99.70 0.29 99.41

ssn 20 9.928 1.166 0.158 88.26 10.15 78.11

ssn 50 9.928 4.834 2.273 51.31 25.79 25.52

ssn 100 9.928 6.739 4.850 32.12 19.03 13.09

ssn 200 9.928 8.228 7.015 17.12 12.22 4.90

ssn 500 9.928 8.963 8.364 9.72 6.03 3.69

stocfor2 5 -39774.587 -40260.837 -40398.400 1.22 0.35 0.88

stocfor2 10 -39774.587 -40160.577 -40257.877 0.97 0.24 0.73

stocfor2 50 -39774.587 -39833.387 -39893.347 0.15 0.15 0.00


165




stocfor2 100 -39774.587 -39796.466 -39831.139 0.06 0.09 -0.03

stocfor2 200 -39774.587 -39800.165 -39817.412 0.06 0.04 0.02

stocfor2 400 -39774.587 -39778.942 -39786.380 0.01 0.02 -0.01

storm 5 15498730.700 15450516.500 15445973.278 0.31 0.03 0.28

storm 10 15498730.700 15492193.000 15490013.260 0.04 0.01 0.03

storm 50 15498730.700 15493522.500 15493363.850 0.03 0.00 0.03

storm 100 15498730.700 15499332.000 15499177.058 0.00 0.00 0.00

weekly lg 5 -2502.560 -2786.158 -3018.460 11.33 9.28 2.05

weekly lg 10 -2502.560 -2683.037 -2824.882 7.21 5.67 1.54

weekly lg 50 -2502.560 -2532.143 -2563.217 1.18 1.24 -0.06

weekly lg 100 -2502.560 -2513.372 -2528.961 0.43 0.62 -0.19

weekly md 5 -964.228 -1179.943 -1307.369 22.37 13.22 9.16

weekly md 10 -964.228 -1067.530 -1146.708 10.71 8.21 2.50

weekly md 50 -964.228 -990.868 -1012.690 2.76 2.26 0.50

weekly md 100 -964.228 -975.047 -986.964 1.12 1.24 -0.11

weekly sm 5 -3617.906 -4056.330 -4374.087 12.12 8.78 3.34

weekly sm 10 -3617.906 -3803.789 -3985.861 5.14 5.03 0.11

weekly sm 50 -3617.906 -3662.871 -3705.332 1.24 1.17 0.07

weekly sm 100 -3617.906 -3646.092 -3666.893 0.78 0.57 0.20

We refer the results in Table A.16 to Section 5.4.4.

Table A.16: Mean Square Error for Bootstrap Estimate with Different Number of Replica-

tions

Instance N z∗ B=0 B=1 B=2 B=3 B=9 B=19 B=39 B=99

20term 5 254313.00 2.89E+06 6.58E+05 9.89E+05 1.49E+06 3.69E+06 6.60E+06 1.24E+07 5.57E+06

20term 10 254313.00 6.99E+04 1.37E+06 1.70E+06 1.22E+06 3.44E+06 3.68E+06 8.18E+06 2.13E+07

20term 50 254313.00 1.11E+04 3.12E+05 3.57E+05 2.17E+05 3.94E+05 4.94E+05 9.19E+05 3.84E+05

20term 100 254313.00 1.52E+04 3.41E+04 4.93E+04 6.11E+04 2.41E+05 1.64E+05 4.57E+05 6.86E+05

20term 200 254313.00 5.29E+03 2.39E+04 4.62E+04 8.75E+04 1.44E+05 1.15E+05 2.47E+05 3.34E+05

4node 1024 5 434.11 1.51E+01 2.23E+00 1.11E+00 1.19E+00 3.16E+00 2.67E+00 9.02E+00 1.67E+01

4node 1024 10 434.11 1.14E+00 9.01E-01 4.94E-01 9.53E-01 1.93E+00 3.41E+00 5.23E+00 1.35E+01

4node 1024 50 434.11 8.32E-04 2.53E-03 4.39E-03 6.09E-03 1.01E-02 1.98E-02 4.33E-02 2.34E-01

4node 1024 100 434.11 2.94E-04 1.67E-03 2.64E-03 4.05E-03 8.55E-03 2.57E-02 1.76E-02 2.99E-02

4node 1024 200 434.11 1.04E-04 4.96E-04 1.27E-03 1.51E-03 3.95E-03 7.48E-03 1.65E-02 5.28E-03

4node 1024 400 434.11 7.87E-05 3.36E-04 5.34E-04 6.08E-04 1.29E-03 2.61E-03 7.43E-03 2.22E-02

4node 16384 5 446.85 1.09E+02 3.98E+01 3.72E+01 4.53E+01 3.70E+01 1.99E+01 1.85E+01 3.57E+01

4node 16384 10 446.85 2.17E+01 2.13E+00 2.86E+00 3.77E+00 3.49E+00 5.08E+00 1.16E+01 1.49E+01


166




4node 16384 50 446.85 1.04E-02 2.56E-01 3.19E-01 1.67E-01 9.06E-02 1.46E-01 5.04E-01 1.89E+00

4node 16384 100 446.85 8.98E-04 6.78E-03 1.14E-02 1.09E-02 1.94E-02 2.26E-02 7.40E-02 2.93E-01

4node 16384 200 446.85 1.61E-04 1.47E-03 2.74E-03 2.08E-03 8.63E-03 1.10E-02 6.03E-03 3.69E-03

4node 16384 400 446.85 2.17E-04 7.38E-04 9.00E-04 1.21E-03 2.18E-03 3.81E-03 1.08E-02 1.38E-02

4node 32768 5 446.85 8.77E+01 1.61E+01 2.09E+01 2.21E+01 2.69E+01 5.90E+01 1.15E+02 2.24E+02

4node 32768 10 446.85 2.14E+01 2.63E+00 2.88E+00 4.84E+00 7.77E+00 1.03E+01 2.59E+01 6.51E+01

4node 32768 50 446.85 2.41E-02 9.51E-02 9.58E-02 7.03E-02 1.99E-01 3.95E-01 5.62E-01 1.08E+00

4node 32768 100 446.85 4.04E-04 5.79E-03 7.83E-03 9.07E-03 1.89E-02 3.81E-02 8.68E-02 1.48E-01

4node 32768 200 446.85 1.37E-04 1.23E-03 1.28E-03 1.86E-03 5.80E-03 3.98E-03 1.39E-02 3.35E-03

4node 4096 5 446.85 9.98E+01 2.11E+01 3.39E+01 3.30E+01 2.76E+01 4.58E+01 7.76E+01 4.68E+01

4node 4096 10 446.85 2.08E+01 4.02E+00 5.65E+00 5.83E+00 1.39E+01 1.79E+01 1.04E+01 1.21E+00

4node 4096 50 446.85 1.55E-02 5.84E-02 9.02E-02 2.05E-01 2.29E-01 2.21E-01 1.34E-01 2.64E-01

4node 4096 100 446.85 3.03E-04 1.53E-03 4.78E-03 5.02E-03 1.30E-02 5.12E-02 8.97E-02 1.54E-01

4node 4096 200 446.85 1.39E-04 7.17E-04 1.34E-03 1.51E-03 9.13E-03 1.30E-02 2.57E-02 2.79E-02

4node 8192 5 446.85 1.04E+02 2.71E+01 3.33E+01 3.06E+01 3.64E+01 5.69E+01 9.78E+01 1.79E+02

4node 8192 10 446.85 2.39E+01 4.04E+00 9.65E+00 1.12E+01 1.97E+01 1.85E+01 6.23E+01 5.60E+01

4node 8192 50 446.85 3.95E-02 2.12E-02 3.08E-02 9.29E-02 1.19E-01 1.18E-01 7.82E-02 2.78E-01

4node 8192 100 446.85 8.85E-04 1.65E-02 1.15E-02 8.77E-03 2.88E-02 2.54E-02 2.81E-02 3.35E-02

4node 8192 200 446.85 1.48E-04 8.14E-04 1.99E-03 2.59E-03 2.99E-03 9.61E-03 2.45E-02 2.94E-02

AIRL1 5 249080.44 1.42E+06 7.58E+05 1.44E+06 2.81E+06 2.52E+06 2.66E+06 5.65E+06 4.48E+06

AIRL1 10 249080.44 3.61E+05 2.67E+05 4.25E+05 6.57E+05 1.82E+06 3.97E+06 1.04E+07 2.39E+07

AIRL1 50 249080.44 4.57E+04 4.88E+04 6.89E+04 1.17E+05 3.66E+05 7.53E+05 8.46E+05 1.51E+06

AIRL1 100 249080.44 7.70E+03 3.14E+04 4.17E+04 5.12E+04 2.45E+05 3.82E+05 3.79E+05 8.56E+04

AIRL1 200 249080.44 1.91E+03 1.41E+04 1.85E+04 3.53E+04 9.98E+04 1.88E+05 2.81E+05 2.34E+05

AIRL1 400 249080.44 9.63E+02 7.66E+03 9.32E+03 1.30E+04 3.35E+04 7.52E+04 1.82E+05 1.10E+06

AIRL1 800 249080.44 4.37E+02 6.18E+03 1.01E+04 1.38E+04 1.93E+04 2.85E+04 4.79E+04 7.09E+04

AIRL2 5 269618.86 1.15E+06 9.50E+05 1.64E+06 2.34E+06 3.87E+06 5.09E+06 1.28E+07 2.11E+07

AIRL2 10 269618.86 1.21E+05 5.59E+05 1.07E+06 1.05E+06 1.93E+06 3.67E+06 6.43E+06 2.97E+07

AIRL2 50 269618.86 5.80E+04 5.43E+05 4.82E+05 5.02E+05 1.86E+06 3.07E+06 2.14E+06 5.43E+05

AIRL2 100 269618.86 1.03E+04 7.72E+04 1.30E+05 1.80E+05 3.28E+05 1.47E+06 1.63E+06 3.12E+06

AIRL2 200 269618.86 9.92E+03 1.41E+05 5.73E+04 5.65E+04 1.65E+05 5.95E+05 3.40E+05 3.93E+05

AIRL2 400 269618.86 2.96E+03 2.16E+04 3.22E+04 3.92E+04 8.30E+04 1.85E+05 3.53E+05 5.77E+05

AIRL2 800 269618.86 2.29E+03 7.94E+03 1.15E+04 1.80E+04 1.92E+04 3.03E+04 2.94E+04 1.23E+05

assets lg 5 -695.66 3.60E+00 2.48E+01 5.47E+01 5.91E+01 8.84E+01 1.03E+02 7.77E+01 1.61E+02

assets lg 10 -695.66 2.43E+00 9.48E+00 2.00E+01 2.07E+01 1.26E+02 4.83E+01 2.34E+02 3.83E+02

assets lg 50 -695.66 8.90E-01 2.90E+00 5.49E+00 6.27E+00 1.94E+01 5.90E+01 7.91E+00 2.53E+01



assets lg 400 -695.66 6.95E-02 6.43E-01 8.54E-01 7.52E-01 1.54E+00 3.48E+00 8.33E+00 3.77E+00

assets lg 800 -695.66 3.82E-02 1.80E-01 2.31E-01 3.15E-01 6.56E-01 1.63E+00 1.01E+00 5.72E+00

assets sm 5 -724.01 2.60E+01 2.32E+01 2.89E+01 3.85E+01 8.73E+01 2.07E+02 2.93E+02 1.95E+02

assets sm 10 -724.01 2.25E+00 9.69E+00 2.25E+01 2.24E+01 7.35E+01 2.20E+02 7.38E+02 1.59E+03

assets sm 50 -724.01 3.96E-01 2.52E+00 3.88E+00 6.48E+00 1.57E+01 3.67E+01 6.79E+01 2.66E+02

assets sm 100 -724.01 2.72E-01 2.76E+00 2.53E+00 3.48E+00 5.89E+00 1.60E+01 1.64E+01 1.91E+01

assets sm 200 -724.01 1.37E-01 7.09E-01 1.35E+00 1.60E+00 2.52E+00 8.14E+00 2.16E+01 4.24E+01

assets sm 400 -724.01 4.67E-02 2.66E-01 3.90E-01 5.11E-01 1.22E+00 3.94E+00 5.66E+00 9.95E+00

assets sm 800 -724.01 3.59E-02 2.12E-01 2.96E-01 3.71E-01 7.14E-01 1.65E+00 3.60E+00 4.80E+00

biweekly lg 5 -7206.16 8.08E+04 2.69E+04 2.23E+04 2.97E+04 1.46E+04 1.96E+04 5.53E+04 1.27E+04






167




biweekly md 5 -1468.56 4.76E+04 5.55E+03 8.89E+03 9.74E+03 6.16E+03 5.89E+03 1.10E+04 6.85E+04





biweekly sm 5 -8498.19 1.96E+05 2.13E+04 7.63E+04 9.83E+04 2.08E+05 4.57E+05 1.35E+06 1.73E+06





chem 5 -12999.04 1.13E+05 7.07E+04 6.04E+04 5.48E+04 8.37E+04 1.35E+05 2.74E+05 7.73E+05

chem 10 -12999.04 3.51E+04 4.31E+04 1.82E+04 3.08E+04 2.71E+04 4.23E+04 3.70E+04 8.09E+04

chem 50 -12999.04 1.04E+04 1.03E+04 1.12E+04 1.38E+04 4.02E+04 7.37E+04 4.51E+04 3.40E+03

chem 100 -12999.04 3.23E+03 3.60E+03 3.60E+03 2.25E+03 5.44E+03 2.15E+03 2.74E+03 3.30E+03

chem 200 -12999.04 1.05E+03 7.60E+02 7.62E+02 8.02E+02 1.12E+03 2.35E+03 4.90E+03 1.56E+04

chem 400 -12999.04 8.49E+02 1.18E+02 2.84E+02 8.31E+02 1.73E+03 2.30E+03 1.16E+03 5.30E+03

dcap233 200 5 641.12 7.87E+01 1.18E+03 1.65E+03 1.59E+03 2.00E+03 3.36E+03 6.17E+03 1.79E+04

dcap233 200 10 641.12 3.10E+01 3.45E+02 9.69E+02 6.02E+02 1.58E+03 2.62E+03 1.50E+03 2.64E+03

dcap233 200 50 641.12 1.73E+01 3.34E+01 7.15E+01 7.55E+01 1.28E+02 3.93E+02 6.23E+02 4.18E+02

dcap233 200 100 641.12 1.09E+01 4.03E+01 3.00E+01 5.99E+01 9.47E+01 2.44E+02 5.74E+02 2.62E+03

dcap233 200 200 641.12 4.07E+00 1.38E+01 1.99E+01 2.06E+01 1.14E+02 8.73E+01 1.56E+02 4.13E+02

dcap233 200 400 641.12 9.81E-01 5.53E+00 1.08E+01 1.55E+01 7.25E+01 1.12E+02 2.31E+02 6.47E+02

dcap233 200 800 641.12 1.80E+00 4.55E+00 7.18E+00 6.67E+00 1.19E+01 1.89E+01 9.91E+00 2.18E+00

dcap233 300 5 553.86 1.49E+02 2.52E+02 4.28E+02 4.92E+02 1.12E+03 3.04E+03 7.93E+03 7.65E+01

dcap233 300 10 553.86 1.97E+01 2.33E+02 1.56E+02 2.01E+02 6.39E+02 9.67E+02 2.17E+03 2.63E+03

dcap233 300 50 553.86 4.12E+00 3.16E+01 2.95E+01 3.64E+01 1.06E+02 1.54E+02 2.18E+02 3.73E+02

dcap233 300 100 553.86 2.23E+00 1.75E+01 2.85E+01 3.24E+01 1.23E+02 1.31E+02 1.66E+02 3.82E+02

dcap233 300 200 553.86 1.75E+00 6.37E+00 1.20E+01 1.55E+01 3.08E+01 7.49E+01 1.40E+02 3.68E+02

dcap233 300 400 553.86 1.05E+00 3.36E+00 4.85E+00 5.46E+00 1.50E+01 2.40E+01 7.74E+01 5.28E+01

dcap233 300 800 553.86 1.04E+00 1.71E+00 2.82E+00 3.43E+00 8.37E+00 1.58E+01 6.00E+01 1.85E+02

dcap233 500 5 587.24 8.31E+01 3.27E+02 5.46E+02 8.11E+02 2.32E+03 3.86E+03 5.78E+03 7.03E+03

dcap233 500 10 587.24 5.84E+01 1.95E+02 6.41E+02 1.08E+03 1.59E+03 2.44E+03 4.22E+03 1.30E+04

dcap233 500 50 587.24 1.45E+01 4.73E+01 5.45E+01 7.99E+01 2.87E+02 5.77E+02 1.03E+03 6.00E+02

dcap233 500 100 587.24 3.49E+00 2.03E+01 3.30E+01 5.62E+01 5.03E+01 9.38E+01 5.24E+01 2.02E+02

dcap233 500 200 587.24 6.19E+00 1.35E+01 1.40E+01 1.91E+01 5.76E+01 1.29E+02 3.10E+02 8.26E+02

dcap233 500 400 587.24 2.06E+00 4.07E+00 7.21E+00 1.22E+01 1.82E+01 4.11E+01 1.16E+02 5.62E+00

dcap233 500 800 587.24 5.39E-01 1.61E+00 4.72E+00 6.06E+00 1.41E+01 4.67E+01 6.33E+01 7.48E+01

dcap243 200 5 1168.94 2.30E+02 2.60E+02 4.91E+02 7.39E+02 2.00E+03 3.52E+03 1.07E+03 1.78E+03

dcap243 200 10 1168.94 5.28E+01 4.10E+02 2.59E+02 3.57E+02 7.17E+02 1.13E+03 3.79E+03 2.65E+03

dcap243 200 50 1168.94 1.27E+01 1.34E+02 8.01E+01 1.47E+02 2.16E+02 4.56E+02 7.15E+02 2.61E+03

dcap243 200 100 1168.94 2.25E+00 1.05E+01 1.78E+01 2.36E+01 7.18E+01 1.51E+02 4.22E+02 5.09E+02

dcap243 200 200 1168.94 1.36E+00 9.21E+00 1.25E+01 1.98E+01 5.45E+01 1.50E+02 3.21E+02 3.74E+02

dcap243 200 400 1168.94 6.41E-01 6.41E+00 7.32E+00 1.20E+01 2.47E+01 3.86E+01 5.77E+01 6.18E+01

dcap243 200 800 1168.94 4.19E-01 2.09E+00 2.97E+00 2.39E+00 9.45E+00 1.44E+01 2.72E+01 6.45E+01

dcap243 300 5 1273.68 4.31E+02 5.12E+02 1.47E+03 9.71E+02 2.52E+03 5.63E+03 9.16E+03 1.06E+04

dcap243 300 10 1273.68 3.76E+01 3.08E+02 3.95E+02 7.06E+02 2.26E+03 3.76E+03 5.55E+03 1.31E+04

dcap243 300 50 1273.68 4.05E+01 8.36E+01 7.41E+01 1.03E+02 3.58E+02 4.69E+02 1.12E+03 3.08E+03

dcap243 300 100 1273.68 4.11E+00 2.24E+01 5.48E+01 5.23E+01 1.22E+02 2.47E+02 5.04E+02 1.59E+03

dcap243 300 200 1273.68 3.75E+00 1.15E+01 2.55E+01 2.07E+01 5.14E+01 5.90E+01 1.41E+02 5.98E+02

dcap243 300 400 1273.68 1.91E+00 4.31E+00 1.51E+01 1.13E+01 4.23E+01 6.76E+01 1.38E+02 1.49E+02

dcap243 300 800 1273.68 3.02E+00 7.46E+00 1.35E+01 8.69E+00 2.48E+01 5.88E+01 6.60E+01 1.40E+01

dcap243 500 5 1048.61 4.81E+01 2.74E+02 4.08E+02 6.37E+02 1.32E+03 1.87E+03 5.73E+03 2.85E+03


168




dcap243 500 10 1048.61 2.22E+01 2.50E+02 1.74E+02 3.03E+02 6.70E+02 1.59E+03 2.86E+03 1.06E+04

dcap243 500 50 1048.61 3.84E+00 2.11E+01 3.49E+01 5.53E+01 1.06E+02 1.78E+02 4.52E+02 6.10E+02

dcap243 500 100 1048.61 3.81E+00 2.87E+01 2.62E+01 3.37E+01 3.57E+01 1.16E+02 2.42E+02 4.53E+02

dcap243 500 200 1048.61 1.03E+00 1.10E+01 1.10E+01 1.67E+01 5.29E+01 1.10E+02 1.70E+02 1.39E+02

dcap243 500 400 1048.61 1.60E+00 3.62E+00 4.77E+00 6.16E+00 2.04E+01 8.75E+01 9.20E+01 6.63E+01

dcap243 500 800 1048.61 2.65E+00 1.89E+00 2.27E+00 2.36E+00 6.18E+00 1.07E+01 3.74E+01 4.20E+01

dcap332 200 5 162.75 2.01E+02 1.25E+02 1.34E+02 1.36E+02 1.89E+02 2.46E+02 1.64E+02 7.15E+01

dcap332 200 10 162.75 4.26E+01 2.42E+01 2.62E+01 3.18E+01 4.09E+01 4.88E+01 9.26E+01 2.19E+02

dcap332 200 50 162.75 2.06E+00 7.85E-01 3.04E+00 3.45E+00 4.16E+00 9.00E+00 1.04E+01 1.90E+01

dcap332 200 100 162.75 1.44E+00 1.19E+00 5.27E-01 7.11E-01 2.06E+00 4.25E+00 1.23E+01 1.77E+01

dcap332 200 200 162.75 6.73E-01 2.10E-01 2.75E-01 3.43E-01 1.00E+00 1.28E+00 3.51E+00 7.94E+00

dcap332 200 400 162.75 9.15E-02 1.23E-01 2.32E-01 3.22E-01 5.34E-01 7.66E-01 2.16E+00 9.06E+00

dcap332 200 800 162.75 1.39E-02 3.81E-02 8.43E-02 1.05E-01 2.47E-01 3.34E-01 1.04E+00 2.31E+00

dcap332 300 5 178.28 1.21E+02 9.75E+01 8.32E+01 1.10E+02 8.39E+01 1.47E+02 8.03E+01 1.75E+02

dcap332 300 10 178.28 4.62E+01 3.11E+01 2.74E+01 2.23E+01 4.87E+01 9.98E+01 1.72E+02 7.64E+01

dcap332 300 50 178.28 3.12E+00 7.34E-01 2.11E+00 1.93E+00 5.86E+00 6.45E+00 9.64E+00 2.71E+00

dcap332 300 100 178.28 4.66E-01 2.79E-01 5.65E-01 8.68E-01 1.06E+00 1.55E+00 2.69E+00 7.84E+00

dcap332 300 200 178.28 3.51E-01 3.52E-01 3.22E-01 4.87E-01 1.10E+00 6.90E-01 1.77E+00 4.13E+00

dcap332 300 400 178.28 9.30E-02 8.88E-02 1.37E-01 1.80E-01 5.75E-01 8.57E-01 1.86E+00 4.22E+00

dcap332 300 800 178.28 6.66E-02 3.16E-02 4.99E-02 7.35E-02 2.26E-01 4.46E-01 1.02E+00 1.82E+00

dcap332 500 5 130.36 3.56E+01 1.00E+01 2.69E+01 3.89E+01 1.11E+02 1.85E+02 1.50E+02 9.68E+01

dcap332 500 10 130.36 6.54E+00 6.39E+00 1.02E+01 1.82E+01 4.15E+01 1.15E+02 4.62E+02 1.63E+02

dcap332 500 50 130.36 2.70E+00 2.82E+00 2.17E+00 2.21E+00 7.05E+00 1.01E+01 1.52E+01 7.64E+01

dcap332 500 100 130.36 2.52E-01 5.42E-01 8.94E-01 1.29E+00 3.53E+00 1.03E+01 2.23E+01 4.67E+00

dcap332 500 200 130.36 8.84E-02 4.16E-01 7.21E-01 1.12E+00 1.24E+00 2.26E+00 6.47E+00 9.11E+00

dcap332 500 400 130.36 2.56E-02 1.20E-01 1.49E-01 1.85E-01 4.94E-01 8.75E-01 2.40E+00 2.46E+00

dcap332 500 800 130.36 1.39E-02 6.06E-02 8.28E-02 8.77E-02 2.71E-01 3.75E-01 4.79E-01 1.37E+00

dcap342 200 5 332.90 1.78E+02 1.62E+02 2.78E+02 4.47E+02 1.08E+03 1.09E+03 3.66E+03 1.45E+04

dcap342 200 10 332.90 3.16E+01 1.34E+02 1.24E+02 2.06E+02 4.55E+02 1.22E+03 1.09E+03 1.94E+03

dcap342 200 50 332.90 2.90E+00 2.26E+01 4.01E+01 8.18E+01 9.39E+01 1.93E+02 5.70E+02 1.14E+03

dcap342 200 100 332.90 5.90E+00 1.30E+01 1.64E+01 2.27E+01 3.01E+01 8.13E+01 1.23E+02 1.16E+02

dcap342 200 200 332.90 6.16E-01 4.54E+00 7.16E+00 1.10E+01 2.59E+01 5.31E+01 1.17E+02 1.56E+02

dcap342 200 400 332.90 3.09E-01 3.04E+00 4.40E+00 5.83E+00 8.75E+00 2.67E+01 5.47E+01 4.74E+00

dcap342 200 800 332.90 1.83E-01 1.85E+00 1.98E+00 2.60E+00 1.40E+01 2.03E+01 1.23E+01 5.08E+01

dcap342 300 5 373.06 2.96E+02 6.35E+02 9.57E+02 8.95E+02 1.10E+03 2.95E+03 6.85E+03 7.30E+03

dcap342 300 10 373.06 4.71E+01 1.40E+02 2.10E+02 2.57E+02 4.74E+02 1.18E+03 2.75E+03 3.80E+03

dcap342 300 50 373.06 1.04E+01 1.01E+02 9.46E+01 1.25E+02 2.11E+02 2.35E+02 6.09E+02 1.85E+03

dcap342 300 100 373.06 2.42E+00 3.44E+01 2.49E+01 3.24E+01 1.19E+02 3.21E+02 4.67E+02 7.19E+02

dcap342 300 200 373.06 1.86E+00 6.13E+00 1.08E+01 1.52E+01 4.26E+01 1.49E+02 2.89E+02 6.35E+02

dcap342 300 400 373.06 4.34E+00 4.57E+00 1.58E+01 1.19E+01 1.28E+01 2.86E+01 2.67E+01 1.09E+02

dcap342 300 800 373.06 2.99E-01 1.56E+00 3.94E+00 4.10E+00 1.32E+01 2.18E+01 8.81E+00 2.35E+01

dcap342 500 5 355.23 3.70E+01 2.06E+02 3.58E+02 5.46E+02 1.51E+03 3.15E+03 6.88E+03 1.31E+04

dcap342 500 10 355.23 3.15E+01 6.76E+01 2.43E+02 1.91E+02 5.74E+02 2.00E+03 4.57E+03 9.15E+02

dcap342 500 50 355.23 1.46E+01 4.00E+01 5.98E+01 5.88E+01 2.72E+02 4.56E+02 4.17E+02 3.25E+01

dcap342 500 100 355.23 8.08E+00 1.69E+01 1.30E+01 1.59E+01 2.98E+01 7.93E+01 8.45E+01 2.37E+02

dcap342 500 200 355.23 1.34E+00 5.19E+00 8.41E+00 1.37E+01 2.24E+01 7.31E+01 1.58E+02 5.69E+02

dcap342 500 400 355.23 1.27E+00 3.78E+00 4.27E+00 8.57E+00 1.75E+01 2.43E+01 7.67E+01 1.77E+02

dcap342 500 800 355.23 1.29E+00 4.76E+00 3.76E+00 2.93E+00 9.88E+00 1.44E+01 4.72E+01 9.38E+01

electricity lg 5 -1818.75 2.75E+04 2.94E+03 3.45E+03 3.41E+03 7.54E+03 1.20E+04 3.56E+04 9.17E+04





169






electricity md 5 -5866.93 2.43E+04 4.08E+03 4.11E+03 5.36E+03 2.45E+04 6.39E+04 2.21E+05 6.31E+05






electricity sm 5 -1440.27 6.78E+03 1.55E+03 4.45E+03 2.83E+03 3.46E+03 1.06E+04 3.19E+03 1.31E+04







gbd 5 1655.62 7.39E+03 4.51E+03 5.88E+03 9.26E+03 1.21E+04 1.76E+04 1.09E+04 2.04E+03

gbd 10 1655.62 7.69E+02 1.67E+03 2.42E+03 3.42E+03 8.30E+03 1.03E+04 3.73E+04 6.05E+04

gbd 50 1655.62 1.67E+02 4.51E+02 5.20E+02 6.54E+02 1.62E+03 2.70E+03 4.70E+03 2.12E+04

gbd 100 1655.62 2.66E+02 3.62E+02 3.15E+02 6.53E+02 9.81E+02 1.64E+03 3.09E+03 2.57E+03

gbd 200 1655.62 3.53E+01 8.48E+01 1.57E+02 2.34E+02 5.57E+02 1.03E+03 2.15E+03 1.24E+04

gbd 500 1655.62 4.91E+00 2.49E+01 5.06E+01 7.00E+01 1.77E+02 3.75E+02 6.70E+02 1.19E+03

LandS 5 225.49 2.70E+01 1.23E+02 8.28E+01 1.04E+02 3.45E+02 4.09E+02 5.21E+02 3.15E+02

LandS 10 225.49 3.31E+00 1.32E+01 2.43E+01 2.94E+01 1.25E+02 2.84E+02 6.13E+02 2.21E+03

LandS 50 225.49 3.95E-01 3.04E+00 3.10E+00 4.10E+00 1.68E+01 3.40E+01 1.30E+01 2.04E+00

LandS 100 225.49 1.83E-01 1.53E+00 2.64E+00 2.30E+00 4.69E+00 1.25E+01 3.88E+01 4.26E+01

LandS 200 225.49 5.37E-01 1.44E+00 1.76E+00 2.26E+00 1.81E+00 3.75E+00 9.97E+00 3.98E+01

LandS 500 225.49 3.45E-02 1.73E-01 3.87E-01 4.31E-01 1.38E+00 2.84E+00 9.54E+00 3.21E+00

PGP1 5 438.27 5.46E+01 8.58E+01 6.77E+01 6.79E+01 4.68E+02 7.63E+02 5.67E+02 1.36E+03

PGP1 10 438.27 5.63E+01 4.94E+01 5.86E+01 4.10E+01 1.29E+02 1.52E+02 2.34E+02 2.44E+02

PGP1 50 438.27 2.51E+01 5.02E+01 5.22E+01 4.84E+01 8.04E+01 8.07E+01 1.60E+02 6.64E+01

PGP1 100 438.27 4.65E+00 1.07E+01 4.08E+00 2.73E+00 1.48E+01 1.68E+01 2.79E+01 3.01E+01

PGP1 200 438.27 2.17E-01 3.31E+00 3.16E+00 3.82E+00 5.99E+00 1.60E+01 1.90E+01 1.08E+02

PGP1 400 438.27 9.23E-01 1.97E+00 2.71E+00 2.15E+00 3.10E+00 5.36E+00 1.42E+01 5.25E+01

PGP3 5 496.23 2.25E+03 1.94E+03 1.86E+03 2.46E+03 2.94E+03 1.63E+03 1.81E+03 3.98E+03

PGP3 10 496.23 1.82E+03 1.47E+03 1.91E+03 1.66E+03 1.44E+03 2.17E+03 2.57E+03 6.55E+03

PGP3 50 496.23 1.99E+02 4.18E+01 7.65E+01 1.05E+02 2.94E+02 4.22E+02 1.38E+03 9.89E+01

PGP3 100 496.23 1.45E+02 6.71E+01 3.56E+01 5.15E+01 4.01E+01 7.91E+01 2.01E+02 7.57E+02

PGP3 200 496.23 2.69E+01 6.57E+00 1.84E+01 3.24E+01 3.59E+01 9.51E+01 1.40E+02 2.67E+02

PGP3 400 496.23 6.61E+00 2.72E+00 4.15E+00 7.57E+00 2.30E+01 3.95E+01 2.40E+01 5.74E+01

PGP3 800 496.23 4.47E+00 1.88E+00 3.31E+00 4.83E+00 2.49E+01 3.76E+01 5.07E+01 3.55E+01

phone 5 36.89 3.03E-03 1.56E-02 2.72E-02 4.57E-02 1.45E-01 3.07E-01 1.04E+00 8.70E-01

phone 10 36.89 1.44E-03 1.70E-02 2.97E-02 4.34E-02 5.93E-02 8.03E-02 8.11E-02 9.08E-02

phone 50 36.89 3.34E-04 3.16E-03 4.01E-03 8.04E-03 1.81E-02 3.48E-02 1.03E-01 1.58E-01

phone 100 36.89 1.47E-04 9.79E-04 1.60E-03 1.62E-03 3.91E-03 1.00E-02 3.33E-02 3.88E-02

phone 200 36.89 9.96E-05 3.68E-04 1.38E-03 2.22E-03 5.34E-03 6.73E-03 2.58E-03 7.57E-03

phone 400 36.89 2.33E-04 3.89E-04 9.13E-04 6.71E-04 8.49E-04 8.74E-04 1.79E-03 1.57E-03

product lg 5 -29862.75 1.52E+05 2.71E+04 2.90E+04 4.41E+04 1.58E+05 1.30E+05 3.62E+05 1.46E+04




product md 5 -28226.79 7.72E+04 5.09E+04 1.01E+05 8.97E+04 2.13E+05 1.63E+05 6.68E+05 7.27E+04



170







product sm 5 -17940.84 5.09E+04 1.87E+04 2.73E+04 5.26E+04 8.93E+04 2.83E+05 5.24E+05 3.30E+04






semi2 5 113.66 4.07E+00 3.62E-01 5.51E-01 5.87E-01 1.79E+00 1.27E+00 2.79E+00 1.29E+01

semi2 10 113.66 1.76E-01 3.25E-01 4.44E-01 3.36E-01 1.01E+00 6.20E-01 9.74E-01 1.65E+00

semi2 15 113.66 1.78E-02 4.66E-02 4.22E-02 6.20E-02 5.18E-02 1.19E-01 8.33E-02 1.92E-03

semi3 5 113.70 2.54E+02 8.86E+01 8.35E+01 9.40E+01 1.36E+02 7.07E+01 1.23E+02 1.24E+02

semi3 10 113.70 7.59E+01 2.62E+01 2.74E+01 2.76E+01 1.40E+01 3.31E+00 4.66E+00 2.97E+00

semi3 15 113.70 2.80E+01 4.16E+00 6.03E+00 9.72E+00 2.93E+01 3.04E+01 2.92E+01 1.01E+02

semi4 5 113.12 4.64E+02 2.88E+02 2.40E+02 2.80E+02 2.77E+02 2.00E+02 1.03E+02 1.31E+02

semi4 10 113.12 1.91E+02 9.27E+01 1.17E+02 1.00E+02 6.78E+01 2.92E+01 7.48E+01 1.40E+02

semi4 15 113.12 1.08E+02 3.19E+01 4.73E+01 4.54E+01 3.98E+01 3.20E+01 2.45E+01 4.61E+01

snip10x10a 5 88.17 7.17E+01 2.68E+01 1.70E+01 2.49E+01 1.42E+01 2.66E+01 5.90E+01 1.08E+02

snip10x10a 10 88.17 1.10E+01 2.52E+00 4.27E+00 4.12E+00 9.77E+00 2.34E+01 7.01E+01 3.72E+02

snip10x10a 50 88.17 1.04E+00 4.57E-01 7.14E-01 1.07E+00 1.50E+00 3.40E+00 7.39E+00 2.58E+01

snip10x10a 100 88.17 4.92E-01 2.10E-01 3.15E-01 4.60E-01 1.66E+00 2.24E+00 5.55E+00 2.74E+00

snip10x10b 5 70.03 3.14E+01 3.55E+01 3.01E+01 3.12E+01 3.27E+01 4.58E+01 8.91E+01 2.87E+02

snip10x10b 10 70.03 1.37E+01 6.95E+00 7.88E+00 7.69E+00 2.21E+01 2.77E+01 2.84E+01 7.01E+01

snip10x10b 50 70.03 8.90E-01 3.25E-01 4.73E-01 8.50E-01 1.28E+00 1.32E+00 3.91E+00 1.08E+01

snip10x10b 100 70.03 9.68E-02 3.36E-01 4.21E-01 3.75E-01 1.19E+00 3.99E+00 9.76E+00 2.27E+00

snip10x10c 5 76.76 6.12E+01 3.93E+01 3.74E+01 3.99E+01 1.44E+01 2.49E+01 1.22E+02 4.40E+02

snip10x10c 10 76.76 2.55E+01 4.71E+00 5.77E+00 4.46E+00 2.52E+01 4.54E+01 6.84E+01 1.11E+02

snip10x10c 50 76.76 8.51E-01 4.37E-01 5.61E-01 1.01E+00 2.02E+00 4.83E+00 4.96E+00 5.56E+00

snip10x10c 100 76.76 4.38E-01 1.95E-01 2.66E-01 3.55E-01 6.28E-01 1.44E+00 4.43E+00 1.48E+01

snip20x20 5 182.44 1.90E+03 5.88E+02 5.91E+02 5.42E+02 3.90E+02 3.73E+02 4.65E+02 2.25E+02

snip20x20 10 182.44 6.00E+02 1.51E+02 9.69E+01 1.15E+02 1.29E+02 2.73E+02 1.00E+02 1.10E+02

snip20x20 15 182.44 3.39E+02 4.51E+01 2.03E+01 3.48E+01 2.32E+01 3.94E+01 8.01E+01 8.61E+01

snip20x20 20 182.44 2.15E+02 1.37E+01 1.55E+01 1.86E+01 9.35E+00 2.07E+01 2.29E+01 5.44E+01

snip30x30 5 227.43 2.57E+03 9.39E+02 8.52E+02 8.07E+02 1.22E+03 1.44E+03 1.89E+03 1.94E+03

snip30x30 10 227.43 8.62E+02 1.06E+02 9.82E+01 1.05E+02 1.56E+02 1.63E+02 1.18E+02 4.04E+02

snip4x9 5 10.65 8.45E+00 6.36E+00 5.98E+00 6.61E+00 7.75E+00 1.49E+01 1.31E+01 1.26E+01

snip4x9 10 10.65 5.08E+00 2.42E+00 2.16E+00 3.06E+00 5.88E+00 9.59E+00 1.30E+01 1.71E+01

snip4x9 50 10.65 4.26E-01 1.55E-01 4.63E-01 5.40E-01 8.03E-01 2.03E+00 2.35E+00 4.53E+00

snip4x9 100 10.65 3.81E-01 1.23E-01 2.59E-01 2.99E-01 5.21E-01 5.42E-01 1.37E+00 2.11E+00

snip4x9 200 10.65 1.27E-01 4.64E-02 1.50E-01 8.84E-02 1.81E-01 4.64E-01 8.72E-01 1.12E+00

snip4x9 400 10.65 4.29E-02 4.17E-02 5.24E-02 4.79E-02 1.55E-01 3.01E-01 2.36E-01 1.97E-01

snip7x5 5 80.23 2.39E+02 8.90E+01 7.97E+01 1.16E+02 9.83E+01 2.64E+02 5.22E+02 3.10E+02

snip7x5 10 80.23 8.13E+01 1.05E+01 3.37E+01 3.87E+01 4.74E+01 2.40E+01 8.62E+01 4.04E+02

snip7x5 50 80.23 1.49E+01 3.32E+00 2.63E+00 5.21E+00 5.39E+00 5.05E+00 9.10E+00 6.73E+00

snip7x5 100 80.23 4.52E+00 1.12E+00 2.89E+00 3.53E+00 9.03E+00 1.33E+01 1.50E+01 6.25E+01

snip7x5 200 80.23 1.63E+00 6.57E-01 5.68E-01 6.41E-01 1.56E+00 5.55E+00 6.77E+00 1.17E+01

snip7x5 400 80.23 3.99E-01 2.11E-01 2.00E-01 3.06E-01 3.72E-01 9.00E-01 1.36E+00 3.86E+00

sslp 10 50 100 5 -398.90 1.33E+02 1.39E+02 1.83E+02 1.74E+02 2.46E+02 2.01E+02 4.14E+02 9.09E+02

sslp 10 50 100 10 -398.90 8.36E+00 1.37E+01 1.77E+01 2.31E+01 6.02E+01 1.53E+02 2.91E+02 2.88E+02

sslp 10 50 100 50 -398.90 4.42E-01 5.01E+00 8.42E+00 1.07E+01 2.04E+01 3.50E+01 1.04E+02 2.63E+01

sslp 10 50 100 100 -398.90 2.07E-01 1.91E+00 1.82E+00 2.60E+00 4.41E+00 1.32E+01 1.91E+01 9.28E+00


171




sslp 10 50 100 200 -398.90 1.11E-01 1.59E+00 2.29E+00 1.82E+00 3.78E+00 8.65E+00 9.80E+00 2.31E+01

sslp 10 50 100 400 -398.90 1.00E-01 2.76E-01 3.85E-01 5.33E-01 2.09E+00 6.31E+00 4.75E+00 1.15E+00

sslp 10 50 1000 5 -395.36 3.81E+01 2.67E+01 1.35E+02 1.88E+02 3.69E+02 4.90E+02 2.91E+02 5.38E+02

sslp 10 50 1000 10 -395.36 9.22E+00 1.15E+01 2.12E+01 3.20E+01 5.29E+01 1.32E+02 9.37E+01 2.67E+02

sslp 10 50 1000 50 -395.36 4.46E-01 3.82E+00 5.56E+00 1.32E+01 2.57E+01 2.53E+01 8.72E+01 2.96E+00

sslp 10 50 1000 100 -395.36 6.57E-01 1.36E+00 1.95E+00 3.39E+00 7.50E+00 1.94E+01 6.74E+01 5.80E+01

sslp 10 50 1000 200 -395.36 1.13E-01 8.48E-01 1.72E+00 2.04E+00 6.98E+00 1.20E+01 2.78E+01 4.37E+01

sslp 10 50 1000 400 -395.36 7.37E-02 3.00E-01 1.04E+00 1.09E+00 1.89E+00 4.98E+00 9.28E+00 2.79E+01

sslp 10 50 2000 5 -390.19 1.99E+01 2.55E+01 3.88E+01 6.06E+01 1.37E+02 2.89E+02 3.04E+02 7.65E+02

sslp 10 50 2000 10 -390.19 9.57E+00 1.82E+01 1.73E+01 2.51E+01 4.81E+01 4.17E+01 9.56E+01 2.88E+02

sslp 10 50 2000 50 -390.19 4.74E-01 2.86E+00 4.73E+00 1.03E+01 1.82E+01 4.65E+01 9.43E+01 9.50E+01

sslp 10 50 2000 100 -390.19 2.73E-01 1.19E+00 1.87E+00 2.49E+00 7.13E+00 1.08E+01 2.23E+01 1.20E+01

sslp 10 50 2000 200 -390.19 8.94E-02 6.43E-01 1.02E+00 1.37E+00 6.00E+00 1.34E+01 1.02E+01 3.00E+00

sslp 10 50 2000 400 -390.19 1.64E-01 3.27E-01 5.00E-01 1.04E+00 1.75E+00 3.22E+00 6.68E+00 3.38E+01

sslp 10 50 50 5 -407.93 3.93E+01 6.72E+01 3.32E+01 3.70E+01 1.98E+02 2.95E+02 3.80E+02 5.85E+02

sslp 10 50 50 10 -407.93 1.40E+01 1.30E+01 2.22E+01 3.06E+01 8.07E+01 2.72E+02 4.08E+02 8.15E+02

sslp 10 50 50 50 -407.93 5.77E-01 2.89E+00 4.48E+00 5.56E+00 1.44E+01 4.62E+01 1.44E+02 3.44E+02

sslp 10 50 50 100 -407.93 9.19E-01 2.19E+00 1.81E+00 2.18E+00 1.10E+01 3.77E+01 5.12E+01 1.47E+02

sslp 10 50 50 200 -407.93 9.05E-02 5.03E-01 1.04E+00 9.79E-01 3.32E+00 9.09E+00 1.89E+01 9.85E+00

sslp 10 50 50 400 -407.93 6.13E-01 1.85E+00 5.66E-01 8.30E-01 1.93E+00 1.97E+00 3.71E+00 8.41E+00

sslp 10 50 500 5 -393.09 5.47E+01 3.96E+01 4.25E+01 5.80E+01 1.73E+02 3.07E+02 1.09E+03 3.32E+03

sslp 10 50 500 10 -393.09 6.07E+00 1.87E+01 1.98E+01 3.04E+01 1.01E+02 2.23E+02 3.91E+02 6.81E+02

sslp 10 50 500 50 -393.09 4.73E-01 3.93E+00 4.02E+00 8.88E+00 1.54E+01 3.75E+01 5.68E+01 1.46E+02

sslp 10 50 500 100 -393.09 7.18E-01 1.31E+00 1.39E+00 2.10E+00 5.35E+00 5.78E+00 1.48E+01 5.52E+01

sslp 10 50 500 200 -393.09 1.09E-01 1.55E+00 1.50E+00 2.11E+00 3.27E+00 6.15E+00 5.71E+00 4.13E+00

sslp 10 50 500 400 -393.09 6.58E-02 3.30E-01 8.07E-01 6.41E-01 2.56E+00 6.83E+00 2.95E+00 3.66E+00

sslp 15 45 10 5 -273.39 2.35E+01 2.76E+01 1.89E+01 3.08E+01 7.89E+01 2.24E+02 4.49E+02 5.58E+02

sslp 15 45 10 10 -273.39 5.90E+00 6.55E+00 1.57E+01 2.25E+01 2.36E+01 1.98E+01 3.16E+01 1.97E+01

sslp 15 45 10 50 -273.39 7.13E-01 1.27E+00 3.57E+00 6.48E+00 1.38E+01 1.57E+01 4.17E+01 3.82E+01

sslp 15 45 10 100 -273.39 9.89E-02 3.69E-01 3.60E+00 2.51E+00 1.86E+00 2.66E+00 4.53E+00 1.57E+00

sslp 15 45 10 200 -273.39 9.67E-02 1.53E+00 1.16E+00 1.92E+00 2.61E+00 5.37E+00 1.81E+01 2.91E+01

sslp 15 45 10 400 -273.39 9.30E-02 2.44E-01 4.96E-01 3.55E-01 1.87E+00 1.33E+00 6.81E-01 1.44E+00

sslp 15 45 15 5 -270.40 4.83E+01 2.89E+01 2.15E+01 3.11E+01 8.82E+01 1.08E+02 2.68E+02 8.30E+02

sslp 15 45 15 10 -270.40 4.06E+00 7.42E+00 1.26E+01 1.75E+01 4.16E+01 7.93E+01 1.90E+02 7.72E+01

sslp 15 45 15 50 -270.40 2.36E-01 3.77E+00 3.33E+00 2.80E+00 6.83E+00 2.07E+01 4.07E+01 5.58E+01

sslp 15 45 15 100 -270.40 1.31E-01 1.65E+00 2.08E+00 1.56E+00 3.25E+00 4.75E+00 1.10E+01 9.11E+00

sslp 15 45 15 200 -270.40 3.23E-01 1.93E+00 1.53E+00 2.38E+00 2.21E+00 3.66E+00 3.94E+00 9.74E+00

sslp 15 45 15 400 -270.40 9.91E-02 4.45E-01 6.35E-01 4.36E-01 3.63E-01 3.74E-01 7.41E-01 4.65E-01

sslp 15 45 5 5 -266.41 2.31E+01 9.53E+00 1.96E+01 2.37E+01 1.03E+02 1.65E+02 3.20E+02 1.79E+03

sslp 15 45 5 10 -266.41 5.54E+00 7.94E+00 1.18E+01 1.78E+01 6.09E+01 8.95E+01 1.60E+02 2.58E+01

sslp 15 45 5 50 -266.41 2.80E-01 1.12E+00 1.65E+00 2.33E+00 5.85E+00 1.27E+01 2.45E+01 8.69E+01

sslp 15 45 5 100 -266.41 1.80E-01 6.54E-01 1.62E+00 1.74E+00 2.96E+00 1.20E+01 1.24E+01 2.85E+01

sslp 15 45 5 200 -266.41 6.15E-02 7.88E-01 8.49E-01 1.07E+00 1.56E+00 5.06E+00 8.43E+00 2.04E+01

sslp 15 45 5 400 -266.41 9.52E-02 1.66E-01 3.51E-01 6.14E-01 1.40E+00 1.46E+00 8.59E-01 1.80E+00

sslp 5 25 100 5 -169.40 8.41E+00 1.03E+01 2.31E+01 2.60E+01 8.82E+01 2.10E+02 1.80E+02 1.20E+02

sslp 5 25 100 10 -169.40 3.90E+00 8.56E+00 8.27E+00 1.52E+01 1.08E+01 2.54E+01 7.64E+01 7.34E+01

sslp 5 25 100 50 -169.40 2.96E-01 1.33E+00 1.55E+00 2.10E+00 6.73E+00 2.16E+01 2.52E+01 8.17E+01

sslp 5 25 100 100 -169.40 2.07E-01 6.78E-01 1.41E+00 1.03E+00 2.49E+00 4.18E+00 1.17E+01 5.51E+00

sslp 5 25 100 200 -169.40 3.56E-02 2.99E-01 3.37E-01 4.93E-01 8.63E-01 2.17E+00 7.24E+00 6.98E+00

sslp 5 25 100 400 -169.40 4.10E-02 1.98E-01 3.10E-01 2.40E-01 6.09E-01 9.86E-01 2.35E+00 8.58E+00

sslp 5 25 100 800 -169.40 2.54E-02 7.58E-02 1.10E-01 1.51E-01 2.52E-01 5.11E-01 1.27E+00 2.80E+00

sslp 5 25 50 5 -163.63 1.69E+01 1.09E+01 1.60E+01 2.11E+01 7.63E+01 2.87E+02 5.80E+02 1.40E+03


172




sslp 5 25 50 10 -163.63 6.17E+00 6.04E+00 9.20E+00 1.24E+01 4.91E+01 8.77E+01 3.66E+01 5.67E+01

sslp 5 25 50 50 -163.63 2.00E-01 7.94E-01 1.58E+00 1.86E+00 5.16E+00 9.76E+00 1.05E+01 1.19E+01

sslp 5 25 50 100 -163.63 3.84E-01 8.55E-01 1.34E+00 8.02E-01 2.52E+00 5.85E+00 1.06E+01 3.01E+01

sslp 5 25 50 200 -163.63 7.79E-02 3.23E-01 4.77E-01 7.05E-01 1.69E+00 5.10E+00 1.43E+01 1.52E+01

sslp 5 25 50 400 -163.63 6.29E-02 5.52E-01 3.81E-01 3.90E-01 9.03E-01 1.86E+00 2.76E+00 4.59E+00

ssn 5 9.93 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01

ssn 10 9.93 9.80E+01 9.62E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01 9.86E+01

ssn 20 9.93 7.68E+01 5.81E+01 6.87E+01 7.17E+01 7.77E+01 9.82E+01 9.86E+01 9.86E+01

ssn 50 9.93 2.60E+01 5.84E+00 6.33E+00 6.77E+00 1.01E+01 1.70E+01 2.35E+01 2.74E+01

ssn 100 9.93 1.02E+01 1.00E+00 1.19E+00 1.89E+00 6.85E-01 1.03E+00 1.02E+00 7.93E-01

ssn 200 9.93 2.89E+00 5.18E-01 3.90E-01 4.25E-01 4.07E-01 1.41E-01 4.30E-01 8.73E-01

ssn 500 9.93 9.34E-01 1.40E-01 1.74E-01 8.73E-02 9.19E-02 2.69E-01 6.14E-01 8.85E-01

stocfor2 5 -39774.59 2.41E+05 5.86E+04 5.94E+04 7.34E+04 1.92E+05 3.04E+05 8.35E+05 3.08E+06






storm 5 15498730.70 2.45E+09 2.13E+09 3.22E+09 3.35E+09 5.41E+09 6.59E+09 5.84E+09 6.95E+09

storm 10 15498730.70 1.06E+08 6.37E+08 7.18E+08 9.93E+08 1.51E+09 4.12E+09 9.36E+09 1.83E+10

storm 50 15498730.70 3.60E+07 1.30E+08 2.03E+08 2.53E+08 5.81E+08 1.15E+09 1.55E+09 6.04E+09

storm 100 15498730.70 5.63E+06 6.29E+07 4.15E+07 6.23E+07 1.77E+08 4.50E+08 3.59E+08 6.85E+07

weekly lg 5 -2502.56 8.12E+04 1.14E+04 1.17E+04 3.13E+04 3.72E+04 2.35E+04 3.67E+04 6.98E+04




weekly md 5 -964.23 4.68E+04 5.68E+03 2.51E+03 6.04E+03 2.80E+04 1.11E+04 1.52E+04 4.33E+04




weekly sm 5 -3617.91 1.93E+05 4.22E+04 2.88E+04 3.91E+04 1.13E+05 2.15E+05 2.97E+05 4.56E+05




173

Bibliography

[1] Shabbir Ahmed. Siplib: A stochastic integer programming test problem library. Avail-

able as http://www2.isye.gatech.edu/∼sahmed/siplib/.

[2] Shabbir Ahmed and Renan Garcia. Dynamic capacity acquisition and assignment un-

der uncertainty. Annals of Operations Research, 124:267–283, 2003.

[3] G. Albeanu, M. Ghica, and F. Popentiu-Vladicescu. On using bootstrap scenario-

generation for multi-period stochastic programming applications. International Jour-

nal of Computers, Communications & Control, III:156–161, 2008.

[4] K. A. Ariyawansa and A. J. Felt. On a new collection of stochastic linear programming

test problems. INFORMS Journal on Computing, 16:291–299.

[5] K. A. Ariyawansa and D. D. Hudson. Performance of a benchmark parallel implemen-

tation of the Van Slyke and Wets algorithm for two-stage stochastic programs on the

Sequent/Balance. Concurrency Practice and Experience, 3:109–128, 1991.

[6] Dash Associates. Xpress-mp reference manual. 1999.

[7] D. Bernholdt, S. Bharathi, D. Brown, K. Chancio, M. Chen, A. Chervenak, L. Cinquini,

B. Drach, I. Foster, P. Fox, J. Garcia, C. Kesselman, R. Markel, D. Middleton, V. Nefe-

dova, L. Pouchard, A. Shoshani, A. Sim, G. Strand, and D. Williams. The earth system

grid: Supporting the next generation of climate modeling research. Proceedings of the

IEEE, 93(3):485–495, 2005.

174

BIBLIOGRAPHY

[8] P. Bickel, F. Gotze, and W. van Zwet. Resampling fewer than n observations: gains,

losses and remedies for losses. Statistica Sinica, 7:1–31, 1997.

[9] J. Birge. Decomposition and partitioning methods for multi-stage stochastic linear

programs. Operations Research, 33:989–1007, 1985.

[10] J. R. Birge and R. Louveaux. Introduction to Stochastic Programming. Springer, New

York, 1997.

[11] J. R. Birge, M. A. H. Dempster, H. I. Gassmann, E. A. Gunn, and A. J. King. A standard

input format for multiperiod stochastic linear programs. COAL Newsletter, 17:1–19,

1987.

[12] J. R. Birge, C. J. Donohue, D. F. Holmes, and O. G. Svintsiski. A parallel implementa-

tion of the nested decomposition algorithm for multistage stochastic linear programs.

Mathematical Programming, 75:327–352, 1996.

[13] K. L. Chung. A Course in Probability Theory. Academic Press, New York, 1974.

[14] A. R. Conn, N. I. M. Gould, and Ph. L. Toint. LANCELOT: A Fortran Package for Large-

scale Nonlinear Optimization (Release A). Springer–Verlag, 1992.

[15] A. R. Conn, N. I. M. Gould, and Ph. L. Toint. Trust-Region Methods. SIAM, 2000.

[16] J. Czyzyk, J. Linderoth, and J. Shen. SUTIL: A utility library for handling stochastic

programs, 2005. User’s Manual.

[17] G. Dantzig, J. Ho, and G. Infanger. Solving stochastic linear programs on a hypercube

multicomputer. Technical Report SOL 91-10, Department of Operations Research,

Stanford University, August 1991.

[18] G. B. Dantzig. Linear Programming and Extensions. Princeton University Press, Prince-

ton, NJ, 1963.

175

BIBLIOGRAPHY

[19] G. B. Dantzig. Parallel processors for planning. Paper presented at the 12th Inter-

national Symposium on Mathermatical Programming, Boston, MA, USA, 5-9 August,

1985.

[20] G. B. Dantzig and P. W. Glynn. Parallel processors for planning under uncertainty.

Annals of Operations Research, 22:1–21, 1990.

[21] M. A. H. Dempster and R. T. Thompson. Parallelization and aggregation of nested

Benders decomposition. Annals of Operations Research, 81:163–187, 1995. proceed-

ings of the APMOD95 Conference.

[22] J.E. Dennis and R.B. Schnabel. Numerical Methods for Unconstrained Optimization and

Nonlinear Equations. Prentice-Hall, Englewood Cliffs,NJ, 1983.

[23] Elizabeth Dolan and Jorge More. Benchmarking optimization software with perfor-

mance profiles. Mathematical Programming, 91:201–213, 2002.

[24] Thain Douglas, Tannenbaum Todd, and Livny Miron. Distributed computing in prac-

tice: The condor experience. Concurrency and Computation: Practice and Experience,

17:323–356, 2005.

[25] Bradley Efron. Bootstrap methods: Another look at the jackknife. The Annals of

Statistics, 7:1–26, 1979.

[26] Bradley Efron and Robert J. Tibshirani. An Introduction to the BootStrap. Chapman &

Hall, New York, 1993.

[27] O. Gunluk F. Barahona, S. Bermon and S. Hood. Robust Capacity Planning in Semicon-

ductor Manufacturing. IBM Research Report RC22196, 2001.

[28] John Forrest, David de la Nuez, and Robin Lougee-Heimer. Clp user guide. Available

as http://www.coin-or.org/Clp/userguide/index.html.

176

BIBLIOGRAPHY

[29] E. Frangiere, J. Gondzio, and J.-P. Vial. Building and solving large-scale stochastic pro-

grams on an affordable, distributed computing system. Annals of Operations Research,

99:167–187, 2000.

[30] Jacek Gondzio and Roy Kouwenberg. High performance computing for asset liability

management. Operations Research, 49:879–891, 2001.

[31] J.-P. Goux, J. T. Linderoth, and M. E. Yoder. Metacomputing and the master-worker

paradigm. Submitted. Available from http://www.cs.wisc.edu/condor/mw/mw.ps,

1999.

[32] Ronald Hochreiter, Clemens Wiesinger, and David Wozabal. Large-Scale Computational

Finance Applications on the Open Grid Service Environment. Lecture Notes in Computer

Science. Springer, Berlin, Germany, 2005.

[33] U. Janjarassuk and J. Linderoth. Reformulation and sampling to solve a stochastic

network interdiction problem. In preparation, 2005.

[34] J.R.Birge and F.V.Louveaux. A multicut algorithm for two-stage stochastic linear pro-

grams. European Journal of Operations Research, 34:384–392, 1988.

[35] P. Kall and J. Mayer. Stochastic Linear Programming: Models, Theory, and Computation.

Springer, New York, 2005.

[36] P. Kall and S. Wallace. Stochastic Programming. John WIley and Sons, New York,

1994.

[37] J. Korycki. On a distributed implementation of a decomposition method for multistage

linear stochastic programs. Optimization, 38(2):173–200, 1996.

[38] Kenneth Lange. Numerical Analysis for Statisticians(Statistics and Computing).

Springer, 2000.

[39] J. T. Linderoth. MW: A software framework for combinatorial optimization on grids.

In Parallel Combinatorial Optimization. Wiley and Sons, New York, 2005.

177

BIBLIOGRAPHY

[40] J. T. Linderoth and S. J. Wright. Implementing a decomposition algorithm for stochas-

tic programming on a computational grid. Computational Optimization and Applica-

tions, 24:207–250, 2003. Special Issue on Stochastic Programming.

[41] J. T. Linderoth and S. J. Wright. Computational grids for stochastic programming.

In S. Wallace and W. Ziemba, editors, Applications of Stochastic Programming, SIAM

Mathematical Series on Optimization, pages 61–77. SIAM, 2005.

[42] J. T. Linderoth, A. Shapiro, and S. J. Wright. The empirical behavior of sampling

methods for stochastic programming. Technical Report Optimization Technical Report

02-01, Computer Sciences Department, University of Wisconsin-Madison, 2002.

[43] F. Louveaux and Y. Smeers. Optimal investments for electricity generation: A stochas-

tic model and a test problem. In Y. Ermoliev and R. J.-B Wets, editors, Numerical tech-

niques for stochastic optimization problems, pages 445–452. Springer-Verlag, Berlin,

1988.

[44] W. K. Mak, D. P. Morton, and R. K. Wood. Monte carlo bounding techniques for

determining solution quality in stochastic programs. Operations Research Letters, 24:

47–56, 1999.

[45] W. K. Mak, D. P. Morton, and R. K. Wood. Monte Carlo bounding techniques for

determining solution quality in stochastic programs. Operations Research Letters, 24:

47–56, 1999.

[46] M. D. McKay, R. J. Beckman, and W. J. Conover. A comparison of three methods for

selecting values of input variables in the analysis of output from a computer code.

Technometrics, 21:239–245, 1979.

[47] Christopher Z. Mooney and Robert D. Duval. Bootstrapping, A Nonparametric Approach

to Statistical Inference. SAGE Publications, Newbury Park, 1993.

[48] J. M. Mulvey and A. Ruszczynski. A new scenario decomposition method for large

scale stochastic optimization. Operations Research, 43:477–490, 1995.

178

BIBLIOGRAPHY

[49] S. S. Nielsen and S. A. Zenios. Scalable parallel Benders decomposition for stochastic

linear programming. Parallel Computing, 23:1069–1089, 1997.

[50] J. Nocedal and S. J. Wright. Numerical Optimization. Springer-Verlag, New York, 1999.

[51] L. Ntaimo and S. Sen. The million variable march for stochastic combinatorial opti-

mization. Journal of Global Optimization, 2004. to appear.

[52] Dimitis N. Politis, Joseph P. Romano, and Michael Wolf. Subsampling. Springer, 1999.

[53] Ruijin Qi and Stavros A. Zenios. On the scalability of data-parallel decomposition

algorithms for stochastic programs. Journal of Parallel and Distributed Computing, 22:

565–570, 1994.

[54] Charles Rosa and Andrzej Ruszczynski. On augmented lagrangian decomposition

methods for multistage stochastic programs. Annals of Operations Research, 64:289–

309, 1996.

[55] M. Russell, G. Allen, G. Daues, I. Foster, E. Seidel, J. Novotny, J. Shalf, and G. von

Laszewski. The astrophysics simulation collaboratory: A science portal enabling com-

munity software development. Cluster Computing, 5(3):297–304, 2002.

[56] A. Ruszczynski. Parallel decomposition of multistage stochastic programming prob-

lems. Mathematical Programming, 58:201–228, 1993.

[57] A. Ruszczynski and A. Shapiro, editors. Stochastic Programming, Handbook in Opera-

tions Research and Management Science. Elsevier, 2003.

[58] ILOG S.A. and ILOG Inc. Ilog cplex 10.0 user’s manual. January 2006.

[59] A. Sartenaer. Automatic determination of an initial trust region in nonlinear program-

ming. SIAM Journal on Scientific Computing, 18(6):1788–1803, 1997.

[60] S. Sen, R. D. Doverspike, and S. Cosares. Network planning with random demand.

Telecommunications Systems, 3:11–30, 1994.

179

BIBLIOGRAPHY

[61] A. Shapiro and T. Homem-de-Mello. On the rate of convergence of optimal solutions

of Monte Carlo approximations of stochastic programs. SIAM Journal on Optimization,

11:70–86, 2000.

[62] A. Shapiro and T. Homem-de-Mello. A simulation-based approach to two-stage

stochastic programming with recourse. Mathematical Programming, 81:301–325,

1998.

[63] Condor Team. Condor homepage, . Available as http://www.cs.wisc.edu/condor/.

[64] MW Team. User’s guide to mw, . Available as http://www.cs.wisc.edu/condor/mw/

usersguide.pdf, 2005.

[65] Huseyin Topaloglu. Stochastic programming data sets. Available as http://legacy.

orie.cornell.edu/∼huseyin/research/sp datasets/sp datasets.html.

[66] R. Van Slyke and R.J-B. Wets. L-shaped linear programs with applications to con-

trol and stochastic programming. SIAM Journal on Applied Mathematics, 17:638–663,

1969.

[67] G. von Laszewski, M. Westbrook, I. Foster, E. Westbrook, and C. Barnes. Using com-

putational grid capabilities to enhance the ability of an x-ray source for structural

biology. Cluster Computing, 3(3):187–199, 2000.

[68] R. J. B. Wets. On parallel processors design for stochastic programs. Working Paper

WP 85-67, International Institute of Applied Systems Analysis, A-2361, Laxenburg,

Austria, 1985.

180

Vita

Udom Janjarassuk was born in Bangkok, Thailand, on October 23, 1974. He received his

Bachelor degree in Control Engineering from King Mongkut’s Institute of Technology Lad-

krabang, Bangkok, Thailand, in 1996. After graduation, he worked as a research assistant

in the National Electronics and Computer Technology Center (NECTEC), Thailand. He be-

came a lecturer in the Department of Industrial Engineering in King Mongkut’s Institute

of Technology Ladkrabang in 1997. He came to the Industrial and System Engineering

department at Lehigh University to pursue his master and PhD degrees in 2003.

181

using computational grids for effective...

Documents