university of floridaufdcimages.uflib.ufl.edu/uf/e0/05/21/58/00001/zhang_y.pdf · 4 acknowledgments...

USING MULTIPLE PREDICTIONS AND MULTIPLE SOURCES OF DATA FOR DESIGN SPACE EXPLORATION

By

YIMING ZHANG

A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2018

To my mom, Cuixia, from whom I have got unconditional love and support

4

ACKNOWLEDGMENTS

First of all, I would like to thank my advisor Dr. Raphael Haftka for his extremely

valuable guidance and support. I would like to thank my co-advisor Dr. Nam-Ho Kim for

his generous sharing and patience. I would also like to thank the members of my

advisory committee, Dr. Sivaramakrishnan Balachandar, Dr. Thomas L. Jackson, Dr,

Herman Lam and Dr. George. Michailidis. I am grateful for their willingness to serve on

my committee and constructive criticism for my work.

I am also grateful to the ones that have collaborated with me and directly

contributed to my work: Dr. Chanyoung Park, Dr. Jaco Schutte, Dr. Siddharth Singh

Thakur, Yash A Mehta, John Meeker, Dr. Waruna Seneviratne, Upul R. Palliyaguru,

Dylan Rudolph, Aravind Neelakantan, Nalini Kumar, Carlo Pascoe, Dr. Vladimir

Balabanov, Dr. Leif Carlsson.

I am thankful to my amazing colleagues from Structural and Multidisciplinary

Optimization Group: Shu Shang, Maria G. Fernandez, Anirban Chaudhuri, Sam Nili,

Dawn An, Na Qiu, Garrett Waycaster, Yiwei Wang, Zhendong Guo, Charles F Jekel Jr,

Sangjune Bae, Ting Dong.

Financial support was gratefully provided by Defense Advanced Research

Projects Agency contract W911QX-13-C-0137 and Predictive Science Academic

Alliance Program contract DE-NA0002378.

5

TABLE OF CONTENTS page

ACKNOWLEDGMENTS .................................................................................................. 4

LIST OF TABLES ............................................................................................................ 9

LIST OF FIGURES ........................................................................................................ 12

ABSTRACT ................................................................................................................... 17

CHAPTER

1 INTRODUCTION .................................................................................................... 19

1.1 Motivation ......................................................................................................... 19 1.2 Function Prediction at Inaccessible Domain using Subsets of Samples ........... 20

1.3 Multi-fidelity Surrogates to Combine Datasets with Different Fidelities ............. 20 1.4 Adaptive Sampling When Sampling Cost is Variable in Design Space ............. 22

1.5 Design Space Sampling by Exploration and Replication .................................. 23 1.6 Approximating Field Data using Proper Orthogonal Decomposition ................. 24 1.7 Objectives ......................................................................................................... 25

1.8 Publications ...................................................................................................... 26 1.9 Outline .............................................................................................................. 27

2 FUNCTION PREDICTION AT ONE INACCESSIBLE POINT USING CONVERGING LINES ............................................................................................ 28

2.1 Background ....................................................................................................... 28 2.2 Principal Idea for Method of Converging Lines ................................................. 30

2.3 One-dimensional Function Estimation using Ordinary Kriging .......................... 32 2.4 Combination of Prediction from Multiple Lines .................................................. 33 2.5 Numerical Properties of Method of Converging Lines ....................................... 37

2.5.1 Algebraic Illustration Functions ................................................................ 37

2.5.2 Method of Converging Lines versus Multi-dimensional Approximation ... 39 2.5.2.1 Extrapolation .................................................................................. 39 2.5.2.2 Interpolation ................................................................................... 41

2.5.3 Applicability to Very High-dimensionality ................................................. 43 2.6 Long-distance Extrapolation and Interpolation of a 2D Drag Coefficient ........... 47

2.6.1 Introduction to the Drag Coefficient Function .......................................... 47 2.6.2 Extrapolation Results............................................................................... 49

2.6.2.1 Extrapolation using the method of converging lines ....................... 49 2.6.2.2 Comparison of 1D and 2D surrogates ............................................ 50

2.6.3 Interpolation Results ................................................................................ 52 2.6.3.1 Interpolation using the method of converging lines ........................ 52 2.6.3.2 Comparison of 1D and 2D surrogates ............................................ 53

2.7 Concluding Remarks ......................................................................................... 54

6

3 MULTI-FIDELITY SURROGATE BASED ON SINGLE LINEAR REGRESSION .... 56

3.1 Background ....................................................................................................... 56

3.2 Linear Regression Multi-Fidelity Surrogate ....................................................... 58 3.2.1 Bayesian Multi-fidelity Surrogate Frameworks ........................................ 58 3.2.2 Proposed Linear Regression Multi-fidelity Surrogate .............................. 59

3.3 Approximation of the Fluidized-Bed Process using LR-MFS ............................ 62 3.3.1 A Benchmark Dataset: Fluidized-Bed Process ........................................ 63

3.3.2 Comparison with the Bayesian MFS using Two-fidelity Dataset .............. 63 3.3.3 Effect of Noise for High-fidelity Dataset ................................................... 66 3.3.4 LR-MFS with Multiple Low-fidelity Datasets ............................................ 67

3.4 Approximation of the Modified Currin Function using LR-MFS ......................... 69

3.4.1 The Modified Currin Function .................................................................. 69 3.4.2 The Effect of Introducing Low-fidelity Dataset ......................................... 71 3.4.3 Comparison with Co-Kriging .................................................................... 72

3.4.4 Fitting High-fidelity Dataset with Different Levels of Noise using LR-MFS .............................................................................................................. 73


4 MULTI-FIDELITY SURROGATE MODELING FOR APPLICATION/ARCHITECTURE CO-DESIGN ...................................................... 77

4.1 Background ....................................................................................................... 78 4.2 Related Research ............................................................................................. 80

4.3 Application and Architecture Models ................................................................. 83

4.3.1 CMT-nek .................................................................................................. 84 4.3.2 CMT-bone and CMT-bone-BE ................................................................. 85 4.3.3 Behavioral Emulation (BE) Simulation ..................................................... 85

4.4 Multi-fidelity Surrogates .................................................................................... 86 4.5 Developing MFS Model ..................................................................................... 88

4.5.1 Design Space .......................................................................................... 89 4.5.2 Validations of BE Simulation Results....................................................... 90

4.6 Evaluating MFS Predictions --- Three Case Studies ......................................... 92 4.6.1 Case Study 1: CMT-nek Predictions from BE Simulations ...................... 93

4.6.2 Case Study 2: CMT-bone Predictions from BE Simulations .................... 96 4.6.3 Case Study 3: CMT-nek Predictions from CMT-bone .............................. 97


5 ON APPROACHES TO COMBINE EXPERIMENTS AND SIMULATIONS ............. 99

5.1 Background ..................................................................................................... 100 5.2 Open-Hole-Tension Tests ............................................................................... 101

5.2.1 Experiments .......................................................................................... 101

5.2.2 Finite Element Simulations .................................................................... 103 5.3 Approaches to Predict Mean Strength of Untested Points Assisted by

Simulations ........................................................................................................ 104

5.3.1 Model Calibration .................................................................................. 105

7

5.3.2 Algebraic Correction (Multi-fidelity Surrogate) ....................................... 106 5.4 Results for Combining Simulations and Experiments ..................................... 107

5.5 Using Tests from One Material System to Predict Strength of Another Material System ................................................................................................. 110

5.6 Predicting B-basis Allowables of Experimental Strength Assisted by Aimulations ........................................................................................................ 112

5.6.1 Background ........................................................................................... 113

5.6.2 Predicting B-basis Allowable at Untested Points using Surrogates ....... 115 5.6.2.1 Linear regression tolerance intervals ........................................... 115 5.6.2.2 B-basis allowables of Open-Hole-Tension tests ........................... 116 5.6.2.3 Predicting B-basis allowable of Open-Hole-Tension tests using

surrogate............................................................................................... 117

5.6.3 Predicting B-basis Allowable using Multi-fidelity Surrogate ................... 118 5.6.3.1 Simple framework of multi-fidelity modeling ................................. 118

5.6.3.2 Predicting B-basis allowable of using multi-fidelity surrogate ....... 119 5.6.3.3 Risky sampling strategy ............................................................... 120

5.7 Concluding Remarks ....................................................................................... 121

6 A STRATEGY FOR ADAPTIVE SAMPLING WHEN SAMPLING COST IS VARIABLE IN DESIGN SPACE ............................................................................ 122

6.1 Background ..................................................................................................... 122 6.2 Adaptive Sampling using Surrogate ................................................................ 125

6.2.1 The Basic Steps in Adaptive Sampling .................................................. 125 6.2.2 Kriging Surrogate .................................................................................. 126

6.3 Proposed Adaptive Sampling Strategy with Varying Sampling Cost ............... 127 6.3.1 Proposed Methodology .......................................................................... 127

6.3.2 Illustration of AS-C using 1D Algebraic Function ................................... 129 6.4 Multivariate Test Function and Algebraic Cost Functions ............................... 131

6.4.1 The Normalized Branin Function ........................................................... 131 6.4.2 Four Algebraic Cost Functions .............................................................. 132

6.5 Numerical Performance of AS-C ..................................................................... 134 6.5.1 Evaluation Plan ..................................................................................... 134

6.5.2 A Typical Case using the AS and AS-C ................................................. 136 6.5.3 Different Cost Functions and Different Initial Samples .......................... 138


7 DESIGN SPACE SAMPLING BY EXPLORATION AND REPLICATION FOR ESTIMATING EXPERIMENTAL STRENGTH ....................................................... 142

7.1 Background ..................................................................................................... 142 7.2 Exploration versus Replication Sampling Schemes for Open-Hole-Tension

Tests .................................................................................................................. 146 7.2.1 Open-Hole-Tension Tests ..................................................................... 146 7.2.2 Experimental Strength Results and Polynomial Fit ................................ 148 7.2.3 Resampling Experimental Strength for Comparing Exploration and

Replication .................................................................................................. 149

8

7.2.4 Identifying the Distribution Type of OHT Strength ................................. 152 7.2.5 Effect of Between-batch Variability ........................................................ 155

7.3 Influence of Distribution Type on Comparison between Sampling Strategies for an Analytical Model of a Composite Plate .................................................... 157

7.3.1 Synthetic Noise using Three Types of Distributions .............................. 159 7.3.2 Fitting Strategy ...................................................................................... 160 7.3.3 Influence of Distribution Type on Sampling Strategies .......................... 161


8 FIELD APPROXIMATION USING SURROGATES AND PROPER ORTHOGONAL DECOMPOSITION ..................................................................... 165

8.1 The Drop Falling for Direct Numerical Simulations of Multiphase Flows ......... 165

8.2 Approximating Flow Field using Proper Orthogonal Decomposition and Kriging ............................................................................................................... 166

8.3 Approximating Extreme Scalars using Field Data and Scalar Data ................ 168 8.4 Predicting Border of Multiphase Flow ............................................................. 171 8.5 Concluding Remarks ....................................................................................... 174

9 CONCLUSION ...................................................................................................... 176

APPENDIX

A ANALYTICAL EXPRESSIONS FOR DRAG COEFFICIENT ................................. 180

B ONE-DIMENSIONAL FUNCTION EXTRAPOLATION USING SURROGATES ... 182

B.1 Background .................................................................................................... 182 B.2 Possible Behaviors of Extrapolation using Surrogates ................................... 184

B.3 An Error Metric for 1D Extrapolation ............................................................... 186 B.4 Surrogate Comparison for Extrapolation ........................................................ 187 B.5 Estimating Extrapolation Distance .................................................................. 189

B.5.1 Effective Extrapolation Distance ........................................................... 189

B.5.2 Identification of Effective Extrapolation Distance using Correlation ...... 190 B.6 Concluding Remarks ...................................................................................... 192

C WEIBULL DISTRIBUTION WITH A HEAVY TAIL ................................................. 193

D SPECIMEN STRENGTH SUMMARY FOR OHT TESTING .................................. 194

E VARIABILITY FOR PREDICTION ACCURACY WHEN COEFFICIENT OF VARIATION IS 0.2 ................................................................................................ 198

LIST OF REFERENCES ............................................................................................. 199

BIOGRAPHICAL SKETCH .......................................................................................... 211

9

LIST OF TABLES

Table page 2-1 Target point, variable range and inaccessible domain of 8D algebraic

function ............................................................................................................... 38

2-2 Target point, variable range and inaccessible domain of 100D Styblinski-Tang function ...................................................................................................... 39

2-3 Target point, variable range and inaccessible domain of drag coefficient function ............................................................................................................... 48

2-4 Mean value of prediction and its uncertainty given by standard deviation for extrapolation of drag coefficient function along three lines ................................. 49

2-5 Interpolation results along three lines for drag coefficient function ..................... 52

3-1 Evaluation plan for the approximation of multi-fidelity dataset from Fluidized-Bed Process following [Le Gratiet, 2013] ............................................................ 65

3-2 Major settings for the modified Currin test function ............................................. 70

3-3 Key factors of the evaluation plan to approximate the modified Currin function . 71

3-4 Relative performance of LR-MFS and Co-Kriging for the approximation of the modified Currin function using 100 sets of HF samples ...................................... 73

4-1 Predicting execution times of CMT models based on a typical set of 12 samples and evaluating the prediction using R-RMSE (%) ................................ 94

4-2 Range of the scale factors for LS-MFS ............................................................... 98

5-1 w/D test matrix .................................................................................................. 102

5-2 Layup test matrix .............................................................................................. 102

5-3 Mean value of experimental strength of composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric (ksi) ................................................. 102

5-4 Relative error and root mean square error at untested points based on 6 samples ............................................................................................................ 108

5-5 Experimental strength of composite laminate from T700SC-12K-50C/#2510 Plain Weave Fabric (ksi). .................................................................................. 110

5-6 Statistics of OHT tests at the 12 experimental locations ................................... 116

10

5-7 B-basis allowables of OHT tests at the 12 experimental locations using non-parametric approach ......................................................................................... 117

6-1 Details of AS and AS-C for the approximation of Forrester function with a budget of up to 4.5 total cost ............................................................................ 130

6-2 Original input variable space for the four cost functions ................................... 134

6-3 Key factors of the evaluation plan ..................................................................... 134

6-4 Details for a typical DOE of AS and ASC for the approximation of Branin function and linear cost function with up to 10 total cost ................................... 137

6-5 Median value of area metrics 2AR

and maxA (in %) using 100 sets of initial

samples ............................................................................................................ 139

6-6 Median values of total samples for AS and AS-C using 100 sets of initial samples ............................................................................................................ 140

7-1 w/ D ratio test matrix. 18 replicates are selected from three pre-preg batches . 147

7-2 Statistical properties of OHT strengths at given w/D with 18 replicates ............ 148

7-3 Relative difference between surrogate prediction and mean strength using quadratic PRS .................................................................................................. 149

7-4 Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens and 24 specimens. Mean values reported are based on 1,000 sets of samples .................................................................................. 151

7-5 Statistics of Welch's t-test between specimens from batch 1 and batch 2 ........ 156

7-6 Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens (3 replicates from 4 configurations for exploration, 4 replicates from three configurations for replication). Mean values reported are based on 1,000 sets of samples ....................................................................... 157

7-7 Parameter settings of composite laminate ........................................................ 159

8-1 Parameter settings ........................................................................................... 171

8-2 Fraction of contribution for first 4 modes to ensemble snapshots ..................... 171

B-1 Average AE for extrapolation of test functions using 30 sets of samples ......... 189

D-1 Specimen strength at w/d=3. Nominal thickness for one ply is 0.0086 inch. ....... 194

D-2 Specimen strength at w/d=4. Nominal thickness for one ply is 0.0086 inch. ........ 195

11



12

LIST OF FIGURES

Figure page 2-1 Illustration of inaccessible domain for surrogate modeling ................................. 28

2-2 Illustration of space-filling sampling (LHS) and Method of converging lines ....... 31

2-3 A typical set of extrapolation results from 8D Dette function. The three lines are from randomly selected vertices to the target vertex. ................................... 40

2-4 Summary of extrapolation results from multiple lines and 8D Kriging based on Dette function ................................................................................................ 41

2-5 A typical set of interpolation results from 8D Dette function ............................... 42

2-6 Summary of interpolation results from multiple lines and 8D Kriging based on Dette function ..................................................................................................... 43

2-7 A typical set of extrapolation results from 100D Styblinski-Tang function ........... 44

2-8 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function ..................................................................................... 45

2-9 A typical set of interpolation results from 100D Styblinski-Tang function ............ 46

2-10 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function ..................................................................................... 46

2-11 Response of drag coefficient on Re and M ......................................................... 48

2-12 Lines and samples selection for extrapolating and interpolation of drag coefficient function .............................................................................................. 48

2-13 Extrapolation results of drag coefficient function using samples from reduced sampling domain ................................................................................................ 50

2-14 Extrapolation results of drag coefficient function using multi-dimensional Kriging and converging lines............................................................................... 51

2-15 Interpolation results of drag coefficient function ................................................. 52

2-16 Interpolation results using multi-dimensional Kriging and converging lines ........ 54

3-1 Function values of the expT (high-fidelity) and 2T (low-fidelity) from Fluidized-

Bed Process ....................................................................................................... 64

3-2 Illustration of the resampling procedure to generate training and test data ........ 65

13

3-3 Performance of the surrogate models from 100 combinations of training samples and test sets for the approximation of fluidized bed process. ............... 66

3-4 Function values of the expT (high-fidelity) with synthetic noise from normal

distribution .......................................................................................................... 67

3-5 RMSE of LS-MFS to approximate HF samples of fluidized bed process with noise. The boxplots were based on 100 combinations of training samples and test sets. ...................................................................................................... 67

3-6 RMSE of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset. The boxplots were based on 100 combinations of training samples and test sets. .................. 68

3-7 Scale factors 𝝆 of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset. .......... 69

3-8 Responses of the modified Currin function in 2D space ..................................... 70

3-9 Effect of introducing low-fidelity samples for prediction of the modified Currin function without noise. PRS indicates the fit based on only HF samples. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples ................................................................................................... 72

3-10 Comparison of Co-Kriging (CK) with LR-MFS for the approximation of the modified Currin function. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples ................................................ 73

3-11 Comparison of LR-MFS with different level of noise for the modified Currin function. 3 and 10 are the number of HF samples. The RMSE is based on 100 sets of samples ............................................................................................ 74

3-12 Estimated noise level of LR-MFS with a different number of HF samples for the modified Currin function. 3 and 10 are the number of HF samples. The median STD is based on 100 sets of samples.................................................... 75

3-13 Conceptual illustration of the major epistemic uncertainty for LR-MFS with different number of HF samples ......................................................................... 75

4-1 Hierarchy of the CMT models ............................................................................. 84

4-2 Design of experiments for CMT-nek, CMT-bone, and BE simulations ................ 90

4-3 Validation of BE simulation against CMT-bone-BE ............................................. 91

4-4 Comparing CMT-bone mini-app and CMT-bone-BE skeleton app trends for various parameter values ................................................................................... 92

14

4-5 Difference between CMT-nek validation runs and multi-fidelity predictions based on CMT-bone simulation .......................................................................... 95

4-6 Difference between CMT-bone validation runs and multi-fidelity predictions based on BE simulation ...................................................................................... 96

4-7 Difference between CMT-nek validation runs and multi-fidelity predictions based on CMT-bone ........................................................................................... 97

5-1 Open-Hole Tension Test Specimen Configuration and observed failure mode 102

5-2 Mesh of finite element model for OHT test ....................................................... 104

5-3 Illustration of experimental results and simulations for OHT tests .................... 104

5-4 Failure criteria for a circular through thickness holes in an infinite plate ........... 105

5-5 Selected points from experimental strength (high-fidelity function) used to predict strength at other points ......................................................................... 108

5-6 Performance of different correction models for predicting OHT strength using FEA. Experiments were performed based on composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric ......................................................... 109

5-7 Experimental strength of OHT tests from two different material systems ......... 111

5-8 Performance of different correction models for predicting OHT strength using FEA or another material. Experiments for prediction were performed based on composite laminate made from T700SC-12K-50C/#2510 Plain Weave Fabric ............................................................................................................... 111

5-9 Illustration of B-basis allowables based on replicates/repeated tests ............... 114

5-10 Typical histograms (out of 12 locations) based on 18 replicates from OHT tests .................................................................................................................. 116

5-11 Overall accuracy (relative RMSE) for PRS tolerance and direct PRS with increasing number of experimental tests .......................................................... 118

5-12 Overall accuracy (relative RMSE) for MFS and direct PRS with increasing number of experimental tests ........................................................................... 120

5-13 Samples corresponding to the worst three sets of predictions using MFS. Solid points indicate selected experiments out of 12 for MFS. ......................... 120

6-1 Basic steps of adaptive sampling for global approximation. One additional sample is added per iteration............................................................................ 126

15

6-2 Flowchart of the adaptive sampling with varying sampling cost. One additional sample is added per iteration at the point with maximum value to improve the global approximation. .................................................................... 128

6-3 One-dimensional algebraic example Forrester function ................................... 129

6-4 Initial samples and surrogate prediction for the Forrester function ................... 130

6-5 Surrogate predictions for adaptive sampling plans for Forrester function example ............................................................................................................ 131

6-6 Response of Normalized Branin function in the input variable space ............... 132

6-7 Response of the four cost functions in normalized input variable space .......... 133

6-8 An illustration of the evolution of accuracy for adaptive sampling with increasing total sampling cost ........................................................................... 135

6-9 Final samples for a typical DOE the approximation of Branin function with linear cost function ............................................................................................ 137

6-10 Evolution of adaptive sampling with increasing samples for a typical DOE for the approximation of Branin function and linear cost function .......................... 138

6-11 The population of area metrics for the approximation of the Branin function with four cost functions ..................................................................................... 139

6-12 The number of samples used for the approximation of the Branin function with four cost functions using AS and AS-C ................................................. 13940

7-1 Replication strategy vs. exploration strategy using two design variables and 64 samples ....................................................................................................... 144

7-2 OHT test specimen configuration and observed failure mode .......................... 147

7-3 Experimental data and quadratic polynomial response surface fitted to the data. ................................................................................................................. 149

7-4 Illustration for one set of resampled experimental strength representing exploration or replication strategy using 12 points ............................................ 150

7-5 K-S test statistic for OHT strength .................................................................... 154

7-6 Probability plots for the fit of pooled OHT tests................................................. 154

7-7 Statistical properties of specimen strength ....................................................... 155

16

7-8 Configuration and failure response of composite laminate. α is a constant to balance between horizontal and vertical loadings. θ is the orientation angle of lamina. .............................................................................................................. 158

7-9 Three distributions with mean value to be 8.5 ksi. The mean values of the five synthetic batches are 8.1 ksi, 8.5 ksi, 8.8 ksi, 8.4 ksi, 9.0 ksi..................... 160

7-10 Mean value of RMSE for relative error based on 1000 sets of samples with different distribution types. Number of samples indicates the total number of tests including 1~7 replicates for each test point. ............................................. 162

8-1 The front and velocity field in a rectangular box at different times .................... 166

8-2 Illustration of three snapshots at different times. Red point denotes location of minimum pressure ........................................................................................ 169

8-3 Kriging prediction of mode coefficients ............................................................. 170

8-4 Predicting minimum pressure from surrogate estimation and field approximation. Blue line is true value, red line is approximation ...................... 170

8-5 Coefficients of first 4 basis modes in variable space using Kriging ................... 172

8-6 Illustration for validation points, samples and variable space ........................... 173

8-7 Comparing simulation and POD prediction at 4 validation points ..................... 174

B-1 Three one-dimensional test functions for extrapolation .................................... 184

B-2 Extrapolation of three one-dimensional test functions ...................................... 184

B-3 Extrapolation of the logrithmic function ............................................................. 185

B-4 Extrapolation of the logrithmic function from different sampling domain (a) Full sampling domain and (b) Reduced sampling domain ................................ 186

B-5 Four test functions for surrogate selection. Domains close to origin are inaccessible domain. ........................................................................................ 188

B-6 Two types of effective extrapolation distance ................................................... 190

B-7 Correlation and distance ratio over effective distance while extrapolating 30 1D functions...................................................................................................... 191

E-1 Approximating the strength of composite laminate with the trade-off between exploration and replication. Variability of mean RMSE with normal noise, coefficient of variation is 0.2. ............................................................................ 198

17

Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

USING MULTIPLE PREDICTIONS AND MULTIPLE SOURCES OF DATA FOR

DESIGN SPACE EXPLORATION

By

Yiming Zhang

May 2018

Chair: Raphael T. Haftka Cochair: Nam-Ho Kim Major: Mechanical Engineering

Design optimization and uncertainty quantification need to examine a large

number of candidate configurations. This is commonly done by doing simulations and/or

experiments at a number of design points and approximating at untested points using

surrogates. The objective of this work is to check on combining multiple ways to predict

by integrating various data sources and numerical models. A series of approaches are

proposed as in the following.

Function prediction towards inaccessible points could be encountered for

simulations beyond validation or unaffordable tests. The method of converging line is

proposed based on multiple 1D surrogates towards the inaccessible point. Multiple

predictions based on 1D surrogates are combined using Bayesian inference to increase

reliability. Converging lines proved to be much more reliable and accurate than the

multi-dimensional surrogate for a drag coefficient function and two algebraic functions.

Then the Linear regression multi-fidelity surrogate (LR-MFS) is proposed to

combine datasets with different fidelities. The proposed LR-MFS is expected to be

efficient (low computational cost) and robust for noisy data. The proposed LR-MFS has

18

been evaluated using two benchmark test functions and applied for: (i) estimating failure

strength of composite laminates. (ii) predicting performance of supercomputers

Current sampling plans implicitly assume that all samples have the same cost to

produce. An adaptive sampling scheme with varying cost (AS-C) is developed to handle

varying sampling cost in the design space. Two sets of surrogates are constructed for

the target function (quantity of interest) and the sampling cost. Comparing with standard

adaptive sampling scheme, AS-C led to more samples and more accurate fitting for the

approximation of selected test function.

Two more topics are discussed after that. (1) The first one is on the trade-off for

design space sampling between exploration versus replication. The study is based on a

set of experiments for the strength of a composite laminate. A resampling procedure is

proposed that compared exploration and replication. (2) The second one is on the

approximation of flow field using surrogate and proper orthogonal decomposition. The

field approximation is evaluated for predicting extreme quantities of interest (e.g.

pressure and border of multi-phase flow).

19

CHAPTER 1 INTRODUCTION

1.1 Motivation

Effective prediction of mechanical system response is often the basis for design.

Experimental tests and physics-based simulation are routinely adopted to approximate

system response at given design variables. The test results (referring to both

experiments and simulations) are scattered in variable space and statistical inference

approaches (e.g. surrogate/machine learning/ data mining) are adopted to predict

response at untested points in the variable space. These methods enable near-real-time

estimation of system responses for informed decision making.

Usually, a single approach is used to combine all the information in order to

make a prediction. In recent days, there has been interest in the simultaneous use of

multiple approaches rather than a single one [1-3]. For example, comparing and

potentially combining multiple surrogates proved to be more accurate or at least

equivalent than a single model [4]. Motivated by the combination of different surrogates,

other sources of predictions are examined in the dissertation to make the most use of

available information and therefore characterize mechanical system effectively. We

check on combining multiple ways to predict by integrating various data sources and

numerical models. Firstly, we explore the possibility to develop and combine surrogate

models based on subsets of data. Secondly, we aim to improve the predicative

capability of multi-fidelity surrogates using an algebraic function to combine datasets

with different fidelities. Thirdly, we examine the scheme to incorporate varying sampling

cost for the design of experiments. Fourthly, the trade-off between exploration versus

replication is discussed for the approximation of experimental strength. At the end, we

20

investigate a combination schemes for approximation of field data and approximation of

scalar quantities.

1.2 Function Prediction at Inaccessible Domain using Subsets of Samples

The focus of this chapter is a strategy for making prediction at a point where a

function cannot be evaluated. The key idea is to take advantage of the fact that

prediction is needed at one point and not in the entire domain. We explore the

possibility of predicting a multi-dimensional function using multiple one-dimensional

lines converging on the inaccessible point. The multi-dimensional approximation is thus

transformed into several one-dimensional approximations, which provide multiple

estimates at the inaccessible point. Kriging model is adopted in this work for the one-

dimensional approximation, estimating not only the function value but also the

uncertainty of the estimate at the inaccessible point. Bayesian inference is then used to

combine multiple predictions along lines. We evaluated the numerical performance of

the proposed approach using an 8-dimensional and a 100-dimensional functions in

order to illustrate the usefulness of the method for mitigating the curse of dimensionality

in surrogate-based predictions. Finally, we applied the method of converging lines to

approximate a two-dimensional drag coefficient function. The method of converging

lines proved to be more accurate, robust and reliable than a multi-dimensional Kriging

surrogate for single-point prediction. The chapter 2 on the method of converging lines is

based on the publication [5-7] co-authored by Chanyoung Park, Nam Ho Kim, and

Raphael T. Haftka.

1.3 Multi-fidelity Surrogates to Combine Datasets with Different Fidelities

Multi-fidelity surrogates (MFS) combine low-fidelity models with few high-fidelity

samples to infer the response of high-fidelity model for design optimization or

21

uncertainty quantification. Most publications in MFS focus on Bayesian frameworks

based on Gaussian Process. Other types of surrogates might be preferred for some

applications. In this work, a simple and yet powerful MFS based on single linear

regression is proposed, termed as Linear Regression Multi-fidelity Surrogate (LR-MFS),

especially for fitting high-fidelity data with noise. The LR-MFS considers the low-fidelity

model as a basis function and identifies unknown coefficients of both the low-fidelity

model and the discrepancy function using a single linear regression. Since the proposed

LR-MFS is obtained from the standard linear regression, it can take advantage of

established regression techniques such as prediction variance, D-optimal design, and

inference. The LR-MFS is first compared with three Bayesian frameworks using a

benchmark dataset from the simulations of a fluidized-bed process. The LR-MFS

showed a comparable accuracy with the best Bayesian frameworks. The effect of

multiple low-fidelity models was also discussed. Then the LR-MFS is evaluated using an

algebraic function with different sampling plans. The LR-MFS bested Co-Kriging for

55%~63% cases with increasing number of high-fidelity (HF) samples. For both

examples, the LR-MFS proved to be better than fitting only HF samples, and robust with

noisy data. The chapter 3 on LR-MFS is based on publication [8] co-authored with

Chanyoung Park, Nam Ho Kim and Raphael T. Haftka. The proposed MFS has been

applied to combine the experiments and simulations for two engineering problems. The

first one is behavioral emulation for application/architecture co-design of the exascale

computer. The collection of dataset and details of the experiments were kindly provided

by Aravind Neelakantan, Nalini Kumar and Herman Lam. The chapter 4 on multi-fidelity

surrogate is based on the publication [9] co-authored with Aravind Neelakantan, Nalini

22

Kumar, Herman Lam, Chanyoung Park, Nam Ho Kim and Raphael T. Haftka. The

second one is predicting the strength of composite laminate with a central hole. The

collection of dataset and details of the experiments were kindly provided by Jaco

Schutte, John Meeker and Upul R. Palliyaguru. The chapter 5 on multi-fidelity surrogate

modeling of composite laminate is based on publication[10, 11] and report co-authored

with Jaco Schutte, John Meeker and Upul R. Palliyaguru, Nam Ho Kim and Raphael T.

Haftka.

1.4 Adaptive Sampling When Sampling Cost is Variable in Design Space

Surrogate models have been developed to infer the response of mechanical

systems based on scattered tests/simulations for efficient design optimization. The

method to determine the number and location of the tests is the sampling plan or design

of experiment (DOE). Effective DOE enables surrogate desirable accuracy while

balancing the testing budgets. Current methods for DOE creation implicitly assume that

all samples have the same cost to produce. But the cost for tests/simulations may vary

in the input variable space because some configurations are more expensive to test or

simulate than others. This work explores the possibility to introduce varying sampling

cost for DOE. An adaptive sampling strategy with varying sampling cost (AS-C) is

proposed. Two sets of surrogates are constructed for the target function (quantity of

interest) and the sampling cost. Then a value metric is defined to estimate the

uncertainty to be reduced per cost. The work studies adaptive sampling for global

approximation and one new sample is added per iteration at the point with maximum

value. The proposed AS-C was evaluated using 1D and 2D algebraic functions. Four

different cost functions and 100 sets of initial samples were produced for evaluation. For

a fixed sampling budget, the AS-C was likely to lead to more samples and better

23

accuracy than the standard adaptive sampling strategy. The chapter 6 on adaptive

sampling is based on the publication [12] co-authored with Chanyoung Park, Nam Ho

Kim and Raphael T. Haftka.

1.5 Design Space Sampling by Exploration and Replication

The industry has to revert to experiments for practical design of composite

laminates when physical model/ simulations are inadequate for desirable accuracy or

require excessive computational resources. Surrogates are used to predict strength of

composite laminates in the design space by conducting an array of tests. It has been

reported that an exploration strategy to test as many different configurations as possible

is more effective than replication of fewer points for reducing test noise. The observation

was based on analytical test functions and synthetic Gaussian noise. This work first

studies real experiments to check whether the previous observation stands. We

examined test results of open-hole-tension (OHT) for composite plates that included 18

replicates per test point. We developed a resampling procedure that compared

exploration and replication and found that exploration proved to be more accurate for

prediction than replication. Secondly, we examined the major source of uncertainty for

surrogate prediction, which is variability of strength. The distribution of experimental

OHT strength was found to be not unambiguously independent and identically

distributed normal distribution as commonly assumed. The variation of specimen

strength is correlated rather than independent at different configurations due to the

between-batch variability. Consequently, the influence of distribution type was then

investigated on an analytical test function with three synthetic distributions. The

exploration strategy proved to be better than the replication strategy for all three

distributions. We found that the exploration strategy allows for higher order polynomial

24

surrogate to be used, which is a key point to improve characterization of a function with

complex dependence on design parameters. The chapter 7 on the design of

experiments is based on the publication [13, 14] co-authored with Jaco Schutte, Waruna

Seneviratne, Nam Ho Kim and Raphael T. Haftka.

1.6 Approximating Field Data using Proper Orthogonal Decomposition

Large-scale mechanical system simulation may need to explore the variable

space defining a flow field to understand physical phenomena, perform systematic

validation, sensitivity analysis or design optimization. Surrogate model is frequently

adopted as one major tool to approximate system response in variable space.

Comparing with common system response, which is a scalar, one specialty of

computational fluid dynamic analysis (CFD) is that the whole flow field may be of special

interest. Therefore, reduced order modeling (ROM) has been developed and became a

popular research topic of computational engineering in past decade.

A typical ROM procedure includes feature extraction using mathematical

decomposition [15-17]. Proper orthogonal decomposition (POD) is one popular

approach for model decomposition. Snapshots (matrix for the field of interest) are first

obtained at different variables (e.g. temporal, spatial of dimensional coordinates). The

POD is then performed to decompose the snapshots as a linear combination of basic

modes. The flow field at untested points can be reconstructed based on the

decomposed modes. The coefficients of the modes are traditionally obtained using

projection methods. ROM has also been used to approximate non-linear governing

equations instead of snapshots of flow field [18]. Jeong et. al [19] integrated machine

learning algorithm to approximate Navier-Stokes equations.

25

Kriging surrogate has been studied [20] to approximate whole flow field based

on incomplete information and illustrated competitive performance with ROM. The

combination of POD with Kriging surrogate has also been considered in [21]. After

obtaining basic modes and corresponding coefficients using POD. Kriging surrogates

were developed to estimate coefficients of modes at untested points. This method

proved to be effective for stationary field prediction under subsonic condition in [11].

For design optimization, we are more interested in extreme quantities. ROM

retains all the information from the simulated field [22, 23] and surrogate mainly

approximate specific quantity of interest. It might be useful to combine predicted

extreme values from surrogate with the reconstructed field for integrated analysis. We

propose one approach to correct predicted field data using POD against the

approximated scalar quantities from surrogates. A corrected flow field could be

conveniently used for both qualitative investigation and quantitative analysis.

1.7 Objectives

The main objectives of this research are to address the following challenges:

• Prediction at inaccessible points using multiple surrogates: The function prediction at inaccessible point is challenging and lacks validation. We propose to develop surrogates from different data sets to increase reliability. We apply the idea of multiple data sets to predictions of a single inaccessible point rather than estimating the whole variable space. Data sets are sampled along different lines converging to the target point.

• Multi-fidelity models for noisy data: The mainstream Bayesian multi-fidelity surrogates (MFS) have been commonly studied to combine multi-fidelity data. However, there is a risk for Bayesian MFS of estimating inappropriate hyper-parameters especially in the presence of noisy data. The least-square MFS (LS-MFS) using linear regression is proposed to handle noisy samples.

• Adaptive sampling for variable sampling cost: Current sampling schemes implicitly assume that all samples have the same cost to produce. However, the cost for tests/simulations may vary in the input variable space. We examine the possibility to introduce varying sampling cost for the adaptive sampling. Two sets

26

of surrogates are constructed for the target function (quantity of interest) and the sampling cost.

• Design space sampling by exploration and replication: The study on the trade-off between exploration and replication has been performed using algebraic functions and synthetic noise. The mathematical assumptions are usually violated to some extent for practical design. We tried to study the design of experiments using physical test following regulations.

• Field approximation: Proper orthogonal decomposition has been associated with surrogate models to predict flow field at untested points/configurations. Flow field is useful to provide physical insights, but the extreme quantities (e.g. scalars) are usually of interest for design optimization. We aim to evaluate the performance of field approximation regarding the extreme quantities regarding scalar values and borders of multi-phase flow.

1.8 Publications

The research has produced the following contributions with respect to:

• Function prediction at inaccessible domain using subsets of samples: The method of converging lines are proposed for function estimation towards one single inaccessible point. A multi-dimensional approximation is decomposed as multiple one-dimensional predictions. The extrapolation of 1D function and noisy data are also examined. [5, 6, 24, 25]

• Multi-fidelity surrogate modeling: The linear regression multi-fidelity surrogate is proposed and evaluated using benchmark dataset. The proposed schemes have been accepted and applied for two engineering applications (1) behavioral emulation of exascale computer and (2) predicting strength of composite laminate. [8-11, 26]

• Adaptive sampling with varying sampling cost: We identify the challenge of design of experiments with varying sampling cost. An adaptive sampling scheme is proposed correspondingly and evaluated on algebraic test functions. The proposed scheme has been applied to guide behavioral simulation of exascale computer. [12, 27, 28]

• Design space sampling by exploration and replication: Experimental strength usually suffers large variation. The trade-off between exploration and replication has been carefully studied using a benchmark composite laminate following regulation. [14, 29]

27

1.9 Outline

The organization of this dissertation is as follows. Chapter 2 elaborates the

approach to combine multiple predictions developed from different subsets of data. The

proposed scheme has been evaluated using algebraic test functions and drag

coefficient function. Chapter 3 presents the linear regression multi-fidelity surrogate to

datasets with variable fidelity. The proposed scheme has been evaluated using both

benchmark dataset and numerical examples. Chapter 4 is the application and

evaluation of the linear regression multi-fidelity on behavioral emulation. Multiple low-

fidelity models are discussed. Chapter 5 investigates the approaches to combine

experiments and simulations including model calibration, multi-fidelity surrogate.

Different combination schemes are evaluated for an open-hole-tension test in composite

laminate. Chapter 6 proposes the strategy for adaptive sampling when the sampling

cost varies in the design space. The strategy has been evaluated using algebraic test

functions. The effect of cost functions is elaborated. Chapter 7 discusses the design

space sampling by exploration and replication for estimating strength of composite

laminate. The experiments are thoroughly analyzed including distribution, uncertainty

and surrogate modeling. Chapter 8 talks the field approximation using surrogate and

proper orthogonal decomposition. The field approximation has been compared with

surrogate only to predict scalar quantities. Chapter 9 closes the dissertation with a

summary of achievements and future research opportunities.

28

CHAPTER 2 FUNCTION PREDICTION AT ONE INACCESSIBLE POINT USING CONVERGING

LINES

2.1 Background

In surrogate modeling, it is common to sample a function at scattered points and

fit the samples with an explicit function to approximate function values at un-sampled

points [30, 31]. The estimation procedure is called interpolation when target points are

inside the convex hull of sampled points, while it is termed extrapolation [32, 33]

otherwise, as shown in Figure 2-1. For one-dimensional samples, for example, convex

hull is set by the lower- and upper- bounds of samples. Surrogate models are routinely

used as interpolation schemes and may be less adequate for extrapolation, which is

commonly encountered while approximating high-dimensional functions [34]. A

surrogate is especially useful to estimate function value when the point of interest

cannot be sampled via simulation or experiment due to extreme conditions, lack of data,

limitations of simulation software or an experiment that is too dangerous to perform. For

example, 16 out of 500 simulations were reported to have failed for an aeroelastic

helicopter blade simulation in Glaz et al. [35] during design optimization.

Figure 2-1. Illustration of inaccessible domain for surrogate modeling. A) Interpolating

inaccessible region and B) extrapolating inaccessible region

B A

29

Several valuable efforts have been initiated towards reliable estimation at

inaccessible points. Neural networks, an advanced framework, has been reported to be

misleading for extrapolation beyond the region of samples [33] and is therefore mostly

used for interpolation. Balabanov et al. [36, 37] demonstrated that regularization

methods can improve extrapolation accuracy beyond sample range even if they

compromise interpolation accuracy. Wilson and Adams [38] proposed a Gaussian

process kernel function for better uncovering data patterns and leading to improved

prediction accuracy of forecasting. Richardson extrapolation has been commonly

adopted to predict response of finite element analysis towards extremely refined mesh

that are unaffordable [39, 40]. A specific polynomial form was used for Richardson

extrapolation to incorporate physical knowledge about convergence order. Hooker [41]

proposed an empirical trend function when predicting towards inaccessible point. In

prognosis for remaining life of mechanical systems, application of Gaussian Process

surrogate and neural network model has been discussed based on heuristic study [42].

Wei-Yin proved that setting a bound for extrapolation using linear model trees could

avoid large error [43]. Besides, numerous works can be found on forecasting one

dimensional time series data at inaccessible points (i.e. future) in financial fields and

weather forecasting, e.g. [44, 45].

Based on this literature, it is challenging to estimate prediction accuracy at

inaccessible points. Surrogate predictions, in particular, risk increasing uncertainty when

prediction points are far from samples [36, 46, 47]. This motivated us to estimate a

function value by creating multiple independent predictions from different data sets.

Introducing multiple predictions may reduce the uncertainty of blind predictions. A new

30

sampling pattern, called the method of converging lines, is proposed in this work. As a

first step towards improving the prediction capability for long-distance prediction using

surrogate model, we explore the possibility of predicting a multi-dimensional function

using multiple one-dimensional lines. The main idea is to select samples along lines

towards the inaccessible point. One-dimensional surrogates are then constructed based

on the samples along each line in order to predict the function value at the inaccessible

point. The method of converging lines transforms multi-dimensional function prediction

into a series of one-dimensional predictions, which provide multiple estimates at the

inaccessible point. Combination of multiple predictions based on Bayesian inference is

proposed to enhance the prediction accuracy. This chapter on method of converging

lines is based on the publication [5-7] co-authored by Chanyoung Park, Nam Ho Kim

and Raphael T. Haftka.

2.2 Principal Idea for Method of Converging Lines

The procedure to select locations where the function is to be sampled is called

design of experiments (DoE). For fitting a surrogate, it is common to select space-filling

DoEs (e.g. Latin Hypercube Sampling (LHS) [48]), which are geared to provide a good

representation of the design domain. The method of converging lines, in contrast,

serves the prediction at a single target point, targetx . This method locates sampling points

along several lines all intersecting at the target point.

The difference between LHS and the sampling scheme of the method of

converging lines in two- and three-dimensional space is illustrated in Figure 2-2. The

target point, targetx , is at the origin, and the domain within a normalized distance of 0.5

from targetx is set to be inaccessible. Three converging lines are shown in Figure 2-2 (B).

31

The orientations of line il may vary with applications in order to generate a good

estimate. Along a given line, local coordinate ( )j

localx of jth sample point (j)x is the Euclidean

distance between (j)x and

targetx as

( )j

localx (

ta et

j)

rgx x (2-1)

where ( )j

localx is a non-negative scalar and zero when (j)

targetx x . The local coordinate,

(j)x , is first normalized to be within [0,1] along each dimension to eliminate the effect of

different scales.

Figure 2-2. Illustration of space-filling sampling (LHS) and Method of converging lines. A) space-filling sampling (LHS) and B) Method of converging lines using 15 samples when the target point is at the origin and the domain at distance of less than 0.5 from the origin is inaccessible

A

B

32

The method of converging lines is expected to be ideal in certain conditions,

where sampling domain is relatively convenient to access in terms of cost and

feasibility,and sampling the target point is impossible or extremely challenging. In this

work, we focused on long-distance prediction, namely, the length of inaccessible

domain equals to that from accessible domain.

2.3 One-dimensional Function Estimation using Ordinary Kriging

The underlying assumption for prediction is that the function behaves similarly in

the accessible domain and inaccessible domain. Therefore, surrogate models providing

acceptable accuracy in sampling domain would be appropriate to estimate the function

at inaccessible points within a certain distance from samples. When this assumption

fails, surrogate prediction is likely to fail, whether we use a multi-dimensional surrogate

or the method of converging lines.

Multiple surrogates have been compared for the extrapolation of one-dimensional

problem as shown in Appendix B. Ordinary Kriging based on correlation kernel is

adopted for one-dimensional modeling due to its superior performance for

approximating noise-free data. Surrogate prediction based on Ordinary Kriging is

reported to avoid large errors at inaccessible points [24]. Kriging assumes that the

distance between sample points reflects a spatial correlation. The value of the function

at a point is correlated to the values at neighboring points based on their separation.

The correlation is strong to nearby points and weak with far away points, but strength

does not change based on location. This unique assumption provides a measure of

uncertainty of the prediction. We adopt the Gaussian model to approximate the

correlation between point x and point s as

33

2

1

exp( ), ( ), ( )n

i i ii

C f f x s

x s θ (2-2)

where n is the dimensionality of variable x and s, i is the hyper-parameter along ith

direction [49]. n is 1 for applying one-dimensional prediction using method of converging

line.

Prediction at an untested point x is formulated as

( ) 2

1

ˆ ˆ( ) ,

exp ( )

T

ni

i k k k

k

y

r s x

x r

(2-3)

where ( )is denotes ith sample point, is the mean of samples. The hyper-parameters

are obtained by maximizing the likelihood that the data comes from a Gaussian

process. Ordinary Kriging is implemented using the surrogate toolbox of Viana and Goel

[50].

The prediction of Ordinary Kriging and the corresponding uncertainty are

expressed as a conditional PDF for a given data set y as |p y x y , which follows a

Student's t-distribution [51]. A Student’s t-distribution is well approximated with a normal

distribution and converges to a normal distribution as the number of samples increases.

In this work, the surrogate prediction with uncertainty is expressed using a normal

distribution for computational efficiency. The conditional PDF of a prediction is defined

as a normal distribution 2ˆ ,N y x x where y x is the Kriging predictor and x is

the standard deviation at x.

2.4 Combination of Prediction from Multiple Lines

For making a prediction in n-dimensional space, one-dimensional surrogate

allows higher accuracy than using an n-dimensional surrogate with the same number of

34

samples. The method of converging lines proposes to combine multiple predictions at

the inaccessible point with independently built multiple one-dimensional surrogates

toward the point to reduce the chance of large error. The true response targety x is

predicted with multiple one-dimensional Kriging surrogates and the prediction of the ith

surrogate and the corresponding prediction uncertainty is defined through a conditional

PDF for the given ith data set as target |i ip y x y , which is a normal distribution having a

mean of the Kriging predictor targetˆ

iy x and a variance of the prediction variance at the

target point targetx .

The multiple predictions can be combined to obtain a single most probable

estimate. Various combination schemes are available to combine multiple predictions

[2, 3]. Here we adopt Bayesian inference [52] to combine Kriging predictions. In the

Bayesian inference, the conditional PDF of the true response from the Kriging based on

ith data set can be interpreted as a likelihood functions as

target target| |i ip y l yx y x y . By assuming independence of prediction uncertainties

between lines, the posterior distribution of the true response is obtained by multiplying

the likelihood functions. With m one-dimensional predictions, the posterior distribution is

expressed as

target target

1 1

| |m m

i i

i i

p y l y

x y x y (2-4)

The posterior distribution in equation Eq. (2-4) can be further derived based on

the approximation using normal distributions as

35

2

target target target target

1 1

2

target target

21

target

2

target target

2

target

ˆ| | ,

ˆ( )exp

2

ˆ( )exp

2

m n

i i i

i i

ni

ii

p y p y y

y y

y y

x y x x x

x x

x

x

x

(2-5)

where 2

target target targetˆ| ,i ip y y x x x is a PDF of a normal distribution at targety x for

the given mean of targetˆ

iy x and standard deviation of targeti x . targety x and target x

are the combined prediction and the standard deviation representing the prediction

uncertainty as

target

2

target

target

2

target

target

2

tar t

1

1ge

1

=1

1=

ˆ

1

ˆ

i

i

i

n

i

i

n

i

n

i

y

y

x

xx

x

x

x

(2-6)

Based on Eq. (2-4) and Eq. (2-5), the posterior distribution target

1

|m

i

i

p y

x y

follows a normal distribution, 2

target targetˆN ,y x x . From the expression of target x

in Eq. (2-6), it can be expected that surrogate prediction from the line with minimum

prediction variance will be most influential on the posterior distribution because the

combined prediction is a weighted average of the predictions based on the prediction

variances.

36

Compared with multi-dimensional approximation, the method of converging lines

is essentially a set of predictions from 1D surrogates. The 1D surrogate is likely to be

more robust than multi-dimensional approximation for two reasons: (a) The number of

hyper-parameters to be estimated for building a one-dimensional surrogate is one while

n hyper-parameters have to be estimated for an n-dimensional surrogate. Therefore, the

cost of evaluating samples to make a reasonable prediction is affordable regardless

dimensionality while the cost to build a n-dimensional surrogate increases excessively

(curse of dimensionality). (b) The 1D surrogate can be easily visualized reducing the

chance that failure of the surrogate fitting will go unnoticed.

One important feature for the method of converging lines is that lines may have

diverse performance. Some lines may be friendly for mathematical modeling, namely

enabling small and reliable prediction variance. These good lines will dominate

combined prediction based on Bayesian method. By potentially improving quality of

surrogate models and combining multiple lines, the method of converging lines is

expected to be more reliable regarding to pointwise estimation. Another bonus feature is

that the number of sampling points required along lines for accurate prediction may not

depend on dimensionality of the function. For example, if we were in 20-dimensional

space, prediction with a standard LHS may require hundreds of points, while it is

possible that three lines with 20 points may still suffice for prediction with the method of

converging lines. The effect of dimensionality will be discussed in section 2.5.3.

Several conditions may lead to large prediction errors even under valid

assumption that the inaccessible domain had a similar trend as that of the accessible

domain:

37

• Long-distance prediction usually risks large errors. Lacking samples near inaccessible domain would make it challenging to determine surrogate parameters and interpret goodness of fit.

• Different surrogate models or parameter settings is a major source of uncertainty. It is more challenging to select appropriate models for predictions towards inaccessible points especially when no prior information for noise is available.

• Some lines may not be trusted due to physical complexity or inappropriate modeling. Outlier prediction could be identified and excluded when most lines have similar predictions, but one line is significantly different. Outlier prediction comes from surrogate model with misleading estimation and prediction variance.

Note, however, that these difficulties are likely to be even more acuter if the

same prediction is attempted with a multi-dimensional surrogate, especially that only a

single estimate is obtained.

2.5 Numerical Properties of Method of Converging Lines

Even though it is tough to determine what function could be approximated and

how to predict towards inaccessible points, we tried to identify and discuss several

important factors based on numerical test functions. We examined the effect of lines

regarding to prediction accuracy, compared prediction from multiple lines with that from

multi-dimensional approximation, and effect of high-dimensionality.

Extrapolation has been reported to be more challenging than interpolation in

general [24]. Therefore, the following analysis distinguishes between these two

scenarios. We adopt Ordinary Kriging for prediction considering it could avoid huge

error regarding to prediction towards inaccessible points [24].

2.5.1 Algebraic Illustration Functions

This section examines properties of method of converging lines based on two

algebraic functions, Dette function and Styblinski-Tang function, from virtual library of

simulation experiments [53]. Dette function given in Eq. (2-7) is an 8D algebraic

38

function proposed by Dette et al. [54]. This function is highly curved in some variables

and less in others. The function is evaluated on the hypercube [0,1]ix for all i = 1, …,

8.

8

2 2 2 21 2 2 2 3 3

4 3

( ) 4( 2 8 8 ) (3 4 ) 16 1(2 1) ln 1i

ji j

f x x x x x x x x

(2-7)

For extrapolation, the target point for prediction was selected to be the vertex

where all eight variables were at their upper value, ix =1, i=1,2,…,8 as seen in Table 2-

1. The inaccessible domain was set to have the same length the as accessible domain

along all the variable directions. For interpolation, the target point was set to be =0.5ix

for i = 1, …, 8, which is the center of variable space. The inaccessible domain was set

to be the hypercube centered at the target point with a width of 0.5. Therefore, the

inaccessible domain had the same length as the accessible domain.

Table 2-1. Target point, variable range and inaccessible domain of 8D algebraic function

Target point targetx Variable range Inaccessible domain

Extrapolation ix =1 [0,1]ix [0.5,1]ix

Interpolation =0.5ix [0,1]ix [0.25,0.75]ix

The second test function is the Styblinski-Tang function, which is a popular

multimodal function for testing optimization algorithms with user-defined dimensionality

as seen in Eq. (2-8). The dimensionality d is set to be 100 to study the effect of high-

dimensionality on prediction. The function is evaluated on the hypercube [-1,3]ix for

all i = 1, …, d. Approximating this test function directly is challenging for classical

surrogate models considering its dimensionality.

39

4 2

1

116 5

2( ) i i i

d

i

x x xf x

(2-8)

For extrapolation, the target point for prediction was selected again to be the

vertex, where all the variables were at their upper bounds, =3ix as seen in Table 2-2.

The inaccessible domain was set to be a hypercube cornered at the target point and

had same length as the accessible domain along all the variable directions. For

interpolation, the target point was set to be =1ix , which was the center of variable

space. The inaccessible domain was set to be the hypercube centered at the target

point with a width of 1. Therefore, the inaccessible domain had the same length as the

accessible domain.

Table 2-2. Target point, variable range and inaccessible domain of 100D Styblinski-Tang function


Extrapolation ix =3 [-1,3]ix [1,3]ix

Interpolation =1ix [-1,3]ix [0,2]ix

2.5.2 Method of Converging Lines versus Multi-dimensional Approximation

We first compared the proposed approach with multi-dimensional approximation

based on Dette function for extrapolation and interpolation respectively.

2.5.2.1 Extrapolation

To start with, the target point for prediction was selected to be the vertex where

all eight variables are at their upper value, ix =1, i=1,2,…,8. For the method of

converging lines, 3 lines were selected towards the target point. The other ends of the

lines were randomly selected from the rest of the 28 vertices of the domain. We selected

100 sets of lines randomly to compensate for the effect of line selection. Inaccessible

40

domain was set to have the same length as accessible domain along all the variable

direction. Six samples were evenly spaced in the accessible domain. A typical set of

extrapolation results based on Ordinary Kriging was shown in Figure 2-3. The function

along line 1 in Figure 2-3 (A) is unimodal like a bowl. Figure 2-3 (B) showed line 2 which

is wavy. Line 3 in Figure 2-3 (C) was close-to-linear and most friendly for approximation.

In the case in Figure 2-3, combined prediction was close to line 3, which had a much

smaller prediction variance. Note that even without knowledge of the true function, the

ability to visualize the data and predictions shown in Figure 2-3 is useful for spotting

which surrogate is likely to be trusted.

Figure 2-3. A typical set of extrapolation results from 8D Dette function. The three lines

are from randomly selected vertices to the target vertex. A) Line 1, B) Line 2 and C) Line 3.

A B

C

41

For multi-dimensional approximation, 18 samples were selected using LHS with

5000 iterations with the Matlab function lhsdesign. Sampling domain was restricted to

be the same as that to generate multiple lines. Inaccessible domain was a hypercube

cornered at target point.

Figure 2-4. Summary of extrapolation results from multiple lines and 8D Kriging based on Dette function

100 sets of predictions were generated using multiple lines and multiple LHS

designs. Prediction accuracy was quantified using absolute percent error and is

summarized using boxplot in Figure 2-4Error! Reference source not found..

Prediction from multiple lines had the error median to be 0.018% and 75% predictions

were less than 0.1%. In contrast, the minimum error of multi-dimensional approximation

was 25%.

2.5.2.2 Interpolation

For interpolation, the target point was set to be =0.5ix for i = 1, …, 8, which is the

center of variable space. Inaccessible domain was set to be the hypercube centered at

the target point with a width to be 0.5. Therefore, the inaccessible domain had the same

42

length as accessible domain. For the method of converging lines, 3 lines were selected

to intersect at the target point. The ends of lines were randomly selected from all

vertices. A typical set of interpolation results using Kriging is shown in Figure 2-5. All the

lines had a bowl shape with different minima. All Kriging models had a similar level of

prediction variance and overestimated function value at the target point slightly. Again,

we note that even without knowing the exact function, figures such as Figure 2-5

provide some protection against the failure of the fitting process.

Figure 2-5. A typical set of interpolation results from 8D Dette function. A) Line 1, B)

Line 2 and C) Line 3.

For multi-dimensional approximation, 18 samples were generated using LHS in

the accessible domain. 100 sets of prediction were generated to estimate randomness

A B

C

43

and summarized in Figure 2-6. Prediction using multiple lines had the error median to

be 3.24% and maximum error to be 3.31%. In contrast, prediction using 8D Kriging had

the error median to be 22.8% and minimum error to be 19.7%. Multiple lines generated

more accurate predictions compared to 8D kriging for this test function.

Figure 2-6. Summary of interpolation results from multiple lines and 8D Kriging based

on Dette function

2.5.3 Applicability to Very High-dimensionality

We adopted Styblinski-Tang function with 100 dimensions to illustrate

applicability to high-dimensionality. The dimension is too high to be approximated by

regular 100-dimensional surrogate models with a reasonable number of samples.

Therefore, the method of converging lines might be the only option for a function with

such a high dimension.

For extrapolation, the target point for prediction was selected again to be the

vertex, where all the variables are at their upper bounds, =3ix . The inaccessible

domain was set to be a hypercube cornered at the target point and had same length as

the accessible domain along all the variable directions. Three lines were randomly

selected from the rest of the 2100 vertices. Six sampling points were evenly spaced in

44

the accessible domain along each line. A typical set of extrapolation results is given in

Figure 2-7. The test function along all three lines was multimodal. Prediction error was

small at domain close to samples and increased with prediction distance in the

extrapolation domain. This example illustrates the difficulty in extrapolating long

distance. Not only the prediction can deteriorate with distance, but for a function that

changes its behavior, even the uncertainty estimate may be flawed.

Figure 2-7. A typical set of extrapolation results from 100D Styblinski-Tang function. A)

Line 1, B) Line 2 and C) Line 3.

100 sets of random lines were generated towards same target point to

compensate for variability. The error of 100 predictions were summarized in Figure 2-8.

The prediction error ranged around 28%. In this case, 100-dimensional surrogates are

A B

C

45

not possible, the performance of the proposed method of converging lines was not

compared with multi-dimensional surrogates, but the prediction error would be huge if

18 samples were able to fit a 100-dimensional surrogate.

Figure 2-8. 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function

For interpolation, the target point is set to be =1ix , which is the center of variable

space. The inaccessible domain is set to be the hypercube centered at the target point

with a width of 1. Therefore, the inaccessible domain has the same length as the

accessible domain. For the method of converging lines, 3 lines are selected toward the

target point. The other end of lines is randomly selected from all vertices. A typical set of

interpolation results is shown in Figure 2-9. All the lines have a bowl shape, and the

Kriging models overestimated function value at the target point. 100 sets of prediction

are generated to compensate for randomness and summarized in Figure 2-10.

Prediction using multiple lines had the error between 7.4% and 9.3%. Based on the

tests using 100D Styblinski-Tang function, the method of lines enables estimation of

high-dimensional function at one point using multiple lines. Such estimation is practically

impossible with a small number of samples using a multi-dimensional surrogate. In

46

particular, it is impossible to fit a Kriging surrogate with 18 points in 100-dimensional

space.

Figure 2-9. A typical set of interpolation results from 100D Styblinski-Tang function. A)

Line 1, B) Line 2 and C) Line 3.

Figure 2-10. 100 sets of extrapolation results from multiple lines based on 100D Styblinski-Tang function

A B

C

47

2.6 Long-distance Extrapolation and Interpolation of a 2D Drag Coefficient

A drag coefficient model of a spherical particle [55] is presented in this section to

demonstrate long-distance function estimation at an inaccessible point using the

method of converging lines.

2.6.1 Introduction to the Drag Coefficient Function

The drag coefficient is a dimensionless quantity that measures the quasi-steady

drag coefficient of a spherical particle in compressible flow. For a given particle, the

drag coefficient, DC , is a function of Mach number, M, and Reynolds number, Re. The

range of applicability for DC in this work is limited to the supersonic (1<M 1.75 ) domain

to avoid the physical discontinuity at M=1. The Reynolds number is limited to

5Re 2 10 to avoid turbulence of the attached boundary layer. It is considered that the

governing physics is consistent within these limits. The analytical expression for DC is

given in DC = (Re, )f M in [55]. The dependence of the drag coefficient on Reynolds

number and Mach number is shown in Figure 2-11, which shows that the behavior can

be made smoother by transforming Re to logarithmic coordinates. The target point,

variable range for analysis and inaccessible domain of drag coefficient function are

summarized in Table 2-3 and plotted in Figure 2-12. For extrapolation, prediction

domain is a square located on the upper right corner as seen in Figure 2-12 (A). For

interpolation, prediction domain is set to be a square in the center of variable range as

seen in Figure 2-12 (B). Length of sampling domain and prediction domain are the

same along each line for both cases.

48

Figure 2-11. Response of drag coefficient on Re and M. A) Natural coordinate and B) log10 coordinate

Table 2-3. Target point, variable range and inaccessible domain of drag coefficient function


Extrapolation log10(Re)=5.301

M=1.75

log10(Re) [2 5.301] ，

M [1 1.75] ，

log10(Re) [3.65,5.301]

M [1.375,1.75]

Interpolation log10(Re) 3.6505

M=1.375

log10(Re) [2 5.301] ，

M [1 1.75] ，

log10(Re) [2.8253,4.4757]

M [1.1875,1.5625]

Figure 2-12. Lines and samples selection for extrapolating and interpolation of drag coefficient function. Red points denote samples. A) Extrapolation and B) Interpolation.

A B

A B

𝛼3 𝛼2

𝛼1 = 0

49

2.6.2 Extrapolation Results

2.6.2.1 Extrapolation using the method of converging lines

Instead of sampling entire range of sampling domain, six uniform samples were

selected from half of the sampling domain as shown in Figure 2-12 (A). This was

because the prediction variance decreased noticeably when samples were closer to the

target point. When the inaccessible domain is in the interpolation region, the sampling

scheme in Figure 2-12 (B) can be used, which will be considered in the following

section. Table 2-4 summarizes statistics of the surrogate predictions. Predictions of

function value along variable M with constant Re (line 3) had the smallest prediction

variance as shown in Table 2-4 and Figure 2-13 (C), indicating better accuracy along

this line.

Table 2-4. Mean value of prediction and its uncertainty given by standard deviation for extrapolation of drag coefficient function along three lines

Line 1 Line 2 Line 3

i 1.021 0.9352 0.9755

i 0.0786 0.0725 0.0003

By substituting extrapolation results of each line from Table 2-4 into Eq. (2-7),

most probable estimation is 2

target 1 target 2 target 3 target| , , N(0.9755,0.0003 )p y y x y x y x .

Note that the posterior distribution was dominated by the prediction along line 3

because the prediction variance (2) was much smaller than for the other lines. The

small prediction variance denotes high reliability of extrapolation. Comparing with true

function value 0.9766, absolute error is 0.001 and relative error is 0.1%. Note also that

50

the less accurate lines (Line 1 and Line 2) provide a reasonable warning of their lack of

accuracy in their prediction uncertainties, i .

Figure 2-13. Extrapolation results of drag coefficient function using samples from reduced sampling domain. A) Line 1, B) Line 2 and A) Line 3.

2.6.2.2 Comparison of 1D and 2D surrogates

One interesting question is how sensitive the prediction is to the location of lines.

The sensitivity to the selection of lines was tested by varying the angle of lines. In

Figure 2-12, the angle between lines and the horizontal axis with constant M = 1.75

were set to be 1 2 3 ，， , respectively, where 1 2 3= 90 0， =45， . In order to test the

effect of lines, 1 varied from 0 to 5 , 2 from 40 to 50 , and 3 from 85 to 90 in

the increment of 1 . 396 sets of three lines were generated based on different

A B

C

51

combinations of 1 , 2 and 3 . Extrapolation results based on each set of lines were

plotted using box-plot in Figure 2-14 (B). All the extrapolation results concentrated near

the true function value, 0.9766, varying from 0.9751 to 0.9756.

Figure 2-14. Extrapolation results of drag coefficient function using multi-dimensional Kriging and converging lines. A) A typical set of samples from LHS and B) accuracy for 100 sets of extrapolations.

In order to examine the performance of multi-dimensional extrapolation, 2-D

Kriging is adopted. 18 samples are selected in the reduced sampling domain using LHS

(5000 iterations maximizing minimum distance) as shown in Figure 2-14 (A). As

converging lines benefitted from the reduced domain, 2-D Kriging also benefited from it

based on our tests. 2-D extrapolation is repeated 20 times to compensate for

randomness of sampling pattern. For a typical 2D extrapolation out of the 20 cases, i.e.

corresponding to the case with median absolute error, the prediction was

2N(0.9869,0.0313 ) . In Figure 2-14 (B) we can see that boxplot of 2-D Kriging prediction

has a median equal to 0.9799 and standard deviation to be 0.0261. 2D Kriging

generated larger variation and bias than converging lines even with 20 sets of data. The

method of converging lines proved to be more reliable and robust for one-point

extrapolation of drag coefficient function.

A B

2 2.5 3 3.5 4 4.5 5 5.5

1

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.82D Samples(18)

log(Re)

M

Border

Reduced sampling domain

Target Point

52

2.6.3 Interpolation Results

2.6.3.1 Interpolation using the method of converging lines

Interpolation using method of converging lines was performed similarly as shown

in 2-11 (b). We selected samples in the half of sampling domain to reduce prediction

variance. Interpolation results are plotted in Figure 2-15 from three lines. Predictions at

target point are provided in Table 2-5.

Table 2-5. Interpolation results along three lines for drag coefficient function

Line 1 Line 2 Line 3

i 0.9408 0.9943 0.9223

i 0.0146 0.0742 5×10-6

Figure 2-15. Interpolation results of drag coefficient function. A) Line 1, B) Line 2 and C)

Line 3.

A B

C

53

As in the case of extrapolation, the prediction variance on Line 3 is by far the

smallest. By substituting prediction results of each line from Table 2-5 into Eq. (2-7),

most probable estimation is 2

target 1 target 2 target 3 target| , , N(0.9223,0.000005 )p y y x y x y x ,

essentially identical to the result of Line 3. Comparing with true function value, 0.9224,

absolute error is 0.0001 and relative error is 0.01%.

2.6.3.2 Comparison of 1D and 2D surrogates

We have more freedom to select lines for interpolation while fixing intersection

angle between lines. Therefore, the intersection angle between line 1, line 2 and line 3

were fixed to be o45 successively. Three lines rotated counterclockwise gradually from

o0 to o45 in 20 steps. 20 sets of 1D interpolation results were then obtained accordingly.

For 2D Kriging, 18 samples are selected in the sample domain using LHS (5000

iterations maximizing minimum distance for the following analysis as shown in Figure 2-

16 (A). For a typical 2D interpolation out of the 20 cases, i.e. corresponding to the case

with median absolute error, the prediction was 2(0.9317,0.0012 )N . It is not only less

accurate than the method of converging lines, but also has a substantial underestimate

of the uncertainty in the prediction in the prediction variance.

Predictions of 1D and 2D Kriging were summarized in Figure 2-16 (B). 1D Kriging

predictions varied from 0.9223 to 0.9232. 2D Kriging prediction generated a median to

be 0.9255, and standard deviation equal to 0.0195. The box plot showed that for some

DOEs the 2D prediction was grossly inaccurate.

54

Figure 2-16. Interpolation results using multi-dimensional Kriging and converging lines.

A) 2D samples (18) and B) Comparison of 2D and 1D interpolation.

2.7 Concluding Remarks

We presented a method for improving the accuracy of function estimation at an

inaccessible point by taking advantage of the fact that prediction is needed at only one

point instead of everywhere. The proposed method of converging lines transforms a

multi-dimensional prediction into a series of 1D predictions. Most probable estimation is

obtained by combining consistent predictions from different lines using Bayesian

inference while assuming all the lines are independent.

One-dimensional prediction was performed using Kriging surrogate model.

Function prediction is only feasible under the assumption that the function behaves

similarly in the sampling domain and prediction domain. We illustrated its applicability to

high dimensional functions with two algebraic examples, 8D Dette function and 100D

Styblinski-Tang function. The proposed approach proved to be more accurate and

reliable than multi-dimensional approximation for the 8D Dette function. The proposed

approach enabled moderately accurate estimation of 100D Styblinski-Tang function with

18 samples. The two examples also illustrated another advantage of the method, which

A B

55

is easy visualization of the fitted surrogate. Finally, the examples illustrated that

combining multiple predictions using Bayesian inference could increase the chance of

obtaining valid estimation.

Next the method was illustrated for a physical example, extrapolation and

interpolation of a two-dimensional drag coefficient. For this example, as well, the

method of converging lines proved to be much more reliable and accurate than 2D

Kriging.

Converging lines were randomly selected in variable space from current study.

Predicative capability for method of converging lines is expected to be improved while

introducing refined strategies to determine location of lines and samples along a line.

Another interesting topic in the future is consistency between multiple predictions along

lines. The extent of consistency between lines may reveal the major source of

uncertainty such as model-form uncertainty, outlier effect or unexpected behavior of

target function at inaccessible domain.

56

CHAPTER 3 MULTI-FIDELITY SURROGATE BASED ON SINGLE LINEAR REGRESSION

3.1 Background

Surrogate models have been applied for various engineering design optimization

and uncertainty quantification problems, which require many physical tests or

simulations [30, 56]. Since physical tests and high-fidelity simulations are usually time-

consuming or expensive, surrogate models can be developed based on scattered tests

to infer the system response [11, 57]. However, performing a number of tests or

simulations needed for fitting an accurate surrogate is often too expensive. Multi-fidelity

surrogates (MFS) can provide a solution for this problem by combining a small number

of accurate simulations or tests with lower accurate models. Multi-fidelity models are

commonly encountered for engineering analysis, such as finite element simulations with

different resolutions and numerical simulations combined with physical tests. Various

frameworks have been proposed to predict mechanical system responses by combining

data from different fidelities for design optimization and uncertainty quantification, e.g.,

see reviews by Fernández-Godino et al.[58] and Peherstorfer et al. [59].

MFS usually introduces a discrepancy surrogate to model the difference between

low- and high-fidelity samples [60]. The popular Bayesian discrepancy-based MFS was

introduced in [61-64] and demonstrated its effectiveness for the approximation of multi-

fidelity datasets. Part of its success compared to earlier MFS was because of scaling

the low-fidelity model in addition to introducing the discrepancy [65]. Model calibration

associated with surrogate is another approach to combine multi-fidelity dataset. The

physical parameters required by the low-fidelity model are first optimized and fitted by a

surrogate to improve agreement between low-fidelity predictions and high-fidelity

57

samples [66-68]. A comprehensive Bayesian MFS model that uses both calibration and

discrepancy was proposed in [66], offering greater flexibility, although this is the most

complex framework. However, the Bayesian framework requires a model for the

uncertainty structure, and has been applied almost exclusively for Gaussian process

(GP). As demonstrated by Viana et al. [3], there are situations where GP is not the most

accurate surrogate, so there is a merit to having an MFS arsenal that includes other

surrogates.

We propose a simple and yet powerful MFS based on regression, or least-

squares fit, that is LR-MFS. The high-fidelity behavior is approximated by a linear

combination of low-fidelity predictions and a discrepancy function. In this work, the

discrepancy function is represented by a linear combination of monomial basis

functions, but the approach is also applicable to other basis functions such as radial

basis functions. The key idea is to consider the low-fidelity model as one additional

basis function in the multi-fidelity model with the scale factor as a regression coefficient.

The design matrix for least-squares estimation consists of both the low-fidelity model

and discrepancy basis functions. Then the scale factor and coefficients of the basis

functions are obtained simultaneously by solving the regression set of linear equations.

LR-MFS is expected to easily use various tools available for regression fits, such as

prediction variance and D-optimal designs. An effective LR-MFS could serve as the

basis for uncertainty propagation [69, 70] and design optimization[71].

The work is organized as follows. Section 3.2 briefs the Bayesian MFS and

presents the proposed LR-MFS using single linear regression. Section 3.3 evaluates the

proposed LR-MFS using a benchmark dataset from a fluidized-bed process. The LR-

58

MFS is compared with three Bayesian frameworks. In section 3.4, the LR-MFS is further

evaluated using an algebraic test function with different levels of noise and different

sampling plans. This chapter on LR-MFS is based on publication [8] co-authored with

Chanyoung Park, Nam Ho Kim and Raphael T. Haftka.

3.2 Linear Regression Multi-Fidelity Surrogate

The basics of Bayesian MFS is discussed first to understand the major

components of an MFS and potential limitations. Then the Linear Regression Multi-

fidelity Surrogate (LR-MFS) is proposed which also includes a scale factor and a

discrepancy function as with the Bayesian MFS.

3.2.1 Bayesian Multi-fidelity Surrogate Frameworks

Kriging[72] or Gaussian process is one of the most popular surrogate models for

design optimization. The Kriging is a stochastic process (a collection of random

variables), such that every finite collection of those random variables has a multivariate

normal distribution, i.e. every finite linear combination of the variables is normally

distributed. Kriging naturally provides the prediction variance at an untested point. Co-

Kriging developed by [Kennedy and O’Hagan, 2000] [61] introduces the correlation

between multi-fidelity dataset. Let us denote a set of sample 1, , Ln

L L LX x x is from

the low-fidelity model Lf x and 1, , Hn

H H HX x x from the high-fidelity model Hf x .

The corresponding function values are 1, , Ln

L L Ly yy and 1, , hn

H H Hy yy ,

respectively. Co-Kriging is made of two sets of correlated Kriging models. The Kriging

surrogate ˆ ( )Lf x is first constructed based on ,L LX y . The second Kriging ˆ ( )Hf x is then

built based on the discrepancies ˆH L Hfy X as the discrepancy function. The scale

59

factor is estimated from maximum likelihood estimation as part of ˆ ( )Hf x . Variations

of the Bayesian multi-fidelity surrogate are developed to improve accuracy and

computational efficiency [62, 73].

Since Kennedy and O’Hagan (2000) introduced the Bayesian multi-fidelity

surrogate, subsequent studies supported that the GP based surrogate generally

provided accurate predictions [74]. However, the use of GP model also embraces

technical difficulties regarding hyper parameter estimation. Finding the hyper

parameters of the GP model is equivalent to finding a global optimum solution for a

highly nonlinear likelihood function [75, 76]. In addition to the computational burden of

likelihood function evaluations, likelihood functions are often plagued with numerical

instability due to covariance matrix inverse operation. Forrester et al. (2007) introduced

co-Kriging that is featured with better computational efficiency. Qian and Wu (2008)

proposed the use of Marko chain Monte Carlo (MCMC) and sample average

approximation algorithm for hyper parameter estimation [62]. Le Gratiet (2013)

proposed an approach to reduce the computational burden by simplifying the

covariance matrix inversion operation [73]. For data with noise, GP based surrogates

tend to underestimate the noise by overly smoothing the prediction [77].

3.2.2 Proposed Linear Regression Multi-fidelity Surrogate

In this work, we propose the least-squares estimation of the scale factor ρ and

the coefficients of the discrepancy function. The linear regression multi-fidelity (LR-

MFS) is built with two surrogates, ˆ ( )Lf x and ˆ( ) x , which are the linear regression fitted

to the low-fidelity data and the discrepancy data as in Eq. (3-1)

60

ˆ ˆ ˆ( ) ( ) ( )H L

f f x x x (3-1)

where the scale factor ρ and discrepancy function ˆ( ) x are obtained together by

minimizing the difference between the scaled low-fidelity samples and the high-fidelity

samples as in Eq. (3-2), where H

d is difference between high-fidelity samples and

scaled low-fidelity samples as defined in the Eq.(3-3). ( HX , Hy ) denotes the high-

fidelity dataset including n samples.

ˆ, ( )

ˆ ˆmin : ( ) ( )T

H H H H

X d X dx

(3-2)

ˆ ( )H L H H

f d yX (3-3)

The MFS in Eq. (3-1) can be expanded with the discrepancy function being

represented by p basis terms (e.g, monomial bases), as

1

ˆ ˆ ˆ( ) ( ) ( )

ˆ= ( )+ X

H L

p

L i i

i

f f

f b

x x x

x x (3-4)

where Xi x denotes the thi basis term, and

ib is the coefficient of X

ix . In the

proposed LR-MFS, the traditional design matrix is augmented by including the low-

fidelity model with unknown scale parameter ρ. In least-squares estimation, the

relationship between high-fidelity samples and model predictions can be given as

Y XB e (3-5)

for LR-MFS, where

61

(1) (1) (1)(1) (1)1

1

1 ( 1) 1 1

( ) ( )( ) ( ) ( )

1

ˆ ( ) X X

, , ,

ˆ ( ) X X

L H H p HH H

n n p p n

n nn n nH HL H H p H

p

fb

fb

x x xy e

Y X B e

y ex x x

(3-6)

In the above equation, Y is the vector of high-fidelity samples, X is the

augmented design matrix, B is the parameter vector and e is the vector of residual

errors. By augmenting the design matrix, the scale parameter and unknown coefficients

of the discrepancy functions are estimated simultaneously. This means that the low-

fidelity model is considered as an additional basis function. An additional advantage of

the proposed LR-MFS is that it is unnecessary to build a surrogate of the low-fidelity

model when low-fidelity simulations are cheap. The method only requires evaluating the

low-fidelity model at the same locations with the high-fidelity samples. The unknown

parameters in LR-MFS are obtained by using the standard regression technique as

1

T T

B X X X Y (3-7)

Linear regression assumes that the residual errors are normal, independent and

identically distributed. The prediction error is estimated from Eq.(3-8)

2ˆ1

T

n p

e e (3-8)

the prediction variance at a point x is obtained from Eq.(3-9)

1

2ˆ ˆ[ ( )]T T

HVar f

x X x X X X x (3-9)

where the X x denotes the design matrix of linear regression at the point x for

prediction.

62

The scale factor implies the level of trend similarity between ˆ ( )H

f x and ˆ ( )L

f x ,

and plays a critical role to approximate multi-fidelity data. Negative values or extremely

large values of indicates a risky prediction, which are likely to be associated with

undesirable low-fidelity models, inappropriate surrogate forms, or inadequate samples.

The discrepancy function ˆ( ) x is likely to be modeled well by a low-order PRS when

Lf x had a similar trend as Hf x .

There are nice properties of LR-MFS. Firstly, the LR-MFS could be easily applied

to more than two-fidelity models by augmenting the design matrix with multiple low-

fidelity models with multiple scale factors as shown in Eq.(3-10), where k is the number

of low-fidelity models. Secondly, it is possible to use other basis functions beyond

monomials to approximate the discrepancy function as in this work. Thirdly, the

established approaches could be directly used to LR-MFS such as (1) handling noise

from different distributions, (2) confidence intervals, prediction intervals and tolerance

limits for conservative estimation, (3) stepwise regression to reduce model-form

uncertainty, (4) Obtaining optimal design of experiments, such as D-optimal designs.

1 1

ˆ ˆ( ) ( )+ Xpk

H j Lj i i

j i

f f b

x x x (3-10)

3.3 Approximation of the Fluidized-Bed Process using LR-MFS

In this section, the proposed LR-MFS was compared with three Bayesian multi-

fidelity surrogates using the dataset from Fluidized-Bed Process. The effect of different

low-fidelity models was also discussed.

63

3.3.1 A Benchmark Dataset: Fluidized-Bed Process

A comparison is made between the proposed LR-MFS and three Bayesian multi-

fidelity surrogates developed by [Kennedy and O’Hagan, 2000][61], [Qian and Wu,

2008][62] and [Le Gratiet, 2013][73]. This case was selected because the results for the

Bayesian implementation are given in [Le Gratiet, 2013][73]. Using these results from

the literature removes possible implementation bias in comparing the proposed

algorithm with existing algorithms. The multi-fidelity surrogates are used to approximate

the simulation a fluidized-bed process [78]. The quantity of interest for approximation is

the temperature of the steady-state thermodynamic operation point for a fluidized-bed

process. There are six variables affecting the quantity of interest: humidity ( rH ), room

temperature ( rT ), temperature of the air from the pump ( aT ), flow rate of the coating

solution ( fR ), pressure of atomized air ( aP ), and fluid velocity of the fluidization air ( fV ).

28 different process conditions were considered with coating solution used for distilled

water at room temperature. Four-fidelity outputs are available: expT , 3T , 2T , 1T . expT is

the experimental response denoted as high-fidelity data, 3T is the most accurate

simulation considering the adjustments for heat losses and inlet airflow. 2T provides

medium accuracy considering adjustment for heat losses. 1T has lowest accuracy

without adjustment for either heat losses or inlet airflow.

3.3.2 Comparison with the Bayesian MFS using Two-fidelity Dataset

We focus on the prediction of expT (high-fidelity) assisted by 2T (low-fidelity) using

two-level multi-fidelity surrogates as from in [Le Gratiet, 2013][73]. The dataset is

collected at 28 points with 6 design variables for both expT and 2T as visualized in

64

Figure 3-1. 2T is highly correlated withexpT having the correlation coefficient 0.99 and

2T overpredicts the value of expT .

Figure 3-1. Function values of the expT (high-fidelity) and 2T (low-fidelity) from Fluidized-

Bed Process

The evaluation is performed according to the plan proposed by [Le Gratiet,

2013][73, 79] as shown in Table 3-1. 20 samples were randomly selected from 2T as LF

samples as shown in Figure 3-2. Then 10 HF samples were randomly selected such

that the HF samples were nested into the LF samples. The left-out 18 points from expT

were used as test set ,T Tx y . Root-mean-square-error (RMSE) was adopted to

evaluate prediction accuracy as in Eq. (3-11). 100 different combinations of training

samples and test sets were produced for comparison. The model from [Kennedy and

O’Hagan, 2000] (KO) [61]was implemented using R CRAN package “approximator”, the

model from [Qian and Wu, 2008] (QW) [62] was implemented using the WinBugs

software, the model from [Le Gratiet, 2013] (LG)[73] was implemented using the R

CRAN package “MuFiCokriging”. Note that KO and LG are based on the same GP

models and their theoretical characteristics are identical. However, they may give

65

different results for the same problem because of the way of implementations and the

performance of their optimizers to estimate hyper parameters.

Table 3-1. Evaluation plan for the approximation of multi-fidelity dataset from Fluidized-Bed Process following [Le Gratiet, 2013]

Steps Procedures

1 Randomly selected 20/28 LF samples and then select 10/20 HF samples

2 The other 18 HF samples were used as test set

3 Generated 100 different combinations of design and test sets

4 Used root-mean-square-error (RMSE) as prediction metrics

Figure 3-2. Illustration of the resampling procedure to generate training and test data

18 2

( ) ( )

1

1 ˆ ( )18

j j

H T T

j

RMSE f

x y (3-11)

The median RMSE for the three Bayesian approaches are taken from [Le Gratiet,

2013] [73, 79] as shown in Figure 3-3. For the construction of LR-MFS, a linear

polynomial response surface (PRS) was fitted to the LF samples and the LR-MFS was

developed with a constant discrepancy. A linear PRS was developed based on only HF

samples to study the effect of introducing LF models. The LR-MFS was more accurate

than the PRS fit by introducing the LF samples. From Figure 3-3, it is clearly shown that

LR-MFS and LG were most accurate and significantly better than the KO and QW

models. The scale factors 𝜌 of LR-MFS was shown in Figure 3-3 (B) and varied

100

sets ⋮

66

between 0.73 and 1.15 which implied the similar trend between HF samples and LF

samples. The median of 𝜌 was 0.9 which was consistent with the observation that LF

model overpredicted HF model.

Figure 3-3. Performance of the surrogate models from 100 combinations of training

samples and test sets for the approximation of fluidized bed process. A) Median RMSE from PRS based on only HF samples, LR-MFS (LR), the model from [Le Gratiet, 2013] (LG), the model from [Kennedy and O’Hagan, 2000] (KO), the model from [Qian and Wu, 2008] (QW). The results for LG, KO, QW were taken from [Le Gratiet, 2013] (LG). B) Scale factors ρ from 100 sets of predictions for LR-MFS.

3.3.3 Effect of Noise for High-fidelity Dataset

High-fidelity samples might be obtained from noisy physical tests or simulations

with noise from discretization errors. Therefore, LR-MFS was further evaluated for HF

samples with perturbed with synthetic noise from a normal distribution 20,2N . The

noise level is comparable with the difference between HF and LF samples. The

evaluation was repeated based on Table 3-1 for the perturbed HF samples. One

realization of the HF samples with noise is shown in Figure 3-4. RMSE of LR-MFS to

approximate HF samples with noise was shown in Figure 3-5. It is seen that LR-MFS

A B

67

was robust with respect to noise. The median RMSE only increased from 2.1 to 2.2

while the standard deviation of the noise was 2.

Figure 3-4. Function values of the expT (high-fidelity) with synthetic noise from normal

distribution

Figure 3-5. RMSE of LS-MFS to approximate HF samples of fluidized bed process with

noise. The boxplots were based on 100 combinations of training samples and test sets. A) RMSE of LR-MFS for the dataset with or without noise and B) estimated noise level (STD).

3.3.4 LR-MFS with Multiple Low-fidelity Datasets

We compared the effect of different low-fidelity dataset using LR-MFS. expT was

the high-fidelity dataset for prediction. 3T , 2T and 1T were each used as the low-fidelity

dataset to build a two-fidelity LR-MFS prediction and then all the 3T , 2T and 1T were

used to build a four-fidelity LR-MFS prediction. The evaluation was performed according

B

LR-MFs no noise LR-MFS with noise

A

68

to Table 3-1 using 100 sets of design of experiments. The RMSEs of the LR-MFS are

shown in Figure 3-6. All the two-level LR-MFS had similar accuracy around 2. The

accuracy of LR-MFS improved with higher fidelity of iT . One interesting observation is

the four-level LR-MFS using all the iT had the worst accuracy and largest variation. The

scale factors 𝜌 are shown in Figure 3-7 for further investigation of the results. For the

two-fidelity LR-MFS, 𝜌 was about 0.9 with small variability. While for the four-fidelity LR-

MFS, all the 𝜌 were associated with significant variability and sometime negative. The

undesirable behavior of 𝜌 was due to the increasing number of parameters for surrogate

training. Considering the limited number of high-fidelity samples, more parameters

would suffer higher risk of overfitting. Regarding the median value, 3T had the highest

fidelity and largest 𝜌 which implied significant contribution to the LR-MFS prediction. For

2T and 1T , the 𝜌 was close to 0 and LR-MFS assigned larger weighs to the dataset with

higher fidelity.

Figure 3-6. RMSE of LR-MFS to approximate HF samples from fluidized bed process

with different low-fidelity dataset and all the low-fidelity dataset. The boxplots were based on 100 combinations of training samples and test sets.

69

Figure 3-7. Scale factors 𝝆 of LR-MFS to approximate HF samples from fluidized bed process with different low-fidelity dataset and all the low-fidelity dataset.

3.4 Approximation of the Modified Currin Function using LR-MFS

In this section, the proposed LR-MFS was evaluated using the modified Currin

function which has a known expression. The effect of different level of noise and

sampling plans were discussed.

3.4.1 The Modified Currin Function

The modified Currin function [53, 80] with two variables was adopted for further

investigation of the proposed LR-MFS approach. The high-fidelity model Hf x is given

in Eq. (3-12). We modified the original low-fidelity model [81] with a larger scale factor

and added a quadratic function as shown in Eq. (3-13). These variations make the

multi-fidelity modeling more challenging. Major settings of the test function are

summarized in Table 3-2. The function values of ( )Hf x vary between 1.2 and 13.8 with

a range 12.6. The ( )Hf x and Lf x are highly correlated with the correlation coefficient

0.99. The responses of Hf x without noise and Lf x are shown in Figure 3-8.

3 2

1 1 1

3 2

2 1 1 1

2300 1900 2092 601( ) 1 exp

2 100 500 4 20H

x x xf

x x x x

x (3-12)

70

1 2 1 2

1 2 1 2

2

1 2

1( ) [ ( 0.05, 0.05) ( 0.05,max 0, 0.05 )]

8

1[ ( 0.05, 0.05) ( 0.05,max 0, 0.05 )]

8

1( 5 7 )

8

Lf f x x f x x

f x x f x x

x x

x

(3-13)

Table 3-2. Major settings for the modified Currin test function

Input variables Range of Hf x Range of Lf x Correlation coefficient

between Hf x and

Lf x

1 2, [0,1]x x [1.1804, 13.7692] [-0.1867, 4.9498] 0.99

Figure 3-8. Responses of the modified Currin function in 2D space

We have evaluated multiple aspects of the LR-MFS as summarized in Table 3-3.

A PRS was fitted to only high-fidelity samples as the baseline to examine the accuracy

of LR-MFS while introducing the low-fidelity samples. Then the LR-MFS was compared

with the Co-Kriging from [Kennedy and O’Hagan, 2000][61]. The Co-Kriging model was

implemented through the ooDACE Toolbox [82]. The effect of sample size was

discussed by comparing the surrogates from 3 and 10 HF samples. Synthetic noise was

added to HF samples with levels of noise 𝑁(0, 0.12), 𝑁(0, 0.22) and 𝑁(0, 0.32). The

71

prediction accuracy was evaluated from RMSE at 100 100 test grid from Hf x . 100

sets of training samples with size 3 and size 10 respectively using LHS. For the MFS

models, the LF samples were obtained directly from Lf x instead of a surrogate to

avoid approximation error in building the low-fidelity surrogate. LF samples were

generated at the same location of HF samples and test grid. It is also possible to build a

ˆ ( )Lf x based on only low-fidelity data for repeated calls of LR-MFS in practical

applications.

Table 3-3. Key factors of the evaluation plan to approximate the modified Currin function

Key factors Plan

Effect of low-fidelity

samples

Compare LS-MFS with PRS fitted to only high-fidelity samples

Comparison with

Bayesian MFS

Compare LS-MFS with Co-Kriging from [Kennedy and O’Hagan,

2000]

Effect of noise for

HF

Add synthetic noise to HF samples from

𝑁(0, 0.12), 𝑁(0, 0.22), 𝑁(0, 0.32)

Effect of sample

size

Investigate surrogates for 3 HF samples and 10 HF samples

Prediction metric Root-mean-square error

Fitting error to only

LF samples

Low-fidelity sample was from 𝑓𝐿(𝐱), instead of a surrogate

Sampling plan Latin Hypercube Sampling, repeated 100 times with different

samples

3.4.2 The Effect of Introducing Low-fidelity Dataset

We first investigated the effect of introducing low-fidelity samples without noise.

For the approximation of 3 HF samples, a constant PRS and LR-MFS with constant

discrepancy were fitted. For the approximation of 10 HF samples, a quadratic PRS and

72

LR-MFS with quadratic discrepancy were fitted. Accuracy of the PRS and LR-MFS is

shown in Figure 3-9. 3 and 10 are the number of HF samples. The median RMSE is

based on 100 sets of samples. LR-MFS was more accurate than PRS by the order of

magnitude, both LR-MFS and PRS improved noticeably with more samples. The

median scale factors of LR-MFS increased from 1.74 to 1.95 with more samples which

indicated an enhanced contribution of low-fidelity dataset.

Figure 3-9. Effect of introducing low-fidelity samples for prediction of the modified Currin function without noise. PRS indicates the fit based on only HF samples. 3 and 10 are the number of HF samples. The median RMSE is based on 100 sets of samples. A) RMSE of surrogates and B) Scale factors of LR-MFS.

3.4.3 Comparison with Co-Kriging

Then the LR-MFS was compared with Co-Kriging without noise. It is seen in

Figure 3-10, LR-MFS was slightly better than Co-Kriging for the median RMSE and has

smaller variation. Co-Kriging had larger scale factors for the approximation of 3 HF

samples and similar scale factors for the approximation of 10 HF samples with LR-MFS.

The relative performance between LR-MFS and Co-Kriging was summarized in Table 3-

4. Among the 100 sets of HF samples, LR-MFS bested Co-Kriging for 55 cases and 63

cases while having 3 HF samples and 10 HF samples respectively.

A B

73

Figure 3-10. Comparison of Co-Kriging (CK) with LR-MFS for the approximation of the

modified Currin function. 3 and 10 are the number of HF samples. The box plot is based on 100 sets of samples. A) Median RMSE of surrogates and B) Median scale factors of MFS.

Table 3-4. Relative performance of LR-MFS and Co-Kriging for the approximation of the modified Currin function using 100 sets of HF samples

Number of HF samples 3 HF samples 10 HF samples

Cases bested by LR-MFS 55 63

Cases bested by Co-

Kriging

45 37

3.4.4 Fitting High-fidelity Dataset with Different Levels of Noise using LR-MFS

HF samples might be associated with significant experimental variation. The

synthetic noise was added to the HF samples to imitate experimental variation from

normal distributions 𝑁(0, 0.12), 𝑁(0, 0.22), 𝑁(0, 0.32). The accuracy of LR-MFS for HF

samples with noise was shown in Figure 3-11. The LR-MFS provided accurate

predictions for all the cases. The LR-MFS was expected to be robust with noise as

inherited from polynomial response surface. The LR-MFS3 didn’t change much with

increasing noise due to the large model-form uncertainty. In Figure 3-11 (B), we can see

that the low-fidelity samples contributed more on LR-MFS10 than that on LR-MFS3. The

estimated noise was computed according to Eq. (3-8) and summarized in Figure 3-12.

A B

74

For LR-MFS3, the estimated noise level was significantly contaminated by model-form

uncertainty. For LR-MFS10, the estimated noise level was close to the true value. The

sources of epistemic uncertainty for LR-MFS prediction were conceptually illustrated in

Figure 3-13. The contribution of uncertainties varied with conditions (e.g. number of

samples). The hazard of model form and noise effect decreased while introducing more

samples. The effect of model form was minuscule for LR-MFS10 and therefore the

estimated noise level was close to the true noise.

Figure 3-11. Comparison of LR-MFS with different level of noise for the modified Currin

function. 3 and 10 are the number of HF samples. The RMSE is based on 100 sets of samples. A) RMSE of the LR-MFS and B) ρ of the LR-MFS

A

B

75

Figure 3-12. Estimated noise level of LR-MFS with different number of HF samples for

the modified Currin function. 3 and 10 are the number of HF samples. The median STD is based on 100 sets of samples.

Figure 3-13. Conceptual illustration of the major epistemic uncertainty for LR-MFS with

different number of HF samples


In this work, the linear regression multi-fidelity surrogate (LR-MFS) was proposed

to combine dataset with different fidelity, especially for high-fidelity with noise. The linear

regression is commonly used, balanced between accuracy, cost and simplicity. The LR-

MFS is less likely to overfit noise by limiting the number of parameters. Comparing with

the heuristic deterministic MFS, the LR-MFS is derived from standard linear regression

Model form Noise effect

Epis

tem

ic u

ncert

ain

ty

PRS10 LR-

MFS3

LR-MFS10 PRS

3

76

with analytical solutions to obtain parameters and therefore promising for theoretical

study. The treasure of linear regression could be applied directly to LR-MFS, such as

prediction variance, optimal DoE. The LR-MFS was demonstrated using polynomial

response surface (PRS) in this work. The proposed LR-MFS was first compared with

three Bayesian frameworks using a benchmark dataset from the simulations of a

fluidized-bed process. The accuracy of Bayesian frameworks varied significantly, and

the LR-MFS was comparable to the best GP-based approach. The effect of multiple

low-fidelity datasets was discussed. Introducing multiple low-fidelity models would suffer

a higher risk of overfitting for limited number of high-fidelity dataset. Then the LR-MFS

was evaluated to approximate a non-linear numerical test function with different

sampling plans. The LR-MFS bested Co-Kriging for 55%~63% cases with increasing

number of high-fidelity(HF) samples. For both examples, the LR-MFS proved to be

better than fitting only HF samples, and robust for HF samples with different level of

noise. LR-MFS could avoid the error for fitting LF data as demonstrated for the

numerical example. The LR-MFS heavily relies on the model form of the discrepancy

function. In the future, the selection of basis terms will be examined using existing

schemes such as stepwise regression.

77

CHAPTER 4 MULTI-FIDELITY SURROGATE MODELING FOR APPLICATION/ARCHITECTURE

CO-DESIGN

The HPC community has been using abstractions such as representative

applications (e.g., mini-apps, proxy-apps, skeleton apps) and architecture models to

enable faster co-design cycles. While abstraction developers often qualitatively verify

the correlation of the app abstractions to the actual (also called parent) application, it is

equally important to quantify this correlation to understand how the co-design results

translate to the parent application. In this work, we propose a multi-fidelity surrogate

(MFS) approach which combines data samples of low fidelity (LF) models (using

representative apps and architecture simulation) with a few samples of a high fidelity

(HF) model (using the parent app). The MFS fitted from a small number of

computationally expensive HF samples and larger number of cheaper LF samples, can

provide a more accurate prediction than using either type of data alone. In this work, we

demonstrate the application of MFS using a compressible fluid flow - application -

compressible multiphase turbulence (CMT)-nek (HF data) and its proxy-app, skeleton-

app, and simulation models (all LF data). We constructed a least-squares MFS based

on polynomial response surface (PRS) to mitigate the difference between the LF and

HF data. Our results show that the relative root-mean-square error (RMSE) between

predictions of MFS and the baseline HF models was 4%, which is significantly better

than 74% RMSE in using purely LF data alone, demonstrating that MFS is a promising

approach for predicting the parent application performance (HF model) using an MFS

while staying within a reasonable computational budget. The collection of dataset and

details of the experiments were kindly provided by Aravind Neelakantan, Nalini Kumar

and Herman Lam. This chapter on multi-fidelity surrogate is based on the publication [9]

78

co-authored with Aravind Neelakantan, Nalini Kumar, Herman Lam, Chanyoung Park,

Nam Ho Kim and Raphael T. Haftka.

4.1 Background

As we approach exascale computing, the next frontier in high-performance

computing, it is important that application developers and system designers co-design

to develop better performing and more energy efficient application codes and

machines[83]. For fast and effective turnaround during the co-design process,

application developers create representative applications which are abstract, smaller,

and self-contained descriptions of their application code (also called parent app) and

only capture the key parameters and features that predominantly influence the outcome

of co-design[84]. To further speed up the co-design process and to enable architecture

design-space exploration (DSE), system architects build simulator models to study the

application performance on various underlying architectures. Behavioral Emulation (BE)

[85] is one such coarse-grained approach for simulation of extreme-scale systems and

applications. While the parent application can be used to drive architecture simulations,

abstract application end-point models are often used to represent the parent app to

speed up the co-design process.

Representative applications, in the form of miniapps, proxyapps, or skeleton

apps, have been developed for many scientific HPC codes and are a necessity in cases

where the actual application cannot be shared with the hardware architects[86-89]. After

development, it is important to validate these representative apps against their parent

apps to ensure they are reasonably accurate representations of the application

behavior. Typically, this qualitative validation is performed by comparing ratio of

computation to communication, weak scaling and strong scaling trends, similarity

79

analysis, etc. Similarly, performance prediction results of architecture simulations are

verified against testbed measurements. After validation, both the representative apps

and simulator models are used as platforms to evaluate tradeoffs for improved

performance, power, and resilience, different programming models, compilers, etc. and

guide the refinement of parent application. Qualitative validation is important; however,

it is also important to determine quantitatively how the improvements in a representative

app or architecture translate to the parent app. To the best of our knowledge, no

solution for quantitative validation of representative applications has been proposed in

the literature.

In general, surrogate models are approximations that are fit to the available data

of a phenomenon of interest, herein the parent app. A high-fidelity surrogate model

(HFM) can be constructed from more accurate and computationally expensive high-

fidelity data (e.g., benchmarking data using parent app); and a low-fidelity model (LFM)

can be constructed from computationally cheaper but less accurate low-fidelity data

(e.g., skeleton apps, simulation results). In this work, we propose the use of a multi-

fidelity surrogate model (MFS) for identifying the relation between parent and

representative apps. These multi-fidelity models are constructed using a combination of

HFMs and LFMs.

The concept of MFS has been extensively studied to approximate the high-

fidelity models (HFMs) assisted by cheaper low-fidelity models (LFMs) [58]. To balance

accuracy and computational cost associated with data collection, the MFS approach

aims to develop a surrogate model based mostly on LF samples assisted with only a

80

few HF samples. Typical multi-fidelity models include finite element analysis with

different resolution, physical tests versus numerical simulations, etc. [11, 58, 90].

In this work, we leverage many of the MFS methods developed in other scientific

domains and adopt and apply them to reduce the computational cost of validation of

representative apps used in the HPC co-design process. After a survey of the related

research in Section 4.2, in Section 4.3 we present an overview of the parent application

case study (CMT-nek), its representative miniapp (CMT-bone) and skeleton app (CMT-

bone-BE), and the Behavioral Emulation approach that we use for performance

modeling and simulation. In Sections 4.4 and Section 4.5, we describe a methodology

for developing an MFS for an application from its mini-app (HFM), skeleton app (LFM),

and a simulator model (LFM). In Section 4.6, we demonstrate the usefulness of the

proposed methodology by applying it to a multi-physics simulation software being

developed for exascale systems - CMT-nek.The results show that the root-mean-

square error (RMSE) between predictions of MFS and the baseline HF models was 4%,

which is significantly better than 74% RMSE in using purely LF data alone,

demonstrating that MFS is a promising approach for predicting the parent application

performance (HF model) using an MFS while staying within a reasonable computational

budget.

4.2 Related Research

Miniapps have become extremely important for exascale DSE and performance

optimization. In [91], which presents a validation methodology, the authors state that

miniapps reduce the DSE time by a factor of a thousand, making them extremely useful

for exploring the design space of the parent application. Heroux, et al. in [92] provide a

81

verification and validation (V&V) methodology for assessing the ability of the miniapp to

effectively represent the performance of their parent application. The authors use the

difference between miniapp and parent app performance as their validation metric and

compare it against a threshold. This approach requires equal number of samples for

both the parent app and the miniapp. Since samples for the parent app are typically

more expensive to obtain, it can be a limiting factor in extensive validation studies over

a large design space. In our proposed approach, constructing an MFS requires much

fewer samples of parent app than of the miniapp, thus considerably reducing the

computational budget of conducting performance validation.

Miniapps are used as a tool to evaluate optimization methods to improve the

performance of the parent application. But improving the performance of the miniapp

does not guarantee the same for the parent app, making it important to know how

representative these miniapps are of their parent app [84]. For example, in [84] the

authors seek to improve the performance of the application on new and future systems

using miniapps. Although the optimizations applied improve the miniapp performance,

the impact on actual application performance is not clear. In our work, an MFS for the

parent application, built using high-fidelity application performance samples and lower-

fidelity miniapp performance samples, can help us draw a relationship between the

performance behavior of the two applications.

Several frameworks have been proposed to realize multi-fidelity modeling in

various science and engineering domains [58] [11, 90]. In [90], a Bayesian framework

has been applied to predict the data of nuclear radiation based on simulations. A

variable fidelity optimization framework has been demonstrated for the design of engine

82

piston[93]. A deterministic framework has been proposed to predict the strength of

composite laminate based on finite element simulations in[11]. A large number of

applications to mechanical systems can be found in[58], which reports that the MFS

reduces computational cost drastically while enabling desirable prediction accuracy.

The MFS has also been adopted as a powerful tool for uncertainty propagation[59].

Various MFS frameworks have been proposed for different engineering

applications. The existing frameworks are mainly different from two aspects: (1) the

form of algebraic function and (2) the scheme to determine MFS parameters. Various

MFS frameworks have been proposed to emphasize different aspects of these two

components. For example, the Bayesian MFS based on a scale factor has been applied

to design buildings[94] and flapping flight [95]. The Bayesian MFS incorporating

discrepancy function has been proposed [62, 90] as a popular MFS framework for

various applications. This Bayesian framework is equivalent to the co-Kriging surrogate

[73] with no prior information. Balabanov et al. [60] used a sequential deterministic MFS

based on the discrepancy function to combine finite element simulation with different

resolutions. Zhang et al.[11, 26] proposed a simultaneous deterministic MFS based on

the discrepancy function to combine experimental strength and finite element simulation

for composite laminate. We can apply this methodology in our HPC community to save

computational cost of parent application.

Sampling schemes for multi-fidelity models have been studied correspondingly.

HF samples are usually a subset of LF samples. One representative all-at-once

sampling strategy is the nested design sampling [96]. First, LF samples are generated

using Latin Hypercube Sampling (LHS). Then the HF samples are generated by

83

maximizing the minimum distance between all existing LF samples. Huang et al. [97]

proposed a sequential sampling scheme for design optimization using Bayesian MFS.

Either LF or HF samples are generated iteratively for design optimization. In our

approach, we used Full Factorial Design (FFD) for sampling as it is convenient for

parametric study. LF samples are available at the configurations of all the HF samples.

In this work, we leverage many of the MFS methods developed in other scientific

domains and adopt and apply them to the HPC domain to reduce the computational

cost of validation of representative apps used in the co-design process.

4.3 Application and Architecture Models

In this section, we give an overview of the parent application under study (CMT-

nek), its representative mini-app (CMT-bone) and skeleton app (CMT-bone-BE); and a

BE simulation approach that we use for performance modeling and simulation. The

relationships among their corresponding models are shown in Figure 4-1. The parent

app, CMT-nek, represents a high-fidelity (HF) model, whereas CMT-bone (a mini-app)

and CMT-bone-BE (a skeleton app) are low-fidelity (LF) models, as compared to CMT-

nek. BE simulation is a modeling and simulation of CMT-bone-BE. Thus, BE simulation

is an even lower fidelity model than CMT-bone-BE and CMT-bone.

In this study, our objective is to first perform validation and uncertainty estimation

of the BE simulation results against test samples of CMT-bone-BE (details in Section

4.3.2). Then a multi-fidelity surrogate model (MFS) is developed using mostly samples

from the low-fidelity BE simulation and a few high-fidelity CMT-nek samples. The MFS

model is then used to predict CMT-nek results (Section 4.6, case study 1). This

experiment is repeated between BE simulation (LF) with CMT-bone (now being an HF,

84

as compared to BE simulation) in case study 2, followed by CMT-nek (HF) and CMT-

bone (LF) for case study 3.

Figure 4-1. Hierarchy of the CMT models

4.3.1 CMT-nek

CMT-nek is being developed at the PSAAP-II Center for Compressible

Multiphase Turbulence (CCMT) at University of Florida to perform simulation of

instabilities, turbulence, and mixing in particulate-laden flows under conditions of

extreme pressure and temperature. CMT has applications in many environmental,

industrial, and national defense and security areas. CMT-nek is being developed from a

production release of petascale code Nek5000 [98], a Gordon Bell prize winning open-

source software for simulating unsteady incompressible fluid flow with thermal and

passive scalar transport. It is a highly scalable code with strong scaling to over a million

MPI ranks on ALCF BG/Q Mira. CMT-nek aims to take advantage of this sustained

performance by inheriting the MPI strategies used in Nek5000; and by hooking into the

Nek5000 repository, leveraging any changes and optimizations made to Nek5000.

85

4.3.2 CMT-bone and CMT-bone-BE

CMT-bone is a mini-app that encapsulates the key data structures and compute

and communication kernels of CMT-nek. While retaining the workflow of CMT-nek,

CMT-bone simplifies the number of variables defined and allocated and also the

number of computation and communication operations performed at each time step in

the simulation. Banerjee et al. and Kumar et al. in [86, 89] have validated mini-app

CMT-bone with its parent app CMT-nek and found that the key compute kernels are

well represented by the proxy application.

CMT-bone-BE is a skeleton app of CMT-nek created to support rapid algorithmic

design-space exploration. It models the computation that happens within every

simulation time step to calculate the partial derivative and exchange data between

nearby spectral element meshes. CMT-bone-BE ignores the initial problem setup

including mesh generation. Mesh generation operations can be abstracted and replaced

with a computation model in BE.

4.3.3 Behavioral Emulation (BE) Simulation

Behavioral Emulation (BE) is a coarse-grained modeling and simulation

approach that aims to provide timely, flexible, and scalable estimates of application

performance on existing and future system architectures. In BE, the complexity of large-

scale system simulation is handled by simultaneously dividing the simulation into

different levels of system abstraction (e.g., device, node, rack, system) and abstracting

the behavior of the components at each of these levels. The coarse-grained component

models mimic or emulate the observed execution behavior of the component instead of

its cycle-accurate operation. There are two basic types of BE models - application BE

objects (AppBEOs) and architecture BE objects (ArchBEOs).

86

BE simulations are used to predict the execution time of CMT-bone-BE and have

computational cost less than that of CMT-bone-BE (and much less than that of CMT-

nek and CMT-bone). In our study, BE simulation results are used to produce the LF

data points which are used to construct MFS models to predict the execution time of

CMT-nek and CMT-bone, using very few data points from these two applications, thus

reducing the overall computational budget of the DSE process.

4.4 Multi-fidelity Surrogates

In engineering applications, it is common to have multiple models with different

fidelities for solving the same problem such as finite element simulations with different

grid resolutions, numerical simulations, and physical experiments. A high-fidelity model

(HFM) represents the physical phenomenon more accurately than the low-fidelity model

(LFM) but it is often very expensive. An MFS based approach uses both high-fidelity

and low-fidelity datasets to approximate the HFM in the design space. An effective MFS

is expected to make accurate prediction with a limited budget for sampling. Fern andez-

Godino et al. [58] reviewed recent developments of MFS especially on effectiveness of

applying MFS to practical design. Peherstorfer et al. [59] summarized the technical

details of MFS for inference and uncertainty propagation.

MFS translates LFM against a few HF samples using an algebraic function.

Typical MFS frameworks include two major components: (1) a model to define the

relation between LFM and HFM, and (2) the scheme to find parameters of the MFS and

associated uncertainty of prediction. The LFM could be translated to HFM through (1) a

constant scale factor, or (2) scaling up the LFM and adding to a discrepancy function.

After determining the form of algebraic function, the MFS could be developed either

87

using Bayesian inference through Gaussian process, or using a least-square regression

minimizing error between fitted model and data.

In this work, we investigate the feasibility of MFS to quantify and mitigate the

difference between high-fidelity parent applications (e.g., CMT-nek) and low-fidelity

simulations (e.g., BE simulation) in the area of co-design of large-scale system. The

least-squares MFS (LS-MFS) (equivalent to the LR-MFS in section 3) [8] was selected

for this feasibility study while balancing complexity and predictive capability.

ˆ ˆ ˆ( ) ( ) ( )H L

f f x x x (4-1)

The LS-MFS is built with two surrogates, ˆ ( )Lf x , a polynomial response surface

(PRS) fitted to low-fidelity data, and ˆ( ) x , the fitted discrepancy data as in Eq. (4-1).

The scale factor and discrepancy function ˆ( ) x are obtained to minimize prediction

error at the high-fidelity samples according to Eq. (4-2) and Eq. (4-3). ( HX , Hy )

denotes the high-fidelity dataset containing n samples.

ˆ, ( )

ˆ ˆmin : ( ) ( )T

H H H H

x d x dx

(4-2)

ˆ ( )H L H H

f d x y (4-3)

The multi-fidelity surrogate using a single linear regression is obtained from Eq.

(4-4) to Eq.(4-8). Y is the vector of high-fidelity samples, X is the augmented design

matrix, B is the vector of unknown coefficients and e is the vector for residual errors.

Xi x denotes the thi monomial/basis, and

ib is the coefficient of X

ix . The obtained

discrepancy function ˆ( ) x is shown in Eq.(4-8).

Y XB e (4-4)

88

(1) (1)

1

1 1 1

( ) ( )

, ,

H H

n p n

n n

H Hp

b

b

y e

Y B e

y e

(4-5)

(1) (1) (1)

1

( 1)

( ) ( ) ( )

1

ˆ ( ) X X

ˆ ( ) X X

L H H p H

n p

n n n

L H H p H

f

f

x x x

X

x x x

(4-6)

1

T T

B X X X Y (4-7)

1

ˆ( )= Xp

i i

i

b

x x (4-8)

The LS-MFS scales up the LFM and adds a polynomial function ˆ( ) x to match a

few high-fidelity samples. The scale factor is critical for prediction. Negative or

extremely large values of indicates a prediction with large error, which is likely to be

associated with undesirable LFMs, inappropriate surrogate forms, or inadequate

samples. ˆ( ) x is supposed to be a low-order polynomial function while assuming the the

LFM has a trend similar to HFM. In our study, we adopted a constant ˆ( ) x for less than

10 HF samples and a linear polynomial function as ˆ( ) x for the rest. We approximated

the execution time in logarithmic coordinate to account for the order-of-magnitude

variation of execution time. ˆ ( )Lf x was developed using a quartic PRS.

4.5 Developing MFS Model

Although various performance metrics can be studied for performance simulation

such as energy consumption and communication times between the processors, the

metric of interest in this work is the total execution time for running a typical

computational fluid dynamics analysis using CMT-nek (HFM), CMT-bone (HFM), CMT-

89

bone BE (LFM), and BE simulation (LFM). All the benchmarking of the CMT models is

performed on the Vulcan HPC platform from Lawrence Livermore National Laboratory

(LLNL) [99]. Vulcan is a 24-rack IBM Blue Gene/Q system based on POWER

architecture that consists of 24,576 nodes and 400TB of compute memory. It is

important to ensure that the HF and LF data are obtained from the same hardware. The

accuracy of the MFS model reflects how representative the LF and HF are of each

other.

4.5.1 Design Space

For CMT-nek, the three main application parameters of concern are Element

Size (ES), Elements per Processor (EPP) and Number of Processors (NP). Application

performance can be affected by changing any one of these parameters. We chose 125

experimental points based on five-level full factorial design. “Five-level” denotes the 5

points/grids selected along each application parameter with similar space, as shown in

Figure 4-2. The design of experiment is ES= [5,9,13,17,21], EPP= [8,32,64,128,256]

and NP=[16,256,2048,16384,131072]. The experimental runs require up to 131,072

processors, 34 million elements and 311 billion computational grid points.

As the model fidelity increases so does the cost of obtaining a test sample. We

obtained data for CMT-bone-BE and BE simulation for the entire design space (125

data points); but for CMT-nek and CMT-bone, data was judiciously obtained from a

subset of the design space. For the runs from LFM and HFM, we made 22 runs from

CMT-nek (HFM), 67 runs from CMT-bone, and 125 runs for both LFMs (CMT-bone-BE

and BE simulation).

90

Figure 4-2. Design of experiments for CMT-nek, CMT-bone, and BE simulations

4.5.2 Validations of BE Simulation Results

Recall from Figure 4-1 that the order of fidelity is as follows: parent app CMT-nek

(highest), mini-app CMT-bone, skeleton app CMT-bone-BE, and BE simulation (lowest).

In the next section, we will use the BE-simulation results (LF) to predict the performance

of CMT-nek (HF parent app). Thus, first it is important to evaluate the accuracy of the

BE simulation. To do so, in this section, we will first validate the accuracy of BE

simulation against skeleton app CMT-bone-BE. We then evaluate the accuracy of CMT-

bone-BE by validating its results against those of mini-app CMT-bone.

We first examined the BE simulation versus CMT-bone-BE. Validation is the

process of comparing the BE simulation results to its respective benchmarking result

using CMT-bone-BE. In this study, we have validated the simulation results for the

entire design space on Vulcan, one of the largest high-performance computing system

available at the Lawrence Livermore National Lab. The validation of the design space

91

covered all the calibration points. But to further evaluate the accuracy of the simulator,

true validation was performed by validating points that are not present in the calibration

set. Polynomial interpolation was used in the simulator to predict the execution time of

the application at these validation points. The obtained simulation results are then

validated by running the actual CMT-bone-BE application on Vulcan for those validation

points.

The validation results of BE simulation against CMT-bone-BE are shown in

Figure 4-3. where the blue points represent the predicted CMT-bone-BE time using BE

simulation and the red points are the validation points obtained by running the

application (CMT-bone-BE) on Vulcan. We validated the simulations up to 128k NP on

Vulcan and predicted the time for 256k NP and 512k NP. The average percentage error

between BE simulation and CMT-bone-BE is 4 %, thus demonstrating the accuracy of

the BE simulator.

Figure 4-3. Validation of BE simulation against CMT-bone-BE

Then we compared CMT-bone-BE versus CMT-bone. The validation of the

skeleton app CMT-bone-BE against mini-app CMT-bone (Figure 4-4 (A) and Fig 4-4 (B),

respectively) is done through comparing their trends under the same experimental setup

92

described above. CMT-bone-BE being the skeleton app, takes less time to execute than

that of the mini-app, CMT-bone, and hence the range of their execution time varies.

Therefore, to make it easier to compare the trend, the execution time is plotted on a

color scale with red being the lowest in the range and blue being highest in the range as

shown in Figure 4-4. The step-wise increase shows that the predicted execution time for

CMT-bone-BE and CMT-bone increases monotonically with ES and EPP and does not

change appreciably with NP, and the color scale on the graphs verify the similarity in

trend between CMT-bone-BE and CMT-bone.

Figure 4-4. Comparing CMT-bone mini-app and CMT-bone-BE skeleton app trends for

various parameter values. A) CMT-bone and B) CMT-bone-BE.

4.6 Evaluating MFS Predictions --- Three Case Studies

Three case studies were used to demonstrate the multi-fidelity surrogate (MFS)

approach in which a surrogate model, based mostly on low-fidelity (LF) samples

assisted with only a few high-fidelity (HF) samples, is used to predict the performance of

a high-fidelity (HF).

• Case 1: Multi-fidelity model based mostly on BE simulation (LF) and few CMT-nek (HF parent app) data points to predict the performance of CMT-nek (HF)

• Case 2: Multi-fidelity model based mostly on BE simulation (LF) and few CMT-bone (relatively HF mini-app) data points to predict the performance of CMT-bone (HF)

A B

93

• Case 3: Multi-fidelity model based mostly on CMT-bone (relatively LF mini-app) and few CMT-nek (HF) data points to predict the performance of CMT-nek (HF)

The setup was same in all three case studies. A subset of high-fidelity data was

selected as the validation/test runs to evaluate predictions while the others were used

as training runs to train LS-MFS (least square MFS). The number of samples increased

gradually from the remaining runs (which excludes the validation runs) to investigate the

effect of sampling plan. For each number of samples, random selection was repeated

20 times to account for the effect of sampling plan. The overall difference was

measured using relative root-mean-square error (R-RMSE) between the LS-MFS

predictions and the validation runs. The relative error is defined in Eq. (4-9) where ˆ( )f x

is the surrogate prediction and ( )f x is the function value at test point. The R-RMSE is

defined in Eq. (4-10) where tN is the number of test points. The relative maximum

difference (R-MD) at the validation runs based on the repeated samples was also

provided to understand individual prediction. In this work, we study LS-MFS using

polynomial response surface as it is robust with noise effect. Other frameworks of multi-

fidelity surrogates are also available such as co-Kriging. The comparison between

different multi-fidelity surrogates is beyond the scope of this work.

ˆ ( ) ( )

( )( )

f ferr

f

x xx

x (4-9)

2

1

( )

R-RMSE

tN

i

i

t

err

N

x

(4-10)

4.6.1 Case Study 1: CMT-nek Predictions from BE Simulations

22 runs of CMT-nek are obtained as shown in Figure 4-2. 10 runs (out of 22)

were selected randomly and fixed as the validation runs. We first examined LS-MFS to

94

approximate CMT-nek (HF model) runs using a typical set of 12 samples as shown in

the Case 1 column of Table 4-1. The R-RMSE of LS-MFS is 4.49% using BE simulation

(LF model) at 10 validation runs. The linear fit using only HF CMT-nek runs was also

developed as a comparison with the R-RMSE to be 131.23% at the validation runs. The

R-RMSE between original BE simulation and CMT-nek ( ˆ ( )Lf x ) at all the 12 points,

without translation, is 66.85%. As mentioned before, BE simulation mimics CMT-bone-

BE and not CMT-nek. Since CMT-bone-BE is just a skeleton app with very few

computational kernels, the percentage difference is high. The LS-MFS was much more

accurate than either the ˆ ( )Lf x or the linear fit to ( HX , Hy ), demonstrating its promise to

compensate the difference between high-fidelity and low-fidelity models.

Table 4-1. Predicting execution times of CMT models based on typical set of 12 samples and evaluating the prediction using R-RMSE (%)

Surrogates Case 1: CMT-nek prediction from BE simulation

Case 2: CMT-bone prediction from BE simulation

Case 3: CMT-nek prediction from CMT-bone

Number of validation runs

10 20 10

LS-MFS 4.49% 5.40% 7.34%

ˆLf x 66.85% 61.26% 18.65%

Linear fit to

,H Hx y

131.23% 1901.91% 131.23%

Residual errors of

ˆLf x

0.77% 0.77% 1.05%

Residual errors of the linear fit to

,H Hx y

18.40% 50.71% 18.40%

Next, we investigated the effect of the sampling plan on prediction accuracy. The

LS-MFS predictions for CMT-nek runs with increasing number of samples were

summarized in Figure 4-5 (A). The LS-MFS was unstable using only 2 CMT-nek

95

samples due to over-fitting and became more accurate with increasing CMT-nek

samples. The R-RMSE was less than 10% with more than 9 CMT-nek samples and

ended with 4.49%. The R-MD was less than 20% with more than 9 CMT-nek samples

and ended with 7% as seen in Figure 4-5(B). The order of ˆ( ) x was changed from

constant to linear for more than 9 CMT-nek samples which was critical for the accuracy

of LS-MFS. We specified the order of ˆ( ) x for simplicity in this feasibility study. The

performance of LS-MFS could be improved by choosing appropriate ˆ( ) x . Another

observation is the large variation of R-RMSE while repeating HF samples. The design of

experiments for HF samples affected LS-MFS noticeably. The evaluation of LS-MFS for

CMT-nek is based on up to 12 samples and may suffer large uncertainty due to the

scarce runs.

Figure 4-5. Difference between CMT-nek validation runs and multi-fidelity predictions

based on CMT-bone simulations. A) Relative RMSE and B) Relative maximum difference.

A B

96

4.6.2 Case Study 2: CMT-bone Predictions from BE Simulations

20 runs (out of 67) were selected randomly and fixed as the validation runs. We

performed LS-MFS for CMT-bone (HF model in this case) based on 20 validation runs

and up to 47 samples. Again, LS-MFS was most accurate comparing to ˆ ( )L

f x and the

linear fit to only ( HX , Hy ) as shown in Case 2 column of Table 4-1. The LS-MFS

predictions were much closer to HFM than the original BE simulations.

Once again, we investigate the effect of the sampling plan on prediction accuracy.

The R-RMSE in Figure 4-6 (A) reduced with increasing CMT-bone samples and ended

with 5.4%. The R-MD in Figure 4-6 (B) oscillated with scarce CMT-bone runs at the

beginning and stabilized around 10%. Both R-RMSE and R-MD reduced noticeably with

the first few samples and stabilized to less than 10% thus proving to be a promising

approach.

Figure 4-6. Difference between CMT-bone validation runs and multi-fidelity predictions

based on BE simulation. A) Relative RMSE and B) Relative maximum difference.

A B

97

4.6.3 Case Study 3: CMT-nek Predictions from CMT-bone

In the final case study, the LS-MFS was developed to predict the high-fidelity

parent app (CMT-nek) from its low-fidelity mini-app (CMT-bone). This helps in

quantitative validation of the mini-app. The setup was same as in case study 1, where

CMT-nek was the HF model. From Case 3 column of Table 4-1, we see that LS-MFS

provides the best fit compared to linear fit. The LS-MFS had less-than 10% R-RMSE in

Figure 4-7 (A). It is worth noting the significant jump between 9 and 10 samples while

changing the order of ˆ( ) x in Figure 4-7 (A) and Figure 4-7 (B). CMT-bone was close to

CMT-nek and a different scheme might be preferred to determine ˆ( ) x .

Figure 4-7. Difference between CMT-nek validation runs and multi-fidelity predictions

based on CMT-bone. A) Relative RMSE and B) Relative maximum difference.

A key observation between case study 1 and 3 is that although the CMT-bone

samples were much closer to the CMT-nek samples, the MFS predictions of CMT-nek

(HF) from BE simulations (LF) were more accurate than the MFS predictions from CMT-

bone (LF) as shown in Table 4-1. Fitting CMT-bone was more challenging considering

the scarce samples (67 runs). BE simulations, on the other hand, had all the 125

A B

98

samples in the design space and thus, lead to better MFS predictions. This is supported

by residual errors of ˆ ( )L

f x from Table 4-1.

In all three cases, the error in prediction ended less than 10%; thus, proving it to

be a valuable approach to use for reducing computational budget in the process of co-

design. The range of scale factor for the three cases are summarized in Table 4-2.

The scale factors are around 1 which indicates that the LF and HF have similar trend.

The major difference between HF and LF are well-compensated by the constant/linear

discrepancy function.

Table 4-2. Range of the scale factors for LS-MFS

Case 1 Case 2 Case 3

Minimum 0.8 0.91 0.5

Maximum 0.98 1.08 1.3


Due to high computational cost, validation samples from HFM are usually

obtained at small scale. But with MFS model, we were able to perform quantitative

validation at a reduced computational budget. In this work, we studied the least-square

MFS (LS-MFS) using polynomial response surface as it is robust with noise effect. For

future work, different multi-fidelity surrogates can be compared. Our ultimate goal is to

predict the performance for exascale computation platform which is essentially long-

range extrapolation far from the validation samples. In the future, we will investigate the

capability of LS-MFS for long-range extrapolation which suffers large uncertainty. We

also noticed the LS-MFS predictions were sensitive with the high-fidelity samples.

Effective design of experiments for validation runs are expected to improve the

accuracy of LS-MFS, which is also a valuable research direction.

99

CHAPTER 5 ON APPROACHES TO COMBINE EXPERIMENTS AND SIMULATIONS

Finite element analysis coupled with failure theory has been widely used for

predicting the failure of composite structures. However, significant discrepancies

between model predictions and experiments may be observed due to modeling errors

and material variability. The problem addressed in this work is how to interpolate or

extrapolate in a parametric design space using results from a limited number of

experiments with the aid of simulations.

In order to make most of simulation and experiments, it should be feasible to

correct model prediction using scattered experimental tests. This modeling procedure

combining both simulation and experiment can be viewed a special case of multi-fidelity

modeling, with experiments being the high-fidelity results and simulations as the lower

fidelity results. In this work, we considered two approaches for combining fidelities. One

approach is to seek an additive (offset) or multiplicative (scalar) correction to the low-

fidelity prediction (here simulation). A second approach is to use experiments to

calibrate a failure criterion used in the simulations.

We compare these two approaches in an example problem which consists of a

composite plate with a hole, which is a benchmark composite structure demonstrating

the effects of a stress concentration. The first approach fits a correction to failure

predictions based on a standard maximum strain failure criterion using a multi-fidelity

surrogate. The second approach calibrates the characteristic distance from the open

hole boundary of the Whitney-Nuismer failure criterion using experiments. These two

approaches are also compared to that of making predictions by fitting a surrogate to

100

experimental results without the aid of simulations. We focused on predicting mean

strength of composite structure.

We also examined the case of using a large number of data points from one

material system to complement small number of data from a second material system.

For this case, we compared fitting the small number of data from the second system to

fitting both using algebraic corrections. The collection of dataset and details of the

experiments were kindly provided by Jaco Schutte, John Meeker and Upul R.

Palliyaguru. This chapter on multi-fidelity surrogate modeling of composite laminate is

based on publication[10, 11] and report co-authored with Jaco Schutte, John Meeker

and Upul R. Palliyaguru, Nam Ho Kim and Raphael T. Haftka.

5.1 Background

Multi-fidelity surrogate models have been proposed to combine computer

simulations with different fidelities. We are mostly interested in this approach for

combining experiments with simulations. Experiments are deemed as high-fidelity

function and simulations are assumed to be low-fidelity function. Significant

discrepancies between model predictions and experiments may be observed due

modeling errors and material variability [100]. The multi-fidelity surrogates are

essentially approaches to correct low-fidelity function against high-fidelity function using

algebraic functions. Different theoretical frameworks are distinct mainly in two features:

form of calibration functions and optimization of calibration parameters [74, 90]. One

approach of multi-fidelity modeling is to seek an additive (offset) or multiplicative (scalar)

correction to the low-fidelity prediction (here simulation) [101, 102]. We adopt a scale

factor and discrepancy function as correction function. A surrogate is first developed

based on simulation. The simulation surrogate is then multiplied by a scale factor and

101

added with a discrepancy surrogate. In contrast to using the sophisticated Bayesian

framework to estimate the correction parameters, we adopted the linear regression

multi-fidelity surrogate (LR-MFS) for multi-fidelity modeling. We apply the framework for

estimating failure strength of composite laminates with assistance of finite element

simulation.

A second approach to incorporate experimental strength with simulation is to

calibrate a failure criterion tailored to the particular structural element by using

experiments [103]. We compare model correction and calibration for a composite plate

with a hole which is a benchmark composite structure [13, 104]. The first approach fits a

correction to failure predictions based on a standard maximum strain failure criterion

using a multi-fidelity surrogate framework. The second approach calibrates the

characteristic distance from the open hole boundary of the Whitney-Nuismer failure

criterion using experiments[105]. These two approaches are also compared to that of

making predictions by fitting experimental results without simulations

5.2 Open-Hole-Tension Tests

5.2.1 Experiments

The experiments were performed by Upul R. Palliyaguru from the Wichita State

University. The OHT test was designed to investigate the effect of an unfilled hole on

the tensile failure strength. The test specimen geometry is shown in Figure 5-1. The

experimental OHT strength was calculated based on the nominal area (i.e. gross cross-

sectional disregarding the hole). The width of the specimen and the diameter of the hole

are denoted by w and D, respectively. Specimens were made of three distinct material

batches.

102

Figure 5-1. Open-Hole Tension Test Specimen Configuration and observed failure

mode.

Two design parameters varied in the tests: w/D and layups. Table 5-1 details the

test matrix for different w/Ds. Table 5-2 listed the test matrix for different layups

quantified by the fraction of ±45° plies. Each structural configuration was composed of

three pre-preg batches, with each batch containing 6 replicates. Mean values of the

experimental strength were summarized in Table 5-3.

Table 5-1. w/D test matrix

w (in.) D (in.) w/D Batches/Replica

tes

No. of Specimens

0.75 0.250 3 3 x 6 18

1.00 0.250 4 3 x 6 18

1.50 0.250 6 3 x 6 18

2.00 0.250 8 3 x 6 18

Total = 72

Table 5-2. Layup test matrix

Layup % 0°/45°/90°

Ply Stacking Sequence Batches/Replicates No. of

Specimens

40/20/40 [0/90/0/90/45/-45/90/0/90/0]s

3 x 6 18

25/50/25 [(45/0/-45/90)2]s 3 x 6 18

10/80/10 [45/-45/90/45/-45/45/-45/0/45/-45]s

3 x 6 18

Table 5-3. Mean value of experimental strength of composite laminate made from

MTM45-1 PWC2 3K PW G30-500 Fabric (ksi) N45 =0.2 N45 =0.5 N45 =0.8

w/D=3 88.67 58.20 37.41 w/D=4 92.08 67.58 42.63 w/D=6 98.52 70.10 49.80 w/D=8 101.65 72.33 51.86

Failure at lateral gage middle

F F

D w

103

5.2.2 Finite Element Simulations

The finite simulations were produced by Jaco Schutte and John Meeker using

the inhouse tool at the Materials Sciences Corporation. The finite element solver

ABAQUS was utilized to predict the strains and stresses in the open hole tension

specimens. The open hole tensile coupon finite element model consisted of linear

continuum brick elements with a linear displacement controlled loading. The laminate

properties were defined by representative volume elements (RVE) assembled of three

sub-cells. Mechanical properties of the laminia MTM45-1 PWC2 3K PW G30-500 Fabric

could be found from [11]. The sub-cells are a transversely isotropic representation of the

resin impregnated fiber bundles, in the form of a set of concentric cylinders, where the

center cylinder consists of the fiber material and the surrounding cylinder the resin

material. Two of the three sub-cells represent each of the orthogonal directions of the

resin impregnated fibers in the fabric and the third sub-cell is a pure cylinder of resin.

Failure was modeled by maximum strain failure criterion applied to the fiber

directions of each sub-cell. The net strains in each sub-cell were calculated from the

state of strain of each element of the global analysis. Failure of the specimen is

assumed when one of the sub-cells exceeds a maximum stain in the fiber direction,

determined with unidirectional test coupons.

Figure 5-2 depicts the mesh of the FE model, including a magnified mesh around

the hole. Critical points of failure were observed at the hole edge. The simulations were

conducted at 221 different combinations of (w/D, N45) uniformly spaced, where w/D

varied between 2 to 10 with a step size of 0.5, and the fraction of 45° plies, N45 varied

between 0.2 to 0.8 with a step size of 0.05. Mean value of experimental strength and

simulation results are shown in Figure 5-3. The strength observed from experiments

104

and simulations are both monotonic in both variables and slightly curved over the

design space.

Figure 5-2. Mesh of finite element model for OHT test. A) Overall mesh of the whole

plate and B) zoomed mesh near the hole.

Figure 5-3. Illustration of experimental results and simulations for OHT tests

5.3 Approaches to Predict Mean Strength of Untested Points Assisted by Simulations

We have examined two approaches to predict experimental strength at untested

points assisted by simulations: model calibration and algebraic correction. In addition, a

surrogate model has been fitted to experiments alone in order to check whether the

A B

105

simulations provide improved accuracy. All three approaches used polynomial response

surface as the surrogate due to its robustness with noisy data.

5.3.1 Model Calibration

The tension and compression strength of composite laminates with holes drops

greatly with increasing hole diameters. This drop cannot be fully predicted with the use

of a classical stress concentration factor. Whitney and Nuismer [105]assumed that

failure of an open hole in an infinite plate under tension or compression can be

determined when the stress at some characteristic distance away from the edge of the

hole reaches the un-notched strength of the material system. Figure 5-4 illustrates the

assumption where an infinite plate containing a hole with radius r is placed under a

uniform tensile stress and y is defined as the stress concentration at a distance x from

the hole. The Figure 5-4 shows a point at distance x from the hole where the stress is

the un-notched strength of the material system 0 .

Figure 5-4. Failure criteria for a circular through thickness holes in an infinite plate

106

The Point Stress Criterion (PSC) [105] gives the ratio of the laminate un-notched

strength 0 to the open hole strength N as function of the radius (r) and the

characteristic distance 0d as shown in Eq. (5-1). The distance 0d can be calibrated by

experiments for a given material system. The un-notched failure strengths 0 for each of

the material systems were obtained with a finite element analysis of the composite

laminate without a hole using max strain failure.

2 4

0 0

2,

2 3

N r

r d

(5-1)

We combined experiments and simulations by fitting the Whitney-Nuismer

characteristic distance as function of 45( / , )w D N . At untested points, N is the quantity

of interest, we can obtain 0 based on finite element analysis of un-notched laminate at

a given 45N . At each of the experimental points the notched strength is available from

mean value of replicates and 0d can be computed using Eq. (5-1). A polynomial

response surface (PRS) was fitted to the characteristic length. Estimated failure

strength was obtained by combining un-notched strength from finite element analysis

with predicted characteristic length.

5.3.2 Algebraic Correction (Multi-fidelity Surrogate)

Simulations at untested points may allow us to improve accuracy of the

prediction. Here we use failure predictions from FEA based on strains at the edge of the

hole that substantially under-predicted the failure strength. We adopt a multi-fidelity

surrogate (MFS) for correcting the predictions. The quantity of interest for prediction is

the mean value of experimental strength. MFS provides a fit combining a small number

of expensive high-fidelity data (here experiments) and a large number of less expensive

107

low fidelity data (here simulations) as function of a design parameter vector x. The high

fidelity data (experimental mean strength) is denoted as ( Hx , Hf ), while the low fidelity

data is ( Lx , Lf ), where Hx is a subset of Lx .

In the simple framework [8] (equivalent to the LR-MFS in section 3), the MFS is

described with two surrogates ˆ ( )Lf x and ˆ( ) x which represent a fit to the low fidelity

data and a discrepancy function or correction, respectively,

ˆ ˆ ˆ( ) ( ) ( ) ( )H Lf f x x x x (5-2)

where ρ is a scalar and ( ) x denotes random noise due to experimental variability and

numerical instability. ρ and ˆ( ) x are obtained simultaneously through optimization as

shown in Eq. (5-3) and Eq. (5-4).

ˆ, ( )

ˆ ˆmin : ( ) ( )T

H H H H

x

x d x d (5-3)

ˆ ( )H L H Hf d x f (5-4)

Polynomial fits are used for surrogates, with the order for the low-fidelity model

being selected using leave-one-out cross validation based on ˆ ( )Hf x at experimental

points. Large scale factor indicates significant contribution of ˆ ( )Lf x in the multi-fidelity

model.

5.4 Results for Combining Simulations and Experiments

We first selected a subset of experimental strength results, shown in Figure 5-5,

for modeling. The other points were used to check the accuracy of the predictions of the

three approaches. We first examined root-mean-square error (RMSE) at untested points

as shown in Table 5-4. For multi-fidelity modeling, all the finite element simulations were

used to develop low-fidelity model.

108

Figure 5-5. Selected points from experimental strength (high-fidelity function) used to predict strength at other points

Table 5-4. Relative error and root mean square error at untested points based on 6 samples

w/D, N45 3, 0.5 4, 0.5 8, 0.5 3, 0.8 4, 0.8 8, 0.8 RMSE

Experiment only 11.6% 0.06% 7.6% 9.1% 1.8% 3.5% 7.0%

MFS with rho=0.5 14.1% 3.5% 9.8% 4.7% 0.65% 0.35% 7.4%

WN with quartic PRS 9.2% 2.2% 0.19% 19.3% 9.7% 1.2% 9.6%

It is seen that expf had the best accuracy, followed by WNf and then by MFSf based

on RMSE. All the three models had reasonable accuracy because of the almost linear

variation of the strength that made six data points sufficient for good fit. Introducing

simulation in this case resulted in some decrease in prediction accuracy. We concluded

that expf should be first choice when enough data is available. MFSf is preferred when the

trends between high- and low- fidelity functions are similar. WNf introduces a physical

model to simplify the quantity of interest for approximation. We will examine the effect

of cross-validation to rank these three surrogate models.

We then further investigated different forms of model correction. Various subsets

of experimental strength were selected for modeling. Root-mean-square error (RMSE)

109

were computed using the mean value at all the experimental points to reflect overall

accuracy. For multi-fidelity modeling, all the finite element simulations were used to

develop low-fidelity model. We examined two partial versions of the multi-fidelity model

as shown in Eq. (5-5) and Eq. (5-6) besides MFSf in Eq. (5-2). They represent the use of

only multiplicative correction or only additive corrections, and were needed when the

number of data from the new system is too low to permit both additive and multiplicative

corrections. The parameters and ˆ( ) x were determined similarly as that from MFSf .

ˆ ˆ( ) ( )Lf f x x (5-5)

disˆ ˆ ˆ( ) ( ) ( )Lf f x x x (5-6)

Figure 5-6. Performance of different correction models for predicting OHT strength using FEA. Experiments were performed based on composite laminate made from MTM45-1 PWC2 3K PW G30-500 Fabric

Mean RMSE of 100 sets of prediction are compared in Figure 5-6. The corrected

models are all better than the FEA simulations. It is seen that ˆ ( )f x could avoid large

error when fewer than 3 samples are available. ˆ ( )f x dominated disˆ ( )f x with varying

110

number of samples. expf and MFSf had similar accuracy when more than 3 samples

were available.

The conclusion from this analysis, is that the response of OHT tests is close to

linear and therefore surrogates fitted to experiments could provide reasonable accuracy

with three or more experimental data points. However, simulations combined with model

correction is needed when only one or two experiments are available.

5.5 Using Tests from One Material System to Predict Strength of Another Material System

Next we compared predicting the OHT strength of a target material system (high-

fidelity system) based on available experimental data on a similar material system (low-

fidelity function). Our goal was to see whether large number of data points from one

system can be used to help improve predictions when only a small amount of data is

available from another material system.

The available 12 points of OHT tests were used to develop a low-fidelity model.

Then we used 6 points from the literature for a different material system as high-fidelity

data shown in Table 5-5. All the OHT tests are illustrated in Figure 5-7. As we wanted to

explore the effect of the number of points from the new system, we considered using

only subsets of the available 6 points. To get meaningful conclusions, we randomly

selected a subset of samples from high-fidelity data for modeling, and repeated the

process 100 times.

Table 5-5. Experimental strength of composite laminate from T700SC-12K-50C/#2510 Plain Weave Fabric (ksi).

45/ ,w D N (3, 0.2) (4, 0.2) (6, 0.2) (8, 0.2) (6, 0.5) (6, 0.8)

Strength 59.6 63.2 66.5 69.3 54.5 41.1

111

Figure 5-7. Experimental strength of OHT tests from two different material systems

Figure 5-8. Performance of different correction models for predicting OHT strength using FEA or another material. Experiments for prediction were performed based on composite laminate made from T700SC-12K-50C/#2510 Plain Weave Fabric

Mean RMSE of 100 sets of prediction are compared in Figure 5-8. Similar to

Figure 5-6, ˆ ( )f x dominated disˆ ( )f x with varying number of samples. expf and MFSf had

similar accuracy when more than 3 samples were available. One interesting observation

112

is that all the corrected models are more accurate based on FEA simulations than those

based on another material.

5.6 Predicting B-basis Allowables of Experimental Strength Assisted by Aimulations

We investigate the prediction of B-basis allowables using surrogate in this

section. The B-basis allowable is widely used as conservative estimation of composite

strength. The various methods for calculating B-basis allowable from replicate tests of a

single structural configuration are well documented. Combining such calculations from

several configurations in order to predict B-basis allowables at design configurations

that lack supporting validation tests is less well explored. Methods for improving B-basis

predictions at untested points by adding simulations (e.g. based on finite element

models) have received almost no coverage in the literature. This section explores the

options of predicting B-basis allowable with surrogate that is built on using experiments

and simulations.

We first explore two options. The first approach is regression tolerance limits,

which uses all replicates in the experiments to develop surrogates for mean values and

corresponding B-basis allowables. This implementation has been derived strictly for

independent and identically distributed (i.i.d.) normal random variables. The second

approach is direct regression, which calculates the B-basis at each test point by existing

single-point estimation and uses only this data to develop surrogates. Different single-

point estimation schemes of B-basis allowable could be adopted based on distributions.

Single-point estimation refers to the calculation based on replicates from a specific

configuration. The regression tolerance limits had noticeable bias for modeling the OHT

tests because the replicates do not satisfy the requirement of i.i.d. normal random

113

variable. In contrast, the direct regression was more flexible to associate with the

selected single-point estimation.

Then the direct regression approach was studied further by incorporating finite

element simulation. We combined simulations with experiments using multi-fidelity

surrogate. A few experiments were selected to correct simulations and predicative

capability of corrected simulation was examined at all the experiments. Fitting

experiments alone is preferable over multi-fidelity surrogate for the OHT tests except

when very few (1 and 2) experiments are available. We also found significant errors for

long-distance extrapolation.

5.6.1 Background

Composite materials have been extensively used for industrial design in past

decades. But large challenges remain for simulating strength of composite laminates

with complicated geometries or progressive damage. Strength of composite laminate is

usually associated with large variability, which makes the simulation even more

challenging. Therefore, substantial testing has to be conducted to measure variability

and estimate the resulting conservative allowables, such as B-basis, empirically. The B-

basis is a value that bounds the true (or population) 10th percentile value with 95%

confidence as shown in Figure 5-9. Computing B-basis allowables for fixed

configuration based on replicated tests have been explored, and many are listed in

Composite Materials Handbook 17. Bhachu et al. [106] compared several common

approaches for fatigue crack growth problems. Romero et al. [107, 108] tested the

performance of normal TI method, Pradlwarter-Schuëller kernel density method,

Johnson method, and non-parametric method.

114

Figure 5-9. Illustration of B-basis allowables based on replicates/repeated tests

When experiments are available for several structural configurations, it may be

possible to estimate B-basis allowable at untested points by interpolation or

extrapolation via a surrogate fit to the available data. Surrogate models, such as

polynomial response surfaces or kriging, are routinely used for approximating system

response as a cheap alternative of expensive simulations or tests [7, 30, 109, 110].

They have been used for fitting experimental data for mean strength [13, 77].

We compared two approaches for predicting B-basis allowable. The first

approach employed all the replicated tests and predicts B-basis allowable at untested

points based on rigorous derivation while assuming the experiments are independent

and identically distributed (i.i.d.) normal random variables. The second approach is

direct modeling of the B-basis allowable at each test point obtained from single-point

estimation and use only this data to develop surrogates. Different single-point

estimation schemes of B-basis allowable, which are estimations based on replicates at

a specific configuration, could be adopted based on distributions for direct modeling.

When it comes to predicting mean strength at untested points, it is common to

use simulations that combine finite element analysis (FEA) and failure theories,

Material

properties

Specimen configuration

Estimated 10th percentile

B-basis allowable: lower 95% C.I.

Tested properties with replicates

115

calibrated or corrected by experiments [103, 111]. To combine simulations and

experiments in surrogate it is common to use multi-fidelity surrogate model [90, 101,

112]. However, this is rarely done for predicting variability or conservative allowable at

untested points.

5.6.2 Predicting B-basis Allowable at Untested Points using Surrogates

5.6.2.1 Linear regression tolerance intervals

A polynomial response surface (PRS) can be fitted when test results are

available at several points, there is clear trend of polynomial dependence with a design

parameter and the standard deviation or coefficient or variation are approximately

constant [113]. PRS-based tolerance method calculates one-sided regression tolerance

limits with confidence level (1-𝛼) for a proportion P of the future replicates are within

certain limits (tolerance limist), and 𝛼=0.05, P=0.9 for B-basis. The details for calculating

B-basis allowables are given in Eq. (5-7) to Eq. (5-9) while assuming that strength

follow independent identical normal distribution.

1,ˆ ˆ( , )i i i iB basis x y y k (5-7)

* * *

;1

1,*

=n p i p

i

i

t n zk

n

(5-8)

2*

2

ˆ=

ˆ. .i

i

ns e y

(5-9)

where 𝑖 denotes the index of untested point (𝑥𝑖, 𝑦𝑖) for prediction, ��𝑖 is the PRS

prediction of the mean value at (𝑥𝑖, 𝑦𝑖), �� is the estimated noise level, calculated by the

root mean square error between PRS predictions and samples. 𝑡𝑛−𝑝;1−𝛼∗ (𝑥) is the (1-𝛼)

th quantile of a non-central t distribution with t degree of freedom and non-centrality

parameter 𝑥. n is sample size, in Eq. (5-8). 𝑛𝑖∗ is called the “effective number of

116

observations. 𝑧𝑝∗ is P-th quantile of standard normal distribution. ˆ. . is e y is the standard

error of PRS at ��𝑖, details for the derivation of PRS could be referred at [114].

5.6.2.2 B-basis allowables of Open-Hole-Tension tests

The statistics of OHT tests are provided in Table 5-6. The standard deviation

(Std) varies from 0.85 ksi to 5.42 ksi, and the coefficient of variation (CV) varies from

0.020 to 0.059. The large variations of STD and CV are against the assumption for

identical distribution. We then examined the distributions of OHT tests. Typical

histograms for OHT tests based on 18 replicates are shown in Figure 5-10. No clear

pattern could be observed for the OHT tests which was also supported by Kolmogorov–

Smirnov test (KS test).

Table 5-6. Statistics of OHT tests at the 12 experimental locations No. 1 2 3 4 5 6 7 8 9 10 11 12

R45 0.2 0.2 0.2 0.2 0.5 0.8 0.5 0.5 0.5 0.8 0.8 0.8

w/D 3 4 6 8 6 6 3 4 8 3 4 8

Mean(ksi) 87.35 91.46 97.44 100.12 71.07 50.30 58.11 67.62 72.39 37.68 42.68 51.84

Std(ksi) 3.55 5.42 5.20 4.90 2.84 1.68 2.48 2.70 3.28 1.34 0.85 1.33

CV 0.041 0.059 0.053 0.049 0.040 0.033 0.043 0.040 0.045 0.036 0.020 0.026

Figure 5-10. Typical histograms (out of 12 locations) based on 18 replicates from OHT

tests. Horizontal axis is the strength of the plate in ksi and vertical axis is the number of replicates.

Consider no clear pattern for the distributions, nonparametric approach was

adopted to compute the B-basis allowables (𝐵𝑛𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐) [115]. If the sample does not

117

demonstrate clear distribution pattern (e.g. due to limited number of samples), the

calculation can be performed according to order statistics. The strength of samples is

first ranked according to the order of magnitude. 𝑥(𝑟) denotes 𝑟𝑡ℎ largest data. 𝑥(1)

denotes the smallest one. B-basis could be computed based on Eq. (5-10) where k is a

factor depending on sample size. B-basis allowables for the OHT tests are calculated in

Table 5-7.

(1)

( )

( )

k

r

r

xB x

x

(5-10)

Table 5-7. B-basis allowables of OHT tests at the 12 experimental locations using non-parametric approach

No. 1 2 3 4 5 6 7 8 9 10 11 12

R45 0.2 0.2 0.2 0.2 0.5 0.8 0.5 0.5 0.5 0.8 0.8 0.8

w/D 3 4 6 8 6 6 3 4 8 3 4 8

Bnonpara

(ksi)

80.30 81.93 89.79 94.47 64.58 45.28 47.09 58.80 63.95 33.54 38.06 45.89

5.6.2.3 Predicting B-basis allowable of Open-Hole-Tension tests using surrogate

The non-identical and non-normal behaviors of replicates may lead to

unexpected predictions using PRS tolerance. But PRS were likely to provide reasonable

approximation considering the response of OHT tests were monotonic and close-to-

linear in the variable space. Therefore, we considered to estimate B-basis allowables at

single point first and then approximate B-basis allowables directly using PRS (Direct

PRS) in the variable space. PRS tolerance and Direct PRS were compared using the

robust test plan. We don’t have the true 10th percentile and distributions for the OHTS

tests. Therefore, 𝐵𝑁𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 were adopted as the quantities of interest for prediction.

The root-mean-square error here is essentially to illustrate the difference between Direct

PRS and PRS tolerance rather than accuracy evaluation. Relative RMSE for PRS

tolerance and direct PRS are given in Figure 5-11. Constant PRS was used for 1-2

118

experiments, linear PRS was used for 3-5 experiments and quadratic PRS was used for

more than 6 experiments. It is seen both approaches had significant error based on 1-2

experiments. The prediction accuracy increased drastically after using linear PRS and

gradually improved by introducing more experiments. The Direct PRS was always

closer to 𝐵𝑁𝑜𝑛𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑟𝑖𝑐 than PRS tolerance. Direct PRS could also be associated with

other point-estimation besides nonparametric approach for robust estimation of B-basis

allowables beyond i.i.d. normal distribution.

Figure 5-11. Overall accuracy (relative RMSE) for PRS tolerance and direct PRS with increasing number of experimental tests

5.6.3 Predicting B-basis Allowable using Multi-fidelity Surrogate

This section discussed predicting B-basis allowable using direct PRS assisted by

finite element simulations. Then the effect of sampling strategy was elaborated for multi-

fidelity modeling.

5.6.3.1 Simple framework of multi-fidelity modeling

Simulations at untested points may allow us to improve accuracy of the

prediction. Here we use failure predictions from finite element simulations based on

strains at the edge of the hole that substantially under-predicted the failure strength. We

adopt a simple framework of multi-fidelity surrogate (MFS) for correcting the finite

119

element simulations[11] (equivalent to the LR-MFS in section 3). MFS provides a fit

combining a small number of expensive high-fidelity data (here experiments) and a

large number of less expensive low fidelity data (here simulations) as function of a

design parameter vector x. The high fidelity data (B-basis allowables) is denoted as (

Hx , Hf ), while the low fidelity data is ( Lx , Lf ).

In the simple framework, the MFS is described with two surrogates ˆ ( )L

f x and ˆ( ) x

which represent a PRS fit to the low fidelity data and a discrepancy function or

correction, respectively,

ˆ ˆ ˆ( ) ( ) ( ) ( )H L

f f x x x x (5-11)

where ρ is a scalar and ( ) x denotes random noise due to experimental variability and

numerical instability. ρ and ˆ( ) x are obtained simultaneously through optimization as

shown in Eq. (5-12) and Eq. (5-13).

ˆ, ( )

ˆ ˆmin : ( ) ( )T

H H H H

x d x dx

(5-12)

ˆ ( )H L H H

f d x f (5-13)

We adopted a cubic PRS as ˆ ( )L

f x to fit all the simulations in the variable space.

For ˆ( ) x , constant PRS was used for 1-2 experiments, linear PRS was used for 3-5

experiments and quadratic PRS was used for more than 6 experiments.

5.6.3.2 Predicting B-basis allowable of using multi-fidelity surrogate

Finite element simulations were introduced to assist estimation of B-basis

allowables at untested point using direct PRS. The high-fidelity samples were B-basis

allowables at selected experiments estimated using nonparametric approach. Low-

fidelity samples were the finite element simulations. The MFS predictions were

120

evaluated using robust test plan and compared with direct PRS as shown in Figure 5-

12. MFS predictions dominated direct PRS for 1-2 experiment and less desirable for the

rest.

Figure 5-12. Overall accuracy (relative RMSE) for MFS and direct PRS with increasing number of experimental tests

5.6.3.3 Risky sampling strategy

The sampling strategy is critical for predictive capability of surrogates. Samples

corresponding to the worst three relative RMSEs using MFS are shown in Figure 5-13.

The selected samples were located on the borders and most domain suffered from

long-distance extrapolation and lead to significant errors.

Figure 5-13. Samples corresponding to the worst three sets of predictions using MFS. Solid points indicate selected experiments out of 12 for MFS.

121


In this work, we investigated the approaches for predicting experimental strength

of composite laminate using multi-fidelity surrogate model. Open-Hole-Tension tests

with two design variables were adopted for the benchmark study. We first examined the

prediction of mean experimental strength assisted using either simulation or historical

data from another material system. Then the multi-fidelity surrogate was used to

prediction B-basis allowables of composite laminate at untested points. The classical

regression tolerance intervals was intended for independent and identically distributed

(i.i.d.) normal variables. The replicates of OHT tests didn’t follow normal distributions

and had different level of variations. The violation of i.i.d. assumptions may lead to large

prediction error. We were motivated to develop PRS based on estimated B-basis

allowables using nonparametric approach (Direct PRS). The Direct PRS was flexible to

associate with various single-point estimation and proved to be more flexible than

regression tolerance intervals while estimating the nonparametric B-basis allowables.

Then we examined the prediction of B-basis allowables using multi-fidelity

surrogate (MFS). MFS predictions were much more accurate than direct PRS for 1-2

experiments and less desirable for the rest. We also found significant errors for long-

distance extrapolation. Hope this effort could guide the prediction for B-basis allowables

of composite laminates.

122

CHAPTER 6 A STRATEGY FOR ADAPTIVE SAMPLING WHEN SAMPLING COST IS VARIABLE IN

DESIGN SPACE

6.1 Background

Design optimization of mechanical systems usually requires extensive

simulations and tests to achieve desirable performance. Surrogate models are often

introduced as an efficient tool to approximate the response of mechanical systems from

scattered simulations/tests in the input variable space. An effective surrogate model

enables a cheap prediction of mechanical responses at a given design

configuration/input. Design optimization based on the surrogate model has been

adopted for numerous mechanical systems with significantly reduced simulation costs

or test period [30, 116]. Besides design optimization, surrogate models have been

applied for various engineering analysis. For example, advanced materials or innovative

designs are emerging which might lack effective theoretical models such as lattice

structure and composite material [11]. Surrogate models may serve as an empirical

model for the fundamental mechanics to enable multi-scale analysis. Surrogates model

is also the key technology for verification, validation and uncertainty quantification such

as the multi-fidelity surrogates which is developed to compensate for the discrepancy

between experiments and simulations [26, 58].

The accuracy of surrogate models strongly depends on the location and number

of samples (experiments and simulations). Methods to select samples are termed

sampling plan or design of experiment (DOE). Desirable DOE enables accurate

surrogate models while balancing the testing budgets. Adaptive sampling is a popular

DOE, which adds samples iteratively with updated surrogates. Various adaptive

123

sampling strategies were developed and proved effective for the approximation of

system response.

One major component of adaptive sampling is the uncertainty metric. Two types

of uncertainty metrics are mainly used for adaptive sampling: model-based prediction

variance and data-driven prediction variance. The prediction error at a point could be

estimated by prediction variance. The prediction variance quantifies the variation of the

potential system response, which is assumed a statistical distribution. The points with

large prediction variance imply risky approximation and could be improved by adding

more samples. The commonly used prediction variance is usually available for a few

types of surrogates by inherently assuming a distribution for the samples. For example,

the Gaussian process [49] assumes a multivariate normal distribution for the samples

and polynomial response surface assumes a normal distribution for the residual

errors[117]. Recent effort has been made to obtain the uncertainty metric for an

arbitrary surrogate based on data-driven approach such as cross-validation (CV). Jin et

al. proposed using the difference between the surrogates from leaving-one-out CV and

the surrogate using full samples to estimate the uncertainty of predictions [118]. Salem

et al. proposed the development of a series of surrogates from leaving-one-out CV.

Then a weighted scheme was used to fit the empirical cumulative distribution function

(ECDF) from the predictions of the surrogates. The prediction variance is then

generated from the ECDF to indicate the estimated uncertainty [119]. Xu et al.

associated surrogate prediction with Voronoi Diagram to determine the uncertainty of

surrogate predictions [120].

124

The current practice of adaptive sampling implicitly assumes that all samples

have the same cost, which is applicable for most applications. However, in some

applications, the cost to obtain samples may substantially vary in the input variable

space. For example, the computational resource of a computational fluid

dynamics(CFD) analysis relies heavily on the mesh density. The mesh density of CFD

analysis changes with Reynold number and Mach number. The effect of sampling cost

might be significant in the high-performance computing (HPC) scenarios. The HPC

architecture and algorithm might change in the input variable space for optimum

performance such as power consumption, an important part of sampling cost, for a

given CFD simulation task [121] [122].

This work explores a DOE strategy with varying sampling cost. We examine the

adaptive sampling for adding samples iteratively. A value function is proposed to

balance the varying sampling cost and expected gain for sampling. In addition, the

actual cost of a sample may not be known in advance. Therefore, two sets of surrogates

are developed: one for the target function and the other for the sampling cost. The value

function is defined as the ratio between the standard deviation of prediction and the

sampling cost estimated from the two surrogates. A new sample is then added at the

point with maximum value. As an initial attempt, the study is based on algebraic test

functions with algebraic cost functions for sampling. We evaluated the proposed

approach based on 1D and 2D algebraic functions, which are convenient for

visualization and discussion.

In the rest of the work, details on the standard adaptive sampling using Kriging

are introduced in section 6.2. The adaptive sampling strategy with varying cost is

125

proposed in section 6.3. Section 6.4 introduces the multivariate algebraic test function

and four algebraic cost functions. Section 6.5 investigates the numerical performance of

the proposed approach. Effect of different cost functions with increasing complexity and

different initial samples are discussed. This chapter on adaptive sampling is based on

the publication [12] co-authored with Chanyoung Park, Nam Ho Kim and Raphael T.

Haftka.

6.2 Adaptive Sampling using Surrogate

6.2.1 The Basic Steps in Adaptive Sampling

Adaptive/sequential sampling refers to the systematic procedure to add the

samples iteratively in order to achieve a cost-effective DOE. We consider here adaptive

sampling that relies on an uncertainty model for the surrogate, and which seeks to

improve the surrogate everywhere rather than toward a target such as the global

optimum. Basic steps for such adaptive sampling are provided in Figure 6-1. The initial

samples are first generated with a part of the sampling budget. Then a surrogate model

is developed with prediction variance estimating the prediction errors in the input

variable space. One additional sample newx is added at the point with maximum

prediction variance as shown in Eq. (6-1), where Var x is the prediction variance of a

surrogate, x denotes the point in the input variable space.

126

Figure 6-1. Basic steps of adaptive sampling for global approximation. One additional

sample is added per iteration.

arg max Varnewx x

x (6-1)

Other criteria are also available for adding samples. For example, expected

improvement might be used to search for the minimum function value among the input

variable space[123]. Multiple points can be added per iteration to take advantage of

parallel computation and reduce the number of iterations. The adaptive sampling and

modeling are iterated until reaching prescribed stopping criterion based on total cost or

estimated accuracy of surrogates[124]. In this work, Kriging is used for surrogate

modeling, and the prediction variance is based on its inherent assumption for

multivariate normal distribution. The stopping criterion adopted here is the total

sampling cost. Namely, the adaptive sampling procedure stops just before the total

sampling cost exceeds a prescribed sampling budget.

6.2.2 Kriging Surrogate

Kriging [117] with a constant trend is adopted in the work for surrogate modeling

as given in Eq. (6-2),

Initial design of experiments with partial budget for sampling

Develop surrogate model and prediction variance

Perform sampling/test at the point with maximum prediction variance

Update surrogate model

Stopping criterion (based on cost, estimated accuracy etc)

Done

No

Yes

127

f Z x x (6-2)

where is the mean value of the samples, Z x is assumed to be a Gaussian

process. The covariance function of the Gaussian process is set to be the squared

anisotropic exponential function and given in Eq. (6-3),

22

, ,

1

, expp

m i m j m

m

Cov Z Z x x

i jx x (6-3)

where ix and jx denote two points from the p dimensional space, 2 is the

process variance, m is the hyper-parameter with m=1,2,…,p. The parameters of Kriging

are obtained according to maximum likelihood estimation. The Kriging predictions

interpolate between samples. The Kriging is based on the assumption of multivariate

normal distribution and naturally provides the prediction variance at an untested point.

Kriging has proved effective to approximate response of various systems and been

used as a major surrogate for adaptive sampling.

6.3 Proposed Adaptive Sampling Strategy with Varying Sampling Cost

6.3.1 Proposed Methodology

The classical adaptive sampling strategy is revised to incorporate the effect of

varying sampling cost. For this purpose, a second surrogate C x is constructed to

approximate the sampling cost in the input variable space using adaptive sampling.

Then a value metric ˆ( )V x is defined as the ratio between the standard deviation of the

prediction x and C x as shown in Eq. (6-4). The ˆ( )V x is evaluated at a grid in the

input variable space. A grid with 100 p points is used in this work for the selection of newx .

The one additional sample newx is generated at the point with the maximum value ˆ( )V x as

128

in Eq. (6-5). The ˆ( )V x indicates the uncertainty to be reduced per cost. x and ˆ( )V x

decrease to 0 after adding the sample at x . The x is used to define ˆ( )V x

considering x is proportional to the confidence interval, which is the key measure of

estimated prediction error. The adaptive sampling with varying cost (AS-C) becomes

classical adaptive sampling (AS) when the cost function is a constant. The flowchart of

AS-C is summarized in Figure 6-2.

ˆˆ( )

ˆV

C

xx

x (6-4)

ˆargmax ( )newx Vx

x (6-5)

Figure 6-2. Flowchart of the adaptive sampling with varying sampling cost. One

additional sample is added per iteration at the point with maximum value to improve the global approximation.

Initial design of experiments with partial budget for sampling

Develop surrogate model to approximate the target function and

to approximate the sampling cost

Perform sampling/test at the point with maximum

Update surrogate models and

Stopping criterion based on total sampling cost

Done

No Yes

Compute the value function from and

129

6.3.2 Illustration of AS-C using 1D Algebraic Function

The popular Forrester function[125] is selected for illustration and preliminary

investigation of the proposed AS-C. The Forrester function is a one-dimensional

algebraic function on [0,1]x as given in Eq. (6-6):

2

6 2 sin 12 4f x x x (6-6)

A linear cost function is introduced in Eq. (6-7). The cost function is always

positive in the input variable space. The response of Forrester function and associated

cost function are visualized in Figure 6-3.

0.1C x x (6-7)

Figure 6-3. One-dimensional algebraic example Forrester function. A) Response of the

objective function and B) cost function for sampling.

For the Forrester function example, the initial design of experiment is 3 samples

at x=0, 0.5 and 1, respectively. The initial samples and surrogates are shown in Figure

6-4 (A) for the target function and Figure 6-4(B) for the cost function. It is clear that the

accuracy of f x is poor due to the sparse initial samples. The cost function has a

simple linear trend and C x matches the true cost function even with just 3 samples.

For a complicated cost function, C x is expected to be less accurate. Comparing with

A B

130

AS, the AS-C is based on two surrogates and expected to suffer from larger uncertainty

when the cost function has a complicated response.

Figure 6-4. Initial samples and surrogate prediction for the Forrester function. A) The

objective function and B) linear cost function

After the construction of surrogate from initial samples, one sample is added per

iteration using Kriging. The sampling budget is set to be up to 4.5. Final predictions

using AS resulted in 7 samples as seen in Figure 6-5(A). And final predictions using AS-

C ended with 8 samples as shown in Figure 6-5(B). Details of the sampling are

summarized in Table 6-1. Comparing with AS, the proposed AS-C enabled more

samples in the input variable space for a fixed budget. This observation might be

general for other test functions. Based on the maximum error, AS-C was more accurate

than the AS. In the following section, the AS-C is investigated further using a 2D test

function with different cost functions and different initial samples.

Table 6-1. Details of AS and AS-C for the approximation of Forrester function with a budget of up to 4.5 total cost

Number of samples Total cost Max error

AS 7 4.0 1.46

AS-C 8 4.1 0.87

A B

131

Figure 6-5. Surrogate predictions for adaptive sampling plans for Forrester function

example. A) AS and B) AS-C.

6.4 Multivariate Test Function and Algebraic Cost Functions

The complexity of cost function has a significant effect on the AS-C. In this

section, a 2D test function is selected for further evaluation of AS-C. Four different cost

functions are introduced to imitate the effect of varying sampling cost.

6.4.1 The Normalized Branin Function

The Branin function is selected as the target function to evaluate AS-C. The

function is in two-dimensional input variable space for convenient visualization and

inspection. The original Branin function is given in Eq. (6-8) and defined in

1 2[ 5,10], [0,15]x x . For the study of varying sampling cost, the input variable space

of Branin function and cost function is normalized within 1 2, [0,1]x x for consistency.

The function value of Branin function and cost function are normalized within

[0.1,1.1]f x for convenient quantitative study. The mapping/scaling of input variable

space and function value is performed through linear transformation. The response of

A B

132

normalized Branin function is shown in Figure 6-6. The Branin function mentioned in the

following analysis refers to the normalized Branin function.

2

2

2 1 1 12

5.1 5 1( ) 6 10 1 cos 10

4 8Braninf x x x x

x (6-8)

Figure 6-6. Response of Normalized Branin function in the input variable space

6.4.2 Four Algebraic Cost Functions

Four cost functions have been adopted to illustrate different cases of varying cost

in the input variable space. The four cost functions are constructed from a linear

function, an exponential function, the Rosenbrock function and the Damper function.

The linear function is shown in Eq. (6-9). The exponential function shown in Eq. (6-10)

increases more drastically in the input variable space. The Rosenbrock function is given

in Eq. (6-11) which has a response with moderate nonlinearity. The Damper function

[126] is given in Eq. (6-12) to imitate the resonance effect. The original input variable

space of the four test functions is provided in Table 6-2 to obtain the response. For the

study of varying sampling cost, the input variable space of cost functions is normalized

within 1 2, [0,1]x x for consistency with the target function. The value of cost function is

normalized within [0.1,1.1]f x for convenient quantitative comparison. 0.1 indicates

133

the basic sampling cost. This has the effect of creating a cost ratio of 1/11 for all the

cost functions between the cheapest and most expensive points. The response of

normalized cost functions are visualized in Figure 6-7.

1 2( ) 100 100 50LinearC x x x (6-9)

1 2( ) exp 5 exp 5 50ExpC x x x (6-10)

2 22

Rosen. 2 1 1( ) 100 1C x x x x (6-11)

2

2

22 2 2 2

2

2 2 2

1 1 2 1 2 1 1 2

11

( )

1 1 1 1 1 11 +4

Damper

xC

Rx x x x x x x

x (6-12)

Figure 6-7. Response of the four cost functions in normalized input variable space. A)

Linear cost function, B) Exponential cost function, C) Rosenbrock cost function and D) Damper cost function.

A B

C D

134

Table 6-2. Original input variable space for the four cost functions

Variable ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x

1x [0,1] [0,1] [ 1.5, 0.5] [1,1.1]

2x [0,1] [0,1] [2,3] [1,1.1]

6.5 Numerical Performance of AS-C

6.5.1 Evaluation Plan

The proposed AS-C was compared with AS using the Branin function. For AS,

only one surrogate was developed to approximate the target function and the true cost

function was used to evaluate the total cost of sampling. The stopping criterion of AS-C

and AS was based on the total sampling cost. The sampling procedure stopped just

before the total sampling cost exceeds 10. The sampling procedure was based on

Kriging with constant trend. The initial samples were generated by the Latin Hypercube

Sampling (LHS) with 5000 iterations and 6 initial samples. 100 sets of initial samples

were generated to account for the effect of data scarcity. The key factors of the

evaluation plan are summarized in Table 6-3.

Table 6-3. Key factors of the evaluation plan

Key factor Plan

Initial sample 100 sets of 6 samples generated from LHS

Cost function Four cost functions with varying complexity

Total sampling cost Evolution for R-squared and maximum error of AS and AS-C

at different total sampling cost

Prediction metric R-squared to measure overall accuracy in the input variable

space; max error to measure local accuracy in the input

variable space; area metric to measure the effect of total

sampling cost

Comparisons

between AS and

AS-C

(1) details of a typical case, (2) median performance with

different initial samples (3) variation with different initial

samples

135

The prediction accuracy of surrogate was examined at a matrix of 100 100 grid

jx , j=1,2,…,10,.000. For a given set of initial samples, two prediction metrics were

adopted to estimate the global and local performance of the surrogates, respectively.

The global accuracy was quantified by the coefficient of determination 2R as given in

Eq. (6-13), where jf x denotes the function value at the jth grid point, ˆjf x denoted

the surrogate prediction, f was the mean of jf x . An 2R of 1 indicates that the

surrogate perfectly fits all the function values. The local performance of surrogates was

quantified by the maximum error maxe in Eq. (6-14). The worst prediction/maximum error

is usually of critical interest in addition to the overall accuracy.

100 100 2

12

100 100 2

1

ˆ

1

j j

j

j

j

f f

R

f

x x

x f

(6-13)

max[1,10000]

ˆmax j jj

e f f

x x (6-14)

Figure 6-8. An illustration of the evolution of accuracy for adaptive sampling with

increasing total sampling cost

2R and maxe measured the prediction accuracy at a given cost. The relative

performance of AS and AS-C might alternate with increasing samples as illustrated in

Total sampling

cost

10

AS

AS-C

136

Figure 6-8, where initialc is the cost of initial samples. The performance of an adaptive

sampling scheme should be interpreted over the total sampling cost. The area metrics

2AR

and maxA were proposed to measure the relative performance of AS and AS-C as

given in Eq. (6-15) and Eq. (6-16), where H is the Heaviside step function defined in

Eq. (6-17). The evolution of adaptive sampling with increasing total sampling cost is

interpolated between samples. 2AR

measures the percentage of times when AS-C is

more accurate then AS over increasing samples with accuracy measured by 2R . maxA

measured the percentage of times when AS-C dominates AS over increasing samples

with accuracy measured maxe . An area metric equal to 50% indicates similar

performance of AS and AS-C. The median and variation of 2AR

and maxA with different

initial samples are also presented in the following sections.

2

10

2 2

-

A = 100%10

initial

AS C AS

c

Rinitial

H R c R c dc

c

(6-15)

10

max,AS max,AS-C

maxA = 100%10

initialc

initial

H e c e c dc

c

(6-16)

0, 0

1, 0

nH n

n

(6-17)

6.5.2 A Typical Case using the AS and AS-C

A typical case of the AS and AS-C to approximate the Branin function and linear

cost function is presented in this section. Six initial samples were generated from LHS.

One additional sample was added per iteration until the total cost of samples was about

to exceed 10. The initial samples and final samples are shown in Figure 6-9 using AS

137

and AS-C. The AS-C allocated more samples around the origin due to the cheap

sampling cost. Details of AS and ASC are summarized in Table 6-4. AS ended with 16

samples and the total cost was 9.5, AS-C ended with 18 samples and the total cost was

9.3. Adding one more sample at the location requested by the sampling algorithm would

exceed the total sampling budget of 10. However, it is clear that cheaper samples can

be added.

Table 6-4. Details for a typical DOE of AS and ASC for the approximation of Branin function and linear cost function with up to 10 total cost

2R maxe 2AR

maxA Total sampling cost Number of samples

AS 0.96 0.1 99.7% 100% 9.5 16

AS-C 0.99 0.05 9.3 18

Figure 6-9. Final samples for a typical DOE the approximation of Branin function with

linear cost function. The solid circles denote the initial samples. A) AS and B) AS-C.

Both AS and AS-C approximated the global response well ( 2R were 0.96 and

0.99, respectively) with AS-C being more accurate. The difference between maxe was

also significant (0.1 vs. 0.05) considering the function value varies between [0.1,1].

Evolution of adaptive sampling with increasing total sampling cost is shown in Figure 6-

10. The 2AR

was 99.7% and the maxA was 100%, namely AS-C was almost always

A B

138

better than AS with increasing total sampling cost. The performance of AS-C oscillated

at the beginning in Figure 6-10. This was mainly due to the large uncertainty of the

Branin surrogate from scarce samples. The cost function was estimated accurately

even with the initial samples.

Figure 6-10. Evolution of adaptive sampling with increasing samples for a typical DOE

for the approximation of Branin function and linear cost function. A) R-squared and B) max error.

6.5.3 Different Cost Functions and Different Initial Samples

The performance of adaptive sampling heavily depends on the quality of

surrogate and initial samples. AS and AS-C were evaluated using the four cost

functions. For the evaluation using each cost function, AS and AS-C were repeated 100

times with different initial samples. Therefore, 100 sets of 2AR

and maxA were generated.

The median value of the 100 2AR

and maxA are used to indicate the general performance

of AS and AS-C.

When 2AR

or maxA is equal to 50%, this indicates similar performance between

AS and AS-C with different total sampling cost. 2AR

and maxA larger than 50% indicate

higher accuracy of AS-C over AS-C. As seen in Table 6-5, 2AR

was close to 100 % for

A B

139

all four cost functions, therefore AS-C proved to be accurate more often, but not

overwhelmingly so. The lowest 2AR

came from the Damper cost function which has a

complicated response and lead to a large uncertainty of estimation. The values of maxA

were also high and indicated preference of AS-C over AS. Similarly, the maxA was lower

for Rosenbrock cost function and Damper cost function which having complicated

responses.

Table 6-5. Median value of area metrics 2AR

and maxA (in %) using 100 sets of initial

samples

Area metric ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x

2AR

93.2 96.9 93.4 89.9

maxA 98.3 93.9 77.8 84.9

Figure 6-11. The population of area metrics for the approximation of the Branin function

with four cost functions. A) Area metric for R-squared and B) area metric for max error.

The collection of 2AR

and maxA was visualized using boxplot in Figure 6-11. Almost

all the values of 2AR

and maxA were larger than 50% namely, AS-C was preferred for the

approximation of the Branin function. The boxplot also reveals significant variability of

A B

140

2AR

and maxA . 6 samples were generated using LHS using 5000 iterations. The number

of initial samples was based on heuristic experience. The performance of adaptive

sampling might be more robust by increasing the number of initial samples or

introducing other advanced schemes for the initial DOE.

AS-C might lead to more total samples than the AS. The total number of samples

was collected for the 100 sets of evaluations to validate this observation. Table 6-6

documents the median value of total samples for AS and AS-C from 100 sets of initial

samples. AS-C resulted in more samples in general. The difference between the

number of samples increased while more samples were used. Compare with AS, AS-C

adopted 3 more samples using the linear cost function and 8 more samples using the

damper cost function. The collection of the total number of samples was visualized

using boxplot in Figure 6-12. The AS-C lead to more samples than AS most time.

Table 6-6. Median values of total samples for AS and AS-C using 100 sets of initial samples

Sampling scheme ( )LinearC x ( )ExpC x Rosen.( )C x ( )DamperC x

AS 16.0 25.0 36.0 27.0

AS-C 19.0 31.0 44.0 33.5

Figure 6-12. The number of samples used for the approximation of the Branin function

with four cost functions using AS and AS-C

141


This work studies the effect of varying sampling cost on the design of

experiments. A strategy for adaptive sampling with varying sampling cost is proposed

(AS-C). Two surrogates are developed to approximate the target function and cost

function, respectively. A value function is defined as the ratio between the standard

deviation of prediction and estimated cost. The value function is used to indicate the

uncertainty for reduction per cost. A new sample is then added at the point with

maximum value. The proposed AS-C method was evaluated using 1D and 2D algebraic

functions and algebraic cost functions. The adaptive sampling stops just before the total

sampling cost exceeding the prescribed budget. Four different cost functions and 100

sets of different initial samples were produced to evaluate the AS-C. AS-C was

compared with standard adaptive sampling (AS) regarding R-squared, maximum error,

and evolution with increasing total sampling budget. For global approximation of the test

function in the whole input variable space, AS-C led to more samples than AS, and AS-

C was more accurate than AS in the large majority of DOEs.

142

CHAPTER 7 DESIGN SPACE SAMPLING BY EXPLORATION AND REPLICATION FOR

ESTIMATING EXPERIMENTAL STRENGTH

7.1 Background

Composite materials have been routinely used in aerospace applications due to

their outstanding capability to be tailored to specific load paths and conditions, resulting

in weight efficient designs. To achieve weight savings, effective and accurate

characterization of structural strength is essential. Significant progress has been

achieved on composite mechanics in past decades. Failure criteria have demonstrated

reasonable accuracy for predicting strength of benchmark structures. For example,

Whitney and Nuismer [103] proposed a failure criterion to predict strength of composite

laminates with a hole. Tsai and Wu [127] proposed a phenomenological material failure

theory which is widely used for anisotropic composite materials. The World Wide

Failure exercises [128, 129] summarized the effort to provide experimental data and

benchmark different modelling strategies for failure criteria of composites.

Various commercial software is available to simulate the response of composite

structures and used for routine analysis. However, each new material system, structural

configuration, and fabrication process requires a large, costly, and time-consuming

program to obtain simulation with reasonable accuracy [104, 130]. Innovative

approaches are being developed to simulate specific structures with certain configures

such as open-hole-tension test [131]. Large challenges still remain for simulating

strength of composite structures with complicated failure mechanisms and variability in

failure response due to progressive damage.

Industry has resorted to empirical approaches where testing has been the focus

for the characterization of structural failure when simulations are inadequate for

143

desirable accuracy or require excessive computational resources. The quantity of

interest for experimental analysis may be as simple as an averaged pass/fail criteria

based on a single load, or may be extended to include mixed mode loadings and more

involved statistical analysis such as strength allowables [132-134]. Handler et al. [135]

discussed the experimental procedures to develop a test database for composite

structures. Carlsson et al. [111] and the Composite Materials Handbook-17 provide in-

depth guideline for systematic experimental analysis.

Experimental results are usually obtained for an array of configurations for

different combinations of important design parameters. Then surrogate models (such

as polynomial response surfaces) are often fit to the data in order to estimate the

structural response over design space. Forrester et al. [125] discussed about

approximating noisy data with Kriging. Glaz et al. [136] adopted multiple surrogate

models for design optimization to reduce vibration of rotor blade problem. Chaudhuri et

al. [137] proposed an adaptive sampling for reliability-based design optimization. In this

context, a surrogate model serves three purposes. Firstly, a surrogate allows for the

statistical averaging or weighed averaging of multiple test results or replicates at the

same configuration (i.e. multi-batch testing). Secondly, it allows interpolation or

extrapolation of the failure response at locations in the design space where test data is

not available. Lastly, it yields a functional representation of a failure criterion which can

easily be incorporated in the design process.

Surrogate prediction consists of two parts: design of experiments and model

development. Design of experiments, which is also termed sampling strategy, refers to

the approach of selecting test configurations. Prediction accuracy of surrogate can be

144

significantly affected by sampling strategy. Wang et. al [116] and Mackman et. al [138]

reviewed popular approaches to generate design of experiments which had significant

effect on optimization results. For design of experiments, we may explore different test

configurations in design space or replicate the same test configuration multiple times.

We would like to examine the tradeoff between exploration and replication when limited

testing resources are available as shown in Figure 7-1. The replication strategy is

represented by 4 × 4 test matrix with 4 replicates at each point, while the exploration

strategy is represented by 8 × 8 test matrix with no replicates.

Figure 7-1. Replication strategy vs. exploration strategy using two design variables and

64 samples

Exploration interrogates the design space to find potential unforeseen failure

modes and unexpected responses, while replication at the same configuration

quantifies experimental scatter and improves accuracy at test points. In the replication

strategy, test variability is considered by averaging replicated tests, which may allow the

use of an interpolating surrogate that passes through the averaged data points. Without

replication, filtering out experimental noise is accomplished by a regression (i.e., least

squares fit) surrogate model in the exploration strategy. In the literature [77, 139] it was

found that exploration may be more efficient than replication in testing structural

elements for a given budget. Exploration is more likely to reduce surprises from un-

𝑥1

𝑥2

𝑥1

𝑥2

Trade-off

145

recognized failure modes, and it yields surrogate models that are more accurate when

applied to approximation at untested designs. However, the results obtained by [77,

139] were with manufactured data to the assumption that experimental strength was a

random variable with normal distribution, independent and identically distributed.

This work first presents a study of how this conclusion stands using real

experimental results produced by the National Institute for Aviation Research (NIAR) on

Open-Hole-Tension (OHT) tests [29, 140] according to ASTM standard [141]. These

OHT experiments were intended to investigate the impact of the coupon width to hole

diameter ratio on the failure of a composite panel. The failure mode and strength were

evaluated at four structural configurations with replicates. We examined the effect of

exploration vs replication on surrogate prediction based on subsets of experimental

results. Different designs of experiments were generated with the same total number of

tests.

The experimental data exhibited deviations from a normal distribution. So we

were prompted to consider other distributions that could be alternatives for

characterization of composite strength, such as the Weibull distribution [142] and

irregular distribution without analytical expression caused by batch to batch variability

[143]. Influence of strength distributions on the sampling methods was investigated

using one analytical function, composite laminate with highly nonlinear failure response.

Polynomial response surface (PRS) was selected for surrogate modeling due to its

robustness and excellence with approximating noisy data. The surrogate toolbox

provided by [72] was adopted to develop PRS prediction.

146

The work is arranged as follows: Section 7.2 introduces the OHT test

configuration and surrogate prediction of OHT tests. Surrogate models are developed

and compared based exploration strategy and replication strategy. In order to

compensate for experimental variability, we resampled subsets of experimental results

emphasizing either exploration or replication and repeated the modeling procedure. In

Section 7.3 we investigate the effect of distribution on sampling plans using one

analytical function and three synthetic distributions. Exploration and replication

strategies are compared. This chapter on the design of experiments is based on the

publication [13, 14] co-authored with Jaco Schutte, Waruna Seneviratne, Nam Ho Kim

and Raphael T. Haftka.

7.2 Exploration versus Replication Sampling Schemes for Open-Hole-Tension Tests

7.2.1 Open-Hole-Tension Tests

The Open-Hole-Tension (OHT) test was designed to investigate the effect of hole

size on the tensile strength of composite laminates. OHT tests were conducted

according to ASTM D5766 [141]. The standard test specimen geometry (w/D=6) is

shown in Figure 7-2. The width of the plate and diameter of the hole are denoted by w

and D, respectively. The plates with a hole were made of three distinct material batches

of Toray T700SC-12K-50C/#2510 plain weave fabric. The specimen laminate ply

orientations were [0/90/0/90/45/-45/90/0/90/0]s with a nominal thickness of 0.172 inch.

Tests were conducted in ambient laboratory conditions with an as-fabricated moisture

content. The tensile strength of the plate with a hole was calculated based on tensile

load and the nominal area (disregarding the hole). Table 7-1 details the test matrix at

four configurations, with w/D = 3, 4, 6, 8. Eighteen replicates were manufactured at

147

each configuration to quantify strength variability. Three pre-preg batches were adopted

to quantify batch to batch variability. All pre-preg batches were manufactured with the

same processing specification but at different dates. Loading rate of the tension tests

was 0.05 in./min. In all the tests, the dimension of “D” is the same, but the specimen

width “w” is different. The smaller lateral gauge may result in higher possibility of

manufacturing defects. Therefore, panels were examined using through-transmission

ultrasonic C-scan prior to machining specimens to guarantee the manufacturing quality.

Figure 7-2. OHT test specimen configuration and observed failure mode. A) Illustration

for the geometry of the composite plate with a hole and A) Typical tension failure mode

of composite laminate with a hole.

Table 7-1. w/ D ratio test matrix. 18 replicates are selected from three pre-preg batches

w (in.) D (in.) w/D Replicates No. of Specimens

0.75 0.250 3 3 x 6 18

1.00 0.250 4 3 x 6 18

1.50 0.250 6 3 x 6 18

2.00 0.250 8 3 x 6 18

Total = 72

Failure at lateral gage middle

F F

D w

A

B

148

For the OHT tests, only a single failure mode was observed, Lateral Gage Middle

and the strength response is likely to be smooth without jump by visual check as seen in

Figure 7-3. The strength of OHT tests was assumed to be a smooth curve perturbed by

random noise. We focused on the prediction accuracy of mean strength with varying

w/D ratio in this work.

7.2.2 Experimental Strength Results and Polynomial Fit

Physical models [111, 144] have been proposed to predict the strength of

composite plate with a hole. We limited ourselves to data-driven approaches only as a

valuable complement to design optimization when the physical models /simulations are

inadequate for desirable accuracy or require excessive computational resources. A

quadratic polynomial response surface (PRS) was selected to approximate the mean

value of failure strength after trying other fits. The PRS was developed using all 72

specimen strength values at four w/D ratios. The OHT strength of each laminate is

shown in Figure 7-3, along with a quadratic PRS fitted to the mean value of specimen

strengths and 95% confidence interval of the prediction variance. The OHT strength

increased gradually with w/D. Statistical information on the replicate OHT strengths

were summarized in Table 7-2, where iS denote tested strength of 18 replicates at

w/D= 3,4,6,8. Detailed experimental results are provided in Appendix D.

Table 7-2．Statistical properties of OHT strengths at given w/D with 18 replicates

Data

set

Mean

(ksi)

Standard

Deviation(ksi)

Coeff. of

variation

(%)

Range of strength

(ksi)

3S 59.64 3.52 5.90 [54.27, 66.33]

4S 63.22 2.79 4.41 [58.66, 68.07]

6S 66.53 3.01 4.52 [61.83, 71.89]

8S 69.27 3.04 4.39 [64.51, 76.24]

149

Figure 7-3. Experimental data and quadratic polynomial response surface fitted to the

data.

In Table 7-3, we can see that the maximum discrepancy between surrogate

prediction and mean strength was 0.8 % or less. Different from the interpolation

surrogate models (i.e., Kriging) that pass all the samples, the PRS filters noise and

preserves monotonicity in design space. Therefore, a quadratic PRS seemed

reasonably accurate to approximate OHT tests and was considered as the true function

value in the following analysis of prediction accuracy. The difference between this PRS

model and any other surrogate models based on subsets of OHT test results were

assumed to be from sampling strategies.

Table 7-3. Relative difference between surrogate prediction and mean strength using quadratic PRS

w/D=3 w/D=4 w/D=6 w/D=8

Relative difference (%)

0.4 0.8 0.5 0.2

7.2.3 Resampling Experimental Strength for Comparing Exploration and Replication

To compare sampling strategies for exploration and replication, we simulated

situations of paucity of data, by resampling partial experimental strength out of the 72

0 2 4 6 8 1045

50

55

60

65

70

75

80

w/d

Str

ength

(ksi)

Quadratic PRS

Estimation

Samples

95% Confi.

150

available experimental results without replacement. For the exploration strategy, we

sampled from all values of w/D, while for the replication strategy we left out one value of

w/D but had more replicates of the other three. This resampling approach was

motivated by bootstrapping [145], which is a nonparametric approach for statistical

inference. Strength of composite material is often associated with large variability (as

seen in Appendix D), the evaluation of sampling strategy should be interpreted in the

context of stochastic effect. Resampling procedure make the most use of limited

experimental strength and is likely to enable insightful understanding of sampling

strategy excluding stochastic effect. All the resampling plans were repeated 1000 times

using Monte-Carlo simulations to account for randomness. We compared surrogate

prediction for the cases of a total of 12 samples and of 24 samples.

Figure 7-4. Illustration for one set of resampled experimental strength representing

exploration or replication strategy using 12 points. A) Illustration of exploration and B) Illustration of replication strategy while omitting w/D=3.

For the exploration strategy, with 12 samples, 3 samples were randomly selected

from each of the 3 4 6 8, , ,S S S S sets. For the replication strategy, 4 samples were selected

from each of , ,i j kS S S sets, where i j k as shown in Figure 7-4. We skipped one of

w/D and used 4 replicates for each of the remaining three. Replication strategy was

A B

151

repeated four times by omitting one w/D in turn. With 24 samples, the same scheme

was used but with twice as many samples for each w/D.

A quadratic PRS was then constructed with the selected samples. The relative

error, ( )err x in Eq. (1) was computed at the four test configurations to quantify the

prediction accuracy,

( / ) ( / )

( / )( / )

i ii

i

f w D f w Derr w D

f w D

(7-1)

where ( / )f w D denoted predicted failure strength and ( / )f w D denoted the true strength

obtained from the previous PRS with all 72 samples. The root-mean-square error

(RMSE) given in Eq. (2) was used to quantify the overall performance of surrogate

prediction.

42

1

( / )

RMSE4

i

i

err w D

(7-2)

Table 7-4. Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens and 24 specimens. Mean values reported are based on 1,000 sets of samples

Sampling

strategy

Exploration Replication omitting specimen from specific w/D

w/D=3 w/D=4 w/D=6 w/D=8

12 samples 2.27 3.07 2.33 2.76 6.75

24 samples 1.57 2.22 1.63 1.89 4.70

Table 7-4 shows the accuracy of surrogate models using exploration and

replication strategies with 12 and 24 total samples. The RMSE of relative error for 1000

sets of samples were computed at four configurations, w/D =3, 4, 6, 8. It is obvious that

for this problem the exploration strategy was more accurate than the replication

strategy. When modeling surrogates using samples from (w/D=4, 6, 8) and (w/D=3, 4,

6), the estimation errors at w/D=3 and w/D=8 were significantly large due to the

152

necessity for extrapolation. Also, the largest RMSE errors among the different sample

selections occurred when extrapolation was required.

The influence of number of samples on prediction accuracy was also observed.

When twice as many samples are used to average the noise, the RMSE error is

expected to be reduced by the square root of 2. Indeed, the errors were reduced by a

factor ranging from 1.38 to 1.45. The error in surrogate prediction strongly depends on

variability of experimental strength, and the results in Table 7-4 show that exploration

filters out the noise in individual measurement as effectively as or more effectively than

replication. We further discuss the effect of strength variability on surrogate models in

the following section to make the study more general.

7.2.4 Identifying the Distribution Type of OHT Strength

In their previous work, Matsumura et al. [77] assumed that the distribution of

strengths at the same configuration was normal, independent and identically distributed

(i.i.d.). Independent and identical normal distributions are frequently used for theoretical

studies of stochastic effects. Here we examined to what extent this assumption holds for

OHT test data. We assume all iS follow the same type of distribution, but possibly with

different distribution parameters for each w/D. Three candidate distributions were

selected to characterize variability of strength: a normal distribution, two-parameter

Weibull distribution, which are commonly used for strength characterization, and a

uniform distribution for comparison purposes.

The iS distribution was analyzed using the Kolmogorov–Smirnov (K-S) test,

which is a non-parametric test to quantify the goodness-of-fit between a given

probability distribution and empirical distribution of samples [146]. The p-value of K-S

153

test indicates the probability that samples do not reject the hypothetical distribution. A

high p-value denotes high probability that samples are from the hypothetical distribution.

Effectiveness of K-S test strongly depend on the number of samples. Therefore, we

pooled iS together (i.e. from different w/Ds) for better estimation before applying the K-S

test. iS distributions had similar coefficients of variation (CV) as seen in Table 7-2, which

implied that it was possible to combine the four sets of iS together to query the

underlying distribution. Samples were transformed into the standard form, namely with

location parameter equal to zero and scale parameter equal to one. Eq. (3) was used to

standardize iS while assuming normal distribution or uniform distributions. Eq. (4) was

used to standardize iS while assuming Weibull distribution.

( )

( )

i i

i

mean

std

i

S SN

S, where i = 3, 4, 6, 8 (7-3)

( )

i

istdi

SW

S, where i = 3, 4, 6, 8 (7-4)

The p-values for pooled strengths were computed for the three candidate

distributions as shown in Figure 7-5. A 5% significance level was adopted to exclude

distributions that were less likely to occur (the red horizontal line in Figure 7-5).

Candidate distributions with a p-value lower than the significance level were rejected.

The figure showed that there was a high chance that OHT strength follows a normal

distribution, a lower chance that it came from the Weibull distribution, and the uniform

distribution was rejected. Probability plots for OHT strengths while assuming normal

distribution and Weibull distribution are provided in Figure 7-6 for further comparison.

The OHT strengths matched both distributions in general except at the tails. The

inconsistency at tails may be due to outliers in the experiments and the inability of

154

statistical distributions to approximate physical phenomenon, as well as the possibility

that the real distribution may be different. We also repeated the analysis of fitting of the

distributions considering samples from each batch independently by removing the bias

associated with batches. The P values using Normal distribution were around 0.9 for all

the three batches which were larger than or comparable with the P values using Weibull

distribution. Therefore, Normal distribution seemed desirable to approximate the

variation of the specimen strength.

Figure 7-5. K-S test statistic for OHT strength.

Figure 7-6. Probability plots for the fit of pooled OHT tests. A) Normal distribution and B)

Weibull distribution

Normal Weibull Uniform0

0.2

0.4

0.6

0.8

1

Candidate distribution

p-v

alu

e f

or

K-S

test

5% significance level

-2 -1 0 1 2

0.0050.01

0.050.1

0.25

0.5

0.75

0.90.95

0.990.995

Normalized Strength

Pro

babili

ty

Probability plot for normal distribution

p-value of Kolmogorov-Smirnov test:

Normal dist.: 0.936

10-0.05

10-0.01

100.03

0.01

0.05

0.1

0.25

0.5

0.75

0.90.950.99

0.999

Normalized Strength

Pro

babili

ty

Probability plot for Weibull distribution

p-value of Kolmogorov-Smirnov test:Weibull dist.: 0.49

A B

155

7.2.5 Effect of Between-batch Variability

We examined the distribution of OHT test results and concluded that the

assumption that structural strength followed i.i.d. normal distribution was not perfectly

satisfied as assumed in Matsumura et al. [14, 15]. The strength of composite laminates

could be associated with significant variability as seen in section 2.3. In practical

designs, specimens may be made from different pre-preg batches. Batch-to-batch

variability could be a major source of strength variation as reported by ASTM standard.

Notable batch-to-batch variability would invalidate the assumption of using i.i.d. normal

random variability to approximate structural strength. Figure 7-7 compares iS obtained

from different batches. Let /Dw

ib be the strength of specimen tested from batch i at w/D,

where i=1,2,3 and w/D=3,4,6,8. At given w/D, maximum values of min ( /Dw

ib ), max ( /Dw

ib

) and mean ( /Dw

ib ) were mostly from batch 2 indicating that specimens made from batch

2 tended to be stronger.

Figure 7-7. Statistical properties of specimen strength

3 4 6 850

55

60

65

70

75

80

w/D

str

ength

(ksi)

OHT test specimen strength

Batch 1

Batch 2

Batch 3

mean of batch 1

mean of batch 2

mean of batch 3

95% C.I.

156

We assumed /Dw

ib followed a normal distribution based on the results of the

previous non-parametric analysis. /Dw

ib was likely to have different mean value with

same w/D ratio. Welch's t-test was adopted to compare central values of two Gaussian

populations [147]. This test assumes that the two populations have normal distributions

and unequal variances. The p-value was used to indicate probability of the null

hypothesis that the two population means are equal (using a two-tailed test). A high p-

value represents high possibility that the two distributions under consideration are

equal. Table 7-5 documents the p-value for Welch's t-test between specimens made

from batch 1 and batch 2. The p-value at w/D= 3,6,8 are smaller than the 5%

significance level; i.e., the assumption that the two samples were from the same

distribution was rejected. The p-value at w/D=4 was 0.15, which is slightly larger than

significance level 0.05. Based on Figure 7-7 and Table 7-5, strengths of specimens from

same batch obviously shared systematic bias. The between-batch variability resulted in

a violation of normality assumption and i.i.d. condition even though iS was close to

normal distribution in overall.

Table 7-5. Statistics of welch's t-test between specimens from batch 1 and batch 2

w/D=3 w/D=4 w/D=6 w/D=8 /D

1

wb /D

2

wb /D

1

wb /D

2

wb /D

1

wb /D

2

wb /D

1

wb /D

2

wb

Mean

value

57.6

ksi

63.3

ksi

63.8

ksi

65.2

ksi

66.7

ksi

69.8

ksi

67.3

ksi

70.7

ksi

p-value 0.0004 0.15 0.01 0.04

It would be interesting to check the effect of between-batch variability on

prediction accuracy. This is an extended study on tradeoff between exploration of more

batches and replication ignoring the between-batch variability. We assumed each pre-

157

preg batch was equally important, and selected at least one sample from each batch to

conserve batch-to-batch variability. The sampling plan conserving batch-to-batch

variability is termed three-batch sampling and can be viewed as emphasizing

exploration, while random sampling may omit a batch and have more replicates in the

other batches. We compared surrogate models based on random sampling and three-

batch sampling. Table 7-6 shows the accuracy of surrogate models using exploration

and replication strategies with 12 specimens. The relative RMSE for 1000 sets of

samples were computed at four configurations, w/D =3, 4, 6, 8. It is seen that we benefit

from both types of exploration, using full set of w/Ds, and all three batches.

Table 7-6. Mean values of relative RMSE error (%) for surrogate prediction at w/D=3, 4, 6, 8 using 12 specimens (3 replicates from 4 configurations for exploration, 4 replicates from three configurations for replication). Mean values reported are based on 1,000 sets of samples

Sampling

strategy

Exploration Replication omitting specimen from w/D=i

w/D=3 w/D=4 w/D=6 w/D=8

Random

sampling

2.27 3.07 2.33 2.76 6.75

Three-batch

sampling

1.62 2.45 1.69 2.14 4.9

7.3 Influence of Distribution Type on Comparison between Sampling Strategies for an Analytical Model of a Composite Plate

The experimental results indicated that the strength of composite laminates was

not a perfectly independent and identically distributed Gaussian random variable.

Therefore, we examined the influence of distribution type on sampling strategies using

an analytical test function. While the results aligned with the previous numerical studies

showing that exploration was better than replication, the OHT case was relatively

benign compared to the numerical examples of Matsumura et al. [77]. First, there is only

one design parameter in the OHT test results; second, the failure stress varies smoothly

158

allowing accurate approximation with low-order polynomials; and third, the variation of

the distribution is relatively small. To obtain further insight, we investigate sampling

strategies and distribution types using one analytical function of the failure load of an

unnotched composite laminate with highly non-linear failure response. The geometry of

the plate and the dependence of the failure load on the two design parameters are

shown in Figure 7-8.

Figure 7-8. Configuration and failure response of composite laminate. α is a constant to

balance between horizontal and vertical loadings. θ is the orientation angle of lamina. A) Illustration of composite laminate and B) Response for critical failure load in the design space.

The composite laminate structure had three ply angles [0°/−θ°/+θ°]s which was

intended to have a more complex failure response as shown in Figure 7-8. The laminate

is subject to forces Nx and Ny along x and y axis defined by parameter α, where Nx

=(1−α)F and Ny = αF. There are two design parameters for this structure: θ ∈ [0°, 90°]

and α ∈ [0,0.5]. Table 7-7 details the material properties and strain allowables. The

strains are predicted by the Classical Lamination Theory (CLT). Critical failure mode

was due to ply axial strain. The failure loads were calculated analytically as in

Matsumara et al. [77].

00.1

0.20.3

0.40.5

20

40

60

80

2

4

6

8

10

12

x 105

Strength from multi-source variability

F

A B

159

Table 7-7. Parameter settings of composite laminate

Properties E1(GPa) E2(GPa) ν12 G12(GPa) Ply

thickness

(μm)

ε1allow ε2allow γ12allow

Quantities 150 9 0.34 4.6 125 ±0.01 ±0.01 ±0.015

The analytical failure loads are perturbed with synthetic noise to imitate

experimental variability. Three types of distribution were considered in this work: normal

distribution, Weibull distribution with heavy tail, and multi-source distribution to imitate

batch-to-batch variability. The numerical study focused on average performance of

surrogates based on the synthetic samples from Monte Carlo simulation. Variability of

the performance of surrogates are attached in Appendix D for reference.

7.3.1 Synthetic Noise using Three Types of Distributions

We considered three types of distributions to simulate strength variation. Normal

and Weibull distributions were considered as two popular distributions to approximate

the structural strength. The normal distribution was set to be ( ( , ), ( , ))N NN ,

where ( , )N was the mean value of structural strength from analytical function at

configuration ( , ) , and ( , )N was obtained assuming constant coefficient of

variation (CV) to be 0.05.

The Weibull distribution was first shaped to follow ( , )W a b with a heavy tail to

amplify the difference with the normal distribution, where the scale parameter a=2, the

shape parameter b=1.5. Then ( , )W a b was scaled through a linear transformation to

imitate the experimental variability of the analytical test function. Details of the Weibull

distribution were given in the Appendix C.

160

The multi-source distribution was proposed to imitate between-batch variability

based on findings from the OHT test results. We first generated 5 random values iM as

the mean strengths of 5 batches from 2(0,0.035 )N and fixed to imitate the between-

batch bias for the following analysis. The mean values were then scaled up through

linear transformation at different configurations ( , ) as shown in Eq. (5).

( , ) ( , ) ( , )i im f M f (7-5)

Synthetic samples were then generated from 2( , ), ( , ) 0.035i iN m m

assuming constant coefficient of variation (CV) to be 0.035. Typical probability density

functions (PDF) of three distributions are illustrated in Figure 7-9 for 8.5 ksi mean value

of strength. The synthetic specimen strength in the design space is assumed to come

from one of the five batches. The multi-source distribution should be skewed due to

between-batch for five specific distributions as seen in Figure 7-9 (C).

Figure 7-9. Three distributions with mean value to be 8.5 ksi. The mean values of the five synthetic batches are 8.1 ksi, 8.5 ksi, 8.8 ksi, 8.4 ksi, 9.0 ksi. A) Normal distribution, B) Weibull distribution and C) Multi-source distribution.

7.3.2 Fitting Strategy

A polynomial response surface (PRS) surrogate was selected because of its

robustness and good performance for predictions based on noisy data. Leave-one-out

cross validation was adopted to select an appropriate polynomial order for each

A B C

6 7 8 9 10

x 104

0

0.005

0.01

0.015

0.02Strength from Normal distribution

Strength

Em

piric

al P

DF

6 7 8 9 10

x 104

0

0.005

0.01

0.015

0.02

0.025

Strength

Em

piric

al P

DF

Strength from Weibull distribution

6 7 8 9 10

x 104

0

0.005

0.01

0.015

Strength

Em

piric

al P

DF

Strength from multi-source variability

161

individual set of samples. PRS assumed constant variance instead of constant

coefficient of variation. Therefore, function values of sampling points were preprocessed

using logarithmic transformation with base 10 to obtain close-to-constant variance.

As in Matsumura et al. [77], our test matrices ranged from 4 × 4 to 7 × 7 with

evenly spaced test points in order to investigate the effect of the density of matrix on the

accuracy of approximation. For each test matrix, we replicated the same test

configuration up to seven times. Each test matrix was generated and fitted 1000 times

to compensate for randomness of samples and obtain the average accuracy. Accuracy

of surrogate prediction was computed using a 20 × 20 matrix of test points (400 points

in total). Relative RMSE from Eq. (4) was adopted to measure overall accuracy of the

surrogate model.

7.3.3 Influence of Distribution Type on Sampling Strategies

We first developed surrogate models to approximate the analytical function

without noise for reference. Figure 7-10 (A) summarizes prediction accuracy of

surrogate prediction at all test matrix without noise. The typical orders for polynomial

response surface selected using cross validation were quadratic, cubic, quartic and

quantic respective for the four sets of test matrix. The composite laminate has a highly

nonlinear response (see Figure 7-8) that greatly benefited from the higher order

polynomials made possible by using more test points. This example corresponds to a

case that is more challenging to fit than the OHT case. The RMSE of relative error

reduced from 0.2 to 0.11 with increasing points for exploration. While modeling the

composite laminate strength, a typical PRS model based on 4×4 matrix was quadratic

and a typical PRS model based on 7 × 7 matrix was quantic. Note that since there is no

noise, RMSE remains constant through replication.

162

Figure 7-10. Mean value of RMSE for relative error based on 1000 sets of samples with

different distribution types. Number of samples indicates the total number of tests including 1~7 replicates for each test point. A) Without noise, B) Noise under Normal distribution, C) Noise under Weibull distribution and D) Noise under multi-source distribution.

Noise was then added to the data, and the influence of noise distribution on

sampling strategies was compared. Figure 7-10 documents the accuracy of surrogate

models for the composite laminate. From Figure 7-10 (B-D), we can observe a clear

advantage of exploration over replication with different distributions. The prediction

accuracy did not change much with increasing replicates for the same configuration set.

This was due to the highly non-linear response that implies that surrogate model error is

the major concern. Increasing replicates could mitigate mostly the effect of randomness

B

C D

A

163

(noise). Allocating samples for exploration enables high-order PRS for closer

approximation to the response. That is, model error was more significant than sampling

error corresponding to experimental variability for this example. We also examined the

influence of magnitude of noise on prediction accuracy in Appendix E. Noise with a

coefficient of variation varying from 0.05 to 0.2 only had a limited effect on predicative

capability of surrogates comparing with model-form.


To predict strength of many composite structural elements, it is necessary to

perform structural strength tests for a matrix of configurations as function of loading,

geometry and material properties. Then a surrogate fit allows interpolation or

extrapolation to be performed to obtain strengths of untested configurations. For a given

number of tests, we must balance between exploration, meaning testing many structural

configurations, and replication of each tested configuration to overcome the effects of

noise. Previous work has shown that exploration is more effective than replication

especially when the functional form of response is complicated, but it was based on

simulated test data assuming independent and identically distributed normal noise.

This work first tested the conclusion favoring exploration using actual test data of

strength of open hole composite plates as function of the ratio of plate width to hole

diameter, w/D. We proposed a resampling procedure based on subsets of experimental

strength to compare exploration and replication. The experimental data consists of 72

samples, indicating failure strengths for three batches of laminates with 6 replicates per

batch and four values of w/D. The two strategies were compared by using 12, 24 or all

72 samples. For each case, testing all four w/D configurations with fewer replicates was

compared with testing only three of the four with more replicates. Using 12 or 24

164

samples randomly drawn from 72 samples, the accuracy of fitted surrogates was

checked 1,000 times in a Monte Carlo sampling strategy. The results showed that

testing all four configurations with fewer replicates led to more accurate predictions than

testing only three configurations with more replicates, thus confirming the previous

simulated results.

We then investigated the major source of uncertainty for prediction, which is the

variability of strength. We found the distribution of OHT strength is not perfectly

independent and identically distributed normal distribution as commonly assumed. This

was likely due to systematic bias in strength for different prepreg batches. To generalize

our observation on sampling strategies, we investigated the influence of distributions

based on synthetic data using non-normal distributions and correlations. Synthetic

samples were generated based on the strength response for an un-notched composite

laminate using normal distribution, Weibull distribution and multi-source distribution.

Prediction of polynomial response surface(PRS) with fixed order/terms had similar

accuracy while increasing replicates for different distributions. Introducing more

replicates did not benefit prediction accuracy noticeably. An exploration strategy

enables a higher order PRS to be used, which is key to increase prediction accuracy for

complex structural behavior.

These results are limited to the accuracy of the mean value of the failure

strength. Future work will focus on estimating the tolerance limits, such as the B-basis

using surrogate models for designing structures with a limited number of available

experimental results. We are also searching for additional composite test types to

investigate the application of surrogate.

165

CHAPTER 8 FIELD APPROXIMATION USING SURROGATES AND PROPER ORTHOGONAL

DECOMPOSITION

In this chapter, we investigate the approximation of flow field using surrogate and

proper orthogonal decomposition (POD). The accuracy for predicting extreme quantities

of interest are evaluated using both field approximation and only surrogate. All the study

is demonstrated using a simple CFD analysis of drop falling implemented in Matlab.

8.1 The Drop Falling for Direct Numerical Simulations of Multiphase Flows

In order to investigate flow field approximation, we adopt a simple Navier-Stokes

solver to simulate a drop falling in a rectangular box [148]. The code for the simulation

was obtained from section 3.4 in [148]. A forward in time, centered in space

discretization is used. The density is advected by a front tracking scheme and a

stretched grid is used, allowing us to concentrate the grid points in specific areas. Four

snapshots at different simulated time have been taken in Figure 8-1 for illustration. The

rectangular box is set to be one-unit length vertically and horizontally. Initial radius of

the spherical drop is 0.15. Density of the flow and drop are set to be 1 and 2

respectively. Viscosity is a constant 0.01(water at 20°C). The viscosity of a fluid is a

measure of its resistance to gradual deformation by shear stress or tensile stress.

This CFD analysis seemed a feasible test function for studying field

approximation due to fast computation and reasonable complexity. Three variables

could be parameterized to apply the methods of snapshots: time, viscosity and density

difference. Complexity of the front increases as time goes.

166

Figure 8-1. The front and velocity field in a rectangular box at different times. A) 0 unit

time, B) 50 unit time, C) 100 unit time and D) 200 unit time.

8.2 Approximating Flow Field using Proper Orthogonal Decomposition and Kriging

Proper orthogonal decomposition(POD) has been proposed to associate with

Kriging surrogate model to realize field prediction [21]. POD is adopted to obtain major

modes for characterizing the flow field and Kriging is used to approximate mode

coefficients. Comparing with traditional projection methods for estimating mode

coefficients, this flow field prediction method is independent of the governing equations.

The numerical instabilities caused by viscosity and compressibility for POD-projection

A B

C D

167

method could be avoided. The POD decomposes the original flow field into a linear

combination of basic modes. We adopt K snapshots to solve basic modes. The modes

could be obtained through optimization procedure that constrains the modes to be

normalized and orthogonal. The scalar value representing flow field at each pixel or

grids make up a matrix ( ) ( )

( , )

k k

i jP p . Here, k denotes indices for snapshots (samples)

and (i, j) are the indices of the pixels, where i=1,2,…,I and j=1,2,3…,J. Each flow field is

a linear combination of M basis modes m and corresponding coefficients ( )k

mc as seen

in Eq.(8-1) , where m is mode index, M=K while all the modes are used for

approximation. The basic modes and mode coefficients ( )k

mc are obtained by minimizing

the root-mean-square difference between the snapshots and predictions as in Eq.(8-2).

The constraints for normalization and orthogonality are provided in Eq.(8-3) and Eq.(8-

4).

( ) ( )

1

Mk k

m m

k

P c

(8-1)

2

( ) ( )

1 1

minK M

k k

m m

k m

P c

(8-2)

subject to

( )

,

1 1

1I J

k

i j

i j

p

(8-3)

and

,

,

( ) ( ) ,

1 if

0 f

m p m p

m p

x x dx

i j

i i j

(8-4)

168

Since ( )m x is normalized only to reflect data pattern, ( )k

mc contains the amplitude

that the corresponding basic modes contribute to the reconstructed flow field. ( ) 2( )k

mc (so

called energy for velocity field) quantifies the contribution of mth mode to the kth flow

field. The POD mode spectrum is often used to describe the fraction (energy function for

velocity field) each mode captures of the total ensemble of snapshots.

( ) 2

1

( )K

k

m m

k

C c

(8-5)

1

/M

m m i

i

Frac C C

(8-6)

The mode order of the POD spectrum is defined according to the magnitude of

mFrac . Most variation of the flow field could be captured from the first few modes.

Coefficients of modes at untested points are approximated using Ordinary Kriging [72].

8.3 Approximating Extreme Scalars using Field Data and Scalar Data

Field approximation using Proper Orthogonal Decomposition (POD) is usually

adopted to build database of expensive CFD for future efficient design optimization.

Various extreme values are the most important quantity of interest for design

optimization. We compare approximation of fields and approximation of scalars

regarding to accuracy of extreme values. Pressure field is usually of critical interest for

engineering analysis. We examine approximations of minimum pressure for the drop

falling simulation with varying time. Considering symmetry of the flow field, only half

domain is adopted for analysis as seen in Figure 8-2.

169

Figure 8-2. Illustration of three snapshots at different times. Red point denotes location of minimum pressure

Four snapshots were adopted at different times for analysis. We extracted four

modes using POD from the snapshots. The mode coefficients at varying time were

estimated using Kriging as seen in Figure 8-3. The minimum pressure was then

obtained from the estimated fields. We also developed a Kriging surrogate to estimate

the minimum pressure directly based on the four samples. The prediction accuracy is

shown in Figure 8-4. For approximation of scalar using Kriging, root-mean-square error

(RMSE) of relative difference with true function is 3.8%, maximum relative difference is

10.3%. For approximation of scalar using POD and Kriging, RMSE of relative difference

with true function is 15.0 %, maximum relative difference is 34.7%. It is obvious that

direct approximation of scalar using Kriging is more accurate.

170

Figure 8-3. Kriging prediction of mode coefficients with varying time. A) 1st mode, B) 2nd

mode, C) 3rd mode and D) 4th mode.

Figure 8-4. Predicting minimum pressure from surrogate estimation and field

approximation. Blue line is true value, red line is approximation. A) Surrogate estimation alone, B) Field approximation using Kriging and POD.

A B

C D

A B

171

8.4 Predicting Border of Multiphase Flow

Besides the minimum pressure, we explore the approach to improve POD

prediction further regarding to approximating other extreme quantities. The border of

multiphase flow is one quantity of interest for engineering analysis. The CFD analysis is

based on incompressible flow, uniform viscosity for the two flows. We transformed the

density field of multiphase flow to be binary value 0 and 1 to indicate different phases.

We then performed field prediction using binary snapshots based on two variables:

viscosity and time for transient simulation. Parameter settings for the CFD analysis are

provided in Table 8-1.

Table 8-1. Parameter settings

Parameters Viscosity Simulation

Time

CFD

grids

Pixels Density

for drop

Density for

flow

Setting 0.005~0.015 150~250 32×32 656×875 2 1

Most variation of the flow field could be captured from the first few modes. We

adopted 99 snapshots and the first 50 modes are adopted for prediction. Contribution of

first 4 modes are summarized in Table 8-2. Mode 1 is most significant compared with

the rest.

Table 8-2. Fraction of contribution for first 4 modes to ensemble snapshots

Coefficients of modes at untested points were approximated using Ordinary

Kriging as shown in Figure 8-5. The coefficients illustrated smooth response and less

likely to have noise.

mFrac m=1 m=2 m=3 m=4

Value 97.5% 0.84% 0.59% 0.22%

172

Figure 8-5. Coefficients of first 4 basis modes in variable space using Kriging. A) Mode

1, B) Mode 2, C) Mode 3 and D) Mode 4.

We proposed to use filter and projection [149] for post-processing of field

prediction. A field value at (i,j) was updated by the average field value within certain

distance from (i, j), namely filter domain as seen in Eq.(8-7). We used a square domain

with side length to be 2R as filter domain. ( )

,

k

f i jp denotes filtered field value at (i,j).

Projection approach is essentially a step function as seen Eq.(8-8). is the threshold

value for binary predictions. and R were empirically estimated based on cross-

validation.

( )

,

1 1( )

,

R RI Jk

i j

i jk

f i j

R R

p

pI J

(8-7)

B A

C D

173

( )

,( )

, ( )

,

1, if

0, if

k

f i jk

pi j k

f i j

pp

p

(8-8)

4 validation points and 99 samples are shown in Figure 8-6. The graphical

complexity increased with reduced viscosity and increasing time. Comparison between

POD prediction and simulation are summarized in Figure 8-7. POD predictions were

almost the same as simulation regarding to drop fronts. We may reduce the number of

samples for this specific case considering the excellent prediction accuracy.

Figure 8-6. Illustration for validation points, samples and variable space

Increasing graphical complexity

Incr

easi

ng

grap

hic

al c

om

ple

xity

(1)

(2)

(3) (4)

174

Figure 8-7. Comparing simulation and POD prediction at 4 validation points. A) Point 1,

B) Point 2, C) Point 3 and D) Point 4.


We have examined the numerical performance of field approximation using POD

and Kriging surrogates. Flow fields are predicted as a linear combination of selected

basis modes generated by POD. The mode coefficients are approximated using Kriging

models with varying variables. We compared predicative capability for extreme quantity

of interest using field approximation and scalar approximation based on a simple CFD

simulation. Field prediction is found to be less accurate with minimum pressure from the

flow field comparing with scalar prediction.

We have also investigated the prediction of border between multiphase flow. The

border between immiscible liquids is important to investigate physical interaction. We

found the approximation using POD prediction is blurred as grey domain. Therefore, we

D C

A B

175

proposed to use filter and projection approach for post-processing. These approaches

proved to be effective to refine border prediction based on numerical tests.

In the future, it should be helpful to investigate the effect of factors on

approximation schemes. Several major factors have been identified from previous

study, such as number of snapshots, number of modes, design of experiments, number

of variables.

176

CHAPTER 9 CONCLUSION

For design optimization and uncertainty quantification, we need to examine large

number of candidate configurations. This is commonly done by doing simulations and/or

experiments at a number of design points and approximating at untested points. Usually

a single approach is used to combine all the information in order to make a prediction.

The objective of this work are to check on combining multiple ways to predict by

integrating various data sources and numerical models. A series of approaches have

been proposed and elaborated in this work.

Firstly, this work proposes to develop surrogates from different data sets. Multiple

predictions based on different data sets are assumed to be independent and combined

using Bayesian inference to increase reliability. We apply the idea of multiple data sets

to predictions of a single inaccessible point rather than estimating the whole variable

space. Data sets are sampled along different lines converging to the target point. One-

dimensional models are then performed individually and combined at target point.

Converging lines proved to be much more reliable and accurate than multi-dimensional

model based on the modeling of a physical function (drag coefficient function).

Secondly, the linear regression multi-fidelity surrogate (LR-MFS) was proposed

to combine dataset with different fidelity, especially for high-fidelity with noise. The linear

regression is commonly used, balanced between accuracy, cost and simplicity. The LR-

MFS is less likely to overfit noise by limiting the number of parameters. The LR-MFS is

derived from standard linear regression with analytical solutions to obtain parameters to

guarantee global optima. The treasure of linear regression techniques could be applied

directly to LR-MFS, such as prediction variance, optimal DoE. The LR-MFS was

177

demonstrated using polynomial response surface (PRS) in this work. The proposed LR-

MFS was first compared with three Bayesian frameworks using a benchmark dataset

from the simulations of a fluidized-bed process. The accuracy of Bayesian frameworks

varied significantly, and the LR-MFS was comparable to the best GP-based approach.

The effect of multiple low-fidelity datasets was also discussed. The three-level fidelity

MFS was inferior to the one-level MFS for limited number of high-fidelity dataset. Then

the LR-MFS was evaluated to approximate a non-linear numerical test function with

different sampling plans. The LR-MFS bested Co-Kriging for 55%~63% cases with

increasing number of high-fidelity(HF) samples. For both examples, the LR-MFS

proved to be better than fitting only HF samples, and robust for HF samples with

different level of noise. LR-MFS could avoid the error for fitting LF data as demonstrated

for the numerical example. The LR-MFS heavily relies on the model form of the

discrepancy function. In the future, the selection of basis terms will be examined using

existing schemes such as stepwise regression. After evaluation using the benchmark

datasets, the proposed scheme has been applied for two engineering applications: (1)

behavioral emulation of exascale computer architectures; (2) predicting strength of

composite laminates with a central hole.

Thirdly, this work studies the effect of varying sampling cost on the design of

experiments. A strategy for adaptive sampling with varying sampling cost is proposed

(AS-C). Two surrogates are developed to approximate the target function and cost

function, respectively. A value function is defined as the ratio between the standard

deviation of prediction and estimated cost. The value function is used to indicate the

uncertainty for reduction per cost. A new sample is then added at the point with

178

maximum value. The proposed AS-C method was evaluated using 1D and 2D algebraic

functions and algebraic cost functions. The adaptive sampling stops just before the total

sampling cost exceeding the prescribed budget. Four different cost functions and 100

sets of different initial samples were produced to evaluate the AS-C. AS-C was

compared with standard adaptive sampling (AS) regarding R-squared, maximum error,

and evolution with increasing total sampling budget. For global approximation of the test

function in the whole input variable space, AS-C led to more samples than AS, and AS-

C was more accurate than AS in the large majority of DOEs.

Fourthly, this work explores the trade-off between exploration and replication for

the design of experiments. Previous work has shown that exploration is more effective

than replication especially when the functional form of response is complicated, but it

was based on simulated test data assuming independent and identically distributed

normal noise. In this work, we first tested the conclusion favoring exploration using

actual test data of strength of open hole composite plates as function of the ratio of

plate width to hole diameter, w/D. We proposed a resampling procedure based on

subsets of experimental strength to compare exploration and replication. The results

showed that testing all four configurations with fewer replicates led to more accurate

predictions than testing only three configurations with more replicates, thus confirming

the previous simulated results. Prediction of polynomial response surface(PRS) with

fixed order/terms had similar accuracy while increasing replicates for different

distributions. Introducing more replicates did not benefit prediction accuracy noticeably.

An exploration strategy enables a higher order PRS to be used, which is key to increase

prediction accuracy for complex structural behavior.

179

At the end, we investigated approximation of flow field and approximation of

scalar quantities. Field data is commonly encountered in engineering analysis (e.g. flow

field) and usually contains rich information. Reconstructing field data at varying

conditions has been extensively studied for effective design optimization. The field data

is assumed to be a linear combination of major modes. Proper orthogonal

decomposition (POD) is adopted here to extract major modes of field data. The mode

coefficients are then approximated using several surrogate models. We found that field

prediction using POD with surrogates is less capable to preserve extreme value such as

maximum pressure from a CFD analysis. In contrast, a surrogate model could

approximate maximum pressure accurately in variable space. Therefore, approximation

schemes for field should depend on the quantity of interest.

180

APPENDIX A ANALYTICAL EXPRESSIONS FOR DRAG COEFFICIENT

The analytic expression for drag coefficient function was cited from [55]. The

range of applicability for cf is 5Re 2 10 , M 1.75 , and Kn 0.01 , where Kn denotes

Knudsen number. 5Re =2 10 is the subcritical Reynolds number above which the

attached boundary layer becomes turbulent. Attention is limited to continuum flows,

namely, Kn<0.01. DC is given in Eq. (A-1) for subcritical ( crM M 0.6 ), supersonic (

1<M 1.75 ) and intermediate ( crM <M 1 ) Mach number regimes.

, , ,

,

,sup

(Re,M)

M(Re) (Re) (Re) if M M

M

(Re,M) if M M 1.0

(Re,M) if 1.0 M 1.75

cr

D

D std D M D std cr

cr

D sub cr

D

C

C C C

C

C

(A-1)

where

1

0.687

, 1.16

24 42500(Re) (1 0.15Re ) 0.42 1

Re ReD stdC

(A-2)

1

0.684

, 0.669

24 483(Re) 1 0.15Re 0.513 1

Re RecrD MC

(A-3)

,sup ,M 1 ,M 1.75 ,M 1 sup(Re,M) (Re) (Re) (Re) (M,Re)D D D DC C C C

(A-4)

0.813 1

,M 1 0.793

24 3550(Re) 1 0.118Re 0.69 1

Re ReDC

(A-5)

0.867 1

,M 1.75 0.634

24 861(Re) 1 0.107Re 0.646 1

Re ReDC

(A-6)

33

,sup

sup ,sup

1 ,sup ,sup1

log Re(M,Re) (M)

j

i

i j i i jj

Cf

C C

(A-7)

2 3

1,sup (M) 2.963 4.392M 1.169M 0.027M 0.233exp (1 M) / 0.011f (A-8)

2 3

2,sup (M) 6.617 12.11M 6.501M 1.182M 0.174exp (1 M) / 0.01f (A-9)

2 3

3,sup (M) 5.866 11.57M 6.665M 1.312M 0.350exp (1 M) / 0.012f (A-10)

1,sup 2,sup 3,sup6.48; 8.93; 12.21C C C (A-11)

181

,sub ,M ,M 1 ,M sub(Re,M) (Re) (Re) (Re) (M,Re)

cr crD D D DC C C C (A-12)

33

,sub

sub ,sub

1 ,sub ,sub1

log Re(M,Re) (M)

j

i

i j i i jj

Cf

C C

(A-13)

2 3

1,sub (M) 1.884 8.422M 13.70M +8.162Mf (A-14)

2 3

2,sub (M) 2.228 10.35M 16.96M +9.840Mf (A-15)

2 3

3,sub (M) 4.362 16.91M 19.84M -6.296Mf (A-16)

1,sub 2,sub 3,sub6.48; 9.28; 12.21C C C (A-17)

182

APPENDIX B ONE-DIMENSIONAL FUNCTION EXTRAPOLATION USING SURROGATES

Surrogate modeling is commonly used to estimate function values efficiently and

accurately at unsampled points. The estimation procedure is called interpolation when

target points are inside the convex hull of sampled points while extrapolation otherwise.

This work explores one-dimensional deterministic function extrapolation using

surrogates. We first define a new error metric, relative average error, for quantifying

overall performance of extrapolation technique. Ordinary Kriging and Linear Sheppard

surrogates proved to be safer on several challenging functions than polynomial

response surfaces, support vector regression or radial basis neural functions. This

reflected that prediction of these surrogates converge to mean value of samples at

points far from samples.

It’s commonly recognized that long-range extrapolation is likely to be less

accurate than short-range extrapolation. Two kinds of effective extrapolation distance

are defined to indicate how far we can extrapolate test functions. We propose using the

correlation between the nearest sample and the prediction point given by Ordinary

Kriging as indicator of effective extrapolation distance. The relationship between

effective extrapolation distance and corresponding correlation over the distance is

examined by several test functions. A large value of correlation is associated with

effective extrapolation distance.

B.1 Background

In surrogate modeling, it is common to sample a function f at several points and

fit them with an explicit function in order to estimate the function at other points[150].

This is often required for optimization or reliability analysis in which thousands of

183

function evaluations are common, and each sample often means expensive simulation

or costly or time-consuming experiment.

Function estimation is defined as interpolation when target points are inside the

convex hull of sampled data points and extrapolation otherwise. For one-dimensional

samples, convex hull is the smallest interval containing the samples. Although many

research results have been reported on the accuracy of surrogate modeling, most

focused on the prediction accuracy in interpolation. Extrapolation is usually associated

with large estimation errors [46] and commonly encountered in three situations:

1. Sampling pattern such as Latin Hypercube sampling is adopted, which typically does not sample at or near the boundaries of sampling region.

2. For function estimation in high-dimensional space, we usually cannot afford enough points to avoid extrapolation. For example, in twenty-dimensional box, more than million points (220) are required.

3. Region of interest changes after samples are collected[36].

Besides the above conditions, extrapolation may be useful when the target points

cannot be sampled via simulation or experiment due to the need to know future events,

inadequacy of simulation software or high cost to perform experiments[151]. As a first

step to explore effective extrapolation scheme in engineering problems, attention is

limited here to one-dimensional (1D) function extrapolation.

This work investigates general issues on extrapolation using surrogates. Section

B.2 illustrates possible behavior of surrogates for extrapolation and potentials of

extrapolation using surrogates. Section B.3 proposes an error metric designed for

extrapolation. Extrapolation of a few examples using five surrogates are compared in

Section B.4. Section B.5 discusses the possibility of estimating extrapolation distance

using surrogates.

184

B.2 Possible Behaviors of Extrapolation using Surrogates

Estimation of several analytical functions using ordinary kriging is presented to

illustrate possible behavior in the extrapolation region, which is assumed here to be

inaccessible. Figure B-1 presents three 1D functions which have different function

behavior between the accessible and inaccessible domains. These three functions are

estimated using Ordinary Kriging in Figure B-2. It is seen extrapolation results

approximate the true function value surprisingly well. Since extrapolation in Kriging is

based on correlation between function values based on distance, we evaluate first the

correlation between inaccessible domain and accessible domain.

Figure B-1. Three one-dimensional test functions for extrapolation. A) Forrester

function, B) ( ) sin( )f x x and C) 2( ) ( 6) /13 2f x x . Vertical line denotes

border of inaccessible domain which is on the left and accessible domain which is on the right.

Figure B-2. Extrapolation of three one-dimensional test functions. A) Forrester function,

B) ( ) sin( )f x x and C) 2( ) ( 6) /13 2f x x . Vertical line denotes border of

C B A

0 0.1 0.2 0.3 0.4 0.5-1

0

1

2

3

41D Forrester function

x

y

-2 0 2-1

-0.5

0

0.5

1sin(x)

x

y

0 2 4 6 8 10 12-1

-0.5

0

0.5

1

1.5

2Extrapolation of -(x-6)2/13+2

x

y

A B C

185

inaccessible domain which is on the left and accessible domain which is on the right.

Denote by r the ratio of length of extrapolation distance to that of accessible

domain. As expected, extrapolation error increases with r as shown in Figure B-3 for the

log function. In this work, the length of inaccessible domain and accessible domain are

set to be equal, which would typically be considered as long-range extrapolation.

Figure B-3. Extrapolation of the logarithmic function ( ) log( )f x x . A) Extrapolation of the

response using Kriging and B) absolute error for extrapolation.

Another factor determining the extrapolation accuracy is how close the samples

to the boundary are. Figure B-4 illustrates that by using samples close to the boundary.

Extrapolation accuracy improves obviously after shifting samples close to inaccessible

domain.

The examples and discussion above identified three issues, which are correlation

between inaccessible domain and accessible domain, relative extrapolation distance

and absolute extrapolation distance, which are important for extrapolation research.

B

0 2 4 6 8 10 12

-4

-2

0

2

4

6

8

x

f(x)

Extrapolation using Kriging

Border

True function

Extrapolation results

Samples

A

0 2 4 60

1

2

3

4

5

x

Absolu

te e

rror

Absolute error for extrapolation

00.20.40.60.81r

186

Figure B-4. Extrapolation of logarithmic function from different sampling domain

( ) log(12.8 )f x x . A) Extrapolation from the full sampling domain, B)

absolute error for extrapolation from the full sampling domain, C) extrapolation from the reduced sampling domain and D) absolute error for extrapolation from the reduced sampling domain.

B.3 An Error Metric for 1D Extrapolation

We denote extrapolation result by ˆ( )f x . The performance of one-dimensional

extrapolation technique may be quantified by various error measures. Relative error

cr ( )e x is

cr

ˆ ( ) ( )( )=

( )

f x f xe x

f x

(B-1)

cr ( )e x may be misleading when the function changes sign. So one often uses the

range of the function instead of the function value for normalization. In addition, for

C

B A

0 2 4 6 8 10 12

0

1

2

3

4

x

f(x)


Border

True function


Samples

0 2 4 6 8 10 12

0

1

2

3

4

x

f(x)


Border

True function


Samples

0 2 4 60

0.02

0.04

0.06

0.08

0.1

0.12

0.14

x

Absolu

te e

rror


0 1 2 3 4 5 60

0.5

1

1.5

2

x

Absolu

te e

rror


D

187

extrapolation, we are often interested in the error in predicting change from a boundary

point bx . This error, ( )ece x is:

ˆ ( ) ( )

( )= 1( ) ( )

bec

b

f x f xe x

f x f x

(B-2)

For example, if based on this year’s record we predict that gas prices will rise

from $4/gallon today to $5/gallon a year from now, and they rise only to $4.50, we may

consider ( )=100%ece x this as rather than cr ( )=11%e x error. Of course, this error measure

will fail if the change in the function is near zero, and for this case an alternate relative

error is defined as ( )re x . Denoting ( )range f as the range of true function in the

extrapolation domain, ( )re x is:

ˆ ( ) ( )

( )=( )

r

f x f xe x

range f

(B-3)

( )re x is used in the following for extrapolation comparisons. We may also use

1 1( ( )) max( ) min( )range f x f f , where 1f denotes function value in the range between

extrapolation point x and the closest sample. In order to evaluate the overall

performance of extrapolation, we use the average error AE in extrapolation domain:

( )b

t

x

r

x

b t

e x dx

AEx x

(B-4)

B.4 Surrogate Comparison for Extrapolation

Surrogates have different performance for interpolation and extrapolation. We

test the performance of five popular surrogates: Ordinary Kriging, Polynomial response

surface (PRS), Radial basis neural network (RBNN), Linear Shepard (moving least

188

squares), and Support vector regression (SVR). Four test functions were extracted from

well-known multidimensional functions taken from [5] and shown in Figure B-5.

Figure B-5. Four test functions for surrogate selection. Domains close to origin are

inaccessible domain. A) Forrester function, B) 1D Braining-Hoo function, C) (c) 1D Ackley function and D) 1D Gramacy & Lee function.

The number of sampling points along each line is 6. Sampling points are

generated using Latin hypercube sampling with 5 iterations, which introduces

randomness in the position of the samples. To average out the effect of the positions of

the sampling points, 30 sets of samples are generated for each test function, and the

mean value of AE for all the sample sets are computed. The extrapolation results of

test functions are listed in Table B-1. All the surrogates except Ordinary Kriging and

Linear Shepard can generate huge errors. Kriging and Linear Shepard do not

extrapolate well, but do not incur huge errors. This is because function estimation using

A B

C D

189

Ordinary Kriging and Linear Shepard are weighted sum of samples and they eventually

revert to the mean of the samples. Ordinary Kriging was then selected for further testing

in the next section.

Table B-1. Average AE for extrapolation of test functions using 30 sets of samples

Surrogate models

Test functions Average AE of Forrester et al.(2008)

function

Average AE of one-dimensional

Branin-Hoo function

Average AE of one-dimensional Ackley function

Average AE of one-dimensional Gramacy & Lee (2009) function

Kriging 0.80 0.13 0.31 1.44 Quadratic PRS 13.36 0.97 2.01 6.13

Cubic PRS 7.54 0.53 4.02 60.22 Quartic PRS 93.62 3.18 8.67 442.14

RBNN 2322.5 0.2 0.54 25612 Linear Shepard 0.44 0.19 0.56 1.85

SVR 20.7 0.26 0.76 188.08

B.5 Estimating Extrapolation Distance

Kriging is based on a correlation structure between points based on their

distance. Large correlation between extrapolation points to closest sample may indicate

reliable extrapolation. We defined two types of effective extrapolation distance and tried

to find the relation between extrapolation distances of test functions and corresponding

correlation over that extrapolation distance.

B.5.1 Effective Extrapolation Distance

Ordinary Kriging assumes that the error at a point is normally distributed with a

mean of zero and a given standard deviation. Error bounds here are set be 95%

confidence interval of this normal distribution. The conservative extrapolation distance d

is defined for measuring how far the error bounds of the surrogate bound the true

function.

190

The second extrapolation distance is denoted as accuracy distance. Accurate

distance is inside conservative distance and in which estimated error bounds of the

points are less than 30% of ( )range f . We make an exception to the requirement of

being within the error bounds when they are very tight, allowing error bounds to be off

by 1% of ( )range f ). Two types of effective extrapolation distance are illustrated in

Figure B-6.

Figure B-6. Two types of effective extrapolation distance. A) Conservative extrapolation

distance and B) accurate extrapolation distance.

B.5.2 Identification of Effective Extrapolation Distance using Correlation

The error estimates get less dependable as we go deeper into the extrapolation

domain. We tried to find certain indicators of effective extrapolation distance based on

Kriging. Prediction of Ordinary Kriging is based on the assumption that correlation

between function values decay with distance at a rate controlled by , with the

correlation r between two points at a distance l being equal to

2( ) expr l l (B-5)

is usually found by maximizing likelihood of observing that the samples come

from Gaussian process. Large means short wavelength, large curvature, fast changing

function, and reverse for small . It is reasonable to expect that as the correlation

0 2 4 6 8-20

0

20

40

60

80

100

120

x

f(x)


Border of inaccessible domain

Conservative bound

Accurate bound

True function

Extrapolation

Sampling points

95% confidence range

0 5 10 15 20 25

0

100

200

300

400

500

600

x

f(x)


Border of inaccessible domain

Conservative bound

Accurate bound

True function

Extrapolation

Sampling points

95% confidence range

A B

191

between function values in the sampling domain and extrapolation domain diminishes,

the reliability of the error estimates deteriorates. Distance corresponds to given small

correlation as possible measure of how far we can go.

We have performed a test to figure out the relation between effective

extrapolation distance and corresponding correlation value. 10 multi-dimensional

functions: Branin-Hoo function, Ackley function, Gramacy & Lee (2009) function,

Hartmann 3-D Function, Hartmann 6-D Function, Sasena Function, Friedman Function,

Zhou (1998) Function, Franke's Function, Dette & Pepelyshev (2010) curved Function.

Figure B-7. Correlation and distance ratio over effective distance while extrapolating 30

1D functions. A) Correlation over accurate distance, B) distance ratio of accurate distance to accessible domain, C) correlation over conservative distance and D) distance ratio of conservative distance to accessible domain.

These functions are commonly used for testing algorithm performance and can

be found from [5]. 3 lines are extracted towards one random vertex from each function.

We use 6 uniformly spaced samples to train Kriging. In Figure B-7, we present

B

0

0.2

0.4

0.6

0.8

1

1

Box plot of correlation at accurate bound

0

0.2

0.4

0.6

0.8

1

1

Box plot of distance ratio at accurate bound

0

0.2

0.4

0.6

0.8

1

1

Box plot of correlation at conservative bound

0

0.2

0.4

0.6

0.8

1

1.2

1

Box plot of distance ratio at conservative boundA

C D

192

corresponding correlation value between effective extrapolation bounds and closest

sample. Accurate bounds are associated with large correlation value. The third quartiles

of correlation values corresponding to accurate bounds and conservative bounds are

both 0.99. The box plots of distance ratios are dispersed and imply extrapolation

accuracy vary with functions.

B.6 Concluding Remarks

This work first illustrated the possibility of extrapolating 1D function using

surrogates and proposed an average error metric designed for quantifying the

performance of extrapolation technique. Testing extrapolation on several challenging

functions indicated that Kriging and Linear Shepard were safer than other surrogates.

We defined two types of effective extrapolation distance and correlation of Ordinary

Kriging has been demonstrated as a possible indicator for effective extrapolation

distance based on the tests of 30 one-dimensional functions.

193

APPENDIX C WEIBULL DISTRIBUTION WITH A HEAVY TAIL

The Weibull distribution was first shaped to follow ( , )W a b with a heavy tail to

amplify the difference with the normal distribution, where the scale parameter a=2, the

shape parameter b=1.5 as seen in Eq. (C-1). Then ( , )W a b was scaled through a linear

transformation to imitate the experimental variability of the analytical test function

according to Eq.(C-2), Eq.(C-3) and Eq.(C-4), where is the strength of the

analytical test function, 1( , )WF p a b is the inverse cumulative distribution at percentile p,

,W a b is the mean of ( , )W a b . ,al from Eq. (C-2) was adopted to scale up the

variability of the synthetic strength and ,bl in Eq.(C-3) was used to guarantee the

mean value of the synthetic strength is ,f .

1

/~ ,

bb

x ab kk W a b e

a a

(C-1)

1 1

0.2 ,,

(97.5 , ) (2.5 , )W W

fal

F a b F a b

(C-2)

, , , ,Wbl f al a b (C-3)

, , + ,s k al bl (C-4)

,f

194

APPENDIX D SPECIMEN STRENGTH SUMMARY FOR OHT TESTING

Table D-1. Specimen strength at w/d=3. Nominal thickness for one ply is 0.0086 inch.

plynorm

norm

tStrength Strength

t

Pre-preg

batch

number

Strength

[ksi]

Ave.

specimen

thickness

[in]

Number of

plies in

laminate

Normalized

Strength

[ksi]

Failure

Mode

1

61.768 0.167 20 60.111 LGM

59.168 0.168 20 57.689 LGM

55.954 0.169 20 54.913 LGM

57.743 0.170 20 57.228 LGM

57.041 0.169 20 56.146 LGM

60.357 0.170 20 59.625 LGM

2

66.554 0.165 20 63.710 LGM

69.057 0.165 20 66.327 LGM

62.830 0.166 20 60.717 LGM

67.037 0.167 20 65.185 LGM

63.317 0.168 20 61.697 LGM

63.750 0.168 20 62.225 LGM

3

58.432 0.171 20 58.149 LGM

54.575 0.171 20 54.269 LGM

63.433 0.170 20 62.819 LGM

59.470 0.170 20 58.605 LGM

56.688 0.168 20 55.249 LGM

59.979 0.169 20 58.817 LGM

195


plynorm

norm

tStrength Strength

t

Pre-

preg

batch

number

Strength [ksi] Ave.

specimen

thickness

[in]

Number of

plies in

laminate

Normalized

Strength

[ksi]

Failure

Mode

1

69.734 0.165 20 66.903 LGM

66.548 0.166 20 64.040 LGM

64.929 0.167 20 62.878 LGM

67.434 0.168 20 65.950 LGM

64.613 0.168 20 63.248 LGM

60.781 0.169 20 59.603 LGM

2

67.778 0.167 20 65.683 LGM

67.386 0.168 20 65.707 LGM

65.272 0.169 20 64.052 LGM

68.817 0.170 20 68.070 LGM

63.226 0.171 20 62.767 LGM

65.002 0.171 20 64.706 LGM

3

62.466 0.166 20 60.336 LGM

65.718 0.167 20 63.763 LGM

60.186 0.168 20 58.658 LGM

61.175 0.169 20 60.138 LGM

63.366 0.170 20 62.678 LGM

59.326 0.171 20 58.815 LGM

196


plynorm

norm

tStrength Strength

t

Pre-preg

batch

number

Strength

[ksi]

Ave.

specimen

thickness

[in]

Number of

plies in

laminate

Normalized

Strength

[ksi]

Failure

Mode

1

65.081 0.171 20 70.113 LGM

70.862 0.170 20 64.881 LGM

66.084 0.169 20 66.797 LGM

68.646 0.167 20 64.996 LGM

66.182 0.169 20 64.522 LGM

65.173 0.170 20 68.600 LGM

2

63.417 0.168 20 68.077 LGM

69.299 0.169 20 71.889 LGM

72.877 0.170 20 69.745 LGM

70.050 0.171 20 69.875 LGM

73.024 0.165 20 68.162 LGM

70.541 0.166 20 71.016 LGM

3

65.243 0.164 20 62.379 LGM

68.869 0.165 20 66.039 LGM

67.953 0.166 20 65.570 LGM

67.890 0.165 20 65.225 LGM

64.027 0.167 20 62.321 LGM

64.980 0.169 20 63.986 LGM

197


plynorm

norm

tStrength Strength

t

Pre-preg

batch

number

Strength

[ksi]

Ave.

specimen

thickness

[in]

Number of

plies in

laminate

Normalized

Strength

[ksi]

Failure

Mode

1

66.696 0.170 20 65.746 LGM

71.789 0.167 20 69.842 LGM

66.359 0.171 20 65.870 LGM

69.543 0.171 20 69.017 LGM

69.239 0.170 20 68.541 LGM

65.602 0.169 20 64.508 LGM

2

69.609 0.167 20 67.484 LGM

72.620 0.168 20 71.072 LGM

77.366 0.170 20 76.242 LGM

67.859 0.169 20 66.754 LGM

73.933 0.169 20 72.522 LGM

71.999 0.168 20 70.185 LGM

3

74.593 0.169 20 73.219 LGM

68.716 0.170 20 67.784 LGM

70.205 0.170 20 69.300 LGM

69.854 0.169 20 68.486 LGM

75.330 0.167 20 73.002 LGM

68.976 0.168 20 67.219 LGM

198

APPENDIX E VARIABILITY FOR PREDICTION ACCURACY WHEN COEFFICIENT OF VARIATION

IS 0.2

Figure E-1. Approximating the strength of composite laminate with the trade-off

between exploration and replication. Variability of mean RMSE with normal noise, coefficient of variation is 0.2. A) 4×4 test matrix, B) 5×5 test matrix, C) 6×6 test matrix and D) 7×7 test matrix.

A

0 50 100 1500.1

0.2

0.3

0.4

0.5

Composite:Variability of RMSE with normal noise

Number of samples

RM

SE

4*4 test matrix

95% Confi. Int.

0 50 100 150 2000.1

0.2

0.3

0.4

0.5


Number of samples

RM

SE

5*5 test matrix

95% Confi. Int.

0 100 200 3000.1

0.2

0.3

0.4

0.5


Number of samples

RM

SE

6*6 test matrix

95% Confi. Int.

0 100 200 300 400

0.1

0.2

0.3

0.4

0.5

Composite:Variability of mean RMSE with normal noise

Number of samples

RM

SE

7*7 test matrix

95% Confi. Int.

B

C D

199

LIST OF REFERENCES

[1] Mack, Y., Goel, T., Shyy, W., Haftka, R., and Queipo, N., "Multiple surrogates for the shape optimization of bluff body-facilitated mixing," Proc. Proceedings of the 43rd AIAA Aerospace Sciencis Meeting and Exhibit, pp. 10-13.

[2] Goel, T., Haftka, R. T., Shyy, W., and Queipo, N. V., 2007, "Ensemble of surrogates," Structural and Multidisciplinary Optimization, 33(3), pp. 199-216.

[3] Viana, F. A., Haftka, R. T., and Steffen Jr, V., 2009, "Multiple surrogates: how cross-validation errors can help us to obtain the best predictor," Structural and Multidisciplinary Optimization, 39(4), pp. 439-457.

[4] Hoeting, J. A., Madigan, D., Raftery, A. E., and Volinsky, C. T., 1999, "Bayesian model averaging: a tutorial," Statistical science, pp. 382-401.

[5] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Function Extrapolation at One Inaccessible Point Using Converging Lines," Proc. ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, pp. V02BT03A027-V002BT003A027.

[6] Zhang, Y., Park, C., Kim, N. H., and Haftka, R. T., 2017, "Function prediction at one inaccessible point using converging lines," Journal of Mechanical Design, 139(5), p. 051402.

[7] Y. Zhang, N. H. K., C. Park, R. T. Haftka, 2015, "One-dimensional Function Extrapolation Using Surrogates," 11th World Congress on Structural and Multidisciplinary OptimizationSydney, Australia.

[8] Zhang, Y., Kim, N.-H., Park, C., and Haftka, R. T., 2017, "Multi-Fidelity Surrogate Based on Single Linear Regression," ArXiv e-prints.

[9] Zhang, Y., Neelakantan, A., Kumar, N., Park, C., Haftka, R. T., Kim, N. H., and Lam, H., "Multi-fidelity Surrogate Modeling for Application/Architecture Co-design," Proc. International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems, Springer, pp. 179-196.

[10] Yiming Zhang, J. M., Jaco Schutte, Nam-ho Kim, Raphael T. Haftka, 2017, "Predicting B-Basis Allowable at Untested Points from Experiments and Simulations of Plates with Holes," 12th World Congress on Structural and Multidisciplinary OptimizationGermany.

[11] Zhang, Y., Meeker, J., Schutte, J., Kim, N., and Haftka, R., "On Approaches to Combine Experimental Strength and Simulation with Application to Open-Hole-Tension Configuration," Proc. Proceedings of the American Society for Composites: Thirty-First Technical Conference.

200

[12] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Effect of Varying Test Cost on Design of Experiments," Proc. 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, p. 1163.

[13] Zhang, Y., Haftka, R. T., Kim, N.-H., Schutte, J. F., and Seneviratne, W. P., "Allocation of Samples Between Exploration and Replication for Open-Hole-Tension Test," Proc. American Society of Composites-30th Technical Conference.

[14] Zhang, Y., Schutte, J. F., Seneviratne, W. P., Kim, N. H., and Haftka, R. T., 2017, "Sampling by Exploration and Replication for Estimating Experimental Strength of Composite Structures," AIAA Journal.

[15] Kerschen, G., Golinval, J.-c., Vakakis, A. F., and Bergman, L. A., 2005, "The method of proper orthogonal decomposition for dynamical characterization and order reduction of mechanical systems: an overview," Nonlinear dynamics, 41(1-3), pp. 147-169.

[16] Bui-Thanh, T., Damodaran, M., and Willcox, K. E., 2004, "Aerodynamic data reconstruction and inverse design using proper orthogonal decomposition," AIAA journal, 42(8), pp. 1505-1516.

[17] Willcox, K., 2006, "Unsteady flow sensing and estimation via the gappy proper orthogonal decomposition," Computers & fluids, 35(2), pp. 208-226.

[18] Galbally, D., Fidkowski, K., Willcox, K., and Ghattas, O., 2010, "Non-linear model reduction for uncertainty quantification in large-scale inverse problems."

[19] Jeong, S., Solenthaler, B., Pollefeys, M., and Gross, M., 2015, "Data-driven fluid simulations using regression forests," ACM Transactions on Graphics (TOG), 34(6), p. 199.

[20] Gunes, H., Sirisup, S., and Karniadakis, G. E., 2006, "Gappy data: To Krig or not to Krig?," Journal of Computational Physics, 212(1), pp. 358-382.

[21] Qiu, Y., and Bai, J., 2015, "Stationary flow fields prediction of variable physical domain based on proper orthogonal decomposition and kriging surrogate model," Chinese Journal of Aeronautics, 28(1), pp. 44-56.

[22] Luo, J., Tang, X., Duan, Y., and Liu, F., "An Iterative Inverse Design Method of Turbomachinery Blades by Using Proper Orthogonal Decomposition," Proc. ASME Turbo Expo 2015: Turbine Technical Conference and Exposition, American Society of Mechanical Engineers, pp. V02BT39A026-V002BT039A026.

[23] Carlberg, K., and Farhat, C., 2009, "An adaptive POD-Krylov reduced-order modeling framework for repeated analyses problems."

201

[24] Yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2015, "One-dimensional Function Extrapolation Using Surrogates," 11th World Congress on Structural and Multidisciplinary OptimisationSydney Australia.

[25] Zhang, Y., Kim, N. H., Park, C., and Haftka, R. T., "Function Extrapolation of Noisy Data using Converging Lines," Proc. AIAA Modeling and Simulation Technologies Conference, p. 2144.

[26] Zhang, Y., Kim, N.-H., Park, C., and Haftka, R. T., 2017, "Multi-Fidelity Surrogate Based on Single Linear Regression," arXiv preprint arXiv:1705.02956.

[27] yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2018, "A Strategy for Adaptive Sampling When Sampling Cost Is Variable In Design Space," Asian Congress of Structural and Multidisciplinary optimization (abstract only)Dalian, China.

[28] yiming Zhang, N.-H. K., Chan-Young Park , Raphael T. Haftka, 2018 "A Strategy for Adaptive Sampling When Sampling Cost Is Variable In Design Space," AIAA Journal (submitted).

[29] Y. Zhang, R. T. H., N.H. Kim, W. Seneviartne and J. F. Schutte, 2015, "Allocation of Samples between Exploration and Replication for Open-Hole-Tension Test," Proceedings of the American Society of Composites- 30th Annual Technical ConferenceEast Lansing, Michigan.

[30] Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R., and Tucker, P. K., 2005, "Surrogate-based analysis and optimization," Progress in aerospace sciences, 41(1), pp. 1-28.

[31] Qian, Z., Seepersad, C. C., Joseph, V. R., Allen, J. K., and Wu, C. J., 2006, "Building surrogate models based on detailed and approximate simulations," Journal of Mechanical Design, 128(4), pp. 668-677.

[32] Brooks, D. G., Carroll, S. S., and Verdini, W. A., 1988, "Characterizing the domain of a regression model," The American Statistician, 42(3), pp. 187-190.

[33] Barnard, E., and Wessels, L., 1992, "Extrapolation and interpolation in neural network classifiers," Control Systems, IEEE, 12(5), pp. 50-53.

[34] Shan, S., and Wang, G. G., 2010, "Metamodeling for high dimensional simulation-based design problems," Journal of Mechanical Design, 132(5), p. 051009.

[35] Glaz, B., Goel, T., Liu, L., Friedmann, P. P., and Haftka, R. T., 2007, "Application of a weighted average surrogate approach to helicopter rotor blade vibration reduction," AIAA Paper, 1898.

202

[36] Balabanov, V., Weckner, O., and Wu, J., 2014, "Reducing Error of Polynomial Approximation Outside of Designated Design Space for Practical Problems."

[37] Hooker, G., and Rosset, S., "DARE: Data-Augmented Regression for Extrapolation."

[38] Wilson, A. G., and Adams, R. P., "Gaussian Process Kernels for Pattern Discovery and Extrapolation," Proc. ICML (3), pp. 1067-1075.

[39] Roache, P. J., 2009, Fundamentals of verification and validation, hermosa publ.

[40] Sidi, A., 2003, Practical extrapolation methods: Theory and applications, Cambridge University Press.

[41] Hooker, G., 2004, "Diagnostics and extrapolation in machine learning," stanford university.

[42] An, D., Kim, N. H., and Choi, J.-H., 2015, "Practical options for selecting data-driven or physics-based prognostics algorithms with reviews," Reliability Engineering & System Safety, 133, pp. 223-236.

[43] Loh, W.-Y., Chen, C.-W., and Zheng, W., 2007, "Extrapolation errors in linear model trees," ACM Transactions on Knowledge Discovery from Data (TKDD), 1(2), p. 6.

[44] George, J. J., 2014, Weather forecasting for aeronautics, Academic press.

[45] Rapach, D. E., and Zhou, G., 2013, "Forecasting stock returns," Handbook of Economic Forecasting, 2(Part A), pp. 328-383.

[46] Chaudhuri, A., and Haftka, R. T., "Optimistic Bias in Surrogate Prediction near Surrogate Optima."

[47] Durelli, A., and Rajaiah, K., 1981, "Quasi-square hole with optimum shape in an infinite plate subjected to in-plane loading," Journal of Mechanical Design, 103(4), pp. 866-870.

[48] Stein, M., 1987, "Large sample properties of simulations using Latin hypercube sampling," Technometrics, 29(2), pp. 143-151.

[49] Rasmussen, C. E., 2006, "Gaussian processes for machine learning."

[50] Viana, F., and Goel, T., 2010, "SURROGATES toolbox user’s guide," ht tp://fchegury. googlepages. com,[retrieved Dec. 2009].

[51] Bernardo, J., Berger, J., Dawid, A., and Smith, A., 1992, "Some Bayesian Numerical Analysis," Bayesian Statistics, 4.

203

[52] Murphy, K. P., 2007, "Conjugate Bayesian analysis of the Gaussian distribution," def, 1(2σ2), p. 16.

[53] Bingham, S. S. D., 2016, "virtual library of simulation experiments: test functions and datasets."

[54] Dette, H., and Pepelyshev, A., 2012, "Generalized Latin hypercube design for computer experiments," Technometrics.

[55] Parmar, M., Haselbacher, A., and Balachandar, S., 2010, "Improved drag correlation for spheres and application to shock-tube experiments," Aiaa Journal, 48(6), pp. 1273-1276.

[56] Forrester, A. I., and Keane, A. J., 2009, "Recent advances in surrogate-based optimization," Progress in Aerospace Sciences, 45(1), pp. 50-79.

[57] Frangos, M., Marzouk, Y., Willcox, K., and van Bloemen Waanders, B., 2010, "Surrogate and reduced-order modeling: a comparison of approaches for large-scale statistical inverse problems [Chapter 7]."

[58] Fernández-Godino, M. G., Park, C., Kim, N.-H., and Haftka, R. T., 2016, "Review of multi-fidelity models," arXiv preprint arXiv:1609.07196.

[59] Peherstorfer, B., Willcox, K., and Gunzburger, M., 2016, "Survey of multifidelity methods in uncertainty propagation, inference," and optimization. Technical Report TR-16-1. Aerospace Computational Design Laboratory, Department of Aeronautics and Astronautics, Massachusetts Institute of Technology, Cambridge, MA, USA.

[60] Balabanov, V., Grossman, B., Watson, L., Mason, W., and Haftka, R., "Multifidelity response surface model for HSCT wing bending material weight," Proc. 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, p. 4804.

[61] Kennedy, M. C., and O'Hagan, A., 2000, "Predicting the output from a complex computer code when fast approximations are available," Biometrika, 87(1), pp. 1-13.

[62] Qian, P. Z., and Wu, C. J., 2008, "Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments," Technometrics, 50(2), pp. 192-204.

[63] Han, Z., Zimmerman, R., and Görtz, S., 2012, "Alternative cokriging method for variable-fidelity surrogate modeling," AIAA journal, 50(5), pp. 1205-1210.

[64] Kuya, Y., Takeda, K., Zhang, X., and Forrester, A. I., 2011, "Multifidelity surrogate modeling of experimental and computational aerodynamic data sets," AIAA journal, 49(2), p. 289.

204

[65] Park, C., Kim, N. H., and Haftka, R. T., "Including ρ in Multi-Fidelity Surrogate Prediction Can Make Discrepancy Extrapolation Accurate by Reducing Bumpiness," Proc. 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, p. 0915.

[66] Kennedy, M. C., and O’Hagan, A., 2001, "Supplementary details on Bayesian calibration of computer models," Internal Report. URL http://www. shef. ac. uk/~ st1ao/ps/calsup. ps.

[67] Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., and Ryne, R. D., 2004, "Combining field data and computer simulations for calibration and prediction," SIAM Journal on Scientific Computing, 26(2), pp. 448-466.

[68] Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H., and Tu, J., 2007, "A framework for validation of computer models," Technometrics, 49(2), pp. 138-154.

[69] Wang, L., Fang, X., Subramaniyan, A., Jothiprasad, G., Gardner, M., Kale, A., Akkaram, S., Beeson, D., Wiggs, G., and Nelson, J., "Challenges in uncertainty, calibration, validation and predictability of engineering analysis models," Proc. ASME 2011 Turbo Expo: Turbine Technical Conference and Exposition, American Society of Mechanical Engineers, pp. 747-758.

[70] Sankararaman, S., Ling, Y., and Mahadevan, S., 2011, "Uncertainty quantification and model validation of fatigue crack growth prediction," Engineering Fracture Mechanics, 78(7), pp. 1487-1504.

[71] Jin, R., Du, X., and Chen, W., 2003, "The use of metamodeling techniques for optimization under uncertainty," Structural and Multidisciplinary Optimization, 25(2), pp. 99-116.

[72] Viana, F., 2010, "SURROGATES toolbox user’s guide," Gainesville, FL, USA. [73] Le Gratiet, L., 2013, "Multi-fidelity Gaussian process regression for computer

experiments," Université Paris-Diderot-Paris VII.

[74] Park, C., Haftka, R. T., and Kim, N. H., 2016, "Remarks on multi-fidelity surrogates," Structural and Multidisciplinary Optimization, pp. 1-22.

[75] Campbell, K., 2002, "Exploring Bayesian model calibration: a guide to intuition," Los Alamos Technical Report LA-UR-02-7175.

[76] Swiler, L. P., and Trucano, T. G., 2005, "Calibration Under Uncertainty," Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States).

[77] Matsumura, T., Haftka, R. T., and Kim, N. H., 2015, "Accurate predictions from noisy data: replication versus exploration with applications to structural failure," Structural and Multidisciplinary Optimization, pp. 1-18.

http://www/

205

[78] Dewettinck, K., De Visscher, A., Deroo, L., and Huyghebaert, A., 1999, "Modeling the steady-state thermodynamic operation point of top-spray fluidized bed processing," Journal of Food Engineering, 39(2), pp. 131-143.

[79] Le Gratiet, L., 2013, "Bayesian analysis of hierarchical multifidelity codes," SIAM/ASA Journal on Uncertainty Quantification, 1(1), pp. 244-269.

[80] Currin, C., Mitchell, T., Morris, M., and Ylvisaker, D., 1991, "Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments," Journal of the American Statistical Association, 86(416), pp. 953-963.

[81] Xiong, S., Qian, P. Z., and Wu, C. J., 2013, "Sequential design and analysis of high-accuracy and low-accuracy computer codes," Technometrics, 55(1), pp. 37-46.

[82] Couckuyt, I., Dhaene, T., and Demeester, P., 2014, "ooDACE toolbox: a flexible object-oriented Kriging implementation," Journal of Machine Learning Research, 15, pp. 3183-3186.

[83] Barrett, R. F., Borkar, S., Dosanjh, S. S., Hammond, S. D., Heroux, M. A., Hu, X. S., Luitjens, J., Parker, S. G., Shalf, J., and Tang, L., 2013, "On the role of co-design in high performance computing," vol, 24, pp. 141-155.

[84] Heroux, M. A., Doerfler, D. W., Crozier, P. S., Willenbring, J. M., Edwards, H. C., Williams, A., Rajan, M., Keiter, E. R., Thornquist, H. K., and Numrich, R. W., 2009, "Improving performance via mini-applications," Sandia National Laboratories, Tech. Rep. SAND2009-5574, 3.

[85] Kumar, N., Pascoe, C., Hajas, C., Lam, H., Stitt, G., and George, A., "Behavioral emulation for scalable design-space exploration of algorithms and architectures," Proc. International Conference on High Performance Computing, Springer, pp. 5-17.

[86] Banerjee, T., Hackl, J., Shringarpure, M., Islam, T., Balachandar, S., Jackson, T., and Ranka, S., "CMT-Bone—A Proxy Application for Compressible Multiphase Turbulent Flows," Proc. High Performance Computing (HiPC), 2016 IEEE 23rd International Conference on, IEEE, pp. 173-182.

[87] Barrett, R. F., Vaughan, C. T., and Heroux, M. A., 2011, "MiniGhost: a miniapp for exploring boundary exchange strategies using stencil computations in scientific parallel computing," Sandia National Laboratories, Tech. Rep. SAND, 5294832.

[88] Karlin, I., Keasler, J., and Neely, J., 2013, "Lulesh 2.0 updates and changes," Lawrence Livermore National Laboratory (LLNL), Livermore, CA.

206

[89] Kumar, N., Sringarpure, M., Banerjee, T., Hackl, J., Balachandar, S., Lam, H., George, A., and Ranka, S., "CMT-bone: a mini-app for compressible multiphase turbulence simulation software," Proc. Cluster Computing (CLUSTER), 2015 IEEE International Conference on, IEEE, pp. 785-792.

[90] Kennedy, M. C., and O'Hagan, A., 2001, "Bayesian calibration of computer models," Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(3), pp. 425-464.

[91] Dosanjh, S. S., Barrett, R. F., Doerfler, D., Hammond, S. D., Hemmert, K. S., Heroux, M. A., Lin, P. T., Pedretti, K. T., Rodrigues, A. F., and Trucano, T., 2014, "Exascale design space exploration and co-design," Future Generation Computer Systems, 30, pp. 46-58.

[92] Barrett, R. F., Crozier, P. S., Doerfler, D., Heroux, M. A., Lin, P. T., Thornquist, H., Trucano, T., and Vaughan, C. T., 2015, "Assessing the role of mini-applications in predicting key performance characteristics of scientific and engineering applications," Journal of Parallel and Distributed Computing, 75, pp. 107-122.

[93] Xiong, Y., Chen, W., and Tsui, K.-L., 2008, "A new variable-fidelity optimization framework based on model fusion and objective-oriented sequential sampling," Journal of Mechanical Design, 130(11), p. 111401.

[94] Ellis, M., and Mathews, E., 2001, "A new simplified thermal design tool for architects," Building and environment, 36(9), pp. 1009-1021.

[95] Zheng, L., Hedrick, T. L., and Mittal, R., 2013, "A multi-fidelity modelling approach for evaluation and optimization of wing stroke aerodynamics in flapping flight," Journal of Fluid Mechanics, 721, pp. 118-154.

[96] Jin, R., Chen, W., and Sudjianto, A., 2005, "An efficient algorithm for constructing optimal design of computer experiments," Journal of Statistical Planning and Inference, 134(1), pp. 268-287.

[97] Huang, D., Allen, T., Notz, W., and Miller, R., 2006, "Sequential kriging optimization using multiple-fidelity evaluations," Structural and Multidisciplinary Optimization, 32(5), pp. 369-382.

[98] Fischer, P. F., Lottes, J. W., and Kerkemeier, S. G., 2008, "nek5000 Web page," Web page: http://nek5000. mcs. anl. gov.

[99] Supercomputer, V., "LLNL webpage."

[100] Knight Jr, N. F., and Reeder, J. R., 2006, "User-defined material model for progressive failure analysis."

http://nek5000/

207

[101] Park, C., Haftka, R. T., and Kim, N.-H., "Experience with Several Multi-fidelity Surrogate Frameworks."

[102] Vitali, R., Haftka, R. T., and Sankar, B. V., 2002, "Multi-fidelity design of stiffened composite panel with a crack," Structural and Multidisciplinary Optimization, 23(5), pp. 347-356.

[103] Whitney, J. M., and Nuismer, R., 1974, "Stress fracture criteria for laminated composites containing stress concentrations," Journal of composite materials, 8(3), pp. 253-265.

[104] Hallett, S. R., Green, B. G., Jiang, W.-G., Cheung, K. H., and Wisnom, M. R., 2009, "The open hole tensile test: a challenge for virtual testing of composites," International Journal of Fracture, 158(2), pp. 169-181.

[105] Nuismer, R., and Whitney, J. M., 1975, "Uniaxial failure of composite laminates containing stress concentrations," Fracture mechanics of composites, ASTM International.

[106] Bhachu, K. S., Haftka, R. T., and Kim, N. H., 2016, "Comparison of Methods for Calculating B-Basis Crack Growth Life Using Limited Tests," AIAA Journal, pp. 1-12.

[107] Romero, V., Swiler, L., and Urbina, A., "An initial comparison of methods for representing and aggregating experimental uncertainties involving sparse data," Proc. 13th AIAA Non-Deterministic Approaches Conference.

[108] Romero, V., Swiler, L., Urbina, A., and Mullins, J., 2013, "A Comparison of Methods for Representing Sparsely Sampled Random Quantities," Sandia National Laboratories report SAND2013-4561 printed September.

[109] Haftka, R. T., and Gürdal, Z., 2012, Elements of structural optimization, Springer Science & Business Media.

[110] Shan, S., and Wang, G. G., 2010, "Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions," Structural and Multidisciplinary Optimization, 41(2), pp. 219-241.

[111] Carlsson, L. A., Adams, D. F., and Pipes, R. B., 2014, Experimental characterization of advanced composite materials, CRC press.

[112] Chen, S., Jiang, Z., Yang, S., Apley, D. W., and Chen, W., 2015, "Nonhierarchical multi‐model fusion using spatial random processes," International Journal for Numerical Methods in Engineering.

[113] Young, D. S., 2010, "Tolerance: an R package for estimating tolerance intervals," Journal of Statistical Software, 36(5), pp. 1-39.

208

[114] Draper, N. R., and Smith, H., 2014, Applied regression analysis, John Wiley & Sons.

[115] Bhachu, K. S., 2015, "Effects of measured data uncertainty on damage tolerant design of aircraft structures," University of Florida.

[116] Wang, G. G., and Shan, S., 2007, "Review of metamodeling techniques in support of engineering design optimization," Journal of Mechanical design, 129(4), pp. 370-380.

[117] Viana, F. A., 2011, "Multiple surrogates for prediction and optimization," University of Florida.

[118] Jin, R., Chen, W., and Sudjianto, A., "On sequential sampling for global metamodeling in engineering design," Proc. ASME 2002 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, American Society of Mechanical Engineers, pp. 539-548.

[119] Salem, M. B., Roustant, O., Gamboa, F., and Tomaso, L., 2015, "Universal Prediction Distribution for Surrogate Models," arXiv preprint arXiv:1512.07560.

[120] Xu, S., Liu, H., Wang, X., and Jiang, X., 2014, "A Robust Error-Pursuing Sequential Sampling Approach for Global Metamodeling Based on Voronoi Diagram and Cross Validation," Journal of Mechanical Design, 136(7), p. 071009.

[121] Rudolph, D., and Stitt, G., "An interpolation-based approach to multi-parameter performance modeling for heterogeneous systems," Proc. Application-specific Systems, Architectures and Processors (ASAP), 2015 IEEE 26th International Conference on, IEEE, pp. 174-180.

[122] Banerjee, T., Gadou, M., and Ranka, S., 2016, "A genetic algorithm based approach for multi-objective hardware/software co-optimization," Sustainable Computing: Informatics and Systems, 10, pp. 36-47.

[123] Jones, D. R., Schonlau, M., and Welch, W. J., 1998, "Efficient global optimization of expensive black-box functions," Journal of Global optimization, 13(4), pp. 455-492.

[124] Chaudhuri, A., 2014, "Implications of optimization cost for balancing exploration and exploitation in global search and for experimental optimization," University of Florida.

[125] J. Forrester, A. I., Keane, A. J., and Bressloff, N. W., 2006, "Design and analysis of" Noisy" computer experiments," AIAA journal, 44(10), pp. 2331-2339.

209

[126] Ramu, P., Kim, N. H., Haftka, R. T., and Queipo, N. V., "System reliability analysis using tail modeling," Proc. Proceedings of the 11th AIAA/ISSMO Multidisciplinary Analysis & Optimization Conference.

[127] Tsai, S. W., and Wu, E. M., 1971, "A general theory of strength for anisotropic materials," Journal of composite materials, 5(1), pp. 58-80.

[128] Hinton, M. J., Kaddour, A. S., and Soden, P. D., 2004, Failure criteria in fibre reinforced polymer composites: the world-wide failure exercise, Elsevier.

[129] Hinton, M., and Kaddour, A., 2013, "Triaxial test results for fibre-reinforced composites: The Second World-Wide Failure Exercise benchmark data," Journal of Composite Materials, 47(6-7), pp. 653-678.

[130] LLorca, J., González, C., Molina‐Aldareguía, J. M., Segurado, J., Seltzer, R., Sket, F., Rodríguez, M., Sádaba, S., Muñoz, R., and Canal, L. P., 2011, "Multiscale modeling of composite materials: a roadmap towards virtual testing," Advanced Materials, 23(44), pp. 5130-5147.

[131] Hallett, S. R., Jiang, W.-G., and Wisnom, M. R., 2009, "Effect of stacking sequence on open-hole tensile strength of composite laminates," AIAA journal, 47(7), pp. 1692-1699.

[132] Rice, R. C., et al. , 2003, "Development of MMPDS Handbook Aircraft Design Allowables," Prepared for the 7th Joint DOD/FAA/NASA Conference on Aging Aircraft.

[133] Tomblin, J., Sherraden, J., Seneviratne, W., and Raju, K., 2002, "A-basis and B-basis design allowables for epoxy-based prepreg Toray T700SC-12K-50C/# 2510 plain weave fabric," Advanced General Aviation Transport Experiments, AGATEWP3, pp. 3-033051.

[134] Mehrez, L., Moens, D., and Vandepitte, D., 2012, "Stochastic identification of composite material properties from limited experimental databases, part I: Experimental database construction," Mechanical systems and signal processing, 27, pp. 471-483.

[135] Jordan J. Handler, J. S. D., Salvatore Liguore, Andrew Parrish and Steven Wanthal, "Development of a High Fidelity Test Database for Failure Prediction Methods Validation," Proc. Proceedings of the American Society for Composites 2014-Twenty-ninth Technical Conference on Composite Materials.

[136] Glaz, B., Goel, T., Liu, L., Friedmann, P. P., and Haftka, R. T., 2009, "Multiple-surrogate approach to helicopter rotor blade vibration reduction," AIAA Journal, 47(1), pp. 271-282.

[137] Chaudhuri, A., and Haftka, R. T., 2014, "Efficient global optimization with adaptive target setting," AIAA Journal, 52(7), pp. 1573-1578.

210

[138] Mackman, T., Allen, C., Ghoreyshi, M., and Badcock, K., 2013, "Comparison of adaptive sampling methods for generation of surrogate aerodynamic models," AIAA journal, 51(4), pp. 797-808.

[139] Matsumura, T., 2013, Effective safety measures with tests followed by design correction for aerospace structures.

[140] Tomblin, J., and Seneviratne, W., 2009, Laminate Statistical Allowable Generation for Fiber-Reinforced Composite Materials: Lamina Variability Method, Office of Aviation Research and Development, Federal Aviation Administration.

[141] International, A., 2011, "ASTM D5766 / D5766M-11, . Standard Test Method for Open-Hole Tensile Strength of Polymer Matrix Composite Laminates,"West Conshohocken, PA.

[142] Dirikolu, M. H., Aktaş, A., and Birgören, B., 2001, "Statistical analysis of fracture strength of composite materials using Weibull distribution," Turkish Journal of Engineering and Environmental Sciences, 26(1), pp. 45-48.

[143] Deo, R. B., and Saff, C. R., 1996, Composite materials: Testing and design, ASTM International.

[144] Wisnom, M., Khan, B., and Hallett, S., 2008, "Size effects in unnotched tensile strength of unidirectional and quasi-isotropic carbon/epoxy composites," Composite Structures, 84(1), pp. 21-28.

[145] Mooney, C. Z., Duval, R. D., and Duval, R., 1993, Bootstrapping: A nonparametric approach to statistical inference, Sage.

[146] Chakravarti, I. M., and Laha, R. G., 1967, "Handbook of methods of applied statistics," Handbook of methods of applied statistics, John Wiley & Sons.

[147] Welch, B. L., 1947, "The generalization ofstudent's' problem when several different population variances are involved," Biometrika, 34(1/2), pp. 28-35.

[148] Tryggvason, G., 2012, "A Front-tracking/Finite-Volume Navier-Stokes Solver for Direct Numerical Simulations of Multiphase Flows," October, 19, pp. 11-30p.

[149] Bendsoe, M. P., and Sigmund, O., 2013, Topology optimization: theory, methods, and applications, Springer Science & Business Media.

[150] Queipo, N. V., Haftka, R. T., Shyy, W., Goel, T., Vaidyanathan, R., and Kevin Tucker, P., 2005, "Surrogate-based analysis and optimization," Progress in aerospace sciences, 41(1), pp. 1-28.

[151] An, D., Choi, J.-H., Kim, N. H., and Pattabhiraman, S., 2011, "Fatigue life prediction based on Bayesian approach to incorporate field data into probability model," Structural engineering & mechanics, 37(4), p. 427.

211

BIOGRAPHICAL SKETCH

Yiming Zhang received his Ph.D. in the Department of Mechanical and

Aerospace Engineering at University of Florida (UF) in the spring of 2018. His work

focused on innovative data analytics to characterize mechanical systems especially on

predictive modeling, uncertainty quantification and optimization. He worked under Dr.

Raphael T. Haftka and Dr. Nam-Ho Kim as a part of the Structural & Multidisciplinary

Optimization Group during the graduate study. Before the Ph.D. study, he received

Master of Science from UF with a focus on topology optimization and Bachelor of

Science from Shanghai Jiao Tong University with a focus on error compensation for

CNC machining tool. He has been a main participant for several multi-disciplinary

projects across university, industry and national labs. He has worked as research

assistant on data analytics for the design and validation of various mechanical systems

including composite laminate, computational fluid dynamics for turbulence flow and

Exascale computer architecture. He has been serving as reviewers for Structural and

Multidisciplinary Optimization Journal.

university of floridaufdcimages.uflib.ufl.edu/uf/e0/05/21/58/00001/zhang_y.pdf · 4 acknowledgments...

Documents