generalized pointwise bias error bounds for response surface approximations

INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN ENGINEERINGInt. J. Numer. Meth. Engng 2006; 65:2035–2059Published online 12 December 2005 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/nme.1532

Generalized pointwise bias error bounds for responsesurface approximations

Tushar Goel∗,†, Raphael T. Haftka, Melih Papila‡ and Wei Shyy§

Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611, U.S.A.

SUMMARY

This paper proposes a generalized pointwise bias error bounds estimation method for polynomial-based response surface approximations when bias errors are substantial. A relaxation parameter isintroduced to account for inconsistencies between the data and the assumed true model. The methodis demonstrated with a polynomial example where the model is a quadratic polynomial while thetrue function is assumed to be cubic polynomial. The effect of relaxation parameter is studied. Itis demonstrated that when bias errors dominate, the bias error bounds characterize the actual errorfield better than the standard error. The bias error bound estimates also help to identify regionsin the design space where the accuracy of the response surface approximations is inadequate. It isdemonstrated that this information can be utilized for adaptive sampling in order to improve accuracyin such regions. Copyright � 2005 John Wiley & Sons, Ltd.

KEY WORDS: response surface approximation; pointwise error estimates; bias error; error bounds;model appraisal; surrogate errors

1. INTRODUCTION AND LITERATURE REVIEW

Response surface approximations (RSAs) are widely accepted for solving optimization prob-lems with high computational or experimental cost, as they offer a computationally lessexpensive way of evaluating designs. Polynomial response surfaces are the most popular amongall response surface approximation techniques. There are numerous applications of polynomial-based RSAs to practical design and optimization problems in different areas. A few examplesare given as follows. Kaufman et al. [1], Balabanov et al. [2, 3], Papila and Haftka [4, 5],

∗Correspondence to: T. Goel, Department of Mechanical and Aerospace Engineering, University of Florida,Gainesville, FL 32611, U.S.A.

†E-mail: [email protected]‡Current address: Sabanci University, Faculty of Engineering and Natural Sciences, Istanbul, Turkey.§Current address: Department of Aerospace Engineering, The University of Michigan, Ann Arbor, MI 48109,

U.S.A.

Contract/grant sponsor: Institute for Future Space Transport

Received 29 April 2005Revised 27 August 2005

Copyright � 2005 John Wiley & Sons, Ltd. Accepted 6 September 2005

2036 T. GOEL ET AL.

and Hosder et al. [6] constructed RSAs for structural weight based on structural optimizationsof high speed civil transport. Madsen et al. [7], Papila et al. [8, 9], Shyy et al. [10, 11] and,Vaidyanathan et al. [12, 13] used RSAs as design evaluators for the optimization of propulsioncomponents including a turbulent flow diffuser, supersonic turbine, swirl coaxial injector ele-ment and liquid rocket injector designs. Redhe et al. [14, 15] and Craig et al. [16] used RSAs indesign of vehicles for crashworthiness. Rais-Rohani and Singh [17] used RSAs to approximatelimit state functions for estimating the reliability of composite structures. Rikards and Auzins[18] developed RSAs to model buckling and axial stiffness constraints while minimizing theweight of composite stiffened panels. Vittal and Hajela [19] proposed using RSAs to estimatethe statistical confidence intervals on the reliability estimates. Kim et al. [20], Keane [21],Jun et al. [22] applied RSAs to optimize wing design. Goel et al. [23] used the RSAs toapproximate the Pareto optimal front in multi-objective optimization of a liquid rocket injectordesign. Hill and Olson [24] applied RSAs to approximate noise models in their effort to reducethe noise in the conceptual design of transport aircrafts.

It is important to ensure the accuracy of RSAs before using them for reliability andoptimization. The accuracy of a RSA, constructed using a limited number of experimentsor simulations, is primarily affected by two factors: (i) non-normal distribution of the noise inthe data; and (ii) inadequacy of the fitting model (bias error).

Limitations on the number of experiments or simulations reflect high cost. Design of experi-ments is used to minimize the number of experiments or simulations. In experiments, noise mayappear due to measurement errors and other experimental errors. Numerical noise in computersimulations is usually small, but it can be high if there are unconverged solutions such asthose encountered in CFD or structural optimization. Quite often the true model representingthe data is unknown and a simple model is fitted to the data. Error in approximation due toan insufficient model is known as modelling error, or bias error. For simulation-based RSAs,bias error is mainly responsible for the error in the prediction.

Prediction variance is an established tool to characterize noise-related prediction errors[25, 26]. It is often inappropriately employed when bias errors are dominant. A good amount ofwork on minimizing the mean squared error averaged over the design domain combining noiseand bias errors has been done [27–30]. The bias component of the averaged or integrated meansquared error can also be minimized to obtain so-called minimum-bias designs (see References[25, Chapter 9; 26, Chapter 6]; [31–34]) which protect against the cases where true model isin general an order higher than the response surface model. However, there is much less workon the point-to-point variation of bias errors. Studying point-to-point bias error can help intwo ways:

1. Identification of regions of large errors.2. Selection of design of experiments–maximum bias error can be minimized rather than the

average error.

An approach for estimating pointwise bounds on bias errors in RSAs was presented by Papilaand Haftka [35]. The traditional decomposition of the mean squared error into the variance andthe square of the bias was followed, but point-to-point rather than averaged over the domain.The bounds do not depend on the data and can be used to identify the regions in spacewhere the response surface approximations may be poor. Papila et al. [36] modified this dataindependent approach and obtained design of experiments that minimized the maximum biaserror. They also extended the approach and developed tightened bounds to account for the data

Copyright � 2005 John Wiley & Sons, Ltd. Int. J. Numer. Meth. Engng 2006; 65:2035–2059

GENERALIZED POINTWISE BIAS ERROR BOUNDS 2037

provided that the assumed true model satisfies the data exactly. In their examples, they assumedthat the true model was a higher degree polynomial than the approximating polynomial. Thisassumption is reasonable when the approximation region is small.

In many practical problems, however, the true function is not known a priori and it may notbe desirable to assume a model that would satisfy the generated data exactly (for example dueto noise in the data). That is the assumed true model is inconsistent with the data, and thisinconsistency manifests itself as discrepancies between the assumed model and the data. Thesediscrepancies may arise due to (i) random errors (noise) in the data, and/or (ii) mismatchof the true function and assumed true model. When the matrix of the equations used in thelinear regression has full rank, these inconsistencies manifest themselves in inaccuracies of thebounds on bias errors. When this matrix is rank deficient, as happens commonly (and notedby Papila et al. [36]), their bounds estimation method cannot appraise the error bounds. Oneobjective of the present paper is to address this shortcoming.

In the current work, the pointwise bias error bound estimation method is generalized toaddress discrepancies between the assumed true model and the data. The proposed methodallows the assumed true model to satisfy the design data within the limit of a small relaxationparameter � rather than satisfying the data exactly. The bounds can be applied to identify thezones where the RSA may have high errors and to guide the placement of additional functionevaluations. The proposed method is demonstrated with the help of an analytical problem withvarious data inconsistencies. The bias error bounds estimation method is compared with theprediction variance based approach to estimate the error field when bias error is the dominanterror. The examples focus on the case when the assumed true model is a polynomial oneorder higher than the fit model. This assumption is reasonable when the design space is smallenough.

The objectives of this paper are:

1. To present the generalized pointwise bias error bounds estimation method.2. To compare bias error bounds with prediction variance.3. To demonstrate the usefulness of bias error bounds for placing additional data points.

The paper is organized as follows. The next section presents the theoretical formulation of thegeneralized pointwise bias error bound estimation method. Section 3 presents implementationdetails and the testing procedure. A two-dimensional polynomial test problem and major settingsfor the bounds estimation are delineated in Section 4. The results of the proposed methodare compared with the original bias error bounds estimation method developed by Papilaet al. [36] and the prediction variance in Section 5. Major conclusions are recapitulated inSection 6.

2. THEORETICAL MODEL FOR ESTIMATING BIAS ERROR BOUNDS

For a given set of data a polynomial response surface approximation has two types of errors:(i) errors due to noise in the data and (ii) errors due to inadequate order polynomial, alsoknown as bias error. Prediction variance can effectively characterize noise errors point-to-point;however, prediction variance is not appropriate when bias error is dominant. Papila et al.[35, 36] developed a pointwise bias error bounds estimation method to estimate bounds on thebias errors when the data was consistent with the assumed true model. A generalized pointwise


2038 T. GOEL ET AL.

bias error bounds estimation method to address the scenario where the data is inconsistent withthe assumed true model is presented in this section.

2.1. Estimation of bias error bounds

Let the true response �(x) at a design point x be represented by a polynomial fT(x)�, wheref(x) is the vector of basis functions and � is the vector of coefficients. The vector f(x) hastwo components: f (1)(x) is the vector of basis functions used in the RSA or fitting model, andf (2)(x) is the vector of additional basis functions which are missing in the linear regressionmodel. Similarly, the coefficient vector � can be written as a combination of the vectors �(1)

and �(2) which represent the true coefficients associated with the basis functions vectors f (1)(x)

and f (2)(x), respectively. That is,

�(x) = fT(x)� = [f (1)f (2)]T

⎡⎣�(1)

�(2)

⎤⎦ = (f (1)(x))T�(1) + (f (2)(x))T�(2) (1)

Assuming normally distributed noise with zero mean and variance �2, the observed responsey(x) at a design point x is given as

y(x) = �(x) + N(0, �) (2)

If there is no noise error, the true response �(x) is the same as the observed responsey(x). Then the true response for Ns design points (x(i), ∀i = 1, Ns) in matrix notation can beexpressed as

y = X� = [X(1) X(2)]⎡⎣�(1)

�(2)

⎤⎦ = X(1)�(1) + X(2)�(2) (3)

where y is the vector of observed responses at the data points, X(1) is the Gramian designmatrix constructed using the basis functions corresponding to f (1)(x), and X(2) is constructedusing the missing basis functions corresponding to f (2)(x). A Gramian design matrix in twovariables (with monomial basis functions), when the RSA model is quadratic and the trueresponse is cubic, is shown in Equation (4).

X =

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

1 x11 x21 x211 x11x21 x2

21

1 x12 x22 x212 x12x22 x2

22

......

......

......

1 x1i x2i x21i x1ix2i x2

2i

......

......

......

1 x1Ns x2Ns x21Ns

x1Nsx2Ns x22Ns

∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣∣︸︷︷︸X(1)

x311 x2

11x21 x11x221 x3

21

x312 x2

12x22 x12x222 x3

22

......

......

x31i x2

1ix2i x1ix22i x3

2i

......

......

x31Ns

x21Ns

x2Ns x1Nsx22Ns

x32Ns

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

︸︷︷︸X(2)

(4)



The predicted response at a design point x, y(x) is given as a linear combination ofthe approximating basis functions vector f (1)(x) and the corresponding estimated coefficientsvector b:

y(x) = (f (1)(x))Tb (5)

The estimated coefficients vector b is evaluated using the data for Ns design points as

b = (X(1)TX(1))−1X(1)Ty (6)

Substituting for y from Equation (3) in Equation (6) gives

b = (X(1)TX(1))−1X(1)T[X(1)�(1) + X(2)�(2)] (7)

which can be rearranged as

b = �(1) + A�(2), where A = (X(1)TX(1))−1X(1)TX(2) (8)

where A is called the alias matrix. Equation (8) can be rearranged as

�(1) = b − A�(2) (9)

Note that, Equations (8) and (9) are valid only when the observed response is same as the trueresponse.

The quality of fit between different RSAs can be evaluated by comparing the standard errordefined as

�a =√

(yTy − bTX(1)Ty)

(Ns − n1)(10)

where n1 is the number of coefficients in the RSA model.The error at the Ns design points used to fit the response surface is given as

e = y − y = y − X(1)b (11)

Substituting for y from Equation (3) and for b from Equation (8) gives

e = X(1)�(1) + X(2)�(2) − X(1)[�(1) + A�(2)] = [X(2) − X(1)A]�(2) (12)

Thus, the error at the Ns design points is a function of coefficients vector �(2) only. Sincethe RSA model coefficients vector b is an unbiased estimate of coefficients vector �(1), it isexpected that the error in the prediction will be a function of coefficients in vector �(2), andthis is indeed observed from Equation (12).

The error at a general point x is the difference of the true response and the predicted responsee(x) = �(x) − y(x). When noise is the dominant source of error, the estimated standard errorees is used as an estimate of the error, and its expression is given as

ees =√

Var[y(x)] =√

�2afT(x)(XTX)−1f(x) (13)

When bias error is dominant, e(x) = eb(x), where eb(x) is the bias error at design point x.Substituting the values from Equations (1) and (5), eb(x) can be given as follows:

eb(x) = �(x) − y(x) = (f (1)(x))T�(1) + (f (2)(x))T�(2) − (f (1)(x))Tb (14)


2040 T. GOEL ET AL.

or when the data is consistent with the assumed true model (i.e. Equation (3) holds true), fromEquation (8)

eb(x) = {(f (2)(x)) − AT(f (1)(x))}T�(2) (15)

This shows that for given data (set of design points), the bias error at a general point xdepends only on the true coefficients vector �(2). In the presence of inconsistencies in the data(i.e. Equation (3) cannot be satisfied), the bias error eb(x) is computed using Equation (14),which is more general.

To estimate the error bounds at a design point x, the absolute bias error |eb(x)| is maximizedover all possible combinations of �(1) and �(2). If the data is consistent with the assumed truemodel (i.e. Equation (3) is satisfied), Papila et al. [36] showed that the bounds on the bias errorat a general design point x can be computed by solving the following optimization problem:

Maximize�(1),�(2)

|eb(x)|

Subject to :X(1)�(1) + X(2)�(2) = y

c(1)l � �(1) � c(1)

u

c(2)l � �(2) � c(2)

u

(16)

where cl represents the vector of lower bounds on the true coefficient vectors �(1) and �(2) andcu represents the vector of upper bounds on the true coefficient vectors �(1) and �(2).

This optimization problem (16) would be a linear programming (LP) problem except that theobjective function involves an absolute value. So, the bias error bounds are computed by solvingtwo LP problems–one for the minimum error min eb(x) and the other for the maximum errormax eb(x). Thus the estimate of the bias error bound is given by |eb(x)| = max(max eb(x),− min eb(x)). It is assumed that the dataset is exactly satisfied by the assumed true model,and the Gramian design matrix X(X = [X(1) X(2)]) is of full rank [36].

Papila et al. [36] demonstrated that the bias error bounds can be used to develop designof experiments prior to the generation of data. They also demonstrated that posterior to datageneration, the bounds characterize the actual worst error field well. This characteristic of biaserror bounds can be used for the adaptive sampling of data points by placing additional pointsat bias error bounds hotspots to improve the approximation locally. Such adaptive sampling isdemonstrated for a polynomial example in Section 5.2.

2.2. Generalized bounds estimation method: introduction of relaxation parameter

The optimization problem given by Equation (16) does not have a solution when the Gramiandesign matrix X is rank deficient and the data is inconsistent with the assumed true model(i.e. Equation (3) cannot be satisfied). Such inconsistency may reflect noise or the fact thatthe assumed true model may not actually be true. These inconsistencies are accommodated byrelaxing the first constraint in the optimization problem (given by Equation (16)) by a smallparameter �, which is a measure of maximum acceptable deviation from the data. Now the bias



error bound at a general point x is estimated by solving the following optimization problem:

Maximize�(1),�(2)

|eb(x)|

Subject to :y − 1� �X(1)�(1) + X(2)�(2) � y + 1�

c(1)l � �(1) � c(1)

u

c(2)l � �(2) � c(2)

u

(17)

The solution largely depends on the value of �. Too small a value of � may not relax theconstraints enough to provide a feasible solution, and too large a value of � may compromisethe usefulness of the data. The minimum relaxation �min for which there exists a solution tothe optimization problem (17) can be obtained by solving the following optimization problem:

Minimize�(1),�(2)

�min

Subject to :y − 1�min �X(1)�(1) + X(2)�(2) � y + 1�min

�min � 0

c(1)l � �(1) � c(1)

u

c(2)l � �(2) � c(2)

u

(18)

With � = �min, the choice of the polynomial coefficients is likely to be unique which may notprovide accurate error bounds. The error bounds can be assessed by increasing the relaxation to

� = k�min, k � 1 (19)

where k is the relaxation coefficient. Here, k = 1.5 is used.If the data has high random noise, or the assumed true model is inadequate to represent

data, or both, the estimate of � from Equation (19) may not give accurate error bounds. Therelaxation parameter � can then be selected as follows:

1. If there is no information on the noise, and confidence in the assumed true model is high,the relaxation parameter � is given by Equation (19). The confidence in the assumed truemodel may be based on knowledge of the physics of the problem or previous experiencewith modelling of similar problems.

2. If the magnitude of the noise can be estimated, the relaxation parameter � is taken asmax(k�min, ��), where �� is the estimated bound on the noise.

3. If the confidence in the assumed true model is low, the relaxation parameter � is chosen asmax(k�min, k

∗�a(x)), where �a(x) is given by Equation (10) and k∗ � 1. For the examplesk∗ = 0.25.

4. If both conditions 2 and 3 are true, � is chosen as max(k�min, k∗�a(x), ��).


2042 T. GOEL ET AL.

Note that, when � = 0, the generalized bounds estimation method given by Equation (17) issame as the original bounds estimation method proposed by Papila et al. [36]. Further in thispaper, the original bounds estimation method is referred to as ‘bounds estimation with � = 0’.

2.3. Selection of bounds on coefficients vectors �(1) and �(2)

Papila et al. [36] demonstrated that the selection of proper bounds on �(1) and �(2) is es-sential to accurately estimate the error bounds. Bounds on �(1) and �(2) are selected in twosteps:

(a) Bounds on �(2): Papila et al. [36] proposed guidelines to select the bounds on the vector�(2). Without additional information, the bounds on all the coefficients in �(2) can betaken equal and c(2)

l = − c(2)u . When all the variables are normalized between 0 and 1,

they suggested using the error magnitude at the design points as an indication of themagnitude of the bounds provided that the error at the data points is not too small.Another suggestion was to use a fraction of the largest coefficient of vector b (excludingthe intercept term). Finally, they suggested considering a fraction of the difference betweenthe maximum function value and the intercept term of the approximation as the magnitudeof the bounds.

(b) Bounds on �(1): can be easily obtained from Equation (9) as follows:

�(1)il = bi −∑

j

|Aij�(2)j l |

�(1)iu = bi +∑

j

|Aij�(2)ju |

(20)

where subscript l represents the lower bound and subscript u represents the upper bound.Equation (20) is valid only for the data consistent with the assumed true model. Whenthe data is inconsistent with the assumed true model, the bounds on �(1) are obtainedby relaxing the bounds from Equation (20) as follows:

�(1)′il = �(1)

il − 0.2|�(1)il |, �(1)′

iu = �(1)iu + 0.2|�(1)

iu | (21)

Note that (a) and (b) assume no knowledge on �(2). In many cases, symmetry or other consid-erations permit us to omit some components of �(2) or to assume that they are small. Sharperbounds can be used by taking advantage of such information.

2.4. Implementation procedure

A stepwise procedure discussing the implementation details is outlined as follows:

Step I1: Identify the number of variables, the basis functions vector in the polynomialresponse surface model (f (1)(x)) and the number of simulations Ns.

Step I2: Select the location of Ns design points (x(i)) (using design of experiment techniques).Step I3: Generate response (yi) at the design points (x(i)).Step I4: Construct the Gramian design matrix X(1). The ith row of the matrix X is given

by f (1)(x(i)).Step I5: Estimate b using Equation (6) and the standard error �a using Equation (10).



Step I6: Identify the missing basis functions vector (f (2)(x)). One approach is to assume thetrue polynomial function to be the lowest order polynomial with more coefficientsthan Ns. When noise is present, the true polynomial may be assumed to be an orderhigher than the polynomial response surface model to avoid modelling of noise bya higher order polynomial. In this study, the assumed true polynomial is taken asone order higher than the response surface model.

Step I7: Select upper and lower bounds on the true coefficients vectors (cl, cu).Step I8: Estimate the minimum relaxation �min from Equation (18).Step I9: Estimate the relaxation parameter � as discussed in Section 1.

Step I10: Compute the bias error bounds at a general point x using Equation (17).

In the above procedure, the first five steps are related to the response surface approximationand are presented for the sake of completeness. However, most often the error estimates areobtained after the response surface model is developed, and only steps I6–I10 are relevant.

3. TESTING PROCEDURE

The proposed generalized bias error bounds estimation method is validated by comparingdifferent error estimates (bias error bounds and standard errors) with actual errors and checkingthe effectiveness of the relaxation parameter �.

The error estimates can be compared directly with the actual errors if the true functionis known. But, the true function is rarely known. Then infinitely many candidate coefficientsvector �c (subscript c denotes that this is a candidate true coefficients vector) can be foundwhich satisfy the data y but yield different responses at unsampled points. Because the errorbound characterizes maximum error, and the standard error characterizes root mean square(rms) error, they can be compared with the actual worst error eAW(x) and the actual rms erroreARMS(x) respectively, obtained by considering a large number of �c.

The actual error at a general point x due to �c is given as

ec(x) = y(x) − y(x) = (f (1)(x))T�(1)c + (f (2)(x))T�(2)

c − (f (1)(x))Tb (22)

Then the actual worst error and actual rms error can be computed as

eAW(x) = max |ec| ∀c = 1, P (23)

eARMS(x) =√

P∑c = 1

e2c

/P (24)

Correlations between the error estimates and the corresponding actual errors provide a mea-sure of goodness of characterization of the error field and are used to compare different errorestimates. The steps in the testing procedure to evaluate different error estimates are enumeratedin Table I. A high value of the correlation coefficient indicates that the error estimates wellcharacterize the actual error field.


2044 T. GOEL ET AL.

Table I. Testing procedure to compare different error estimates.

Step no. Description

Step T1 Select Ns design points x(i), ∀i = 1, Ns

Step T2 Generate response data y at the design points by randomly selecting the coefficients vectorb and the true coefficient vector �(2) (see Equation (12)). It is tempting to select the dataor the errors randomly, but this may correspond to large values of �(2) or be inconsistentwith the true function and hence avoided (see Section 4.2 for details)

Step T3 Determine the bounds on the true coefficients vector cl and cu

Step T4 Estimate the relaxation parameter �

Step T5 Compute P candidate true polynomial coefficients vectors �c which satisfy the data. Onemethod of generating �c is described in Section 4.5

Step T6 For a large number of points in design space:

(a) Compute generalized bias error bounds using Equation (17)(b) Compute bias error bounds with �= 0 using Equation (16)(c) Calculate standard errors using Equation (13)(d) Estimate actual worst errors eAW(x) using Equation (23)(e) Estimate actual rms errors eARMS(x) using Equation (24)

Step T7 Determine the correlation coefficients

(a) r(|eb(x)|max,�=0, eAW) between actual worst errors and error bounds obtained with �= 0.(b) r(|eb(x)|max, eAW) between actual worst errors and generalized error bounds(c) r(ees(x), eARMS) between standard errors and actual rms errors

4. QUADRATIC POLYNOMIAL MODEL EXAMPLES

The proposed method is demonstrated using a quadratic polynomial. Different cases of dis-crepancy or inconsistency between the data and the assumed true model are considered. Theproblem description and different settings are discussed in this section.

4.1. Problem description

The two-variable polynomial example has a cubic polynomial as the assumed true model, anda quadratic polynomial response surface is fit to the data.

f(x) = [1 x1 x2 x21 x1x2 x2

2 x31 x2

1x2 x1x22 x3

2 ]T

f (1)(x) = [1 x1 x2 x21 x1x2 x2

2 ]T and f (2)(x) = [x31 x2

1x2 x1x22 x3

2 ]T

The true function, the assumed true model, and the response surface model are given asfollows:

True function : y(x) = (f (1)(x))T�(1) + (f (2)(x))T�(2) + Cx21x2

2

Noise : N(0, D)(25)



Table II. Summary of different test problems used to demonstrate thebounds estimation method.

Cases C D

Case 1: Data consistent with assumed true model 0.00 0.00Case 2: Inconsistency due to incomplete model 0.02 0.00Case 3: Inconsistency due to both noise and incomplete model 0.01 0.0017Case 4: Inconsistency due to noise 0.00 0.0034

Assumed true model : y(x) = (f (1)(x))T�(1) + (f (2)(x))T�(2) (26)

Response surface model : y(x) = (f (1)(x))Tb (27)

where �(1) = [�(1)

1 �(1)2 �(1)

3 �(1)4 �(1)

5 �(1)6 ]T; �(2) = [�(2)

1 �(2)2 �(2)

3 �(2)4 ]T; b = [b1 b2 b3 b4

b5 b6]T.The variable N(0, D) is a normally distributed random variable which is independently

generated at the data points. The basis function x21x2

2 is used to create the discrepancy betweenthe assumed true model and the true function. Different combinations of C and D introduceinconsistencies between the data and the assumed true model. In all test cases, the coefficients C

and D are selected such that the primary bias error (between the assumed true model and RSA)is larger than the inconsistencies in the data, that is, the coefficient vector �(2) was larger thanthe coefficients C and D. A brief summary of all test cases is given in Table II. The range ofthe design variables is x ∈ [0, 1].

4.2. Selection of design points and generation of response vector (y)(Steps T1 and T2, Table I)

Two sets of experimental designs were used to construct the response surface models. Thefirst design, shown in Figure 1(a), has a Gramian design matrix for the assumed true modelwith full rank. The second design was obtained by adding one point in the centre of designdomain to the first design, and the Gramian design matrix for the assumed true model wasrank deficient (Figure 1(b)). It should be noted that while minimum bias designs [25] are bettersuited when bias errors dominate. The design more used here to illustrate the proposed methodwas selected because it is a good example to contrast adaptive sampling based on predictionvariance and bias error bounds.

There can be infinitely many combinations of response data (y) at the design points, andin this study five representative datasets (labeled A–E) were selected by randomly picking thevectors b and �(2). The errors in approximation are expected to be smaller than the response,hence �(2) is smaller compared to �(1) and b. The response surface coefficients vector band the true coefficients vector �(2) were selected randomly between [−1, 1] and [−0.2, 0.2],respectively. To allow easy duplication of the results, the random values of the coefficientswere rounded to two decimal places (Table III).

For each dataset, the vector �(1) was evaluated using Equation (9), and the actual responsevector y was calculated using Equation (25). The estimated coefficients vector b was thenrecomputed for cases 2–4 to account for inconsistencies between the data and the assumedtrue model and is referred as b′.


2046 T. GOEL ET AL.

(a) (b)

(0.0, 1.0) (0.5, 1.0) (1.0, 1.0)

(0.0, 0.5) (1.0, 0.5)

(0.0, 0.0) (0.5, 0.0) (1.0, 0.0)

(0.0, 1.0) (0.5, 1.0) (1.0, 1.0)

(0.0, 0.5) (1.0, 0.5)

(0.0, 0.0) (0.5, 0.0) (1.0, 0.0)

(0.5, 0.5)

Figure 1. Design of experiments used to construct the response surfaces for polynomial exampleproblem: (a) first experimental design; and (b) second experimental design.

Table III. Coefficients vectors b and �(2) for five randomly selected datasets.

Coefficient Dataset A Dataset B Dataset C Dataset D Dataset E

b1 0.23 −0.88 −0.97 0.68 −0.61b2 0.58 −0.29 0.49 −0.96 0.36b3 0.84 0.63 −0.11 0.36 −0.39b4 0.48 −0.98 0.86 −0.24 0.08b5 −0.65 −0.72 −0.07 0.66 −0.70b6 −0.19 −0.59 −0.16 0.01 0.40

�(2)1 0.17 −0.12 0.14 0.08 −0.05

�(2)2 0.17 0.04 0.01 −0.03 0.14

�(2)3 −0.04 −0.09 −0.12 −0.08 0.14

�(2)4 0.16 −0.12 0.07 −0.12 0.04

4.3. Bounds on the true coefficient vector � (Step T3, Table I)

The bounds on �(2) were the same as the range, that is, [−0.2, 0.2]. The bounds on the vector�(1) were estimated using the expressions given by Equations (20) and (21) with the estimatedcoefficient vector b′.

4.4. Relaxation parameter � (Step T4, Table I)

The minimum relaxation �min was obtained by Equation (18). The coefficient of noise �� issame as the value of coefficient D. The standard error �a was obtained from the responsesurface approximation. The relaxation coefficient k in Equation (19) was taken as 1.5, and k∗(Step 3, following Equation (19)) was taken as 0.25. The relaxation parameter � for each casewas selected following the rules discussed in Section 2.



4.5. Candidate true model coefficient vector �c (Step T5, Table I)

To uniquely determine the true coefficient vector �, data for 10 independent design points isrequired. However, both sets of data points generate only eight independent equations and thechoice of true coefficient vector � was not unique. Instead P = 1000 candidate polynomials�c = [�(1)

c �(2)c ] were generated to estimate the pointwise actual worst error and actual rms

error. Note that Equation (25) is used to compute the actual response.For the selected design of experiments, Equation (12) indicated that the error vector e

depends only on the coefficients �(2)2 and �(2)

3 . Thus the coefficients �(2)1 and �(2)

4 can beselected randomly without affecting the response data. This was utilized to generate candidatetrue coefficient vectors as follows:

1. Generate random values of coefficients �(2)c1 and �(2)

c4 between [−0.2, 0.2].2. Select coefficients �(2)

2 and �(2)3 from a corresponding dataset (Table III). Thus,

�(2)c = [�(2)

c1 , �(2)2 �(2)

3 �(2)c4 ].

3. Obtain the candidate true coefficients vector �(1)c from �(1)

c = b − A�(2)c .

A more general method to create candidate true coefficient vectors is described inAppendix A.

The method was implemented in MATLAB� [37]. Different errors were estimated at auniform structured grid of 21 × 21 points in design space, and the correlation coefficients werecomputed (Steps T6–T7, Table I). The results are presented in next section.

5. RESULTS

In this section results comparing different error estimates (generalized bias error bounds, boundswith � = 0, and standard error) and the corresponding actual errors are presented. Also the effectof the increase in the relaxation parameter on the bias error bounds is studied.

5.1. First design: eight-design points (Figure 1(a))

The correlation coefficients between different error estimates and the actual errors for all fivedatasets and different cases are summarized in Table IV. The correlation between the bias errorbounds and the actual worst errors was high and the variation in the correlation coefficientover different datasets was small. The correlation coefficient deteriorated in the presence ofinconsistencies between the data and the assumed true model. On the other hand, the correlationbetween the standard error and the actual rms error was much weaker. The results suggest thatwhen the bias error dominates, the prediction variance-based approach (standard error) is noteffective in characterizing the actual error field, but the bias error bounds represent the actualworst error field very well. The correlation between the error bounds and the actual worsterror for two methods of estimating bias error bounds were both good. When the assumed truemodel was correct, the correlation coefficients for two methods were also the same because themagnitude of the relaxation parameter was insignificant. However as the true model deviatedfrom the assumed true model, the bounds estimation with � = 0 performed better than thegeneralized bounds.


2048 T. GOEL ET AL.

Table IV. Coefficients of correlations and relaxation parameters for firstexperimental design (Figure 1(a)).

RelaxationCase no. Dataset parameter � r(|eb(x)|max,�=0, eAW) r(|eb(x)|max, eAW) r(ees(x), eARMS)

1 Consistent data A 2.54E-14 0.999 0.999 0.300B 2.12E-15 0.999 0.999 0.428C 8.57E-15 0.999 0.999 0.377D 5.24E-15 0.999 0.999 0.438E 3.90E-15 0.999 0.999 0.324

2 Model A 4.88E-03 0.995 0.920 0.280inconsistency

B 2.35E-03 0.992 0.967 0.472C 2.66E-03 0.993 0.962 0.433D 1.55E-03 0.993 0.984 0.483E 5.77E-03 0.994 0.891 0.302

3 Noise + model A 4.69E-03 0.975 0.915 0.326inconsistency

B 2.70E-03 0.963 0.949 0.521C 3.15E-03 0.947 0.940 0.517D 1.92E-03 0.942 0.946 0.556E 4.85E-03 0.974 0.894 0.329

4 Noise A 3.40E-03 0.959 0.914 0.338inconsistency

B 3.40E-03 0.922 0.907 0.496C 3.40E-03 0.968 0.936 0.417D 3.40E-03 0.947 0.930 0.526E 3.40E-03 0.968 0.931 0.330

(r(|eb(x)|max,�=0, eAW) is the correlation coefficient between the bias error bounds with � = 0, corresponding tooriginal bounds estimation method by Papila et al. [36]) and the actual worst errors, (r(|eb|(x)max, eAW)) is thecorrelation coefficient between the generalized bias error bounds and the actual worst errors, r(ees(x), eARMS)

is the correlation coefficient between the standard error and the actual rms (average) error.

When the assumed true model and the true model were the same, bias error bounds over-estimated the actual worst errors (by 5%). When the assumed true model was inaccurate,particularly in the presence of noise, bounds with � = 0 underestimated the actual worst errorsat some points (on an average about 7%) and overestimated them at other points (average by7%), while generalized bounds mostly overestimated the errors (average by 20%).

A typical distribution pattern of different errors is shown in Figure 2. For Case 3, DatasetC, where the data had both noise and model inconsistencies, Figure 2(a)–(e) show the contourplots of the actual worst errors, the generalized bias error bounds, the bias error bounds with� = 0, the actual rms error and the standard error. It can be seen from Figure 2(a)–(c) that theactual worst errors and the bias error bounds had similar distribution. This was reflected in thehigh correlation between the actual worst errors and the bias error bounds. The location andmagnitude of the maximum bias error bound was also found to match well with the maximumactual worst error. The pattern of the standard errors (Figure 2(d)) and the actual rms (average)



Figure 2. Different errors in the design space for quadratic-cubic problem (Case 3, Dataset C) onfirst experimental design, actual worst and rms values are taken over 1000 random polynomials:(a) actual worst error field; (b) bias error bounds (� �= 0) field; (c) bias error bounds (� = 0) field;

(d) actual rms error field; and (e) estimated standard error field.


2050 T. GOEL ET AL.

errors (Figure 2(e)) were dissimilar, which supports the weak correlation. The location of themaximum error predicted using estimated standard error estimates did not correspond well withthe location of maximum actual rms error. Also, the standard errors underestimated the errorsin the region where the actual rms error was high, but overestimated the low actual rms errorregions.

5.2. Adaptive sampling: local improvements by adding data point based on bias error bounds

One useful application of error estimates is to identify the location of additional points forsampling. Since bias error bounds give a very good estimate of the actual worst error field itcan be used to sample additional points. It was observed from Figure 2(b) and (c) that thereare four ‘hotspots’ (high error zones). An additional point (x1 = 0.8, x2 = 0.25) was placed atthe location of one hotspot to study the improvements in the prediction capabilities by adaptivesampling. The Gramian design matrix for this experimental design has full rank.

This set of nine design points was approximated using the response surface, and differenterrors were estimated. The distribution of errors in design space is presented for Case 3, DatasetC in Figure 3(a)–(d). It can be seen by comparing Figure 3(a) to 2(a) and Figure 3(d) to2(d) that the actual worst errors and actual rms errors reduced in the vicinity of the sampledpoint. The bias error bounds (Figure 3(b) and (c)) also reflected the reduction in errors nearthe additional sampled data point. Also, the pattern of the distribution of the actual worst errorsand bias error bounds from the two methods was similar, and consequently, high correlationcoefficients between the actual worst errors and error bounds were obtained.

The results clearly indicate the local improvements in the prediction capabilities by samplingthe data points at the locations where the bias error bounds are high. The adaptive samplingcan be used with any design of experiment technique.

5.3. Effect of rank deficiency: the second design (nine experimental points, Figure 1(b))

Next, the effect of rank deficiency was studied with the help of the second design of experi-ments. Notice that the point added in the centre coincides with the location of the maximumstandard error and corresponds to an adaptive sampling based on the standard error hotspots.With this set of points, the Gramian design matrix for the assumed true model was rankdeficient. The comparison of different error estimates and the actual errors is presented inTable V.

When the assumed true model was same as the true function model (Case 1) both methodsof estimating bias error bounds characterized the actual error field very well. However, whenthere were inconsistencies between the data and the assumed true model (Cases 2–4), therewas no solution to the linear programming problem (Equation (16)) with � = 0. In contrast,the generalized bounds estimated the error bounds well as evidenced by the high correlationsbetween the actual worst errors and the bias error bounds. This clearly demonstrates that therelaxation parameter is necessary in the presence of inconsistencies and rank deficiency. Forthis case also, standard errors did not characterize the actual rms errors well.

The error fields using different estimators in the design space are shown in Figure 4. ForCase 3, Dataset C, Figure 4(a)–(d) show the contour plots of the actual worst errors, thegeneralized error bounds, the actual rms error and the standard error. It can be seen fromFigure 4(a) and (b) that the actual worst errors and the generalized bias error bounds have asimilar distribution. The standard errors reduced in the centre (compare Figure 2(e) and 4(d)),



Figure 3. Different error estimates in the design space after adding one point to first ex-perimental design at the generalized bias error bounds ‘hotspot’ (Case 3, Dataset C) (localimprovements in prediction) actual worst and rms values are taken over 1000 random polynomi-als: (a) actual worst error field; (b) bias error bounds (� �= 0) field; (c) bias error bounds (� = 0)

field; and (d) actual rms error field.

but the pattern of the actual rms errors (Figure 4(c)) and the standard errors (Figure 4(d))remained dissimilar. Also, the standard errors could not characterize the high actual averageerror location.

The results presented in this section clearly establish the importance of the relaxationparameter. It is also demonstrated that when bias errors dominate, adaptive sampling basedon the bias error bounds is more effective than adaptive sampling based on the standard errors.

While the proposed method is demonstrated using a two-dimensional example, the methodcan be easily scaled to higher dimensions. However the user is cautioned against the possibilityof getting very large magnitude of worst-case errors for high dimension spaces due to geometric


2052 T. GOEL ET AL.

Table V. Coefficients of correlations and relaxation parameters for secondexperimental design (Figure 1(b)).

RelaxationCase no. Dataset parameter � r(|eb(x)|max,�=0, eAW) r(|eb(x)|max, eAW) r(|ees(x)|max, eARMS)

1 Constant data A 1.72E-14 0.999 0.999 0.384B 4.40E-15 0.999 0.999 0.521C 1.22E-14 0.999 0.999 0.461D 1.03E-14 0.999 0.999 0.529E 5.97E-15 0.999 0.999 0.428

2 Model A 3.98E-03 — 0.993 0.346inconsistency

B 1.93E-03 — 0.997 0.552C 2.18E-03 — 0.997 0.505D 1.27E-03 — 0.998 0.545E 4.72E-03 — 0.993 0.396

3 Noise + model A 3.99E-03 — 0.990 0.361inconsistency

B 1.84E-03 — 0.979 0.531C 2.23E-03 — 0.985 0.498D 1.70E-03 — 0.991 0.537E 3.92E-03 — 0.987 0.419

4 Noise A 3.40E-03 — 0.970 0.353inconsistency

B 3.40E-03 — 0.946 0.530C 3.40E-03 — 0.949 0.463D 3.40E-03 — 0.965 0.515E 4.85E-03 — 0.975 0.4081

(r(|eb(x)|max,�=0, eAW) is the correlation coefficient between the bias error bounds with � = 0, and the actualworst errors, (r(|eb|(x)max, eAW)) is the correlation coefficient between the generalized bias error bounds andthe actual worst errors, r(ees(x), eARMS) is the correlation coefficient between the standard error and the actualrms error, for Cases 2–4 bounds with � = 0 failed to appraise error bounds.

increase in the number of coefficients in the assumed true polynomial. In such scenarios, theworst-case bias errors would not reflect the overall magnitude of the true errors.

5.4. Effect of increase in relaxation parameter

As discussed earlier, while a small value of the relaxation parameter � may not give a feasiblesolution or may underestimate the error bounds, a large value of � may cause the fitted responsesurface to represent the data very poorly, leading to overestimation of the bias error bounds.Both cases may have an adverse effect on the correlation between the bias error bounds andthe actual worst errors. The effect of the relaxation parameter � on the correlation between theactual worst error and the generalized bias error bounds and the maximum bias error bound wasstudied. Figure 5 shows the results using Case 3, Dataset C and second experimental design.Figure 5(a) shows that the coefficient of correlation between the bias error bounds and the



Figure 4. Different errors in the design space for quadratic-cubic problem (Case 3, Dataset C) onsecond experimental design, (impact of rank deficiency) actual worst and rms values are taken over1000 random polynomials: (a) actual worst error field; (b) bias error bounds (� �= 0) field; (c) actual

rms error field; and (d) estimated standard error field.

actual worst errors reduced linearly with an increase in �. Figure 5(b) shows that the maximumbias error bound increased linearly with �. While these results indicate that it is best to use thelowest possible �(�min), it was also observed that for � = �min, the error bounds underestimatedthe actual errors at a large number of points (average underestimate was about 2%).

6. CONCLUDING REMARKS

In this work, a method is presented to estimate the error bounds for polynomial responsesurface approximation when the bias error is dominant. The proposed method generalizes the


2054 T. GOEL ET AL.

Figure 5. Effect of relaxation parameter on correlation coefficient and maximum biaserror bounds (Case 3, Dataset C) using second experimental design: (a) effect of � oncorrelation between generalized bias error bounds (� �= 0) and actual worst errors; and

(b) effect of � on maximum bias error bounds.

bounds estimation method developed by Papila et al. [36] while addressing the issues of in-consistencies between the data and the assumed true model (i.e. the data cannot be reproducedby the assumed true model). The proposed method was compared with the bounds estimationwhere � = 0 and the prediction variance-based method of appraising errors using the polyno-mial examples. The examples focused on assumed true model being a polynomial one orderhigher than the fit model. This is a reasonable assumption when the region of interest issmall enough. When the assumed true model is same as the true model, or the assumedtrue model has bias error, both methods of estimating the bias error bounds (with � = 0 and� �= 0) were able to characterize the actual error field very well with a small advantage to theoriginal method. However, bounds estimation with � = 0 did not work in the presence of arank deficiency in the Gramian design matrix and inconsistencies between the data and theassumed true model, while the generalized bounds estimation method worked well in all cases.When bias error was dominant, the standard error did not characterize the actual error fieldwell. As demonstrated here, the bias error bound estimates were reasonable even when theassumed true model was not exact; however, it should be noted that as the deviation fromthe true function increases, the ability to capture the effect of missing basis functions willdeteriorate.

The location of the maximum bias error bound and the maximum actual worst case errorcorrelated very well in the polynomial examples. This fact was exploited to demonstrate localrefinements in the approximation by adaptive sampling that is, sampling an additional datapoint at the location of a bias error bounds hotspot. In contrast, adaptive sampling based onthe standard error hotspots did not help in refining the response surface. Thus, when bias errorsdominate, the bias error bounds appear to characterize the actual worst error field better thanthe standard errors characterize the actual average error field.

The effect of an increase in the relaxation parameter was studied to assess its impact on thebias error bounds estimates. For the example studied, small value near the smallest possiblerelaxation parameter, worked the best.



NOMENCLATURE

A alias matrixb vector of estimated coefficients of basis functionsbj estimated coefficients of basis functionscl, cu vectors of bounds on the coefficients vector �C coefficient of missing quartic term in the true modelD coefficient of noise in the true modele vector of true errors at the data pointseb(x) true bias error at design point xees(x) estimated standard error at the design point xeARMS(x) actual root mean square (rms) error at design point xeAW(x) actual worst error at design point xf(x), f (1)(x), f (2)(x) vectors of basis functions at xfj (x) basis functionsk relaxation coefficientN(�, �) normal distribution with mean � and standard deviation �Ns number of design data pointsn1 number of basis functions in the regression modeln2 number of missing basis functions in the regression modelP number of true candidate polynomialsr correlation coefficientX, X(1), X(2) Gramian design matricesx design pointx1, x2, . . . , xn design variablesx

(i)1 , x

(i)2 , . . . , x

(i)n design variables for ith design point

y vector of observed responsesy(x) observed response at design point xy vector of predicted responsesy(x) predicted response at design point xy(x) assumed true response at design point x�, �(1), �(2) vectors of basis function coefficients�j , �(1)

j , �(2)j coefficients of basis functions

�c, �(1)c , �(2)

c candidate vectors of basis function coefficients�� estimated bound on the noise� relaxation parameter�min minimum relaxation�(x) true mean response at x�2 noise variance�a standard error1 vector with all elements equal to one

Subscripts

c candidate true coefficient vectorl lower boundu upper bound


2056 T. GOEL ET AL.

Superscripts

(i) entity corresponding to ith design point(1) entity corresponding to the regression model(2) entity corresponding to the missing terms in the regression model

APPENDIX A: GENERATION OF CANDIDATE TRUE POLYNOMIALCOEFFICIENTS VECTORS

While the method presented in the main body of the paper is useful for the selected designof experiment, a generalized method to generate many candidate true polynomial coefficientsvectors is developed by using the properties of singular vectors of the Gramian matrix. Sincethe true polynomial coefficients vectors satisfy the data exactly, we can have a candidate truecoefficients vector �0 such that

X�0 = y (A1)

Also it is known that a null singular vector V (singular vector associated with the nullsingular value) of a (Gramian design) matrix X satisfies the property XV = 0. Thus candidatetrue polynomial coefficients vectors �c can be generated by using a linear combination ofvectors �0 and V, as follows:

X�0 + �XV = y ⇒ X(�0 + �V) = y

�c = �0 + �V(A2)

where � is a scalar constant and V is a null singular vector of Gramian design matrix X.Arbitrary selection of � yields different candidate true polynomial coefficient vectors. Whenthere are Nsv null singular vectors of the Gramian design matrix X, Equation (A2) can bewritten as

X�0 +Nsv∑i=1

�iXVi = y ⇒ X

(�0 +

Nsv∑i=1

�iVi

)= y

�c = �0 +Nsv∑i=1

�iVi

(A3)

Random selection of Nsv scalar values �i will generate the candidate true polynomial coefficientsvectors. It is important to note that the selection of bounds on �i is very important. Arbitraryselection of bounds on �i may create the vector �c which either violates the bounds on �c ordoes not explore the complete feasible design space. One method of selecting the bounds is toidentify the largest positive value of �max

i with which the vector �0 can be perturbed such thatall components of the vector (�0 +�max

i Vi ) satisfy the upper constraint on the coefficients (cu).Similarly find the smallest negative number −�min

i with which the vector �0 can be perturbedsuch that all components of the vector (�0 + −�min

i Vi ) satisfy the lower constraint on thecoefficients (cl).



The coefficients vector �0 used in above analysis can be easily obtained by finding a vectornear the centre of the feasible � domain, which satisfies the data exactly. This is done bysolving the following optimization problem:

Minimize�

‖� − �cen‖

subject to X� = y

cl � � � cu

(A4)

where �cen = 0.5(cl + cu).Finally a family of candidate true coefficient vectors can be easily generated using Equation

(A4) with −�mini ��i � �max

i and ensuring that the constraint cl � � � cu is satisfied.

ACKNOWLEDGEMENTS

The present efforts have been supported by the Institute for Future Space Transport, under the NASAConstellation University Institute Program (CUIP) with Ms Claudia Meyer as program monitor. Authorsalso acknowledge the reviewers for their thoughtful suggestions.

REFERENCES

1. Kaufman M, Balabanov V, Burgee SL, Giunta AA, Grossman B, Haftka RT, Mason WH, Watson LT. Variable-complexity response surface approximations for wing structural weight in HSCT design. ComputationalMechanics 1996; 18(2):112–126.

2. Balabanov V, Kaufman M, Knill DL, Haim D, Golovidov O, Giunta AA, Grossman B, Mason WH,Watson LT, Haftka RT. Dependence of optimal structural weight on aerodynamic shape for a high speedcivil transport. Proceedings of 6th AIAA/NASA/USAF/ISSMO Symposium on Multidisciplinary Analysis andOptimization, Bellevue, WA, AIAA-96-4046, September 1996; 599–612.

3. Balabanov V, Giunta AA, Golovidov O, Grossman B, Mason WH, Watson LT, Haftka RT. Reasonable designspace approach to response surface approximation. Journal of Aircraft 1999; 36(1):308–315.

4. Papila M, Haftka RT. Uncertainty and wing structural weight approximations. Proceedings of 40thAIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Material Conference, St. Louis, MO,AIAA-99-1312, April 1999; 988–1002.

5. Papila M, Haftka RT. Response surface approximations: noise, error repair and modeling errors. AIAA Journal2000; 38(12):2336–2343.

6. Hosder S, Watson LT, Grossman B, Mason WH, Kim H, Haftka RT, Cox SE. Polynomial response surfaceapproximations for the multidisciplinary design optimization of a high speed civil transport. Optimizationand Engineering 2001; 2(4):431–452.

7. Madsen JI, Shyy W, Haftka RT. Response surface techniques for diffuser shape optimization. AIAA Journal2000; 38(9):1512–1518.

8. Papila N, Shyy W, Griffin LW, Huber F, Tran K. Preliminary design optimization for a supersonic turbinefor rocket propulsion. Proceedings of 36th AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit,Huntsville, Alabama, AIAA-2000-3242, July 2000.

9. Papila N, Shyy W, Griffin LW, Dorney DJ. Shape optimization of supersonic turbines using globalapproximation methods. Journal of Propulsion and Power 2002; 18(3):509–518.

10. Shyy W, Tucker PK, Vaidyanathan R. Response surface and neural network techniques for rocket engineinjector optimization. Journal of Propulsion and Power 2001; 17(2):391–401.

11. Shyy W, Papila N, Vaidyanathan R, Tucker PK. Global design optimization for aerodynamics and rocketpropulsion components. Progress in Aerospace Sciences 2001; 37:59–118.


2058 T. GOEL ET AL.

12. Vaidyanathan R, Papila N, Shyy W, Tucker PK, Griffin LW, Haftka RT, Fitz-Coy N. Neural networkand response surface methodology for rocket engine component optimization. Proceedings of 8thAIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, Long Beach, CA,AIAA-2000-4480, September 2000.

13. Vaidyanathan R, Tucker PK, Papila N, Shyy W. Computational-fluid-dynamics based design optimization forsingle element rocket injector. Journal of Propulsion and Power 2004; 20(4):705–717 (also presented in41st Aerospace Sciences Meeting and Exhibit, Reno, NV, AIAA-2003-0296, January 2003).

14. Redhe M, Forsberg J, Jansson T, Marklund PO, Nilsson L. Using the response surface methodology andthe D-optimality criterion in crashworthiness related problems: an analysis of the surface approximationerror versus the number of function evaluations. Structural and Multidisciplinary Optimization 2002; 24(3):185–194.

15. Redhe M, Eng L, Nilsson L. Using space mapping and surrogate models to optimize vehicle crashworthinessdesign. Proceedings of 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and OptimizationConference, Atlanta, GA, AIAA-2002-5536, September 2002.

16. Craig KJ, Stander N, Dooge DA, Varadappa S. MDO of automotive vehicle for crashworthiness and NVHusing response surface models. Proceedings of 9th AIAA/ISSMO Symposium on Multidisciplinary Analysisand Optimization Conference, Atlanta, GA, AIAA-2002-5607, September 2002.

17. Rais-Rohani M, Singh MN. Efficient response surface approach for reliability estimation of compositestructures. Proceedings of 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and OptimizationConference, Atlanta, GA, AIAA-2002-5604, September 2002.

18. Rikards R, Auzins J. Response surface method in design optimization of carbon/epoxy stiffener shells.Proceedings of 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization Conference,Atlanta, GA, AIAA-2002-5654, September 2002.

19. Vittal S, Hajela P. Confidence intervals for reliability estimated using response surface methods. Proceedingsof 9th AIAA/ISSMO Symposium on Multidisciplinary Analysis and Optimization Conference, Atlanta, GA,AIAA-2002-5475, September 2002.

20. Kim Y, Lee D, Kim Y, Yee K. Multidisciplinary design optimization of supersonic fighter wing usingresponse surface methodology. Proceedings of 9th AIAA/ISSMO Symposium on Multidisciplinary Analysisand Optimization Conference, Atlanta, GA, AIAA-2002-5408, September 2002.

21. Keane AJ. Wing optimization using design of experiments, response surface and data fusion methods. Journalof Aircraft 2003; 40(4):741–750.

22. Jun S, Jeon Y, Rho J, Lee D. Application of collaborative optimization using response surface methodologyto an aircraft wing design. Proceedings of 10th AIAA/ISSMO Multidisciplinary Analysis and OptimizationConference, Albany, NY, AIAA-2004-4442, August–September 2004.

23. Goel T, Vaidyanathan R, Haftka RT, Queipo NV, Shyy W, Tucker PK. Response surface approximationof pareto optimal front in multi-objective optimization. Proceedings of 10th AIAA/ISSMO MultidisciplinaryAnalysis and Optimization Conference, Albany, NY, AIAA-2004-4501, August–September 2004.

24. Hill GA, Olson ED. Application of response surface-based methods to noise analysis in the conceptualdesign of revolutionary aircraft. Proceedings of 10th AIAA/ISSMO Symposium on Multidisciplinary Analysisand Optimization Conference, Albany, NY, AIAA-2004-4437, August–September 2004.

25. Myers RH, Montgomery DC. Response Surface Methodology-Process and Product Optimization usingDesigned Experiments. Wiley: New York, 1995; 208–279.

26. Khuri AI, Cornell JA. Response Surfaces: Designs and Analyses (2nd edn). Marcel Dekker Inc.: New York,1996; 207–247.

27. Box GEP, Draper NR. The choice of a second order rotatable design. Biometrika 1963; 50(3):335–352.28. Draper NR, Lawrence WE. Designs which minimize model inadequacies: cuboidal regions of interest.

Biometrika 1965; 52(1–2):111–118.29. Kupper LL, Meydrech EF. A new approach to mean squared error estimation of response surfaces. Biometrika

1973; 60(3):573–579.30. Welch WJ. A mean squared error criterion for the design of experiments. Biometrika 1983; 70(1):205–213.31. Venter G, Haftka RT. Minimum-bias based experimental design for constructing response surface in structural

optimization. Proceedings of 38th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and MaterialsConference, Kissimmee, FL, AIAA-97-1053, Part 2, April 1997, 1225–1238.

32. Montepiedra G, Fedorov V. Minimum bias design with constraints. Journal of Statistical Planning andInference 1997; 63(1):97–111.



33. Palmer K, Tsui KL. A minimum bias Latin hypercube design. Institute of Industrial Engineers Transactions2001; 33(9):793–808.

34. Qu X, Venter G, Haftka RT. New formulation of a minimum-bias central composite design and Gaussquadrature. Structural and Multidisciplinary Optimization 2004; 28(4):231–242.

35. Papila M, Haftka RT. Uncertainty and response surface approximations. Proceedings of 42ndAIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Material Conference, Seattle, WA, AIAA-2001-1680, April 2001.

36. Papila M, Haftka RT, Watson LT. Pointwise bias error bounds and min-max design for response surfaceapproximations. AIAA Journal 2005; 43(8):1797–1807.

37. MATLAB�. The Language of Technical Computing. Version 6.5 Release 13. � 1984–2002. The MathWorks,Inc.: Natick, MA.


generalized pointwise bias error bounds for response surface approximations

Documents