Contents lists available at ScienceDirect

Journal of Statistical Planning and Inference

Journal of Statistical Planning and Inference 144 (2014) 123–132

0378-37

http://d

n Corr

E-m

journal homepage: www.elsevier.com/locate/jspi

Multi-level orthogonal arrays for estimating main effectsand specified interactions

Ryan Lekivetz a,n, Boxin Tang b

a JMP Division of SAS, Cary, NC, USA 27513b Department of Statistics and Actuarial Science, Simon Fraser University, Burnaby, BC, Canada V5A 1S6

a r t i c l e i n f o

Available online 3 November 2012

Keywords:

Compromise plan

Requirement set

Robustness

58/$ - see front matter & 2013 Elsevier B.V

x.doi.org/10.1016/j.jspi.2012.10.015

esponding author. Tel.: þ1 919 531 6827.

ail address: ryan.lekivetz@jmp.com (R. Lekiv

a b s t r a c t

In some situations, the experimenter would like to study factors at more than two levels,

such as when curvature has the potential to occur within the experimental region. In this

paper we consider selecting orthogonal arrays with more than two levels, when there is a

particular set of two-factor interaction components of interest to the experimenter.

As designs with more than two levels have additional complications with level

permutations, results are provided that aid in the search for efficient designs that have

robust properties. We also examine the existence and construction of such designs

through saturated orthogonal arrays.

& 2013 Elsevier B.V. All rights reserved.

1. Introduction

Orthogonal arrays are widely used for industrial experiments for identifying active effects among a large group ofeffects. Two-level factorial designs are most common, for estimating linear main effects and interactions. With enoughknowledge of the experimental region, the experimenter may want to investigate the curvature of effects during theexploration of identifying active effects. In a factorial design, this requires more than two levels of the factors. Ourdiscussion in this paper focuses on three-level designs, but the results hold in general.

We consider the situation in which the experimenter wants to estimate linear and quadratic main effects and a subsetof the interactions that are presumably important. The concept of estimation of a specified set of effects dates back toAddelman (1962). In this paper we refer to this set of specified effects as a requirement set as in Greenfield (1976). Xu et al.(2004) considered similar three-level designs for factor screening and interaction detection. Whereas their purpose was toensure the estimability of smaller subsets of factors, our design problem is for a requirement set. Recent work related toestimation of specified effects is that of Tang and Zhou (2009) for two-level designs, while Jones and Nachtsheim (2011)studied three-level designs that minimize aliasing.

While our primary goal is estimation of the requirement set, it may not be reasonable to assume that the interactionsnot estimated are negligible. For estimating a requirement set, the postulated model consists of all those effects in therequirement set. If the effects that are not estimated are nonnegligible, they can bias the estimates of the effects in themodel. To account for this, in addition to finding designs that estimate the effects in the requirement set, we also considerminimizing the contamination on the effects in the requirement set, from the interactions not estimated.

The remainder of the paper is structured as follows. Section 2 describes the model and criteria used for ranking designs.Section 3 develops theoretical results that apply to designs for our problem in regards to level permutations, and Section 4

. All rights reserved.

etz).

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132124

the existence of designs through orthogonal arrays. In Section 5 we develop a search algorithm that makes use of theresults from Sections 3 and 4, and demonstrate its use in finding designs. We conclude with final remarks in Section 6.

2. Optimality criteria

Given a factor with q levels, the main effect for this factor has q�1 degrees of freedom and can be broken up into q�1orthogonal contrasts. We adopt the approach from Xu and Wu (2001) which uses normalized main effect contrasts so thatall factorial effects have the same variance. For example, for a three-level factor, the linear and quadratic main effects arecoded as ð�1,0,1Þ �

ffiffiffi3p

=ffiffiffi2p

and ð1,�2,1Þ=ffiffiffi2p

, respectively. For convenience, our discussion refers to the linear andquadratic contrasts for the three-level coding, but the results presented hold for any set of orthogonal contrasts. For aq1-level factor and a q2-level factor using orthogonal contrasts, a two-factor interaction has ðq1�1Þ � ðq2�1Þ orthogonalcomponents, each corresponding to one degree of freedom. In a two-level design, this implies that a two-factor interactioncorresponds to one component and, as such, one degree of freedom. For a three-level design, a two-factor interaction hasfour orthogonal components corresponding to linear-by-linear, linear-by-quadratic, quadratic-by-linear, and quadratic-by-quadratic effects.

In this paper we consider experiments in which the requirement set consists of the grand mean, all main effects, andcertain two-factor interaction components. For a requirement set S, define the core set, C(S), as the subset of S whichincludes all two-factor interaction components in S and all main effects for which a main effect component occurs in atwo-factor interaction component in S. A design is said to support a requirement set if all effects in the requirement set areestimable. One such model of interest consists of the main effects of all factors and a subset of the linear-by-linearinteraction components. This model can be written as

y¼ b0þXrþk

i ¼ 1

bixilþXrþk

i ¼ 1

biixiqþX

1r io jr r

bijxilxjlþE, ð1Þ

where y is the response, E the error term, xil and xiq the linear and quadratic contrast coefficients for factor i and bi, bii theircorresponding effects, bij the linear-by-linear two-factor interaction for factors i and j, and b0 is the grand mean.The number of factors in the core set is r, while the number of factors outside of the core set is k. For example, consider thecase of r¼ k¼ 2 for the model given in (1). In this case the requirement set is S¼ fb1,b11,b2,b22,b3,b33,b4,b44,b12gwhile thecore set is CðSÞ ¼ fb1,b11,b2,b22,b12g, where the effects of the factors that are not involved in any interaction are removed.The core set will play an important role in both the existence and search for designs that support a requirement set. Ifmultiple non-isomorphic designs support a requirement set, a distinction is made between these designs using one ormore optimality criterion, some of which will now be discussed.

Consider a model including all effects in the requirement set,

Y ¼ XMbMþE, ð2Þ

where Y is the vector of n observations, bM is the vector of effects in the requirement set, XM is the corresponding matrix ofcontrast coefficients, and E is the vector of n independent random errors. The matrix XM is also referred to as the modelmatrix. Define

M¼ XTMXM=n¼ ½mij�, ð3Þ

as the information matrix. A D-optimal design maximizes 9M9, the determinant of M, and minimizes the volume of thejoint confidence region on the vector of regression coefficients. For a design that supports S, its D-criterion is defined as

Dcrit ¼ 9XTMXM=n91=p

,

where p is the number of parameters in the model (i.e. the size of the requirement set). If Dcrit ¼ 1, the columns of XM areorthogonal.

There are other optimality criteria that use the information matrix. These include the Eðs2Þ criterion, where

Eðs2Þ ¼X

1r io jrp

m2ij=

p

2

� �,

and E-optimality, which maximizes the minimum eigenvalue of the information matrix, and minimizes the maximumpossible variance of a normalized linear function of bM . Another common criterion is A-optimality which minimizestrðM�1

Þ, the trace of the inverse of the information matrix, and minimizes the average variance of the estimates of theregression coefficients.

In this paper, our first means of ranking designs is by the D-criterion. For designs having the same D-criterion value, wefurther distinguish them by measuring their robustness to nonnegligible effects outside of the requirement set.

To get an estimate for bM in Model (2), we use the least squares estimator, b̂M ¼ ðXTMXMÞ

�1XTMY . This estimator is

unbiased for bM if Model (2) is true. While the model we fit is (2), assume that the true model is actually

Y ¼ X0b0þX1b1þX2b2þE, ð4Þ

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132 125

where X1 and X2 correspond to the matrix of contrast coefficients for the main effects, b1, and two-factor interactions, b2,

X0 the vector of ones with b0 being the grand mean, and E the error term. Note that XM contains the columns of X0, X1, anda subset of columns from X2. An alternative way to write (4) is then

Y ¼ XMbMþX2ob2oþE, ð5Þ

where b2o refers to the two-factor interaction components from b2 outside of the requirement set, and X2o thecorresponding matrix of contrast coefficients.

In order to differentiate between designs having the same D-criterion, we can minimize the contamination ofnonnegligible two-factor interactions outside of the model. Due to restrictions on the run size, even if we have reason tobelieve Model (4) is appropriate, the model with all the main effects and two-factor interactions may not be estimable. Ifwe fit Model (2) when Model (4) is the truth,

Eðb̂M Þ ¼ bMþC2ob2o:

In this paper we call C2o ¼ ½cij� ¼ ðXTMXMÞ

�1XTMX2o the alias matrix. Because of b2o, estimation of bM is contaminated by the

nonnegligible two-factor interactions outside of the model, implying that C2o should be small. One way of minimizing thiscontamination is through the minimum contamination criterion, that minimizes JC2oJ

2¼P

c2ij (Tang and Deng, 1999; Xu

and Wu, 2001; Steinberg and Bursztyn, 2001).In Eq. (5), X2o is the matrix of contrast coefficients corresponding to all two-factor interactions outside of the

requirement set. In practice, linear-by-linear two-factor interactions are more often active than other interactions (Xuet al., 2004), so one could consider the contamination of linear-by-linear two-factor interactions not in the requirementset. In this case, we would minimize JC2lJ

2, with

JC2lJ2¼ JðXT

MXM�1XT

MX2lJ2, ð6Þ

where X2l is the subset of columns from X2o that correspond to the linear-by-linear components outside of the model. If welet X2q be the matrix consisting of the remaining columns from X2o whose components contain a quadratic effect (i.e.X2q ¼ X2o\X2l), we have

JC2qJ2¼ JðXT

MXM�1XT

MX2qJ2: ð7Þ

To minimize the contamination from the two-factor interactions containing a quadratic main effect, we would minimizeJC2qJ

2. Combining (6) and (7), we have

JC2oJ2¼ JC2lJ

2þJC2qJ

2: ð8Þ

Our consideration of nonnegligible effects has been through two-factor interactions. In general, we can consider i-factorinteractions through Ci ¼ ðX

TMXMÞ

�1XTMXi, where Xi is the matrix of contrast coefficients for the i-factor interactions and

measure the contamination of these effects with JCiJ2. If we use the hierarchical ordering principle on the effects outside of

the requirement set, we can sequentially minimize JC2oJ2,JC3J

2, . . . as a way to distinguish between designs.

3. Level permutations

We now look at the effect of permuting the levels of a factor on the properties of a design. For multi-level designs,because of the multiple components for main effects and interactions, level permutations of factors can impact a design,where it is possible that one set of level permutations supports a requirement set while another does not for the samedesign. Remark 2.3.1 from Dey and Mukerjee (1999) states that their main results on factorial designs do not depend onthe choice of level permutations. While this holds in situations where we are interested in all components of interactions,this invariance to level permutations may no longer hold if our requirement set contains only certain interactioncomponents. In this section, we examine the influence of level permutations on the optimality criteria in Section 2. Theresults focus on level permutations on factors outside of the core set, as it will be shown that such level permutationsretain many design properties. In the study of level permutations, we note here the difference between combinatorial andgeometric isomorphism. Designs are said to be geometrically isomorphic (Cheng and Ye, 2004; Katsaounis et al., 2007) ifone can be obtained from the other via permutations of row and column permutations and reversing the level labelordering for one or more of the factors. Combinatorially isomorphic designs (Schoen et al., 2010) allow any permutation ofsymbols within columns. For the situation considered here, geometrically equivalent designs will give the same results asthey preserve statistical properties of the designs, while combinatorially isomorphic designs may not have the sameproperties and give different results. Some geometric non-equivalent designs may be equivalent in our case due to theseparation of factors in and out of the core set. Throughout this paper we assume that the design is an orthogonal array,implying main effects are orthogonal and the lack of orthogonality with effects in the requirement set comes through thetwo-factor interaction components in the requirement set.

Throughout this paper, our consideration of level permutations is on their effect to the matrix of contrast coefficients.We will make use of the fact that a level permutation on a factor can be performed via an orthonormal matrix Q1, whereQT

1Q1 ¼ I. Let A denote the n� ðq�1Þ matrix containing the q�1 vectors of contrast coefficients for the main effects of afactor for the n runs of the design. Let An consist of the vectors of contrast coefficients for the same factor after a level

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132126

permutation. The relationship between An and A can be expressed with the ðq�1Þ � ðq�1Þ orthonormal matrix Q1 as

An¼ AQ1: ð9Þ

From Eq. (9), we have a matrix Q1 to perform a level permutation on a factor in the model. Consider a level permutationon a factor outside of the core set. Denote XM as the model matrix before the level permutation, and Xn

M as the modelmatrix after the level permutation. If we let

Q ¼Q1 0

0 Ip�ðq�1Þ

!,

we can relate XM and Xn

M through

Xn

M ¼ XMQ , ð10Þ

where, without loss of generality, the first q�1 columns of XM correspond to the main effect components of the levelpermuted factor. The information matrix after the level permutation, Mn, is then

Mn¼ ðXMQ ÞT ðXMQ Þ=n¼QT MQ , ð11Þ

where M is the information matrix before the level permutation.At this point, it is important to make the distinction between level permutations of factors inside the core set versus

factors outside of the core set. A level permutation of a factor inside the core set causes permutations of the two-factorinteraction components in the requirement set, which may not allow us to write Xn

M in the form given in (10). When we areable to write Mn in the form of (11), then Mn and M have the same eigenvalues. This provides the following result.

Theorem 1. When the levels of a factor outside of the core set are permuted, the D, A, E, and Eðs2Þ criteria remain the same.

Theorem 1 implies that many criteria are not impacted by level permutations outside of the requirement set, inparticular the D-criterion. For our purposes of ranking designs, it does not address the contamination of the two-factorinteraction components outside of the requirement set. This connection is not immediately obvious, since measurement ofthis contamination is based on XM and X2o, both of which are affected by level permutations. We address this relationshipin the following theorem.

Theorem 2. When the levels of a factor outside of the core set are permuted, the contamination of the two-factor interaction

components outside of the model, measured by JC2oJ2, remains the same.

The proof of Theorem 2 is provided in Appendix. Combined with Theorem 1, Theorem 2 implies that level permutationson factors outside of the core set have no impact on our ranking of designs. The results in this section assume that we havea design that supports a requirement set, but offer no insight into how to find such a design. The next section discusses theexistence of designs that support a requirement set.

4. Existence of designs for a requirement set through orthogonal arrays

Tang and Zhou (2009) showed that for two-level designs, the existence of an orthogonal array that supports arequirement set is equivalent to that of an orthogonal array that supports its core set. For designs with greater than twolevels, a similar result holds, but the fact that main effects and interactions have multiple components needs to be takeninto account. While the requirement set may contain only a subset of the interaction components, all main effectcomponents are in the requirement set.

We consider removing columns representing main effect components from a matrix of contrast coefficientscorresponding to a saturated orthogonal array, such that the main effect components occupy the same space as theinteraction components in the core set. The remaining main effect components can be used to estimate effects outside ofthe core set. If we identify any one main effect component to remove, we must remove all remaining main effectcomponents for that factor. The results are presented based on three-level designs, as they are most powerful in this case,but the arguments are applicable for greater than three levels.

Given a saturated orthogonal array W and requirement set S, let D be the corresponding contrast matrix where eachcolumn of W is replaced by normalized main effect contrasts. We will show that so long as there is a subset D1 of D, thatsupports the core set C(S), there exists a partition D¼ ðD1,D2,D3Þ, where D2 is the set of columns in S that are not in C(S) andD3 are the columns to be removed from D that occupy the same space as the vectors corresponding to interactioncomponents. We will denote the number of factors in D1, D2, and D3 as m1, m2, and m3, respectively, and m¼m1þm2 thetotal number of factors in the requirement set.

Theorem 3. For a requirement set S with m factors and e two-factor interaction components, if a contrast matrix from a

saturated orthogonal array that supports the core set C(S) with m1 factors exists, then a contrast matrix from the orthogonal

array that supports S exists, provided mrðn�1Þ=2�e.

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132 127

Proof. Consider a matrix D constructed from a saturated three-level orthogonal array, W , where each column is replacedby normalized main effect contrasts, such as the coding for linear and quadratic main effects, and a column of all 1’s addedto the array.

Let X1c ¼ ð1,D1Þ, where 1 is the column of all 1 s corresponding to the grand mean and D1 a subset of D that supportsC(S). The model matrix for C(S) is then ðX1c ,X2cÞ, where X2c corresponds to the interaction components in the core set.The partition D¼ ðX1c ,D2,D3Þ forms the column space of D. If we let Dn

¼ ðD2,D3Þ, by the properties of block matrices,

det½ðX1c ,X2cÞTðX1c ,X2cÞ�

¼ detðXT1cX1cÞdetðXT

2cX2c�XT2cX1cðX

T1cX1cÞ

�1XT1cX2cÞ

¼ detðXT1cX1cÞdetðXT

2cDnDnT X2cÞ

¼ n2m1þ1�edetðXT2cDnDnT X2cÞ: ð12Þ

Since D1 supports C(S), detðXT2cDnDnT X2cÞ40, so XT

2cDnDnT X2c must be of rank e. Recalling that rankðAATÞ ¼ rankðAÞ, this also

implies that XT2cDn has rank e.

The dimensions of XT2cDn is e� ð2m2þ2m3Þ. In order for X2c to be estimated, it must be that ð2m2þ2m3ÞZ2m3Ze. Then

XT2cDn must contain a subset of 2m3 columns that has rank e, corresponding to m3 factors (each with two main effect

components) from Dn. As the columns in D3 are pairs of main effect components for factors, the preceding bound ism3Zde=2e. It is possible that there exists a subset of mn

3om3 columns such that XT2cDn has a subset of 2mn

3 columns withrank e. Let Dn

3 be a subset of 2m3 columns from Dn, so that rankðXT2cDn

3Þ ¼ e. This also gives us rankðXT2cDn

3DnT3 X2cÞ ¼ e and

detðXT2cDn

3DnT3 X2cÞ40: ð13Þ

With Dn¼ ðDn

2,Dn

3Þ, let X ¼ ðX1,Dn

2Þ be the model matrix for the requirement set S. Similar to (12), we have

det½ðX,X2cÞTðX,X2cÞ� ¼ n2mþ1�edetðXT

2cDn

3DnT3 X2cÞ, ð14Þ

which is greater than 0 by (13), and implies that ðD1,Dn

2Þ supports S. If there exists mn

3 as described above, S canaccommodate additional factors. &

From Eq. (14), we also get the following result.

Corollary 1. The D-criterion is maximized when detðXT2cDn

3DnT3 X2cÞ is maximized.

Theorem 3 provides a lower bound on the maximum number of factors outside of the core set that can be supported. Inthe formation of Dn

3 in the proof of Theorem 3, the worst-case scenario is that the e two-factor interaction components takeup an e-dimensional space that corresponds to e different factors occupying a 2e-dimensional space. However, it ispossible that the e two-factor interaction components occupy the 2en-dimensional space from enoe different factors, aconsideration we return to in Section 5 when searching for designs. Theorem 3 and Corollary 1 provide insight which willbe useful in the next section to aid in searching for designs.

It is clear that level permutations for factors outside of the core set or model have no effect on the results of Theorem 3and Corollary 1. This is not true for factors in the core set, as level permutations change X2c. Even if the core set issupported by a D1 after a level permutation of a factor within it, the corresponding D2 and D3 will not be the same. Inaddition, it is possible that level permutations on the core set allow additional factors to be considered outside of the coreset or an improvement to the D-criterion. This implies that when searching for designs, different level permutations offactors in the core set should be considered, as will be done in the next section.

5. Searching for designs with high D-criterion values and robust properties from orthogonal arrays

In this section, the previous results are applied to search for designs that support a requirement set, S, if a saturatedorthogonal array of the required run size is available. Let r be the number of factors in the core set, and k the number offactors outside of the core set. For an orthogonal array W with n runs and m columns having q levels each, replace eachcolumn by a set of q�1 orthogonal contrasts to create the matrix D. Our goal is to find the best subset of rþkrm factors toform the design. By Theorem 1, level permutations outside of the core set do not need to be considered. When we refer tochoosing a factor from f1, . . . ,mg, we mean the set of orthogonal contrasts of that factor, and by performing a levelpermutation, we mean considering the effect of said level permutations on the contrast matrix D. Corollary 1 allows theidentification of the best choice of factors to be removed once a set of factors for the core set has been specified. The searchproceeds as follows:

1.

Let c¼ ðc1, . . . ,crÞ, a subset of size r from f1, . . . ,mg, denote the factors in the core set. 2. Let pc ¼ ðpc1

, . . . ,pcrÞ denote a set of level permutations on the factors in the core set.

3.

Denote D1 as the set of columns from D formed by the factors given by c, after performing the level permutations uponthe factors as given by pc. With the main effect components in the core set specified, X2c is the set of contrastcoefficients for the interaction components in the requirement set.

4.

If C(S) is supported by D1,

TabFul

C

(2

(2

(3

(2

(7

(1

(4

(3

(5

(3

(4

(1

(2

(1

(1

(5

(2

(8

(1

(2

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132128

(a) Take a subset of size m�r�k from f1, . . . ,mg\c to be the factors from the saturated orthogonal array not to be used inthe design, and use these to form Dn

3 from D. The design under consideration is ðD1,Dn

2Þ, where Dn

2 is comprised of theremaining columns from D not in D1 or Dn

3.(b) Calculate detðXT

2cDn

3DnT3 X2cÞ.

(c) Repeat Steps 4a and 4b for all ð m�rm�r�kÞ possible subsets from the remaining columns.

le 1l Sear

ore

, 3, 7

, 11,

, 5, 1

, 5, 8

, 8, 1

, 11,

, 6, 1

, 12,

, 7, 1

, 8, 1

, 6, 1

, 6, 1

, 10,

, 4, 9

, 3, 6

, 10,

, 9, 1

, 10,

, 2, 1

, 6, 8

5.

Repeat Steps 2–4 for all possible level permutations pc. 6. Repeat Steps 1–5 for all c � f1, . . . ,mg such that 9c9¼ r.

By Corollary 1, the D-optimal design among those from the saturated orthogonal array is the design for whichdetðXT

2cDn

3DnT3 X2cÞ in Step 4b is maximized. Recall that Theorem 3 gives a lower bound on the number of factors that can be

supported outside of the core set. Provided there are sufficient degrees of freedom, the algorithm allows us to search fordesigns in which k is greater than the bound in Theorem 3.

The algorithm as presented is applicable to search for D-optimal designs. If we want to minimize the contaminationfrom the two-factor interaction components outside of the requirement set, JC2oJ

2 can be calculated in Step 4b, orcalculated for those designs with the largest D-efficiencies after the search is complete. Theorems 1 and 2 imply that we donot need to consider any level permutations for the factors outside of the core set, as this will have no influence on theD-criterion or contamination from the two-factor interaction components outside of the model.

We now apply the above algorithm to a special case of the second order model, with a requirement set that containstwo-factor interaction components for a subset of the factors in the requirement set. The algorithm will be used to searchfor designs with high D-criterion values that minimize the contamination from the two-factor interaction componentsoutside of the model. The model of interest in this section is the second order model (1) as introduced in Section 2

y¼ b0þXrþk

i ¼ 1

bixilþXrþk

i ¼ 1

biixiqþX

1r io jr r

bijxilxjlþE, ð15Þ

where we are interested only in the linear-by-linear two-factor interaction effects from a subset of r factors out of m¼ rþk

factors.Consider the case r¼3 and k¼8. If we were to consider all possible column choices and level permutations, we would

need ð133 Þ � ð

108 Þ � 311

¼ 2,279,881,890 different combinations. Theorems 1 and 2 allow us to restrict level permutations tothe core set, giving 13

3 � ð108 Þ � 33

¼ 347,490 different combinations. Only three level permutations per factor in the core setare considered, as reflection of a factor around the center level retains the projection properties of the design, including D-criterion, as discussed in Xu et al. (2004). With / indicating the levels before and after level permutation, we refer topermutation 0 as ð012Þ/ð012Þ, permutation 1 as ð012Þ/ð120Þ and permutation 2 as ð012Þ/ð201Þ.

Using the catalog of 27-run orthogonal arrays with 13 three-level factors of Evangelaras et al. (2011) we denote the24th and 52nd designs from the catalog as D24 and D52. We consider r¼3 and k¼8, searching for designs in which therequirement set contains the linear and quadratic main effect components for 11 factors, and a subset of size 3 of thesefactors in which the linear-by-linear two-factor interaction components are in the requirement set. Tables 1 and 2 showthe top 20 designs found from each parent design in terms of the D-criterion. We rank the designs firstly by D-criterion,followed by contamination of the two-factor interaction components outside of the model. We also include the

ch for D24 with r¼3, k¼8.

Outside Perm D-crit JC2oJ2 JC2lJ

2

) (4, 5, 6, 8, 9, 10, 11, 13) (1, 1, 0) 0.90751 210.3599 126.0924

12) (1, 3, 4, 5, 7, 9, 10, 13) (1, 1, 0) 0.90751 211.7624 121.6943

2) (1, 2, 4, 6, 7, 9, 11, 13) (0, 2, 1) 0.90751 211.9441 124.1502

) (1, 3, 6, 9, 10, 11, 12, 13) (1, 2, 2) 0.90751 211.9999 127.5223

3) (1, 2, 3, 4, 6, 9, 11, 12) (2, 0, 1) 0.90751 212.1622 121.0393

12) (2, 3, 4, 5, 7, 8, 9, 10) (1, 2, 0) 0.90751 212.3426 128.7172

1) (1, 2, 3, 8, 9, 10, 12, 13) (2, 0, 2) 0.90751 212.6203 116.1502

13) (1, 4, 5, 6, 7, 8, 9, 11) (0, 2, 0) 0.90751 212.7159 128.2826

0) (1, 3, 4, 6, 8, 11, 12, 13) (0, 1, 1) 0.90751 212.7527 124.0225

1) (1, 2, 4, 5, 6, 9, 10, 12) (1, 1, 0) 0.90751 212.7751 123.4190

3) (1, 2, 5, 8, 9, 10, 11, 12) (1, 0, 2) 0.90751 212.7822 122.4536

0) (2, 3, 4, 7, 9, 11, 12, 13) (0, 1, 0) 0.90751 212.8084 121.4064

13) (3, 4, 5, 7, 8, 9, 11, 12) (2, 1, 2) 0.90751 212.8944 117.4062

) (2, 3, 5, 6, 7, 8, 10, 13) (2, 0, 2) 0.90751 212.9416 121.6931

) (2, 4, 5, 7, 9, 10, 12, 13) (0, 2, 2) 0.90751 212.9611 122.0260

11) (1, 3, 4, 6, 7, 8, 9, 13) (0, 0, 2) 0.90751 212.9723 124.3190

3) (1, 3, 5, 7, 8, 10, 11, 12) (2, 2, 0) 0.90751 213.0855 130.5637

12) (1, 2, 3, 4, 5, 6, 7, 13) (1, 2, 2) 0.90751 213.1078 127.5364

2) (3, 4, 5, 6, 7, 9, 10, 11) (2, 0, 0) 0.90751 213.1779 113.1100

) (1, 3, 4, 5, 10, 11, 12, 13) (0, 1, 2) 0.90751 213.1784 122.3386

Table 2Full search for D52 with r¼3, k¼8.

Core Outside Perm D-crit JC2oJ2 JC2lJ

2

(1, 3, 11) (4, 5, 6, 7, 8, 9, 10, 13) (2, 0, 2) 0.91642 212.5225 119.0060

(1, 3, 11) (2, 4, 5, 6, 7, 8, 10, 13) (2, 0, 2) 0.91642 212.6413 122.6324

(3, 7, 9) (1, 2, 4, 6, 8, 10, 12, 13) (0, 2, 2) 0.91642 212.6723 127.5303

(3, 7, 9) (1, 2, 4, 6, 8, 11, 12, 13) (0, 2, 2) 0.91642 212.6723 130.0322

(2, 3, 10) (4, 5, 7, 8, 9, 11, 12, 13) (2, 0, 2) 0.91642 212.8388 121.9882

(2, 3, 10) (1, 4, 5, 8, 9, 11, 12, 13) (2, 0, 2) 0.91642 212.9387 126.4317

(1, 4, 7) (3, 5, 8, 9, 10, 11, 12, 13) (0, 1, 0) 0.91018 211.0688 114.7868

(1, 4, 7) (2, 3, 5, 8, 9, 11, 12, 13) (0, 1, 0) 0.91018 211.0787 123.9678

(4, 10, 11) (1, 2, 3, 6, 8, 9, 12, 13) (1, 1, 1) 0.91018 212.2993 130.6163

(4, 10, 11) (1, 2, 3, 6, 7, 8, 12, 13) (1, 1, 1) 0.91018 212.3142 133.0460

(2, 4, 9) (3, 5, 6, 7, 8, 10, 11, 13) (1, 1, 0) 0.91018 214.3700 125.4501

(2, 4, 9) (1, 3, 5, 6, 7, 8, 10, 13) (1, 1, 0) 0.91018 214.5439 127.2921

(1, 2, 13) (3, 4, 6, 8, 9, 10, 11, 12) (2, 2, 1) 0.90474 211.5843 105.1457

(1, 2, 13) (3, 4, 6, 7, 8, 10, 11, 12) (2, 2, 1) 0.90474 211.5843 107.5666

(7, 10, 13) (2, 3, 4, 5, 6, 8, 9, 11) (0, 1, 1) 0.90474 212.1420 124.6016

(7, 10, 13) (1, 2, 3, 4, 5, 6, 8, 9) (0, 1, 1) 0.90474 212.1420 125.6868

(9, 11, 13) (1, 3, 4, 5, 7, 8, 10, 12) (0, 1, 1) 0.90474 212.9785 125.3592

(9, 11, 13) (1, 2, 3, 4, 5, 7, 8, 12) (0, 1, 1) 0.90474 212.9785 126.7352

(1, 2, 13) (3, 4, 6, 8, 9, 10, 11, 12) (0, 1, 1) 0.90474 213.8150 121.8106

(1, 2, 13) (3, 4, 6, 7, 8, 10, 11, 12) (0, 1, 1) 0.90474 213.8150 122.5939

Table 3Full search for D24 with r¼4, k¼6.

Core Outside Perm D-crit JC2oJ2 JC2lJ

2

(1, 4, 5, 9) (3, 6, 7, 10, 11, 12) (0, 0, 0, 1) 0.81496 162.3934 89.64550

(3, 5, 12, 13) (1, 2, 7, 8, 10, 11) (0, 0, 0, 1) 0.81496 162.3934 93.77505

(5, 7, 10, 11) (2, 3, 4, 8, 9, 12) (0, 0, 2, 0) 0.81496 162.6169 94.57458

(1, 3, 6, 10) (4, 5, 8, 11, 12, 13) (0, 0, 0, 2) 0.81496 162.6169 95.74347

(4, 8, 10, 12) (1, 2, 5, 6, 9, 11) (0, 2, 2, 0) 0.81496 162.8417 99.40190

(1, 2, 11, 12) (3, 6, 8, 9, 10, 13) (0, 2, 0, 2) 0.81496 163.6982 81.27873

(1, 7, 8, 13) (2, 3, 4, 5, 6, 10) (0, 0, 2, 1) 0.81496 163.9439 97.01499

(6, 7, 9, 12) (1, 2, 3, 4, 11, 13) (0, 0, 1, 0) 0.81496 164.0664 94.07995

(4, 6, 11, 13) (1, 3, 5, 7, 8, 9) (0, 0, 0, 1) 0.81496 164.0664 94.51895

(5, 7, 10, 11) (2, 3, 4, 8, 9, 12) (2, 0, 2, 0) 0.81496 164.2559 88.50985

(1, 2, 11, 12) (3, 6, 8, 9, 10, 13) (0, 2, 0, 0) 0.81496 164.2900 76.67385

(2, 3, 4, 7) (1, 5, 9, 10, 12, 13) (2, 0, 0, 0) 0.81496 164.2900 86.31036

(2, 5, 6, 8) (4, 7, 9, 10, 11, 13) (2, 0, 0, 2) 0.81496 164.5147 78.08652

(3, 5, 12, 13) (1, 2, 7, 8, 10, 11) (1, 0, 0, 1) 0.81496 165.0664 97.94988

(2, 9, 10, 13) (1, 4, 6, 7, 8, 12) (2, 1, 2, 1) 0.81496 165.1687 101.91022

(3, 8, 9, 11) (2, 5, 6, 7, 12, 13) (0, 2, 1, 0) 0.81496 165.6169 91.77722

(1, 7, 8, 13) (2, 3, 4, 5, 6, 10) (0, 0, 2, 2) 0.81496 165.8136 92.62801

(2, 5, 6, 8) (4, 7, 9, 10, 11, 13) (2, 0, 0, 0) 0.81496 165.9630 81.70841

(2, 3, 4, 7) (1, 5, 9, 10, 12, 13) (2, 0, 0, 1) 0.81496 165.9975 81.86348

(3, 8, 9, 11) (2, 5, 6, 7, 12, 13) (0, 2, 1, 1) 0.81496 166.2218 96.82280

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132 129

contamination from only the linear-by-linear two-factor interaction components outside of the model. The core columnrefers to the set of columns from Di (i¼ 24,52) that form the r factors in the core set, and outside the columns used for thek remaining factors. The set of level permutations for the factors in the core set is denoted by perm, the D-criterion byD-crit, and the contamination from the two-factor interaction components outside of the model JC2oJ

2, with the linear-by-linear contribution being JC2lJ

2. Tables 3 and 4 show the top designs from a complete search for r¼4 and k¼6 for D24 andD52.

Comparing Tables 1–4, we see that D24 and D52 provide different designs according to the D-criterion for the second-ordermodel (15). The highest D-criterion comes from D52, and the highest D-criterion for D24 has less contamination from the two-factor interaction components measured by JC2oJ

2. Also, the number of possible designs that give the same D-criterion differsbetween the parent designs. For D24, there tends to be a large number of level permutations and factor assignments that givethe same D-criterion in comparison to D52. If we were to restrict ourselves to a random selection of core sets rather than allpossibilities, we are more likely to miss designs with a high D-criterion value among D52 versus D24.

The designs D24 and D52 were chosen to highlight the differences that can arise between different parent designs.In Tables 5 and 6 we show further design results using a search method that will be discussed in a further publication.The Parent column in Tables 5 and 6 is in regards to the design from Evangelaras et al. (2011).

Table 4Full search for D52 with r¼4, k¼6.

Core Outside Perm D-crit JC2oJ2 JC2lJ

2

(2, 5, 11, 13) (3, 4, 6, 8, 9, 12) (0, 1, 2, 0) 0.83792 162.4883 97.00187

(6, 7, 11, 13) (3, 4, 5, 8, 10, 12) (0, 2, 0, 0) 0.83792 163.0312 88.83047

(9, 10, 12, 13) (3, 4, 5, 6, 8, 11) (2, 0, 0, 0) 0.83792 164.0350 88.85841

(1, 5, 10, 13) (3, 4, 6, 7, 8, 12) (1, 0, 2, 0) 0.83792 165.1516 95.71076

(9, 10, 12, 13) (3, 4, 5, 6, 8, 11) (0, 0, 1, 2) 0.83792 165.1856 91.56477

(1, 6, 9, 13) (2, 3, 4, 5, 8, 12) (2, 1, 1, 0) 0.83792 165.4647 93.95153

(2, 7, 12, 13) (1, 3, 4, 5, 6, 8) (2, 1, 1, 0) 0.83792 166.0109 87.22564

(6, 7, 11, 13) (3, 4, 5, 8, 10, 12) (1, 0, 0, 2) 0.83792 166.1894 98.86032

(1, 6, 9, 13) (2, 3, 4, 5, 8, 12) (0, 0, 1, 2) 0.83792 167.0529 90.68273

(2, 5, 11, 13) (3, 4, 6, 8, 9, 12) (0, 0, 1, 2) 0.83792 167.3160 93.69656

(2, 7, 12, 13) (1, 3, 4, 5, 6, 8) (1, 1, 0, 2) 0.83792 167.3691 89.23705

(1, 5, 10, 13) (3, 4, 6, 7, 8, 12) (1, 1, 1, 2) 0.83792 169.3557 96.54786

(2, 3, 6, 7) (4, 5, 8, 9, 12, 13) (0, 1, 1, 2) 0.82637 164.8869 98.48818

(1, 3, 9, 12) (4, 5, 6, 7, 8, 13) (1, 1, 2, 1) 0.82637 164.9460 95.64872

(2, 3, 11, 12) (4, 5, 6, 8, 10, 13) (2, 1, 0, 0) 0.82637 166.1974 87.74581

(1, 3, 6, 10) (4, 5, 8, 11, 12, 13) (2, 1, 0, 0) 0.82637 166.5262 86.34181

(3, 5, 7, 11) (1, 4, 6, 8, 12, 13) (1, 0, 1, 2) 0.82637 167.4242 91.53866

(3, 5, 9, 10) (2, 4, 6, 8, 12, 13) (1, 1, 1, 2) 0.82637 169.0701 95.40400

(1, 4, 5, 10) (3, 6, 7, 8, 12, 13) (2, 0, 0, 1) 0.82474 164.6268 86.66257

(2, 4, 5, 11) (3, 6, 8, 9, 12, 13) (2, 0, 1, 1) 0.82474 164.8185 89.98030

Table 5Using search method for r¼3, k¼8.

Parent Core Outside Perm D-crit JC2oJ2 JC2lJ

2

129 (2, 10, 11) (1, 4, 5, 6, 7, 9, 12, 13) (2, 1, 1) 0.93953 210.8255 117.1609

127 (2, 9, 11) (1, 4, 6, 7, 8, 10, 12, 13) (2, 1, 1) 0.93953 210.8255 118.2195

125 (4, 9, 11) (1, 2, 6, 7, 8, 10, 12, 13) (2, 1, 1) 0.93953 210.8255 119.9672

126 (5, 10, 13) (1, 3, 4, 7, 8, 9, 11, 12) (0, 0, 2) 0.93953 210.8255 122.2694

95 (5, 9, 11) (1, 3, 4, 7, 8, 10, 12, 13) (0, 0, 2) 0.93953 210.8255 122.2694

129 (7, 9, 11) (1, 3, 4, 5, 6, 8, 10, 13) (1, 0, 0) 0.93953 212.0503 118.9779

97 (4, 5, 9) (1, 3, 6, 7, 8, 10, 11, 12) (2, 2, 0) 0.93953 212.0503 119.7267

124 (3, 7, 11) (1, 2, 5, 6, 9, 10, 12, 13) (2, 1, 2) 0.93953 212.0503 121.6325

114 (2, 10, 11) (1, 4, 5, 7, 8, 9, 12, 13) (0, 0, 0) 0.93953 212.0503 122.7779

128 (10, 11, 12) (2, 3, 4, 5, 7, 8, 9, 13) (1, 0, 0) 0.93953 212.0503 124.2831

113 (5, 7, 9) (1, 3, 4, 8, 10, 11, 12, 13) (1, 1, 2) 0.93953 212.2744 123.5556

128 (7, 12, 13) (1, 3, 4, 5, 6, 8, 9, 11) (0, 2, 2) 0.93953 212.3291 123.8918

95 (8, 11, 12) (1, 3, 4, 5, 6, 7, 9, 10) (0, 2, 0) 0.93953 212.4985 126.1110

126 (9, 11, 13) (1, 3, 4, 5, 6, 7, 8, 10) (0, 0, 2) 0.93953 212.4985 126.1110

128 (5, 11, 13) (1, 2, 3, 6, 7, 8, 9, 12) (0, 2, 2) 0.93953 212.8868 122.7850

97 (5, 11, 12) (2, 3, 4, 6, 7, 8, 9, 10) (2, 1, 1) 0.93953 213.0562 124.5567

115 (5, 11, 12) (2, 3, 4, 6, 7, 8, 9, 10) (2, 1, 1) 0.93953 213.0562 124.6173

127 (2, 7, 10) (1, 4, 5, 6, 8, 9, 11, 13) (2, 2, 2) 0.93953 213.2750 128.2857

125 (4, 7, 10) (1, 2, 5, 6, 8, 9, 11, 13) (2, 2, 2) 0.93953 213.2750 130.1711

96 (4, 7, 11) (1, 2, 6, 8, 9, 10, 12, 13) (0, 1, 1) 0.93953 213.7233 118.9792

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132130

6. Discussion

Xu et al. (2004) suggested using G2 and G-aberration in choosing designs for second-order models for factor screeningand interaction detection. The criteria of G and G2-aberration do not take a specific requirement set into consideration.While minimum G and G2-aberration designs may perform well when we consider finding designs that are applicableunder a variety of models with two-factor interaction components involving fewer factors, in this situation there arespecific two-factor interaction components to be estimated. The use of G and G2-aberration is to give designs that remainefficient over many subsets of factors. However, when we have a particular set of effects to be estimated, minimum G andG2-aberration designs may not be the best choice.

The requirement set problem considered here does require some prior knowledge about the effects. The fact that we aretrying to estimate the curvature suggests enough information that the experimenter is within a region that curvature ispossible. In terms of the two-factor interactions, we are assuming those in the requirement set have a different level ofimportance to the experimenter such that they want to ensure the estimation of these effects. This may be fromknowledge suggesting negligibility of certain interactions, or simply experimental objectives making certain interactionsof greater interest.

Table 6Using search method for r¼4, k¼6.

Parent Core Outside Perm D-crit JC2oJ2 JC2lJ

2

2 (5, 7, 11, 12) (2, 4, 6, 8, 9, 13) (1, 1, 1, 1) 0.88509 161.1687 98.15536

76 (7, 10, 11, 13) (1, 2, 5, 6, 8, 9) (0, 2, 0, 0) 0.86761 163.2900 93.78200

76 (9, 10, 11, 13) (2, 4, 6, 7, 8, 12) (1, 2, 1, 0) 0.86761 163.5130 93.93308

76 (7, 9, 10, 13) (1, 4, 5, 6, 11, 12) (2, 2, 2, 0) 0.86761 163.5984 92.42728

75 (1, 2, 9, 13) (4, 5, 6, 8, 10, 11) (2, 2, 2, 1) 0.86761 164.1561 89.39627

75 (1, 4, 9, 13) (2, 5, 6, 7, 11, 12) (1, 0, 2, 1) 0.86761 164.6894 96.30732

75 (2, 4, 9, 13) (1, 6, 7, 8, 10, 12) (0, 1, 2, 1) 0.86761 169.2443 92.07690

66 (9, 10, 11, 13) (1, 2, 5, 6, 7, 12) (1, 0, 1, 1) 0.85940 162.2075 92.83496

66 (7, 9, 10, 13) (2, 4, 5, 6, 8, 11) (2, 2, 0, 1) 0.85940 165.3944 92.85750

66 (7, 10, 11, 13) (1, 4, 6, 8, 9, 12) (0, 0, 0, 1) 0.85940 166.8548 89.62653

68 (6, 8, 12, 13) (2, 4, 5, 7, 10, 11) (1, 0, 2, 2) 0.85302 164.7261 90.92066

68 (6, 7, 12, 13) (1, 2, 8, 9, 10, 11) (1, 0, 2, 1) 0.85302 166.6398 94.51921

67 (2, 4, 6, 13) (1, 5, 7, 10, 11, 12) (1, 1, 1, 2) 0.85302 167.1338 96.88900

67 (1, 2, 6, 13) (4, 7, 8, 9, 10, 12) (0, 2, 1, 2) 0.85302 167.6260 86.36760

68 (6, 7, 8, 12) (1, 4, 5, 9, 10, 13) (1, 2, 1, 2) 0.85302 167.8491 91.30406

67 (1, 4, 6, 13) (2, 5, 8, 9, 10, 11) (1, 2, 1, 2) 0.85302 168.4627 88.96247

69 (1, 2, 9, 13) (3, 4, 5, 7, 11, 12) (1, 2, 2, 1) 0.85045 163.6013 95.86586

70 (7, 8, 10, 12) (2, 3, 4, 9, 11, 13) (2, 1, 2, 0) 0.85045 165.6422 93.10338

69 (1, 4, 9, 13) (2, 3, 7, 8, 10, 12) (2, 0, 2, 1) 0.85045 166.2363 92.74255

70 (7, 10, 12, 13) (1, 2, 3, 5, 8, 9) (1, 2, 0, 2) 0.85045 166.8771 90.22760

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132 131

While much of the attention in this paper has been placed on three-level designs, all of the results hold for different setsof orthogonal contrasts, and for orthogonal contrasts used on factors with more than three levels. Depending on the natureof the interaction components in the core set, it is possible that the search can be simplified even further if an orthonormalmatrix can be used on the interactions, such as when we consider all interaction components for a pair of factors.The results also apply to mixed level designs. This is particularly useful if the effects of interest vary among the factors.For example, in the second order model studied in this paper, if the experimenter is not interested in curvature for thefactors outside of the core set, these factors can be set at two levels. Our searches were based on a saturated orthogonalarray. If we have an orthogonal array available that is not saturated, the results of this paper still apply. In this case, thecolumns in the null space of the design are automatically in D3. Relaxation of the restriction to orthogonal designs is alsoan interesting direction of research.

Acknowledgements

This research is supported by the Natural Science and Engineering Council of Canada. The authors would like to thankan editor and two referees for their helpful comments.

Appendix A

Proof of Theorem 2. After the level permutation, JC2oJ2¼ JðXT

MXM�1XT

MX2oJ2 becomes JCn

2oJ2¼ JðXnT

M Xn

M�1XnT

M Xn

2oJ2. In

order to show JC2oJ2¼ JCn

2oJ2, we will look at the relationship between Xn

M and XM and Xn

2o and X2o. We will make use ofthe fact that

JCJ2¼ trðCT CÞ:

Let Q be the orthonormal matrix from (10) such that Xn

M ¼ XMQ . Since QT¼ Q�1, we can simplify the terms of JCn

2oJ2

involving Xn

M to

ðXnTM Xn

M�1XnT

M ¼ QTðXT

MXM�1XT

M :

As the matrix X2o corresponds to the contrast coefficients for all two-factor interaction components outside of the model,it can be partitioned into sets of two-factor interaction components corresponding to pairs of the m factors in the model.We denote this partition as X2op for p 2 ð1, . . . ,ðm2ÞÞ, where p is an index for a pair of factors from the design. For p indexing apair of factors in the core set, X2op may not contain the full set of ðq�1Þ2 two-factor interaction components, as some ofthese components may be in the requirement set. From this partition of X2o, we have that

JC2oJ2¼X

p

JðXTMXMÞ

�1XTMX2opJ

2: ðA:1Þ

Consider i for which X2oi corresponds to a set of two-factor interaction components for a pair of factors that have not hadlevels permuted. After the level permutation, we have Xn

2oi ¼ X2oi. Looking at the effect of this subset of X2o on JCn

2oJ2 after

R. Lekivetz, B. Tang / Journal of Statistical Planning and Inference 144 (2014) 123–132132

level permutation,

JQTðXT

MXM�1XT

MX2oiJ2¼ JðXT

MXM�1XT

MX2oiJ2

ðA:2Þ

showing that the contamination is unchanged for the pair of factors indexed by i.The remaining columns of X2o correspond to two-factor interaction components a pairs of columns in which one

involves a level permutation. Let X2oj correspond to the n� ðq�1Þ2 subset for such a pair (all of the interaction componentsare outside of the requirement set, otherwise the factor with a level permutation would be in the core set). Let Xn

2oj be thesame matrix after level permutations, and in a similar fashion to (9), there is a ðq�1Þ2 � ðq�1Þ2 orthonormal matrix Q2 suchthat Xn

2oj ¼ X2ojQ2j. The influence on JCn

o2J2 is then

JQTðXT

MXM�1XT

MX2ojQ2jJ2¼ JðXT

MXM�1XT

MX2ojJ2, ðA:3Þ

which is also unchanged from the level permutation. Combining (A.2) and (A.3) into (A.1), we have the required result.&

References

Addelman, S., 1962. Symmetrical and asymmetrical fractional factorial plans. Technometrics 4, 47–58.Cheng, S.-W., Ye, K.Q., 2004. Geometric isomorphism and minimum aberration for factorial designs with quantitative factors. The Annals of Statistics 32,

2168–2185.Dey, A., Mukerjee, R., 1999. Wiley series in probability and statistics: probability and statistics. In: Fractional Factorial Plans. Wiley.Evangelaras, H., Koukouvinos, C., Lappas, E., 2011. 27-run nonisomorphic three level orthogonal arrays: identification, evaluation and projection

properties. Utilitas Mathematica 84, 75–87.Greenfield, A.A., 1976. Selection of defining contrasts in two-level experiments. Applied Statistics 25, 64–67.Jones, B., Nachtsheim, C.J., 2011. Efficient designs with minimal aliasing. Technometrics 53, 62–71.Katsaounis, T.I., Dingus, C.A., Dean, A.M., 2007. On the geometric equivalence and non-equivalence of symmetric factorial designs. Journal of Statistical

Theory and Practice 1, 101–115.Schoen, E.D., Eendebak, P.T., Nguyen, M.V.M., 2010. Complete enumeration of pure-level and mixed-level orthogonal arrays. Journal of Combinatorial

Designs 18, 123–140.Steinberg, D.M., Bursztyn, D., 2001. Discussion of ‘‘factor screening and response surface exploration’’. Statistica Sinica 11, 596–599.Tang, B., Deng, L.-Y., 1999. Minimum G2-aberration for nonregular fractional factorial designs. The Annals of Statistics 27, 1914–1926.Tang, B., Zhou, J., 2009. Existence and construction of two-level orthogonal arrays for estimating main effects and some specified two-factor interactions.

Statistica Sinica 19, 1193–1201.Xu, H., Cheng, S.-W., Wu, C., 2004. Optimal projective three-level designs for factor screening and interaction detection. Technometrics 46, 280–292.Xu, H., Wu, C.F.J., 2001. Generalized minimum aberration for asymmetrical fractional factorial designs. The Annals of Statistics 29, 549–560.