Dr. John Mellor-Crummey

Department of Computer ScienceRice University

johnmc@cs.rice.edu

2k Factorial Designs and2kr Designs with Replications

COMP 528 Lecture 12 24 February 2005

2

Goals for Today

Understand• 2k factorial designs

—sign table method—properties—analysis—allocating variation

• 2kr designs with replications—error estimation—confidence intervals for effects

• Multiplicative models

3

2k Factorial Designs

• Determine effects of k factors, each at two levels• Total of 2k experiments required• Total of 2k effects

—k main effects

— two-factor interactions

— three-factor interactions

— j factor interactions, j ≤ k!

k

2

" # $ % & '

!

k

3

" # $ % & '

!

kj" # $ % & '

4

Example: a 23 Design

Evaluate impact of memory, cache size, # processors

Model performance using a non-linear equation in xA and xB

Solve using sign table method

!

y = q0

+ qA xA + qB xB + qAB xA xB + qAC xA xC +

qBC xB xC + qABC xA xB xC

21# processors, C6MB4MBCache Size, B2GB1GBMemory Size, A

Level 1Level -1Factor

5

Calculating Effects with a Sign Table

qABCqBCqACqABqCqBqAq0

32411-1-1-1-111

BC

86111111150-1-1-111-1158-11-11-111461-111-1-11

216

-11-11

AC

540

1-1-11

AB

20160

-1-1-1-1C

total/8151040total84080320

34102214y

-111111-111-111-1-1-11

ABCBAI

6

Calculating Effects

Effects—SSA/SST = 23(102)/4512 = .18 = 18%—SSB/SST = 23(52)/4512 = .04 = 4%—SSC/SST = 23(202)/4512 = .71= 71%—SSAB/SST = 23(52)/4512 = .04 = 4%—SSAC/SST = 23(22)/4512 = .01 = 1%—SSBC/SST = 23(32)/4512 = .02 = 2%—SSABC/SST = 23(12)/4512 = .00 = 0%

!

SST = 23(qA

2+ qB

2+ qC

2+ qAB

2+ qAC

2+ qBC

2+ qABC

2)

!

SST = 23(10

2+ 5

2+ 20

2+ 5

2+ 2

2+ 3

2+1

2) = 4512

qABCqBCqACqABqCqBqAq0

13252051040

7

2kr Factorial Designs with Replications

8

2kr Factorial Designs

• Cannot estimate errors with 2k factorial design—no experiment is repeated

• To quantify experimental errors—repeat measurements with same factor combinations—analyze using sign table

• 2kr design—r replications of 2k experiments (each of 2k factor combinations)

• Model

(e = experimental error)

!

y = q0

+ qA xA + qB xB + qAB xA xB + e

9

Computing Effects for a 22r Design

• Factor combination = sign table row• For each factor combination i, r replicants (here, r = 3)• = mean observation for ith factor combination

• Solve for qi coefficients using sign table and values

qABqBqAq0

total/459.521.541total20388616477244815

1111-11-11-1-1111-1-11

ABBAI

817575192825514845121815

!

y i. yij

!

y i.

!

y i. =1

ryij

j

"

!

y i.

10

Estimating Experimental Errors for a 22r Design

• Model predicts following value for factor combination i

• Errors

• Total error across row is 0 by design• Corollary: total error (across all rows) is also 0

!

ˆ y i = q0

+ qA xAi + qB xBi + qAB xAixBi

!

eij = yij " ˆ y i= yij " q

0+ qA xAi + qB xBi + qAB xAixBi

= yij " y i.

11

Sum of Squared Errors for a 223 Design

!

SSE = eijj=1

r

"i=1

22

"

77244815

1111-11-11-1-1111-1-11

ABBAI

817575192825514845121815

!

y i. yij

4-2-2-54130-3-330

eij

!

SSE = 02 + 32 + "3( )2

+ "3( )2

+ 02 + 32 +

12 + 42 + "5( )

2

+ "2( )2

+ "2( )2

+ 42 =102

12

Mean Response for a 22r Design

• Model for 22r design

• Sum across all 22r observations =

• Since sum of x’s, their products and errors are all 0!

yij = q0

+ qA xAi + qB xBi + qAB xAixBi + eij

!

yiji, j

" = q0

i, j

" + qA xAii, j

" + qB xBii, j

" + qAB xAixBii, j

" + eiji, j

"

!

yiji, j

" = q0

i, j

" = 22rq

0

!

y ..

=1

22r

yij

i, j

" =1

22r22rq

0= q

0 mean response

13

SST and SSE for a 22r Design

!

SST = yij " y ..( )2

=i, j

# yij

2"

i, j

# y ..

2

i, j

#

!

SST = SSY " SS0 = SSA + SSB + SSAB + SSE

!

SSE = SSY " SS0 " SSA " SSB " SSAB

!

SSE = SSY " 22r(q

0

2+ qA

2+ qB

2+ qAB

2)

!

SS0 = y ..

2

i, j

" = q0

2

i, j

"

!

SSY = yij2

i, j

" = q0

2

i, j

" + qA2xAi2

i, j

" + qB2xBi2

i, j

" +

qAB2xAi2xBi2

i, j

" + eij2

i, j

" + product terms (0 by design of sign table)

!

SSY = SS0 + SSA + SSB + SSAB + SSE

14

Allocating Variation for a 22r Design

• Fraction of variation attributed to A = SSA/SST• Fraction of variation attributed to B = SSB/SST• Fraction of variation attributed to interaction of A & B =

SSAB/SST• Fraction of variation attributed to error = SSE/SST

15

Variance for q0

• Effects computed from a sample are random variables• Confidence intervals for effects

—compute from variance of sample estimates

• If errors ~ , model implies that y ~• Consider effect q0

—q0 = linear combination of normally distributed variables—therefore, q0 is normally distributed as well

• Variance of q0

!

N(0,"e

2)

!

N(y ..," e

2)

!

q0

=1

22r

yiji, j

"

!

var(q0) =

" e

2

22r

16

Confidence Intervals for Effects in 22r Designs

• Estimate variance of errors from SSE

• Denominator = 22(r-1) = DOF = # independent terms in SSE—error terms for r replications all add to 0—only r-1 are independent

• Std dev of effects

• Confidence intervals for effects• Effect is significant if its confidence interval does not contain 0

!

se

2=

SSE

22(r "1)

= MSE

!

sq0

= sqA = sqB = sqAB =se

22r

!

qi ± t[1"# / 2;22 (r"1)]

sqi

17

Returning to our Example …

Memory/Cache Study• Standard deviation of errors

• Standard deviation of effects

• Confidence intervals for effects—90% confidence:

• No confidence intervals include 0 ⇒ all significant

!

se

=SSE

22(r "1)

=102

8= 12.75 = 3.57

!

sqi = se / 22r = 3.57 / 12 =1.03

qABqBqAq0

59.521.541

!

qi ± t[1"# / 2;22 (r"1)]

sqi = qi ± (1.86)(1.03) = qi ±1.92

!

t[1"# / 2;22 (r"1)]

= t[.95;8] =1.86

q0= (30.08,42.91), qA= (19.58, 23.41), qB= (7.58, 11.41), qAB= (3.08, 6.91)

18

Assumptions

• Model errors are statistically independent• Errors are additive• Errors are normally distributed• Errors have constant standard deviation• Effects of factors are additive• observations are independent and normally distributed with

constant variance!

"e

19

Visual Tests for Verifying Assumptions

• Independent errors—scatter plot of residuals vs. predicted responses

– should not have any trend– magnitude residuals < magnitude responses/10 ⇒ ignore trends

—plot residuals as a function of experiment number– trend up or down implies other factors or side effects

• Normally distributed errors—normal quantile-quantile plot of errors should be linear

• Constant standard deviation of errors—scatter plot of y for various levels of factors—if spread at one level significantly different than at another

– need transformation

• What to do—if any test fails or ymax/yminis large, investigate multiplicative

model

20

Multiplicative Models for 22r Experiments

• Additive model

• Not applicable if effects don’t add—example: execution time of workloads

– factor A: ith processor speed = vi instructions/sec– factor B: jth workload size = wj instructions

—the two effects multiply: execution time yij = vi x wj

• Use logarithm to produce additive model—log(yij) = log(vi) + log(wj)

• Better modelwhere

!

yij = q0

+ qA xAi + qB xBi + qAB xAixBi + eij

!

y'ij = q0 + qA xAi + qB xBi + qAB xAixBi + eij

!

y'ij = log(yij )

21

Interpreting a Multiplicative Model

• uA is ratio of MIPS rating of two processors• uB ratio of size of the two workloads• Antilog of additive mean q0 yields geometric mean!

uA =10qA ,uB =10

qB ,uAB =10qAB ,

!

˙ y =10q0 = (y

1y

2...yn )

1/ n where n = 22r

!

log(y) = q0 + qA xA + qB xB + qAB xA xB + e

!

y =10q010

qAxA10qBxB10

qABxA xB10e

22

Example: Execution Times

• Analysis using an additive modelfactor A = proc speed; factor B = instructions

qABqBqAq0

total/425.54-26.04-26.0426.55total102.17-104.15-104.15106.190.0131.0031.003

104.170

1111-11-11-1-1111-1-11

ABBAI

.0118.0126.01481.122.933.9551.0721.047.891147.9079.5085.10

!

y i. yij

23

Visual Tests for Additive Model

24

Why is This Model Invalid?

• Physical consideration—workload & processor effects don’t add

• Large range for y— ymax/ymin = 147.90/.0118 = 12,534 ⇒ log transform

– taking arithmetic mean of 104.17 and .013 is inappropriate

• Residuals are not small compared to response• Spread of residuals is larger at larger value of response

—suggests log transformation

25

Analysis using a Multiplicative Model

Data after log transformationfactor A = proc speed; factor B = instructions

qABqBqAq0

total/4.03-0.97-0.97.03total.11-3.89-3.89.11-1.890.000.002.00

1111-11-11-1-1111-1-11

ABBAI

-1.93-1.90-1.83.05-.03-.02.03.02-.052.171.901.93

!

y i. yij

26

Visual Tests for Multiplicative Model

• Conclusion: Multiplicative model is better than additive model

27

Variation Explained by the Models

Additive Multiplicative

• With multiplicative model: interaction almost 0• Little unexplained variation: .2%

.2%10.8%e(-.02,.07)*0.0%.03(15.35,35.74)29.0%25.54AB

(-1.02,-.93)49.9%-.97(-36.23,-15.84)30.1%-26.04B

(-1.02,-.93)49.9%-.97(-36.23,-15.84)30.1%-26.04A

(-.02,.07)*.03(16.35,36.74)26.55I

ConfInterval

%varEffectConfInterval

%varEffectFactor

* = not significant