for discussion today (when the alarm goes off) survey your proceedings for just one paper in which...
TRANSCRIPT
For Discussion Today(when the alarm goes off)
• Survey your proceedings for just one paper in which factorial design has been used or, if none, one in which it could have been used effectively.
© 2003, Carla Ellis
Experimental Design
(l1,0, l1,1, … , l1,n1-1) x (l2,0, l2,1, … , l2,n2-1) x …
x (lk,0, lk,1, … , lk,nk-1)
k different factors, each factor with ni levelsr replications
Factor 1
Factor k
Factor 2
(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …
x (lk,0, lk,1, … , lk,n-1)
2k-p Fractional Factorial Designs
Factor 1
Factor k
Factor 2
Only considering 2 levels of k factorswith 2k-p experiments for suitable p
(l1,0, l1,1, … , l1,n-1) x (l2,0, l2,1, … , l2,n-1) x …
x (lk,0, lk,1, … , lk,n-1)
2k-p Fractional Factorial Designs
Factor 1
Factor k
Factor 2
Only considering 2 levels of k factorswith 2k-p experiments for suitable p
© 1998, Geoff Kuenning
2k-p FractionalFactorial Designs
• Introductory example of a 2k-p design
• Preparing the sign table for a 2k-p design
• Confounding
• Algebra of confounding
• Design resolution
Why and When to Use
• Getting by on fewer experiments than 2k
• When you know certain interactions are negligible and you can “overload” them (confound).
• When you want to focus in on what further experiments to do (where it matters) without wasting time and effort.
© 1998, Geoff Kuenning
Introductory Exampleof a 2k-p Design
• Exploring 7 factors in only 8 experiments:
Run A B C D E F G1 -1 -1 -1 1 1 1 -12 1 -1 -1 -1 -1 1 13 -1 1 -1 -1 1 -1 14 1 1 -1 1 -1 -1 -15 -1 -1 1 1 -1 -1 16 1 -1 1 -1 1 -1 -17 -1 1 1 -1 -1 1 -18 1 1 1 1 1 1 1
© 1998, Geoff Kuenning
Analysis of 27-4 Design
• Column sums are zero:
• Sum of 2-column product is zero:
• Sum of column squares is 27-4 = 8• Orthogonality allows easy calculation of
effects:
x jiji 0
x x j lij ili 0
qy y y y y y y y
A 1 2 3 4 5 6 7 8
8
© 1998, Geoff Kuenning
Effects and Confidence Intervals for 2k-p
Designs• Effects are as in 2k designs:
• % variation proportional to squared effects• For standard deviations, confidence intervals:
– Use formulas from full factorial designs– Replace 2k with 2k-p
q y xk p i ii
1
2
© 1998, Geoff Kuenning
Preparing the Sign Table for a 2k-p Design
• Prepare sign table for k-p factors
• Assign remaining factors
© 1998, Geoff Kuenning
Sign Table for k-p Factors
• Same as table for experiment with k-p factors– I.e., 2(k-p) table– 2k-p rows and 2k-p columns– First column is I, contains all 1’s– Next k-p columns get k-p selected factors– Rest are products of factors
© 1998, Geoff Kuenning
Assigning Remaining Factors
• 2k-p-(k-p)-1 product columns remain
• Choose any p columns– Assign remaining p factors to them– Any others stay as-is, measuring
interactions
© 1998, Geoff Kuenning
Confounding
• The confounding problem
• An example of confounding
• Confounding notation
• Choices in fractional factorial design
© 1998, Geoff Kuenning
The Confounding Problem
• Fundamental to fractional factorial designs• Some effects produce combined influences
– Limited experiments mean only combination can be counted
• Problem of combined influence is confounding– Inseparable effects called confounded effects
© 1998, Geoff Kuenning
An Example of Confounding
• Consider this 23-1 table:
• Extend it with an AB column:
I A B C1 -1 -1 11 1 -1 -11 -1 1 -11 1 1 1
I A B C AB1 -1 -1 1 11 1 -1 -1 -11 -1 1 -1 -11 1 1 1 1
© 1998, Geoff Kuenning
Analyzing theConfounding Example
• Effect of C is same as that of AB:
qC = (y1-y2-y3+y4)/4
qAB = (y1-y2-y3+y4)/4
• Formula for qC really gives combined effect:
qC+qAB = (y1-y2-y3+y4)/4
• No way to separate qC from qAB
– Not a problem if qAB is known to be small
© 1998, Geoff Kuenning
Confounding Notation
• Previous confounding is denoted by equating confounded effects:C = AB
• Other effects are also confounded in this design:A = BC, B = AC, C = AB, I = ABC– Last entry indicates ABC is confounded
with overall mean, or q0
© 1998, Geoff Kuenning
Choices in Fractional Factorial Design
• Many fractional factorial designs possible– Chosen when assigning remaining p signs– 2p different designs exist for 2k-p experiments
• Some designs better than others– Desirable to confound significant effects with
insignificant ones– Which usually means low-order with high-order
© 1998, Geoff Kuenning
Rules ofConfounding Algebra
• Particular design can be characterized by single confounding– Traditionally, the I = wxyz... confounding
• Others can be found by multiplying by various terms– I acts as unity (e.g., I times A is A)– Squared terms disappear (AB2C becomes
AC)
© 1998, Geoff Kuenning
Example:23-1 Confoundings
• Design is characterized by I = ABC• Multiplying by A gives A = A2BC = BC• Multiplying by B, C, AB, AC, BC, and ABC:
B = AB2C = AC, C = ABC2 = AB,AB = A2B2C = C, AC = A2BC2 = B,BC = AB2C2 = A, ABC = A2B2C2 = I
• Note that only first line is unique in this case
© 1998, Geoff Kuenning
Generator Polynomials
• Polynomial I = wxyz... is called generator polynomial for the confounding
• A 2k-p design confounds 2p effects together– So generator polynomial has 2p terms– Can be found by considering interactions
replaced in sign table
© 1998, Geoff Kuenning
Example of Findinga Generator Polynomial
• Consider a 27-4 design
• Sign table has 23 = 8 rows and columns
• First 3 columns represent A, B, and C
• Columns for D, E, F, and G replace AB, AC, BC, and ABC columns respectively– So confoundings are necessarily:
D = AB, E = AC, F = BC, and G = ABC
© 1998, Geoff Kuenning
Turning Basic Terms into Generator
Polynomial• Basic confoundings are D = AB, E =
AC, F = BC, and G = ABC
• Multiply each equation by left side:I = ABD, I = ACE, I = BCF, and I = ABCGorI = ABD = ACE = BCF = ABCG
© 1998, Geoff Kuenning
Finishing theGenerator Polynomial
• Any subset of above terms also multiplies out to I– E.g., ABD times ACE = A2BCDE = BCDE
• Expanding all possible combinations gives the 16-term generator (book is wrong):I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = CEFG= ABCDEFG
© 1998, Geoff Kuenning
Design Resolution
• Definitions leading to resolution
• Definition of resolution
• Finding resolution
• Choosing a resolution
© 1998, Geoff Kuenning
Definitions Leadingto Resolution
• Design is characterized by its resolution• Resolution measured by order of confounded
effects• Order of effect is number of factors in it
– E.g., I is order 0, and ABCD is order 4
• Order of confounding is sum of effect orders– E.g., AB = CDE would be of order 5
© 1998, Geoff Kuenning
Definition of Resolution
• Resolution is minimum order of any confounding in design
• Denoted by uppercase Roman numerals– E.g, 25-1 with resolution of 3 is called RIII
– Or more compactly, 2III
5-1
© 1998, Geoff Kuenning
Finding Resolution
• Find minimum order of effects confounded with mean– I.e., search generator polynomial
• Consider earlier example: I = ABD = ACE = BCF = ABCG = BCDE = ACDF = CDG = ABEF = BEG = AFG = DEF = ADEG = BDFG = ABDG= CEFG = ABCDEFG
• So it’s RIII design
© 1998, Geoff Kuenning
Choosing a Resolution
• Generally, higher resolution is better• Because usually higher-order
interactions are smaller• Exception: when low-order interactions
are known to be small– Then choose design that confounds those
with important interactions– Even if resolution is lower
© 1998, Geoff Kuenning
One Factor Experiments
• If there’s only one important categorical factor• But it has more than two interesting
alternatives– Methods work for two alternatives, but they reduce
to 21 factorial designs
• If the single variable isn’t categorical, examine regression, instead
• Method allows multiple replications
© 1998, Geoff Kuenning
What is This Good For?
• Comparing truly comparable options– Evaluating a single workload on multiple
machines– Or with different options for a single
component– Or single suite of programs applied to
different compilers
© 1998, Geoff Kuenning
What Isn’t This Good For?
• Incomparable “factors”– Such as measuring vastly different
workloads on a single system
• Numerical factors– Because it won’t predict any untested
levels
© 1998, Geoff Kuenning
An Example One Factor Experiment
• You are buying an authentication server to authenticate single-sized messages
• Four different servers are available• Performance is measured in response
time– Lower is better
• This choice could be assisted by a one-factor experiment
© 1998, Geoff Kuenning
The Single Factor Model
• yij = j + eij
• yij is the ith response with factor at alternative j
• is the mean response
• j is the effect of alternative j
• eij is the error term
j 0
© 1998, Geoff Kuenning
One Factor Experiments With
Replications• Initially, assume r replications at each
alternative of the factor
• Assuming a alternatives of the factor, a total of ar observations
• The model is thus
y ar r eij
j
a
i
r
jj
a
ijj
a
i
r
11 1 11
© 1998, Geoff Kuenning
Sample Data for Our Example
• Four alternatives, with four replications each (measured in seconds)
A B C D
0.96 0.75 1.01 0.93
1.05 1.22 0.89 1.02
0.82 1.13 0.94 1.06
0.94 0.98 1.38 1.21
© 1998, Geoff Kuenning
Computing Effects
• We need to figure out and j
• We have the various yij’s
• So how to solve the equation?
• Well, the errors should add to zero
• And the effects should add to zero
eij
j
a
i
r
11
0
j
j
a
0
1
© 1998, Geoff Kuenning
Calculating
• Since sum of errors and sum of effects are zero,
• And thus, is equal to the grand mean of all responses
y arij
j
a
i
r
11
0 0
111ary yij
j
a
i
r
..
© 1998, Geoff Kuenning
Calculating j
• j is a vector of responses– One for each alternative of the factor
• To find the vector, find the column means
• For each j, of course• We can calculate these directly from
observations
yr
yj ij
i
r
.
11
© 1998, Geoff Kuenning
Calculating a Column Mean
• But we also know that yij is defined to be
• So,
y eij j ij
11r
r r ej ij
i
r
yr
ej j ij
i
r
.
11
© 1998, Geoff Kuenning
Calculating the Parameters
• Remember, the sum of the errors for any given row is zero, so
• So we can solve for j -
yrr r
j j. 1
0
j
j j jy y y
. . ..
© 1998, Geoff Kuenning
Parameters for Our Example
Server A B C D
Column .9425 1.02 1.055 1.055
Mean
Subtract from column means to get parameters
Parameters
-.076 .002 .037 .037
© 1998, Geoff Kuenning
Estimating Experimental Errors
• Estimated response is• But we measured actual responses
– Multiple ones per alternative
• So we can estimate the amount of error in the estimated response
• Using methods similar to those used in other types of experiment designs
y j j yj
© 1998, Geoff Kuenning
Finding Sum of Squared Errors
• SSE estimates the variance of the errors
• We can calculate SSE directly from the model and observations
• Or indirectly from its relationship to other error terms
SSE eijj
a
i
r
2
11
© 1998, Geoff Kuenning
SSE for Our Example
• Calculated directly -
SSE = (.96-(1.018-.076))2 + (1.05 - (1.018-.076))2 + . . . + (.75-(1.018+.002))2 + (1.22 - (1.018 + .002))2 + . . . + (.93 -(1.018+.037))2
= .3425
© 1998, Geoff Kuenning
Allocating Variation
• To allocate variation for this model, start by squaring both sides of the model equation
• Cross product terms add up to zero
y e e eij j ij j ij j ij2 2 2 2 2 2 2
y e
cross products
iji j i j
ji j
iji j
2 2 2 2
, , , ,
© 1998, Geoff Kuenning
Variation In Sum of Squares Terms
• SSY=SS0+SSA+SSE
• Giving us another way to calculate SSE
SSY yiji j
2
,
SS arj
a
i
r0 2
11
2
SSA rjj
a
i
r
jj
a
2
11
2
1
© 1998, Geoff Kuenning
Sum of Squares Terms for Our Example
• SSY = 16.9615
• SS0 = 16.58256
• SSA = .03377
• So SSE must equal 16.9615-16.58256-.03377– Which is .3425– Matching our earlier SSE calculation
© 1998, Geoff Kuenning
Assigning Variation
• SST is the total variation
• SST = SSY - SS0 = SSA + SSE
• Part of the total variation comes from our model
• Part of the total variation comes from experimental errors
• A good model explains a lot of variation
© 1998, Geoff Kuenning
Assigning Variation in Our Example
• SST = SSY - SS0 = 0.376244
• SSA = .03377
• SSE = .3425
• Percentage of variation explained by server choice
10003377
37628 97%
.
..
© 1998, Geoff Kuenning
Analysis of Variance
• The percentage of variation explained can be large or small
• Regardless of which, it may be statistically significant or insignificant
• To determine significance, use an ANOVA procedure – Assumes normally distributed errors . . .
© 1998, Geoff Kuenning
Running an ANOVA Procedure
• Easiest to set up a tabular method• Like method used in regression models
– With slight differences
• Basically, determine ratio of the Mean Squared of A (the parameters) to the Mean Squared Errors
• Then check against F-table value for number of degrees of freedom
© 1998, Geoff Kuenning
ANOVA Table for One-Factor Experiments
Compo- Sum of Percentage of Degrees of Mean F- F-
nent Squares Variation Freedom Square Comp Table
y ar
1
SST=SSY-SS0 100 ar-1
A a-1 F[1-; a-,a(r-1)]
e SSE=SST-SSA a(r-1)
SSY yij 2
y.. SS ar0 2
y y ..
SSA r j 2 MSASSA
a
1MSA
MSE
100SSE
SST
MSESSE
a r
( )1
100SSA
SST
© 1998, Geoff Kuenning
ANOVA Procedure for Our Example
Compo- Sum of Percentage of Degrees of Mean F- F-
nent Squares Variation Freedom Square Comp Table
y 16.96 16
16.58 1
.376 100 15
A .034 8.97 3 .011 .394
2.61
e .342 91.0 12 .028
y..
y y ..
© 1998, Geoff Kuenning
Analysis of Our Example ANOVA
• Done at 90% level
• Since F-computed is .394, and the table entry at the 90% level with n=3 and m=12 is 2.61,– The servers are not significantly different
© 1998, Geoff Kuenning
One-Factor Experiment
Assumptions• Analysis of one-factor experiments makes the
usual assumptions– Effects of factor are additive– Errors are additive– Errors are independent of factor alternative– Errors are normally distributed– Errors have same variance at all alternatives
• How do we tell if these are correct?
© 1998, Geoff Kuenning
Visual Diagnostic Tests
• Similar to the ones done before– Residuals vs. predicted response– Normal quantile-quantile plot– Perhaps residuals vs. experiment number
© 1998, Geoff Kuenning
Residuals vs. Predicted For Our
Example
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.9 0.95 1 1.05 1.1
© 1998, Geoff Kuenning
What Does This Plot Tell Us?
• This analysis assumed the size of the errors was unrelated to the factor alternatives
• The plot tells us something entirely different– Vastly different spread of residuals for different
factors
• For this reason, one factor analysis is not appropriate for this data– Compare individual alternatives, instead– Using methods discussed earlier
© 1998, Geoff Kuenning
Could We Have Figured This Out
Sooner?• Yes!
• Look at the original data
• Look at the calculated parameters
• The model says C & D are identical
• Even cursory examination of the data suggests otherwise
© 1998, Geoff Kuenning
Looking Back at the Data
A B C D
0.96 0.75 1.01 0.93
1.05 1.22 0.89 1.02
0.82 1.13 0.94 1.06
0.94 0.98 1.38 1.21
Parameters
-.076 .002 .037 .037
© 1998, Geoff Kuenning
Quantile-Quantile Plot for Example
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5
© 1998, Geoff Kuenning
What Does This Plot Tell Us?
• Overall, the errors are normally distributed
• If we only did the quantile-quantile plot, we’d think everything was fine
• The lesson - test all the assumptions, not just one or two
© 1998, Geoff Kuenning
One-Factor Experiment Effects Confidence
Intervals• Estimated parameters are random variables
– So we can compute their confidence intervals
• Basic method is the same as for confidence intervals on 2kr design effects
• Find standard deviation of the parameters– Use that to calculate the confidence intervals– Typo in book, pg 366, example 20.6, in formula for
calculating this– Also typo on pg. 365 - degrees of freedom is a(r-1), not r(a-
1)
© 1998, Geoff Kuenning
Confidence Intervals For Example Parameters
• se = .158
• Standard deviation of = .040
• Standard deviation of j = .069
• 95% confidence interval for = (.932, 1.10)
• 95% CI for = (-.225, .074)
• 95% CI for = (-.148,.151)
• 95% CI for = (-.113,.186)
• 95% CI for = (-.113,.186)
© 1998, Geoff Kuenning
Unequal Sample Sizes in One-Factor Experiments
• Can you evaluate a one-factor experiment in which you have different numbers of replications for alternatives?
• Yes, with little extra difficulty
• See book example for full details