exploring best practises in design of experiments

Copyright © 2014, SAS Institute Inc. All rights reserved.

Exploring Best Practises inDesign of ExperimentsA Holistic Approach to DOE, Increasing Robustness, Efficiency and Effectiveness


Contents

Background to DOE

Why Use DOE?

Tips for Effective DOE with Classical Designs

Definitive Screening

Case Studies 1-3

Role of Statistical Modelling and DOE in Learning

Holistic DOE

Case Study 4


BACKGROUND TO DESIGN OF EXPERIMENTS (DOE)


FATHER OF DOE RONALD A. FISHER

Rothamstead Experimental Station, England – Early 1920’s


FISHER’S FOUR DESIGN PRINCIPLES

1. Factorial Concept - rather than one-factor-at-a-time

2. Randomization - to avoid bias from lurking variables

3. Blocking - to reduce noise from nuisance variables

4. Replication - to quantify noise within an experiment


AGRICULTURAL IMPACT

US corn yields

Cornell University, http://usda.mannlib.cornell.edu/MannUsda


WHY USE DOE?


Typical ProcessThe properties of products and processes are often affected by many factors:

In order to build new or improve products and processes, we must understand the relationship between the factors (inputs) and the responses (outputs).

Typical

Process

Machine

Operator

Temperature

Pressure

Humidity

Yield

Cost

…

Inputs

Factors

Outputs

Responses


Traditional One-Factor-at-a-Time A common approach is one-factor-at-a-time experimentation.

Consider experimenting one-factor-at-a-time to determine the values of temperature and time that optimise yield.


Traditional One-Factor-at-a-Time


Traditional One-Factor-at-a-Time

One-factor-at-a-time experimentation frequently leads to sub-optimal solutions.

Assumes the effect of one factor is the same at each level of the other factors, i.e. factors do not interact.

In practice, factors frequently interact.


Interaction between factors


Experimental Design Most efficient way of investigating relationships.

Runs (factor combinations) chosen to maximize the information

Ideally balanced for ease of analysis and interpretation


ITERATIVE AND SEQUENTIAL NATURE OF CLASSICAL DOE


TIPS FOR EFFECTIVE DOE WITH CLASSICAL DESIGNS


Stages of Experimental Design

Designing an experiment involves much more than just selecting the sequence of experimental runs:

Historically, improper planning is the most common cause of failed experiments.

Plan Design Conduct Analyse Confirm


Some Planning Steps

Review what we know

• Have peer discussions

Determine new questions to answer

Identify factors and ranges to investigate

Define responses

• Easy and precise to measure


Common Experimental Objectives

Identify Important Factors

Screening Design

Classical Fractional Factorial

OptimiseProcess

RSM DesignClassical Central

Composite

OptimiseIngredients

Mixtures

Classical Simplex & Extreme Vertices


Sequential Experimentation Reduces Total Cost



Screening Design


OptimiseProcess


Composite

OptimiseIngredients

Mixtures



Sequential Experimentation



Screening Design


OptimiseProcess


Composite

OptimiseIngredients

Mixtures


Definitive Screening Design Simplifies Experimental Workflow


Sequential Experimentation



Screening Design


OptimiseProcess


Composite

OptimiseIngredients

Mixtures


Definitive Screening Design

Optimal Design Manages Experimental Constraints


Determining the Appropriate Factors

Determining the factors to be included in your experiment is a critical part of planning.

• Exploring too many factors may be costly and time consuming.

• Exploring too few may limit the success of your experiment.

Prior knowledge and analysis of existing data are useful aids to identifying and prioritising factors for study. Other methods may include:

• Brainstorming

• Ishikawa


Selection of Factor Range is Critical With Two Level Designs …



By experimenting at the two settings in

yellow, X would be declared unimportant



By using half and often times much less than than

half the factor range X is declared important



By using half and often times much less than than

half the factor range X is declared important

Often leads to narrow factor ranges

to force linear relationships but

consequence is high risk of

determining sub-optimal solution


Determining the Appropriate Responses

Selection of your responses will also be critical to the success of your experiment. Whenever possible:

• Choose variables that correlate to internal or external customer requirements

• Find responses that are easy to measure

• Make sure your measurement systems are precise, accurate, and stable

Analysis of current data, prior knowledge, measurement systems analysis are useful aids.


DEFINITIVE SCREENING


Fractional Factorials: Complex workflow from many factors to optimum settings

Tempting to miss out

middle step which can

result in selection of

wrong factors and decisions


Definitive Screening Design

Identifies active main effects, uncorrelated with other effects.

May identify significant quadratic effects, uncorrelated with main effects and at worst weakly correlated with other quadratic effects.

If few factors turn out to be important, can identify significant two-way interactions uncorrelated with main effects and weakly correlated with other higher order effects.

One stage experiment if three or fewer factors important:

• progress straight to full quadratic model

• optimise process with no further experimentation

• otherwise augment DSD for optimization goals


New Class of Screening Design

Three-level screening design

• 2m + 1 runs based on m fold-over pairs and an overall center point, where m is number of factors

• the values of the ±1 entries in the odd-numbered runs are determined using optimal design.


Use of Three Level DesignsAdvantageous

Scientists and engineers are uncomfortable using two-level designs

• Restricting factor ranges may result in sub-optimal solutions

• Scientific/engineering judgment suggests relationships nonlinear over wide ranges

Investigators frequently have an opinion regarding the “best” levels of each factor for optimizing a response

• Experimental region centered at these levels.

• Two-level design might screen out an important factor when experimental region centred at “best”

• Adding centre points allows test for curvature

• However ambiguity over factors causing curvature

• DSD avoids ambiguity by making it possible to uniquely identify the source(s) of curvature.


CASE STUDIES


Case Study 1: Optimising a Chemical Process

Why Consider Definitive Screening Designs?


Background

Five factors

One response yield

Goal optimise yield

Keep total cost of experimentation to minimum

Contrast traditional approach of main effect screening design plus augmentation to RSM with DSD


Traditional screening approach correlates main effects with two factor interaction effects

Cost constraint and inexperience with such designs can lead to missed DOE steps

Investigator missed step of augmenting main effect design to separate correlated interaction effects from assumed important main effects

Resulted in wrong set of factors selected for RSM design which results in wrong solution

Background


Traditional Approach with Missed Step


Resolution III Design Perfectly Correlates Main Effects With Interaction Effects


Model Interpretation

Fitted Model

Y = b0 + b1*X1 + b2*X2 + b3*X3 + Error

Correct Interpretation of Fitted Model

Y = b0 + b1*(X1+X2X3) + b2*(X2+X1X3) + b3*(X3+X1X2) + Error


Missed Step Augments Initial Design to Separate Main Effects From Interactions


Model Interpretation of Augmented Design

Correct Interpretation of Model Fitted to Augmented design

Y = b0 + b1*X1 + b2*X2 + b3*X3 + b12*X1X2 + b13*X1X3 + b23*X2X3 + Error

Allows clear separation of main and interaction effects

This step was missed in case study prior to modelling curvature


DSD results in correct identification of important factors due to non correlated main and two factor interaction effects

Because just three factors are important DSD results in one step design:

• In addition to correctly identifying correct factors

• DSD requires no augmentation to identify optimal settings of important factors

Background


CASE STUDY 1


Conclusions

Fractional factorial designs can lead to selection of wrong factor set

Complex workflow for avoiding this risk which may be misunderstood or not applied by users new to DOE

May lead to conclusion that DOE does not work for us!

DSD simplifies DOE process and removes risk of selecting wrong factor set

Provides one step DOE when three or fewer important factors

• Sufficient to identify correct factor set and determine best settings of selected factors


Case Study 2: Optimising Reaction Conditions for Chemical Methods

Augmenting Definitive Screening Designs


Background

How effective are DSD when more than three factors are important?

Use example from literature

• Response-Surface Co-optimization of Reaction Conditions in Clinical Chemical Methods, Gopal S. Rautela, Ronald 0. Snee,’ and Warren K. Miller, CLINICAL CHEMISTRY, Vol. 25, No. 11, 1979

• CCF RSM in six factors

• Five factors are important

• Use model from this experiment to contrast CCF with augmented DSD


Aspartate Aminotransferase Assay: http://www.chem.qmul.ac.uk/iubmb/enzyme/EC2/6/1/1.html

Six factors (reagent conditions): tris(hydroxymethyl)aminomethane, pH, L-aspartic acid, pyridoxal-5’-phosphate, 2-oxoglutarate, and malate dehydrogenase

Response: aspartate aminotransferase activity measured for human serum with above normal activity at 30C

Goal: select reagent conditions that maximise aspartate aminotransferase activity

Example selected to stress DSD when >3 factors are important, in this case 5 factors are important.

Background


CASE STUDY 2


Conclusions

When >3 factors are important, augmenting DSD works

When >3 factors are important, an augmented DSD approach is more efficient than classical Response Surface Designs


CASE STUDY 3


Case Study 3: Optimising Yield

What About Constrained Factor Spaces?


Background

From chapter 5 of Goos & Jones

Chemical reaction

Goal: maximise yield

2 factors: Temperature and Time


Background

Expert knowledge tells us

• Certain conditions will give poor results (hence, constraints)

• Behaviour very non-linear

We will show

• Design where prior knowledge is ignored.

• Fitting the design to the problem


Example of Process Constraint


Shrink Experimental Range to Factorial


Optimal Design: Use Actual Factor Range


… optimal designs allow investigation of complete factor space properly adjusted for constraints

Typical

Process

Machine

Operator

Temperature

Pressure

Humidity

Yield

Cost

…

Inputs

Factors

Outputs

Responses

Optimal Design: Fit to Model

Model

Y = f(X)

The process is not seen as a black box anymore…


CASE STUDY 3


Conclusions

Custom Design permits studying any:

• combination of factors with or without constraints,

• number of factor levels,

• blocking structure.

Build your design to suit the problem instead of fitting the problem into a design


Case Study 4: Designing Products People Want to Buy

Holistic DOE


ROLE OF STATISTICAL MODELLING AND DOE IN LEARNING


Data Sources

DOE and/or observational (historical)

Potential problems with observational data:

• X’s are correlated – identification of “best” model difficult

• Outliers (potential or real) - bias model estimation

• Missing data cells – result in loss of whole data rows with traditional least squares based analysis

• Range over which X’s varied may be limited –restricting model usefulness

• May not have measured all relevant X’s

In some situations these can also be issues with DOE datasets


WHAT IS HOLISTIC DOE?


Holistic DOE Approach: Integrating Statistical Modelling and DOE

Learning is incremental and effective statistical modelling of observational data aids design of next experiment.

Analysis approach needs to manage real (messy) data simply

• Correlated X’s, outliers, missing cells

• Quickly deliver “best” current model to revise with new DOE data

• Aid better analysis of new experimental data when unexpected occurs

• Build models based on individual datasets and aggregated data

Good statistical modelling integrated with DOE helps reduce total learning time, effort and cost

It would be a shame to not use pre-existing data that comes for free


Holistic DOE Example


Background

PC retailer is observing appreciably retail price variation in its laptop computer line.

Goals:

• Investigate factors associated with retail price variation.

• Perform further experimentation in key factors to optimise and standardise pricing across stores.


CASE STUDY 4


Conclusions

Analysis of prior data helps identify factors and ranges to use in next DOE.

Analysis of prior data helps reduce risk and increase efficiency and effectiveness of future experiments.

DOE is not just for science and engineering.


Holistic DOE: Integrated Statistical Modelling and DOE

Supports wide range of user skills

Exploratory analysis and statistical modelling of historical messy data simplifies and shortens whole DOE process.

Next generation DOE enables more staff to apply DOE with reduced learning and implementation effort

Interact with model predictions to build consensus

Integrated simulation capabilities enables rapid progression from models to decisions

Drag and drop charts help monitor processes and identify potential causes of issues

Manage risk better by correctly identifying signal from noise

exploring best practises in design of experiments

Software

sas institute

effective doe

use doe

time experimentation

factor combinations

design of experiments

factors inputs

father of doe ronald