test design optimization in systems engineering€¦ · test design optimization helps minimize...
TRANSCRIPT
Test Design Optimization
in Systems Engineering
ITEA’s SOS Conference
28 Jan 2016
El Paso, TX
16-TDOSE-1A
Mark J. Kiemele, Ph.D.President and Co-Founder
Air Academy Associates
Office: 719-531-0777
Cell: 719-337-0357
www.airacad.com
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Testing has always been with us …...
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Medieval times saw the proliferation of launching devices.
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Medieval Cannon (combining gunpowder, molten metal, and wheels)
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
1903: First Flight Test by Wright Brothers
Why Kitty Hawk, North Carolina?
• The land was flat and they thought it a perfect place to fly and land.
• There were sand dunes there so they had a better chance to survive in the event of a crash.
• There were no trees in the area to crash into.
• They needed a steady wind.
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
1947: Bell X-1, first airplane to break the sound barrier
(flown by Chuck Yeager)
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
First wheels on suitcases: 1970 (long handle in 1987)
©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
The Ultimate in Test: Invention of the Light Bulb by Thomas Edison
“I have not failed. I’ve just found 10,000 ways that won’t work.”
Edison, 1879
Page 8©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Goals
What is the Motivation for Test Design Optimization?
What is Test Design Optimization?
• Three Primary Reasons for Testing
• What Influences a Test Design?
• Examples of Optimal Test Design
• What is different about DOE?
• Simple Rules of Thumb for Selecting a Test Design
Where does it fit in with Systems Engineering?
Page 9©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Motivation for Test Design Optimization
Technical Debt
• Work which has been done but which has not been shown to
add value
• Associated with “batch” production in manufacturing
Examples:
• Getting far into design without concept validation by the customer
• Getting far into development without proper hardware testing
• Test Debt: the difference between testing everything and what
we can test most effectively and efficiently and still generate the
same knowledge gain as if we had tested everything
Technical debt has to be paid, and it can be expensive.
Test Design Optimization helps minimize technical debt by
minimizing test debt.
Page 10©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
What is Test Design Optimization?
More effective testing
• Performing the right test
• For the right purpose
• At the right time
More efficient testing
• Using resources (time, people, and test materials) wisely
• Removing non-value activities
Minimizing the test debt
Page 11©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Definition of a Testing Process
Y1
X3
X4
X5
X6
X7
X2
X1
Y2
Y3
Page 12©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Inputs
Input Variable
Input Factor
Input Parameter
Indicator Variable
Independent Variable
Outputs
Output Variable
Response Variable
Process Performance Measure
Critical to Customer (CTC)
Critical to Quality (CTQ)
Dependent Variable
x y
IPO Terminology
Page 13©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
A test matrix where each row (run) specifies a test case,
namely a combination of the levels of each of the factors
(inputs) that are being tested.
Run
1
2
3
.
.
X1 X2 X3 X4 Y1 Y2 . . . . . . Y SY
Inputs
A = X1
B = X2
D = X4
C = X3
YOutputs
.
.
.
.
.
.
PROCESS
What is a Test Design?
Page 14©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
The Test Design is Dependent on:
• The purpose of the test
• The number of factors to be tested
• The number of levels to be tested for each factor
• The number of test resources available
• Other considerations and constraints unique to
the test scenario
Page 15©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Three Major Reasons for Testing
• Modeling
- For building functions that can be used to predict
outcomes, assess risk, and optimize performance. These
include the ability to evaluate interaction and higher order
effects.
• Performance Verification and Validation
- For confirming that a system performs in accordance with
its specifications/requirements and to get great test coverage
at low cost. Detecting and isolating bugs fall in this category.
• Screening
- For testing many factors in order to separate the vital few
factors from the trivial many.
Page 16©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Motivation for DOE from Dr. Gilmore (DOT&E)(from his 26 June 2013 memo on Flawed Applications of DOE)
1. One of the most important goals of operational testing is to
characterize a system’s effectiveness over the operational envelope.
2. I advocate the use of DOE to ensure that test programs are able to
determine the effect of factors on a comprehensive set of
operational mission-focused and quantitative response variables.
3. Future test plans must state clearly that data are being collected to
measure a particular response variable (possibly more than one) in
order to characterize the system’s performance by examining the
effects of multiple factors … and clearly delineating what statistical
model (e.g., main effects and interactions) is motivating … the
variation of the test.
4. Confounding factors must be avoided.
5. Another pitfall to avoid is relying on binary metrics as the primary
response variable.
Page 17©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Inputs
Input Variable
Input Factor
Input Parameter
Indicator Variable
Independent Variable
Outputs
Output Variable
Response Variable
Process Performance Measure
Critical to Customer (CTC)
Critical to Quality (CTQ)
Dependent VariableProcess
Characterization:
y = f(x)
y is a function of x (a transfer function)
y is related to x
True Scenario: y = f(x) + error
x y
IPO Terminology
Page 18©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Performance
(# home page loads/sec)
CPU
RAM Amount
HD Size
VM
Cost
($)
Performance
Tuning
Web-Based Application Testing
OS
Page 19©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Performance Tuning Terminology
Factors/Inputs
(X’s)
Levels
(Choices)
Performance/Outputs
(Y’s)
CPU Type
CPU Speed
RAM Amount
HD Size
VM
OS
Itanium, Xeon
1 GHz, 2.5 GHz
256 MB, 1.5 GB
50 GB, 500 GB
J2EE, .NET
Windows, Linux
# home page loads/sec
Cost
Which factors are important? Which are not?
Which combination of factor choices will optimize performance?
What Test Design would you choose to help you answer these questions?
Page 20©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Famous Quote
“All experiments (tests) are
designed experiments;
some are poorly designed,
some are well designed.”
George Box (1919-2013), Professor of Statistics, DOE Guru
Page 21©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Design of Experiments (DOE): A Subset of All Possible Test Design Methodologies
The Set of All Possible Test Design
Methodologies (Combinatorial Tests)
Orthogonal or
Nearly
Orthogonal
Test Designs
(DOEs)
One
Factor
At a
Time
(OFAT)
Best Guess
(Oracle)
Boundary Value Analysis
(BVA)
Equivalence Partitioning (EP)
Decision
Tables
Page 22©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Design of Experiments (DOE)
• “Interrogates” the process
• Changes “I think” to “I know” (with some level of confidence)
• It is the science of test and the key link between test &
evaluation.
• Used to identify important relationships between inputs and
outputs
• Identifies important interactions between process variables
• Can be used to optimize a process
• An optimal data collection methodology
Page 23©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Modeling Flight
Characteristics
of New 3-Wing
Aircraft
Pitch )
Roll )
W1F )
W2F )
W3F )
INPUT OUTPUT
(-15, 0, 15)
(-15, 0, 15)
(-15, 0, 15)
(0, 15, 30)
(0, 15, 30)
Six Aero-
Characteristics
Value Delivery: Reducing Time to Develop New Technologies
Patent Holder: Dr. Bert Silich
What test design would you choose to help you build the models on the next page?
Page 24©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
CL = .233 + .008(P)2 + .255(P) + .012(R) - .043(WD1) - .117(WD2) + .185(WD3) + .010(P)(WD3) -
.042(R)(WD1) + .035(R)(WD2) + .016(R)(WD3) + .010(P)(R) - .003(WD1)(WD2) -
.006(WD1)(WD3)
CD = .058 + .016(P)2 + .028(P) - .004(WD1) - .013(WD2) + .013(WD3) + .002(P)(R) - .004(P)(WD1)
- .009(P)(WD2) + .016(P)(WD3) - .004(R)(WD1) + .003(R)(WD2) + .020(WD1)2 + .017(WD2)2
+ .021(WD3)2
CY = -.006(P) - .006(R) + .169(WD1) - .121(WD2) - .063(WD3) - .004(P)(R) + .008(P)(WD1) -
.006(P)(WD2) - .008(P)(WD3) - .012(R)(WD1) - .029(R)(WD2) + .048(R)(WD3) - .008(WD1)2
CM = .023 - .008(P)2 + .004(P) - .007(R) + .024(WD1) + .066(WD2) - .099(WD3) - .006(P)(R) +
.002(P)(WD2) - .005(P)(WD3) + .023(R)(WD1) - .019(R)(WD2) - .007(R)(WD3) + .007(WD1)2
- .008(WD2)2 + .002(WD1)(WD2) + .002(WD1)(WD3)
CYM= .001(P) + .001(R) - .050(WD1) + .029(WD2) + .012(WD3) + .001(P)(R) - .005(P)(WD1) -
.004(P)(WD2) - .004(P)(WD3) + .003(R)(WD1) + .008(R)(WD2) - .013(R)(WD3) + .004(WD1)2
+ .003(WD2)2 - .005(WD3)2
Ce = .003(P) + .035(WD1) + .048(WD2) + .051(WD3) - .003(R)(WD3) + .003(P)(R) - .005(P)(WD1)
+ .005(P)(WD2) + .006(P)(WD3) + .002(R)(WD1)
Aircraft Equations
Page 25©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
EW Test Example with 6 Factors(18 orthogonal test cases)
Page 26©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
EW Test Example (cont.)(Marginal Means Analysis)
WHAT
IS THE
GOAL?
SCREENINGVALIDATE PERFORMANCE
(TEST COVERAGE)
Rules of Thumb for Selecting a Test Design(based on the objective of the test)
MODELING
Notes:
1. “Mixed” factors means a combination of quantitative and qualitative (categorical)
2. “Mixed” levels means that not all factors have the same number of levels (settings)
3. “K” = Number of Factors and “L” = Number of Levels
4. “OA” = Orthogonal Array; “PAVO” = Pairwise Value Ordering
5. Software such as DOE Pro™, HD ToolsTM, rdExpertTM Lite, Pro-TestTM and
Quantum XLTM generate some or all of these designs
* DS and LHS are sampling techniques to generate representative samples
according to a specified distribution and a specified sample size
* Representative samples do not give orthogonal designs. They are often
used for getting test coverage, validating performance/ determining
capability, or creating noise combinations for test
© Copyright Air Academy Associates, LLC.
DOE Pro™ software is copyright Air Academy Associates, LLC and Digital Computations, Inc.
HD ToolsTM is a trademark of Air Academy Associates, LLC and software is copyright SigmaXL.rdExpertTM Lite software is copyright Phadke Associates, Inc.Pro-TestTM software is copyright Digital Computations, Inc.Quantum XLTM software is copyright SigmaZone.com.
2-Level Designs:
L12(6 ≤ K ≤11)
3-Level Designs:
L18(4 ≤ K ≤ 8)
High Factor/High LevelDesigns (K ≥ 9 and L ≥ 5):
Nearly Orthogonal Latin
Hypercube Designs
(NOLHDs) with K*L runs
Mixed Factor/Mixed Level Designs:HTT (OA or PAVO)
2-Level Designs:Full Factorial(K ≤ 4)
Fractional
Factorial
(K = 5)
3-Level Designs:Full Factorial(K ≤ 3)
CCD (3 ≤ K ≤ 5)
BB (3 ≤ K ≤ 4)
Mixed Factor/Mixed Level Designs:Full FactorialHTT (OA or PAVO, with select interactions only)
Fixed Number of Samples:Descriptive Sample (DS)*Latin Hypercube Sample (LHS)*
Not a Fixed Number of Samples:HTT (OA or PAVO)
Page 28©2016 Air Academy Associates Do Not Reproduce.
Integrating Test Design into Systems Engineering
FL
OW
ING
RE
QU
IRE
ME
NT
S D
OW
N
Customer Needs
System
Requirements
Sub-system
Requirements
Module
Requirements
Parts
Requirements
Parts
Performance
Module
Performance
Sub-system
Performance
System
Performance
Customer Acceptance
FL
OW
ING
CA
PA
BIL
ITY
UP
Design & Development
Page 29©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Critical Parameter Management and COIs
– A Critical Operational Issue (COI) is linked to operational effectiveness and
suitability.
– It is typically phrased as a question, e.g.,
Will the system detect the threat in a combat environment at
adequate range to allow for successful engagement?
y2 (engagement)
y1 (detect)
x1 x2 x3 (ranges) x4 (threat type) x5 x6
Page 30©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Good Test Design Enables Critical Parameter Management (CPM)
CPM is a systems engineering best practice that is extremely useful in managing, analyzing, and reporting technical product performance.
Page 31©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Where can Big Data help?
We already have DT&E and OT&E, but we also need continuous monitoring of
the system throughout its entire lifecyle to detect, predict, and prevent emergent
behavior. Big Data analytics can help us continuously test & evaluate (CT&E).
Page 32©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Guidelines for More Effective and Efficient Testing
Use continuous response variables whenever possible.
Eliminate confounding and aliasing of factors by using orthogonal or
nearly orthogonal designs.
Orthogonality means balance, both vertical and horizontal balance,
in the test design matrix or covering array.
Orthogonality provides the following capabilities:
• Can evaluate each of the factors independently, which implies a
cause-and-effect relationship can be established
• Can build linear and non-linear models, including interaction effects,
for prediction and risk assessment
• Provides better test domain coverage, due to the balance in the
designs, than non-orthogonal designs
• Not only detects defects but can also isolate them
Namely, use DOE.
Great data is never an accident. It happens by design.
Page 33©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Thank You
Questions
Colorado Springs, Colorado
Page 34©2016 Air Academy Associates, LLC. Do Not Reproduce.
Simplify, Perfect, Innovate
Air Academy Associates, LLC1650 Telstar Drive, Ste 110
Colorado Springs, CO 80920
Toll Free: (800) 748-1277 or (719) 531-0777
Facsimile: (719) 531-0778
Email: [email protected]
Website: www.airacad.com
For More Information, Please Contact