1
Can You Measure It?Planning a Successful “Round
Robin”
Paul SchiffelbeinASTM Committee Week
14 June 2006
2
Overview
• Measurement system analysis (Round robin, Interlaboratory Study (ILS), Interlaboratory Comparison (ILC), Gage Study) description, definitions and motivation
• Round robin guidelines
• Round robin logistics
• Data Analysis
• Precision and bias statement
3
Purpose of Measurement System Analysis
• Ensure the measurement system has adequate accuracy: precision and bias
• Identify sources of measurement variation (and make improvements, if necessary)
• Compare several test devices, test methodologies, test locations (linearity, bias, sensitivity)
• Quantify measurement variability for reference
4
Actual Process Variation
Measurement Variation
Observed Process Variation
Long-term Process
Variation
Short-term Process
Variation
Variation within a Sample
Variation due to Labs,
Operators,
Devices,Time
Variation
due to Gage
Bias
Linearity
“Short-term”
Stability
“Long-term”
Measurement Process Variation
Simple “precision”
“bias”
5
ISO 5725-1, ASTM E177
• Accuracy: The closeness of agreement between a test result and the accepted reference value
– “Accuracy,” when applied to a set of test results, involves a combination of random componentsrandom components and a common systematic error or bias componentbias component
• Bias: the difference between the expectation of test results and an accepted reference value
– Bias is the total systematic error as contrasted to random error. There may be one or more systematic components contributing to the bias.
6
ISO 5725-1, ASTM E177
• PrecisionPrecision: The closeness of agreement between independent test results obtained under stipulated conditions.
– Precision depends only on the distribution of random errors and does not relate to the true value or the specified value.
• Precision can be decomposed into short- and long-term (or narrow and wide) components. RepeatabilityRepeatability and reproducibilityreproducibility are used to quantify this concept.
7
ISO 5725-1
Repeatability: Precision under repeatability conditions
• independent test results • the same test method• identical test items• the same laboratory• the same operator• the same equipment • within short time interval
– Repeatability standard deviation: The standard deviation of test results obtained under repeatability conditions
– Repeatability limit (“r”): the absolute difference between two test results obtained under repeatability conditions should be less than or equal to this value
8
ISO 5725-1
Reproducibility: Precision under reproducibility conditions
• independent test results • the same test method• identical test items• different laboratories• different operators• different equipment • longer time interval
– Reproducibility standard deviation: The standard deviation of test results obtained under reproducibility conditions
– Reproducibility limit (“R”): the absolute difference between two test results obtained under reproducibility conditions should be less than or equal to this value
9
Actual Process Variation
Measurement Variation
Observed Process Variation
Long-term Process
Variation
Short-term Process
Variation
Variation within a Sample
Variation due to Labs,
Operators,
Devices,Time
Variation
due to Gage
Bias
Linearity
“Short-term”
“repeatability”
Stability
“Long-term”
ASTM/ISO Usage
Simple “reproducibility”
“bias”
10
Actual Process Variation
Measurement Variation
Observed Process Variation
Long-term Process
Variation
Short-term Process
Variation
Variation within a Sample
Variation due to Labs,
Operators,
Devices,Time
Variation
due to Gage
Bias
Linearity
“Short-term”
“repeatability”
Stability
“Long-term”“reproducibili
ty”
Gage R&R/Auto Industries Usage
Simple “Gage”
“bias”
11
Round Robin Guidelines
• E 691 “Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method”
• D 2904 “Standard Practice for Interlaboratory Testing of a Textile Test Method that Produces Normally Distributed Data”
12
Round Robin Guidelines
• The design should be as simple as possible in order to obtain estimates of within- and between-laboratory variability that are free of secondary effects
• Study should include a minimum of five laboratories
• A minimum of two operators should be used per laboratory
• When multiple instruments within a laboratory are used, tests must be made on all equipment to establish the presence or absence of the equipment effect.
13
Logistics: D 885 Case Study
1 Motivation: The current version of D 885 was written for traditional tensile testing machines. Automated tensile testers are now being used in high-tenacity fiber testing, and need to be included in this standard. The current study will include three types of automated testers, as well as parallel testing on traditional tensile test devices. The objective of this study is to quantify test precision of traditional and automated testers, as well as any bias seen between the device types. The study will only address aramid materials.
14
Logistics: D 885 Case Study
2 Responsibility: Task group D13.19 (Tire Cord and Fabrics) has overall responsibility of the ILS. Dawn Caullwine (chair) will act as overall coordinator for conducting the ILS. The coordinator will supervise the distribution of materials and protocols to the laboratories and receive the test result reports from the laboratories.
15
Logistics: D 885 Case Study
3 Study Design: Nine materials will be tested on each of three types of automated tensile test devices. Two devices of each type will be used. Yarn will be supplied in pre-twisted state for testing. Untwisted yarn will also be provided to Statimat labs, so testing can be performed both on pre-twisted yarn, and yarn automatically twisted by the test machine. Two laboratories will also test the materials using traditional methods for reference. Each of those laboratories will use two operators.
16
Logistics: D 885 Case Study
Automated Testers
Sigma 500 Statimat Uster
Lab AAustria
Lab BAustria
Lab CEurope
Lab DUSA
Lab E Lab F
Pre-twisted Pre-twistedStatimat-twisted
Statimat-twisted
17
Logistics: D 885 Case Study
4 Materials: The ILS will include the following nine materials (All samples are shipped twisted and ready to test, except for Statimat laboratories, which receive both twisted and untwisted samples):– Kevlar®: 195 denier– Kevlar®: 600 denier– Kevlar®: 1140 denier– Nomex®: 200 denier– Nomex®: 1600 denier– Technora®: 550 denier– Twaron: 840 denier– Twaron: 1500 denier– Twaron: 3000 denier
18
Logistics: D 885 Case Study
5 Test Determinations and Test Results: The number of test determinations required for a test result is specified in each individual test method. For the purpose of this study, each laboratory will make one hundred (100) determinations (breaks) for each material.
19
Logistics: D 885 Case Study
5 Test Determinations and Test Results: The following properties (and associated measurement units) will be recorded:
– Break strength (BS) N– Elongation at break (EB) %– Modulus between 300 mN/tex and 400 mN/tex (MOD)
CN/tex– FASE @ 0.3% N– FASE @ 0.5% N– FASE @ 1.0% N
Use nominal linear density for modulus calculation.
20
Logistics: D 885 Case Study
6 Details: The test method being studied is D885-03. Specify the type of equipment used, including manufacturer, model, and software program. Samples should be conditioned as per D885 7.1 to moisture equilibrium in an environmentally controlled room for a minimum of 16 hours, at RH 55 +/- 2% and temperature 24 +/- 1 degrees C (72 +/- 2 degrees F).
Specific for Para-Aramid:– Gauge length: 500mm– Crosshead rate: 250mm/min. (50%)
Etc.
21
Logistics: D 885 Case Study
7 Data: Data should be entered into the Excel workbook provided. Label the workbook with your laboratory name, fill in the test data, and send the completed file to:
Paul SchiffelbeinDuPont Engineering, Quality Management & Technology
[email protected]/774-2417
22
Logistics: D 885 Case Study
23
Logistics: D 885 Case Study
24
A Better Way?: Internet Entry for F 2130 RR
Figure courtesy of Dr. Anugrah Shaw, UMES
25
Data Structure Guaranteed!
26
Logistics: D 885 Case Study
8 Data Analysis: Material = Kevlar® 1420d
Variability Chart for BF(N)
BF
(N)
250
260
270
280
290
300
310
320
330
340
350S
pru
an
ce
A
Sp
rua
nce
B
TF
A
TT
QR
S
Le
nzin
g
Ma
insite
Sp
rua
nce
TF
A
Sp
rua
nce
TF
A
TT
QR
SInstron Sigma500StatimatStatimat PTUster
Laboratory within Device
27
Logistics: D 885 Case Study
8 Data Analysis: S
td D
ev
0
5
10
15
20
Spr
uanc
e A
Spr
uanc
e B
TFA
TT
QR
S
Lenz
ing
Mai
nsite
Spr
uanc
e
TFA
Spr
uanc
e
TFA
TT
QR
S
Instron Sigma500 Statimat Statimat PTUster
Laboratory within Device
Material = Kevlar® 1420d
Variability Chart for BF(N)
28
Repeatability and Reproducibility
Use between- and within-laboratory variance components:
R = 2L + 2
r
Where:
R is the reproducibility standard deviation
r is the repeatability standard deviation
L is the square root of the inter-laboratory
(device, operator, etc.) variance component
29
REML Variance Component EstimatesRandom Effect Var Ratio Var
ComponentStd Error 95% Lower 95% Upper Pct of Total
Device&Random 0.6823116 66.35 62.607 18.903 1854.81 31.5Laboratory[Device]&Random 0.4806723 46.74 26.848 19.480 224.123 22.2
Residual 97.25 46.2Total 210.36 100.0
Logistics: D 885 Case Study
8 Data Analysis:
Repeatability variance Reproducibility variance(one estimate)
Reproducibility variance(another estimate)
30
Logistics: D 885 Case Study
9 Precision and Bias Statement:
13. Precision and Bias
13.1 Interlaboratory Test Data – An interlaboratory test was conducted in 2006 using
commercially available tensile testers from … . Nine materials were included in
the study, and are listed in table _. The study structure is shown in Figure _.
Variance components were computed for individual tensile determinations, and are
summarized in Table _.
31
Logistics: D 885 Case Study
9 Precision and Bias Statement:
13.2 Precision - Repeatability and reproducibility deal with the variability of test
results obtained under specified laboratory conditions. Repeatability concerns the
variability between independent test results obtained within a single laboratory in
the shortest practical period. Those results are obtained by a single operator with
a specific set of test apparatus using test specimens (or test units) taken at random
from a single quantity of homogeneous material obtained or prepared for the
interlaboratory study (ILS). Reproducibility deals with the variability between
single test results obtained in different laboratories, each of which has applied the
test method to test specimens (or test units) taken at random from a single
quantity of homogeneous material obtained or prepared for the ILS.
32
Logistics: D 885 Case Study
9 Precision and Bias Statement:
Method repeatability is defined as the "maximum difference" that can
"reasonably" be expected between two test results obtained on the same material
when the test results are obtained in the same laboratory. Repeatability standard
deviation is taken to be the square root of the “determination” variance
component, and represents within-operator precision. Method reproducibility is
defined as the "maximum difference" that can "reasonably" be expected between
two test results obtained on the same material when the test results are obtained
from different laboratories. The total, or reproducibility, standard deviation, is
formed by taking the square root of the sum of intra- and inter-laboratory variance
components.
33
Logistics: D 885 Case Study
9 Precision and Bias Statement:
Table _ contains repeatability and reproducibility standard devations and
maximum critical differences for single determinations and specified averages of
determinations for the single operator case (repeatability), within-laboratory case,
and between-laboratory case (reproducibility). Two values or averages of
observed values are considered significantly different at the 95% probability level
if the difference between them exceeds the appropriate critical difference in the
table.
34
Logistics: D 885 Case Study
9 Precision and Bias Statement:
13.3 Bias – The procedure in this test method produces a test value that can be definedonly in terms of a test method. There is no independent referee method by which biasmay be determined. The test method has no known bias.
35
Summary
• Explicitly state objectives for your study, and make sure all participants understand them
• Write protocol covering responsibilities, timing, samples and sample handling, test equipment and set-up, sampling plan, properties, units, etc.
• Details are important! Be specific, explicit, and discuss ahead of time with task group to avoid surprises. Communicate!
• Set up data collection to be as “goof proof” as possible, and to facilitate subsequent analysis
36
Thank you