quantifying risk of end result specifications

1

Quantifying Risk of End Result

Specifications

CalAPA Fall Conference October 25 – 26, 2017

Sacramento, CA

Tony Limas

Granite Construction Inc.

Today’s Discussion

2

Specification Tolerances – Should We care? Evolution of Specifications Best Practice for Establishing Specification Limits Types of Risk Measuring Risk - Examples Risk vs. Number of Observations Questions…

Managing Risk

3

Statistics – Ugh…

4

Specification Tolerances When specifications contain unreasonable or

unattainable material tolerances it is likely that

a contractor, providing a product using all the

care and skill normally exercised within the

industry, will fail to meet the specified

acceptance requirements. Such specifications

are said to be unbalanced assigning excessive

risk to the contractor and thus not suitable for

use.

FHWA - NHI Course No. 13442

5

Evolution of Specifications

Pre 1958

Acceptance Primarily based on Inspections vs Test Results

Specification tolerances Primarily based on Anecdotal or

shoot from the hip criteria

Contractors Struggled to Meet Acceptance Limits

Post 1958

1958 AASHTO Road Test Collected “Real Time” Test Data

Variability of Material Properties Better Understood

Information was used to establish End-Result Spec Limits 6

Evolution of Specifications (con’t)

Specifications Must:

Recognize Total Variability of Materials and Construction

Properties (Standard Deviation)

Must Apply Reasonable Risk to All Parties

7

Types of Risks

Buyer’s Risk β = Risk of Accepting “Bad” Material

Seller’s Risk α = Risk of Rejecting “Good” Material

FHWA Recommended Seller’s Risk (α): 5.0% Max.

Typically 2s About the Mean

8

9

Are Risks

Determine Buyer’s and

Seller’s Risk of

Proposed Spec Limits

Modify Spec limits,

Sample Size

and/or lot Size

Finalize The Specifications

No

Yes Acceptable

Yes

Specification

Development

Process

Measuring Risk Standard Deviation Is the measure of dispersion of a set of data from its mean. It measures

the absolute variability of a distribution; the higher the dispersion or variability,

the greater is the standard deviation and greater will be the magnitude of the

deviation of the value from their mean.

10

Measuring Risk

Transportation Related Material Properties Are:

Symmetrically/Normally Distributed About the Mean

Mean

Test Results Test Results

11

Measuring Risk (con’t) N

o. o

f S

am

ple

s

12

Mean

Material Properties Distribution PWL N

o. o

f S

am

ple

s

-3s -2s -1s +1s +2s +3s

68%

95%

99.7%

Air Voids (S) = 0.75

Target ± 0.75%

13

PWL = 68%

Target

Sellers Risk (α) N

o. o

f S

am

ple

s

-3s -2s -1s +1s +2s +3s

68%

95%

99.7%

Air Voids (s) = 0.75

Target ± 0.75%

16%

14

Sellers Risk

α = 32%

Sellers Risk (α) N

o. o

f S

am

ple

s

-3s -2s -1s +1s +2s +3s

68%

95%

99.7%


Target ± 1.5%

2.5%

15

Seller’s Risk

α = 5%

Target

Sellers Risk (α) N

o. o

f S

am

ple

s

-3s -2s -1s +1s +2s +3s

68%

95%

99.7%


Target ± 1.5%

16

Target Mean

Seller’s

Risk >5.0%

Risk (α) Evaluation Examples

Based on Specification Tolerances (vs SD)

USL = Upper Specification limit

LSL = Lower Specification Limit

Mean or Target Value

Standard Deviation: Population (S) or Sample (s)

Z Score Chart for Normal Distribution

17

Risk Evaluation Examples

Z Score Chart

18

Risk (α) Evaluation Examples

Z Score Chart

(USL - x̄)/S= Z score

(LSL - x̄)/S= Z score

Z of ≥1.96 = ≤ 5.0% Risk

19


Caltrans Proposed Binder Content Tolerance

CT Proposal ± 0.3% (1 observation)

Evaluate the Risk Associated with the Proposed Limits

20


What is the Variation for Binder Content?

Based on Statewide Pooled Data from QC/QA Projects

Population Standard Deviation (S) = 0.20

21

22

Seller’s Risk (With 0.3% Tolerance)

S

(n=1)x̄ PWL

Risk

(α)

0.20 5.0 86 14%

Binder Content

1.7 3.7 4.7 5.0 5.3 6.3 7.3

Upper limitLower limit

Target

5.0%

16%7%7%

NTS

86%

23

Sellers (other) Risk(With 0.3% Tolerance)

S

(n=2)*x̄ PWL

Risk

(α)

0.141 5.0 96 4%

Binder Content

1.7 3.7 4.7 5.0 5.3 6.3 7.3

Upper LimitLower Limit

Target

5.0%

16%2%2%

NTS

96%

*Avg. of Two Independent Samples

15

Balancing Risk and Cost A

gen

cy a

nd

/or

Co

ntr

act

or

Ris

k

Dir

ect

Co

st (

$)

1 2 3 4 5 6 7

Number of Test Samples (n)

Total Variability (SD)

Variability = variability + variability + variability

(sampling) (test method) (mat./const.)

S2QC/QA = S2

s + S2t + S2

m/c

3

26

Sellers Risk(With 0.4% Tolerance)

S

(n=1)x̄ PWL

Risk

(α)

0.20 5.0 95 5%

Binder Content

3.8 4.2 4.6 5.0 5.4 5.8 6.2


Target

5.0%

16%2.5% 1.72.5%

NTS

95%

27

Seller’s Risk(With -0.3 +0.5% Skewed Tolerance)

S

(n=1)PWL

Risk

(α)

0.20 5.0 92 8%

Binder Content

3.8 4.2 4.7 5.0 5.5 6.0 6.5


Target

5.0%

16%1% 1.77%

NTS

49%43%

x̄

28

Seller’s Risk(With -0.3 +0.5% Skewed Tolerance)

S

(n=1)PWL

Risk

(α)

0.20 5.1 95 5%

Binder Content

4.7 5.1 5.5


Mean

5.1%

16%2.5%2.5%

NTS

47.5%47.5%

x̄Target Value + 0.10

29

Skewed Binder Content Effect on Volumetrics

Air Voids

- 4.0 +

Design Binder

5.0%

16%

NTS

Target Binder + 0.105.1%

Percent Defective

3

30

Seller’s Risk(With 0.4% Tolerance)

S

(n=1)x̄ PWL

Risk

(α)

0.20 5.0 95 5%

Binder Content

3.8 4.2 4.6 5.0 5.4 5.8 6.2


Target

5.0%

16%2.5% 1.72.5%

NTS

95%

Evaluating Risk Examples (con’t)

Local Agency Relative Density Specification Local Agency 92 – 97 % (n=1)

Contractors’ Could Not Meet Specifications

Spec was Evaluated to Determine Contractors Risk

31


What is the Variation for Relative Density?

Sample Standard Deviation (s) = 1.84

Based on <30 observations from projects

32

9

33


S

(n=1)x̄ PWL

Risk

(α)

1.84 94.5 82 18%

Relative Density92.0 94.5 97.0


Target

16%9%9%

NTS

82%

34


S

(n=1)x̄

PW

L

Risk

(α)

1.84 94.5 98 2.0%



Target

16%1%1%

NTS

98%

Option 1= Open Spec Band ±2.0%

Relative Density Specifications

35

Relative Density Pay Factor

97.1 0r Higher (Over-asphalted mix) 90% Pay Factor

92-97% (Ideal) 100% Pay Factor

89 – 91.9 (Marginal Air Voids) 85% Pay Factor

88.9 Or Less Reject (RQL)

Pay Factors

For all asphalt concrete pavement subject to acceptance testing, the

finished asphalt concrete pavements that do not conform to the

specified relative compaction requirements will be paid for using the

following pay factors:

36

Buyers Risk (β)(With 4.5% Tolerance)

S

(n=1)x̄

Buyer’s

Risk (β)

1.84 88.9 28%

Relative Density

88.9 90.0

Lower Limit

16%28%

NTS

RQL

Reject

Rejectable Quality Limit = 89.9

37

Buyers Risk (β)

88.9 LSL Target USL

16%Buyers

Risk (β)

NTS

AQL

RQL

α

38


S

(n=2)*x̄ PWL

Risk

(α)

1.30 94.5 98 2%



Target

16%1%1%

NTS

98%

Avg. of Two Independent Samples

Risk vs Number of Observations (n)

The myth of the Single Representative Sample:

“The idea persists that a test on a single sample shows

the "true" quality of the material, and that if any test result

is not within some limit, there is something wrong with the

material, construction, sampling or testing. Thus, terms

such as investigational, check, and referee samples are

in common use to either confirm or document these

"failures.“ Nature dislikes identities; variation is the rule.

Therefore, any acceptance or process control sampling

must account for variability of materials or construction.

Multiple sampling accomplishes this objective”

FHWA - NHI Course No. 13442

39

Risk Vs Number of Test A

gen

cy a

nd

/or

Co

ntr

act

or

Ris

k

1 2 3 4 5 6 7

Number of Test Samples (n)

Best Practice:

Never make a decision to reject material

based on a single observation!

40

3

41

Seller’s Risk(QC/QA Multiple Observations)

S

(n=1)x̄ PWL

Risk

(α)

0.20 5.0 95 5%

Binder Content

3.8 4.2 4.6 5.0 5.4 5.8 6.2


Target

16%2.5% 1.72.5%

NTS

95%

42

Buyer’s/Seller’s Risk (Standard Spec With Single Observation)

Asphalt Content

16%

NTS

RQL Population

.

AQL Population

Single Test Specification

FHWA Peer Review Team Recommendation:

For other items without pay factors it is recommended that if one

test falls outside the specification limit then another test will

be taken. If the specification limit is met on the subsequent

test, production continues without any penalties.

If the second consecutive test falls outside the specification limit,

production will cease until the contractor demonstrates that the

specification limit can be met.

43

44

Sellers Risk (Test Result with Check or Referee Test)

Asphalt Content

16%

NTS

RQL Population

.

.

.

AQL Population

45

Sellers Risk (Population Defined with Additional Test)

Asphalt Content

16%

NTS

RQL Population

.

AQL Population

.

.

.. .. .....

...

Solution To Single Test Dilemma

The 2007 FHWA Peer Review Team Recommendations:

Pay equations need to be established for binder content for standard

and method specifications

Penalties should be commensurate with the performance of the

pavement.

46

And Remember

Never Reject Material Based on a Single Observation!

47

quantifying risk of end result specifications

Engineering