research matters nick higham february 25, 2009 school of … · 2019. 6. 4. · the square root of...

Research Matters

February 25, 2009

Nick HighamDirector of Research

School of Mathematics

1 / 6

Probabilistic Versus Worst-CaseRounding Error Analysis

Nick HighamSchool of Mathematics

The University of Manchesterwww.maths.manchester.ac.uk/~higham

Advances in Numerical Linear AlgebraUniversity of Manchester, May 29-30, 2019

Joint work with Theo Mary

http://www.manchester.ac.uk

http://www.maths.manchester.ac.uk/our-research/research-groups/numerical-analysis-and-scientific-computing/numerical-analysis/

http://www.maths.manchester.ac.uk/~higham/

http://www.maths.manchester.ac.uk/

http://www.man.ac.uk

http://www.maths.manchester.ac.uk/~higham

James Hardy Wilkinson (1919–1986)

Nick Higham Probabilistic Rounding Error Analysis 2 / 28

Today’s Floating-Point Arithmetics

Type Bits Range u = 2−t

bfloat16 half 16 10±38 2−8 ≈ 3.9× 10−3

fp16 half 16 10±5 2−11 ≈ 4.9× 10−4

fp32 single 32 10±38 2−24 ≈ 6.0× 10−8

fp64 double 64 10±308 2−53 ≈ 1.1× 10−16

fp128 quadruple 128 10±4932 2−113 ≈ 9.6× 10−35

fp* forms all IEEE standard.bfloat16 used by Google TPU.bfloat16 will be in Intel Xeon Cooper Lake (2020).


Rounding

fl(x) denotes rounding x to the nearest fl pt number.

TheoremFor x ∈ R, excluding overflow and underflow,

fl(x) = x(1 + δ), |δ| ≤ u.

u := 12β

1−t is the unit roundoff, or machine precision.

The normalized nonneg numbers for β = 2, t = 3:

0 0.5 1.0 2.0 3.0 4.0 5.0 6.0 7.0


Model for Rounding Error Analysis

For x , y ∈ F

fl(x op y) = (x op y)(1 + δ), |δ| ≤ u, op = +,−, ∗, /.

Also for op =√

.

Wilkinson (1960):


Weaknesses of the Model

Worst-case bound on |δ|.

Does not account for some modern hardwarefeatures.

Weaker than fl(x op y) being correctly rounded.

Does it matter?Yes.Yes.No (usually).


Model vs Correctly Rounded Resulty = x(1 + δ), with |δ| ≤ u does not imply y = fl(x).

β = 10,t = 2

x y |δ| u = 12101−t

9.185 8.7 5.28e-2 5.00e-29.185 8.8 4.19e-2 5.00e-29.185 8.9 3.10e-2 5.00e-29.185 9.0 2.01e-2 5.00e-29.185 9.1 9.25e-3 5.00e-29.185 9.2 1.63e-3 5.00e-29.185 9.3 1.25e-2 5.00e-29.185 9.4 2.34e-2 5.00e-29.185 9.5 3.43e-2 5.00e-29.185 9.6 4.52e-2 5.00e-29.185 9.7 5.61e-2 5.00e-2


Hardware Features That Reduce Error

Fused multiply-add (FMA) may be available (NVIDIAV100), with just one rounding error:

fl(x + y ∗ z) = (x + y ∗ z)(1 + δ), |δ| ≤ u.

Reduces error bound by factor 2.

80-bit registers on Intel chips may be exploited bythe compiler in evaluating inner products, etc.

“Tensor units” with block FMA on Google TPU andNVIDIA Volta and Turing GPUs.


Traditional Bounds Are Pessimistic (1)Traditional bounds are worst-case bounds and arepessimistic on average. For Uniform [−1,1] data:

Matrix–vector product (fp32)

10 1 10 2 10 3 10 410 -8

10 -7

10 -6

10 -5

10 -4

10 -3

Solution of Ax = b (fp32)

10 1 10 2 10 3 10 410 -8

10 -6

10 -4

10 -2


Traditional Bounds Are Pessimistic (2)Matrix–vector product (fp16)

10 0 10 1 10 2 10 310 -4

10 -3

10 -2

10 -1

10 0


10 0 10 1 10 2 10 3

10 -1

10 0

Traditional bounds do not provide a realistic picture of thetypical behavior of numerical computations


The Rule of Thumb

“In general, the statistical distribution ofthe rounding errors will reduceconsiderably the function of n occurring inthe relative errors. We might expect ineach case that this function should bereplaced by something which is no biggerthan its square root.”

— Wilkinson, 1961


Statistical Argument

Consider sum of rounding errors s =∑n

i=1 δi , |δi | ≤ u.

Worst-case bound |s| ≤ nu is attainable—unlikely!

Assume δi are independent random variables ofmean zeroCentral limit theorem: for sufficiently large n,

s/√

n ∼ N (0, σ2), σ ≤ u,

hence |s| ≤ λ√

nu , with λ a small constant, holds withhigh probability (e.g., 99.7% with λ = 3).

s is only linear part of error.What n is “sufficiently large”?


Objective

Fundamental lemma in backward error analysis (H, 2002)If |δi | ≤ u for i = 1 : n and nu < 1 then

n∏i=1

(1 + δi) = 1 + θn,

where|θn| ≤ γn :=

nu1− nu

= nu + O(u2).

The basis of most rounding error analyses.We seek an analogous result with a smaller, butprobabilistic, bound on θn.We will focus on backward error results.


Probabilistic Model of Rounding Errors

In the computation of interest, the quantities δ infl(a op b) = (a op b)(1 + δ), |δ| ≤ u, op ∈ {+,−,×, /}

are independent random variables of mean zero.

“There is no claim that ordinary rounding andchopping are random processes, or thatsuccessive errors are independent. Thequestion to be decided is whether or notthese particular probabilistic models of theprocesses will adequately describe whatactually happens.”

— Hull & Swenson, 1966


Why Might the Model Not be Realistic?

In some cases δ ≡ 0, e.g., integer operands in x + y ,xy , or an operand a power of 2 in xy , x/y .

Pairs of operands might be repeated, so different δare the same.

Non-pathological examples can be found whererounding errors are strongly correlated (Kahan).

If an operand comes from an earlier computation it willdepend on an earlier δ and so the new δ will dependon a previous one.


The Key IdeasTransform the product into a sum by taking the logarithm:

S = logn∏

i=1

(1 + δi) =n∑

i=1

log(1 + δi).

Hoeffding’s concentration inequalityLet X1, . . . , Xn be random independent variables satisfying|Xi | ≤ ci . Then the sum S =

∑ni=1 Xi satisfies

Pr(|S − E(S)| ≥ ξ) ≤ 2exp(− ξ2

2∑n

i=1 c2i

).

Apply to Xi = log(1 + δi)⇒ requires boundinglog(1 + δi) and E (log(1 + δi)) using Taylor expansions.Retrieve the result by taking the exponential of S.


Probabilistic Error Bound

Theorem (H & Mary, 2018)Let δi , i = 1 : n, be independent random variables of meanzero such that |δi | ≤ u. For any constant λ > 0, the relation∏n

i=1(1 + δi) = 1 + θn holds with

|θn| ≤ γn(λ) := exp(λ√

nu +nu2

1− u

)− 1

≤ λ√

nu + O(u2)

with prob of failure P(λ) = 2exp(−λ2(1− u)2/2

)

Key features:Exact bound, not first order.nu < 1 not required.No “n is sufficiently large” assumption.Small values of λ suffice: P(1) ≈ 0.27, P(5) ≤ 10−5.


Probabilistic Error Bound

Theorem (H & Mary, 2018)Let δi , i = 1 : n, be independent random variables of meanzero such that |δi | ≤ u. For any constant λ > 0, the relation∏n

i=1(1 + δi) = 1 + θn holds with

|θn| ≤ γn(λ) := exp(λ√

nu +nu2

1− u

)− 1

≤ λ√

nu + O(u2)

with prob of failure P(λ) = 2exp(−λ2(1− u)2/2

)Key features:

Exact bound, not first order.nu < 1 not required.No “n is sufficiently large” assumption.Small values of λ suffice: P(1) ≈ 0.27, P(5) ≤ 10−5.


Inner Products

TheoremLet y = aT b, where a,b ∈ Rn. No matter what the order ofevaluation the computed y satisfies

y = (a + ∆a)T b,

|∆a| ≤ γn(λ)|a| ≤ λ√

nu|a|+ O(u2),

with probability of failure nP(λ).

Now a factor n in front of P(λ).Can choose any λ > 0.Analogous result for matrix–vector products.


LU Factorization

TheoremThe computed LU factors from Gaussian elimination onA ∈ Rn×n satisfy LU = A + ∆A, where

|∆A| ≤ γn(λ)|L||U|, |γn(λ)| ≤ λ√

nu + O(u2)

holds with probability of failure (n3/3 + n2/2 + 7n/6)P(λ).

Want probabilities independent of n! Fortunately:

O(n3)P(λ) = O(1) ⇒ λ = O(√

logn)

⇒ error grows no faster than√

n logn u. . . and the constant hidden in the big O is small:

n3

3P(13) ≤ 10−5 for n ≤ 1010.


Probabilities of Success for A = LU Bounds

λ n = 102 n = 103 n = 104

7.0 9.9998e-01 9.8474e-01 −1.4265e+017.5 1.0000e+00 9.9959e-01 5.9320e-018.0 1.0000e+00 9.9999e-01 9.9156e-018.5 1.0000e+00 1.0000e+00 9.9986e-019.0 1.0000e+00 1.0000e+00 1.0000e+00


MATLAB Experiments

Simulate fp16 and fp8 with MATLAB functionchop (H & Pranesh, 2019).

Compare the bounds γn and γn(λ) with thecomponentwise backward error εbwd (Oettli–Prager):

Matrix–vector product y = Ax : εbwd = maxi|y−y |i(|A||x |)i

Solution to Ax = b via LU factorization:εbwd = maxi |Ax − b|i/(|L||U||x |)i

A and x are chosen.

Random entries are Uniform [−1,1] or Uniform [0,1].

Codes available online: https://gitlab.com/theo.andreas.mary/proberranalysis


https://gitlab.com/theo.andreas.mary/proberranalysis

https://gitlab.com/theo.andreas.mary/proberranalysis

Random [−1,1] Entries


10 1 10 2 10 3 10 410 -8

10 -7

10 -6

10 -5

10 -4

10 -3

Linear system Ax = b (fp32)

10 1 10 2 10 3 10 410 -8

10 -6

10 -4

10 -2

Probabilistic bound (λ = 1) much closer to the actualerror.But for [−1,1] entries it’s still pessimistic


Random [0,1] EntriesMatrix–vector product (fp32)

10 1 10 2 10 3 10 410 -8

10 -7

10 -6

10 -5

10 -4

10 -3

Linear system Ax = b (fp32)

10 1 10 2 10 3 10 410 -8

10 -6

10 -4

10 -2

Prob bound has λ = 1⇒ P(λ) pessimistic . . .. . . but γn bound itself can be sharp and successfullycaptures the

√n error growth

⇒ the bounds cannot be improved without furtherassumptions


Low Precisions, Random [−1,1] EntriesMatrix–vector product (fp16)

10 0 10 1 10 2 10 310 -4

10 -3

10 -2

10 -1

10 0


10 0 10 1 10 2 10 3

10 -1

10 0


Low Precisions, Random [0,1] EntriesMatrix–vector product (fp16)

10 0 10 1 10 2 10 310 -4

10 -3

10 -2

10 -1

10 0


10 0 10 1 10 2 10 3

10 -1

10 0

Importance of the probabilistic bound becomes evenclearer for lower precisions


Real-Life MatricesSolution of Ax = b (fp64), b from Uniform [0,1],

for 943 matrices from SuiteSparse collection (λ = 1).

101

102

103

104

10-16

10-15

10-14

10-13

10-12


Example: Rounding Errors Nonzero MeanInner product of two very large nonnegative vectors:

si+1 = si + aibi ⇒ si+1 = (si + aibi)(1 + δi)

100

102

104

106

108

10-10

10-5

100

Top: 1 ≤ n ≤ 106

Bottom: 106 ≤ n ≤ 108

Explanation: si ⇑ and at some point it is so large thatsi+1 = si ⇒ δi = −aibi/(si + aibi) < 0


Example: Rounding Errors Nonzero MeanInner product of two very large nonnegative vectors:

si+1 = si + aibi ⇒ si+1 = (si + aibi)(1 + δi)

100

102

104

106

108

10-10

10-5

100 Top: 1 ≤ n ≤ 106

Bottom: 106 ≤ n ≤ 108

Explanation: si ⇑ and at some point it is so large thatsi+1 = si ⇒ δi = −aibi/(si + aibi) < 0


Conclusions

Given first rigorous justification of Wilkinson’s “takethe square root of the constant in error bound” rule ofthumb—and our results hold for any n!Experiments show prob bounds give betterpredictions than deterministic ones for both randomand real-life matrices and can be sharp. Consistentwith

The fact that rounding errors are neitherrandom nor uncorrelated will not in itselfpreclude the possibility of modelling themusefully by uncorrelated random variables.

— Kahan, 1996

H & Mary, A New Approach to Probabilistic RoundingError Analysis, MIMS EPrint 2018.33, Nov 2018.


http://eprints.maths.manchester.ac.uk/2689/

http://eprints.maths.manchester.ac.uk/2689/

References I

N. J. Higham.Accuracy and Stability of Numerical Algorithms.Society for Industrial and Applied Mathematics,Philadelphia, PA, USA, second edition, 2002.ISBN 0-89871-521-0.xxx+680 pp.

N. J. Higham and T. Mary.A new approach to probabilistic rounding error analysis.MIMS EPrint 2018.33, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Nov. 2018.22 pp.Revised March 2019.


References II

N. J. Higham and S. Pranesh.Simulating low precision floating-point arithmetic.MIMS EPrint 2019.4, Manchester Institute forMathematical Sciences, The University of Manchester,UK, Mar. 2019.17 pp.

T. E. Hull and J. R. Swenson.Tests of probabilistic models for propagation of roundofferrors.Comm. ACM, 9(2):108–113, 1966.


References III

W. Kahan.The improbability of probabilistic error analyses fornumerical computations.Manuscript, Mar. 1996.

J. H. Wilkinson.Error analysis of direct methods of matrix inversion.J. Assoc. Comput. Mach., 8:281–330, 1961.


Example: Rounding Errors Not IndependentInner product of twoconstant vectors:

si+1 = si + aibi = si + c⇒ si+1 = (si + c)(1 + δi)

⇒ δi = θ is constant withinintervals [2q−1; 2q]

10 2 10 3 10 4

10 -6

10 -5

10 -4

10 -3

10 -2

2q−1 2q

×

si si+1 si+2 si+3

+c

××θ

+c

××θ

+c

××θ


research matters nick higham february 25, 2009 school of … · 2019. 6. 4. · the square root of...

Documents