the practice of standards formalization

The Practice of Standards Formalization

Victor KuliaminInstitute for System Programming (ISP RAS)

Moscow, Russia

Outline• ISP RAS Background

– OS Testing – Compiler Testing– Protocol Testing– Hardware Testing

• ISP RAS Technologies (brief summary)• Linux Standard Base Support Activity

– LSB Infrastructure– LSB conformance testing– Detailed: Math conformance testing

April 21, 2023 Practice of Standards Formalization 2/28

ISP RAS Background• Operating System testing

– Regression test suite for Nortel Switch OS 1994-2000

– POSIX conformance test suite for real-time OS 2005-…– Linux Standard Base conformance test suites 2005-…– ARINC 653 2008-…

• Compiler testing– Part of C expressions dynamic semantics 2000-2003– Static semantics specification and test suites 2002-2004– Optimizing units testing in gcc and Intel compilers 2001-2003

• Protocol testing– Testing Microsoft Research IPv6 2000-2001– Test suite for Microsoft Mobile IPv6 2002-2003– Test suite for IPsec 2004-…

• Hardware testing– MIPS-based processors with DSP extensions 2006-…


ISP RAS Technologies• Model based testing

– KVEST (1996)• RSL specifications• Protel target language

– UniTESK (2001)• Specifications in extensions of target languages

(C – 2001, Java – 2001, C# – 2003)• Concurrency testing extension (2001)• Compiler front-end testing support (2004)• Hardware testing support (2006)• Combinational test generation (2007)• Math extension (2007)

• Static analysis


Oracle

State modelState model

Model Based Testing General Scheme


System under test

Behavior model

Test action generator

Coverage metric

12%

Coverage criteria

36%57%87%

Linux Standardization• What to do with 550 distributions of Linux?• Linux Standard Base

– Binary interface standard– Supported by Free Standards Group– Includes

• StandardsPOSIX, X/Open Curses, Open GL, Large File Support, …

• LibrarieslibXML, gtk, Qt, JDK, Perl, …

– 45000 functions in C• 2000 described accurately (POSIX)• 7000-9000 have good description of main functionality• Others have very poor or just no description


ISP RAS Activity for LSB Support

• LSB Infrastructure development– DB of distributions, libraries, profiles, operations, etc.– Conformance checking and certification of distributions

• Static analysis tools• Test suites

– Conformance checking of applications• Monitoring tools• Test suites

• LSB evolution support– Analysis DB and information system

• Linux driver verification


Test Development Levels


2000 accurately described– the only target for formalization– UniTESK conformance testing

7000-9000 partially well-defined – manual test development

specialized massive automated test construction technology

45000

UniTESK API Test Development Basics

• API partitioned into logical modules (classes)• Each module state is modeled• Module operations described with stateful contracts

– Preconditions (on state and operation parameters)– Postconditions (on pre-state, post-state, operation

parameters and results)– Invariants (on state, hold when no op is working)

• Structure of postcondition gives test coverage criteria• Coverage-targeted FSM abstraction for module• Testing – automatic on-the-fly exploration of FSM


Math Library Standards

• IEEE 754 (Floating-point arithmetics)FP numbers, basic operations

• ISO 9899 (C language and libraries)56 real + 16 complex functions

• IEEE 1003.1 (POSIX)63 real + 22 complex functions

• ISO 10697.1-3 (Language independent arithmetics)Elementary real and complex functions


IEEE 754 Floating-Point Numbers


• Normal : E > 0 & E < 2k –1 X = (–1)S·2(E–B)·(1+M/2(n–k–1))

• Denormal : E = 0 X = (–1)S·2(–B+1)·(M/2(n–k–1)) • Exceptional : E = 2k –1

– M = 0 : +, –– M ≠ 0 : NaN

sign

k+1 n-1

0

exponent mantissa

0 1 1 1 1 1 1 0 1 0 01 0 0 0 0 0 0 0 0 00 0 0 0 0 00 0 0 0

0 1 k

n, k

S E MB = 2(k–1) –1

2(–1)·1.1012 = 13/16 = 0,8125

0, -0

1/0 = +, (–1)/0 = –0/0 = NaN

n = 32, k = 8 – float (single precision)n = 64, k = 11 – doublen = 79, k = 15 – extended doublen = 128, k = 15 – quadruple

1/2(n-k-1) – 1 ulp

• Operations: +, –, *, /, sqrt , fma (2008), type conversions, remainder

• Correct rounding – 4 rounding modes– to +– to –– to 0– to the nearest

• NaN and infinite results• Exception flags

– INVALID : Incorrect arguments (NaN result)– DIVISION-BY-ZERO : Infinite result (precise ±∞)– OVERFLOW : Too big result (approximate ±∞)– UNDERFLOW : Too small (or denormal) result– INEXACT : Inexact result

IEEE 754 Computations


0

ISO C and POSIX Requirements

• ISO/IEC 9899 (C language) : 54 real functions – Exact values : sin(0) = 0, log(1) = 0, …– DIVISION-BY-ZERO flag : log(0), atanh(1), pow(0,x), Г(-n)– NaN results and INVALID flag outside of domains

• IEEE 1003.1 (POSIX) : 63 real + 22 complex– All IEEE 754 flags (except for INEXACT) for real functions– errno setting: Domain error, Range error – If x is denormal

f(x) = x for f(x)~x in 0 (sin, asin, sinh, expm1…)– In overflow HUGE_VAL should be returned

(value of HUGE_VAL unspecified)– Sometimes non-NaN results on NaN arguments

fmax(NaN, x) = x, pow(NaN, 0) = 1


Inconsistency with rounding modes

Source of non-interoperability

glibc : +∞MSVCRT : max double (1.797693134862316e+308)Solaris libc : max float (3.402823466385289e+38)

Inconsistency with IEEE 754

Example of POSIX RequirementsNAME sin, sinf, sinl - sine function SYNOPSIS #include <math.h> double sin(double x); float sinf(float x); long double sinl(long double x);DESCRIPTIONThese functions shall compute the sine of their argument x, measured in radians.An application wishing to check for error situations should set errno to zero and call feclearexcept(FE_ALL_EXCEPT) before calling

these functions. On return, if errno is non-zero or fetestexcept(FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW | FE_UNDERFLOW) is non-zero, an error has occurred.

RETURN VALUEUpon successful completion, these functions shall return the sine of x.If x is NaN, a NaN shall be returned.If x is ±0, x shall be returned.If x is subnormal, a range error may occur and x should be returned.If x is ±Inf, a domain error shall occur, and either a NaN (if supported), or an implementation-defined value shall be returned. ERRORSThese functions shall fail if:Domain Error The x argument is ±Inf. If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno shall

be set to [EDOM]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the invalid floating-point exception shall be raised.

These functions may fail if:Range Error The value of x is subnormal If the integer expression (math_errhandling & MATH_ERRNO) is non-zero, then errno

shall be set to [ERANGE]. If the integer expression (math_errhandling & MATH_ERREXCEPT) is non-zero, then the underflow floating-point exception shall be raised.


ISO 10697 Requirements Real and complex elementary functions (no erf, gamma, j0, y1, … ) Only symmetric rounding modes (no rounding to + or to –)

• Preservation of sign• Preservation of monotonicity• Inaccuracy 0.5-2.0 ulp• Evenness and oddity• Exact values : cosh(0) = 1, log(1) = 0, …• Asymptotics near 0 : cos(x) ~ 1, sin(x) ~ x, …• Relations : expm1 <= exp, cosh >= sinh, atan <= ↓( π/2 ) , …


for sin, cos, tan – small arguments only

Summary of Requirements• Domain boundaries and poles (+ flags)• Exact values, limits and asymptotics• Preservation of sign and monotonicity• Symmetries

Evenness, periodicity, others : Г(1+x) = x·Г(x) • Relations and range boundaries• Precision

Correct rounding (according to mode) Computational accuracy Interoperability and portability of libraries and applications Feasible – ~ia64 (Intel), crlibm (INRIA)


Requirements TestedExtension of IEEE 754 on all library functions

• Correctly rounded results for 4 modes– Except for ones contradicting to range boundaries

• Infinite results in overflow and precise infinity cases– In overflow rounding to 0 returns the biggest finite number

• NaN results outside of function domain (and for NaN args)• Exception flags

– INVALID (and EDOM for errno) : Incorrect arguments– DIVISION-BY-ZERO (and ERANGE for errno) : Infinite result – OVERFLOW (and ERANGE for errno) : Too big result – UNDERFLOW (and ERANGE for errno) : Too small result ( + dnr)– INEXACT : Inexact result


Test Data Sources

• Bit structure of FP numbers– Boundaries

• 0, -0, +, -, NaN • Least and greatest positive and negative, normal and denormal

– Mantissa patternsFFFFFFFFFFFFF16 FFFFF1111000016 555550000FFFF16

Both arguments and values of a function

• Intervals of uniform function behavior

• Points hard to compute correctly rounded result


rint(262144.25)↑ = 2621440100000100010000000000000000000100000000000000000000000000000000x10000010001xxxxxxxxxxxxxxxxxx0100000000000000000000000000000000

Intervals and Boundaries• Neighbourhoods of 0, ±∞• Poles and overflow points• Zeroes and extremes• Tangents and asymtotics

– horizontal and diagonal


max

0

Table Maker Dilemma


tan(1.11011111111111111111111111111111111111111111000111112·2-22) = 1.1110000000000000000000000000000000000000000101010001 0 178 010…2·2-22

sin(1.11100000000000000000000000000000000000000111000010002·2-19) = 1.1101111111111111111111111111111111111100000010111000 067 11101…2·2-19

j1(1.10000000000000000000000000000000000000000000000000112·2-23) = 1.0111111111111111111111111111111111111111111111101000 094 11001…2·2-22

Rounding to the nearestf = x.xxxxxxxxxx|011111111...1xx...f = x.xxxxxxxxxx|100000000...0xx...

Rounding to 0, +, -f = x.xxxxxxxxxx|00000000...0xx...f = x.xxxxxxxxxx|11111111...1xx...

?!0,5 ulp

Number of Hard PointsProbabilistic evaluationUniform independent bits distribution• Total N = 2(n-k-1) values• ~N·2-m have m consecutive equal bits

Real data for sin on exponent -16


Eval. 0, +, - N

54 0.5 0 1

53 1 1 2

52 2 4 4

51 4 6 6

50 8 10 12

49 16 19 21

48 32 32 37

47 64 70 67

46 128 142 106

45 256 280 239

44 512 547 518

43 1024 1073 996

42 2048 2103 1985

41 4096 4187 4040

40 8192 8325 8142

3386417804515981120643892082331156599120239393299838035242121518428537554064774221620930267583474709602068045686026362989271814411863708499869721322715946622634302011697632972907922558892710830616034038541342154669787134871905353772776431251615694251273653 · π/2 = 1.0110101011000101101100100110001011001010000111111110 1 857 011…2·2849

sin(1.01101010110001011011001001100010110010100001111111112·2849) =1.11111111111111111111111111111111111111111111111111 1 690110…2·2-1

j = 15

sqrt(1.00100101011001010110010111001010110111001011111101002) =

1.0001001000001111100110011001111010011001001101110100 0 150 000…2

F(x) = f(x) – a·x – b = c1x2 + c2x3 + c3x4 + …

F(x) = c1(G(x) )2, G(x) = x + d1x2 + d2x3 +…

G(x) = y x = H(y), H is the reversed series

xm = H(sqrt(m/c12z)) F(xm) – a·xm – b = m/2z

Hard Points Calculation• Exhaustive search• Continued fractions (Kahan, 1983)• Dyadic method (Tang, 1989; Kahan, 1994)• Reduced search (Lefevre, 1997)• Lattice reduction (Gonnet, 2002; Stehle, Lefevre, Zimmermann, 2003)• Integer secants method (2007)


...2921

1

115

17

13π

Feasible only for single precision numbers

X ≈ N·π; X = M·2m; 2(n – k – 1) <= M < 2(n – k)

π ≈ (2m·M)/Nsqrt(N·2m) ≈ M + ½; 2(n-k-1) <= M, N < 2(n-k) 2(m+2)·N = (2·M + 1)2 – j (2·M + 1)2 = j (mod 2(m+2))

2–z

Test Suite Composition• Hard points

– double• Some hard points with ≥ 48 additional bits can be found in crlibm tests

http://lipforge.ens-lyon.fr/projects/crlibm• Calculated (some) hard points with ≥ 40 additional bits for

sqrt, cbrt, sin, asin, cos, acos, tan, atan, sinh, asinh, cosh, tanh, atanh, exp, log, exp2, expm1, log1p, erf, erfc, j0, j1

– float (single precision)• All hard points with ≥ 17 additional bits for sqrt, cbrt, exp, sin, cos

– extended double• All with ≥ 53 additional bits for sqrt, some for sin, exp

• Test suites developed– double : 58 real variable POSIX functions

• Correct values calculated by Maple and MPFR


sqrt exp sin atan lgamma j1

Boundary 20 20 20 20 20 20

Intrevals 106 1622 3674 4242 11680 24538

Patterns 141009 138451 331744 155008 121502 109036

Hard points 170170 28587 62342 95512 0 29436

Other 84820 0 4616 0 229 5664

Total 396125 168680 402396 254782 133431 168694

Tested Libraries


ID Processor arch Library OS

x86 i686 glibc 2.5 Linux Fedora

ia64 ia64 glibc 2.4 Linux Debian

x86_64 x86_64 glibc 2.3.4 Linux RHEL

s390 s390 glibc 2.4 Linux Debian

ppc64 ppc64 glibc 2.7 Linux Debian

ppc32 ppc32 glibc 2.3.5 Linux SLES

sparc UltraSparc III Solaris libc Solaris 10

VC8 x86_64 MS Visual C 2005 Windows XP

VC6 i686 MS Visual C 6.0 Windows XP

Test Results: Details


x86

ia64

x86_64

s390

ppc64

ppc32

sparc

VC6

VC8

x86

ia64

x86_64

s390

ppc64

ppc32

sparc

VC6

VC8

j1 y0 y1

log10

tgamma

log2

lgamma

log1p

j0

exp2

atan erf

expm1 log

erfc

fabs logb sqrt cbrt exp

sin cos tan asin acos

trunc

asinh

rint

acosh

nearby int

atanh

ceil

sinh

floor

cosh

round

tanh

rint(262144.25)↑ = 262144

Exact

1 ulp errors*

2-5 ulp errors

6-210 ulp errors

210-220 ulp errors

>220 ulp errors

Errors in exceptional cases

Errors for denormals

Completely buggy Unsupported

logb(2−1074) = −1022expm1(2.2250738585072e−308) = 5.421010862427522e−20

exp(−6.453852113757105e−02) = 2.255531908873594e+15

sinh(29.22104351584205) = −1.139998423128585e+12

cosh(627.9957549410666) = −1.453242606709252e+272sin(33.63133354799544) = 7.99995094799809616e+22

sin(− 1.793463141525662e−76) = 9.801714032956058e−2

acos(−1.0) = −3.141592653589794

cos(917.2279304172412) = −13.44757421002838

erf(3.296656889776298) = 8.035526204864467e+8

erfc(−5.179813474865007) = −3.419501182737284e+287

to nearestto –∞

to +∞to 0exp(553.8042397037792) = −1.710893968937284e+239

Implementations with Same Results


ceil floor round trunc rint nearby int fabs logb sqrt cbrt exp exp2 expm1 log log10 log2 log1p

x86

ia64

x86_64

s390

ppc64

ppc32

sparc

VC6

VC8

sinh cosh tanh asinh acosh atanh sin cos tan asin acos atan erf erfc tgamma lgamma j0 j1 y0 y1

x86

ia64

x86_64

s390

ppc64

ppc32

sparc

VC6

VC8

Unsupported

Conclusion

• Formalization can uncover numerous issues in mature industrial standards like POSIX (and more in implementations)

• But it may be not only ineffective but even impossible


Thank you!Questions?

[email protected]/~kuliamin

Institute for System Programming, Software Engineering Departmentwww.unitesk.comwww.linuxtesting.org

April 21, 2023 28/28Practice of Standards Formalization

the practice of standards formalization

Documents

test suites2002

posix conformance test

math conformance testing

hardware testing support

combinational test generation

concurrency testing

testing general scheme

distributions of linux