Floating-PointFloating-Pointandand

High-Level Languages High-Level Languages

Programming LanguagesProgramming Languages

Fall 2003Fall 2003

Floating-Point, the BasicsFloating-Point, the Basics

Floating-point numbers are Floating-point numbers are approximations of real numbers, but approximations of real numbers, but they are not real numbers.they are not real numbers.

Typical format in a machine isTypical format in a machine is sign exponent mantissa sign exponent mantissa

Exponent determines range availableExponent determines range availableMantissa determines precisionMantissa determines precisionBase is usually 2 (rarely 16, never 10)Base is usually 2 (rarely 16, never 10)

The Notion of PrecisionThe Notion of Precision

Precision is relative. Large numbers have Precision is relative. Large numbers have less absolute precision than small less absolute precision than small numbersnumbers

For example if we have a 24 bit mantissa, For example if we have a 24 bit mantissa, then relative precision is 1 in 2**24then relative precision is 1 in 2**24

For 1.0, this is an absolute precision ofFor 1.0, this is an absolute precision of1.0*2**(-24).1.0*2**(-24).

For 100, this is an absolute precision ofFor 100, this is an absolute precision of100*2**(-24).100*2**(-24).

Representing NumbersRepresenting Numbers

Some numbers can typically be Some numbers can typically be represented exactly, e,g. 1.0, 2**(-represented exactly, e,g. 1.0, 2**(-13),13),2**(+20) [assume 24 bit mantissa].2**(+20) [assume 24 bit mantissa].

But other numbers are represented But other numbers are represented only approximately or not at allonly approximately or not at all

Problems in RepresentationProblems in Representation

2**(-9999999)2**(-9999999)Too small, underflows to 0.0Too small, underflows to 0.0

2**(+9999999)2**(+9999999)Too large, error or infinityToo large, error or infinity

0.10.1Cannot be represented exactly inCannot be represented exactly inbinary (repeating fracion in binary)binary (repeating fracion in binary)

145678325785.25145678325785.25Representable in binary, but 24-bit Representable in binary, but 24-bit

mantissamantissatoo small to represent exactlytoo small to represent exactly

Floating-Point OperationsFloating-Point Operations

Result may be representable exactlyResult may be representable exactlyr = 81.0;r = 81.0;s = 3.0;s = 3.0;x = r / s;x = r / s;

Machines typically have a floating-Machines typically have a floating-point division instruction point division instruction

But it may not give the exact result But it may not give the exact result

Floating-Point OperationsFloating-Point Operations

Result may not be representable Result may not be representable exactlyexactly

r = 1.0;r = 1.0;s = 10.0;s = 10.0;t = r / s;t = r / s;

Result cannot be precisely corrrectResult cannot be precisely corrrectWill it be rounded to nearest bit, or Will it be rounded to nearest bit, or

perhaps truncated towards zero, or perhaps truncated towards zero, or perhaps even more inaccurate, all perhaps even more inaccurate, all are possible.are possible.

Unexpected ResultsUnexpected Results

Let’s look at this codeLet’s look at this codea = 1.0;a = 1.0;b = 10.0;b = 10.0;c = a / b;c = a / b;if (c == 0.1) printf (“hello1”);if (c == 0.1) printf (“hello1”);if (c == 1.0/10.0) printf if (c == 1.0/10.0) printf

(“goodbye”);(“goodbye”);if (c == a/b) printf (“what the %$!”);if (c == a/b) printf (“what the %$!”);

We may get nothing printed!We may get nothing printed!

Why was Nothing Printed?Why was Nothing Printed?

if (c == 0.1) …if (c == 0.1) … In this case, we have stored the result In this case, we have stored the result

of the run-time computation of 0.1, but of the run-time computation of 0.1, but it’s not quite precise, in c.it’s not quite precise, in c.

The other operand has been converted The other operand has been converted to a constant by the compiler.to a constant by the compiler.

Both are good approximations of 0.1Both are good approximations of 0.1But neither are accurateBut neither are accurateAnd perhaps they are a little bit And perhaps they are a little bit

differentdifferent

Why Was Nothing Printed?Why Was Nothing Printed?

if (c == 1.0 / 10.0) …if (c == 1.0 / 10.0) …The compiler may compute 1.0/10.0 The compiler may compute 1.0/10.0

at compile time and treat it as at compile time and treat it as though it had seen 0.1, and get a though it had seen 0.1, and get a different resultdifferent result

Really ends up being the same as Really ends up being the same as last caselast case

Why Was Nothing Printed?Why Was Nothing Printed?

if (c == a/b)if (c == a/b)Now surely we should get the same Now surely we should get the same

computation.computation.Maybe not, compiler may be clever Maybe not, compiler may be clever

enough to know that a/b is 0.1 in one enough to know that a/b is 0.1 in one case and not in the other.case and not in the other.

Now Let’s Get Something Now Let’s Get Something Printed!Printed!

Read in value of a at run timeRead in value of a at run timeRead in value of b at run timeRead in value of b at run timeCompiler knows nothingCompiler knows nothingNow we will get some output or else!Now we will get some output or else!

c = a / b;c = a / b;if (c == a/b) printf (“This will if (c == a/b) printf (“This will

print!”);print!”);

Still Nothing Printed!!!Still Nothing Printed!!!

How can that beHow can that beFirst a bit of backgroundFirst a bit of backgroundTypically we have two or more Typically we have two or more

different precisions of floating-point different precisions of floating-point values, with different length mantissasvalues, with different length mantissas

In registers we use only the higher In registers we use only the higher precision form, expanding on a load, precision form, expanding on a load, rounding on a store.rounding on a store.

What Happened?What Happened?

c = a / b;c = a / b;if (c == a/b) printf (“This will print!”);if (c == a/b) printf (“This will print!”);

First compute a/b in high precisionFirst compute a/b in high precisionNow round to fit in low precision c, Now round to fit in low precision c,

loosing significant bitsloosing significant bitsCompute a/b in high precision into a Compute a/b in high precision into a

register, load c into a register, register, load c into a register, expandingexpanding

Comparison does not say equalComparison does not say equal

Surprises in PrecisionSurprises in Precision

Let’s compute x**4Let’s compute x**4Two methods:Two methods:

Result = x*x*x*x;Result = x*x*x*x;Result = (x*x)**2Result = (x*x)**2

Second has only two multiplications, Second has only two multiplications, instead of 3, must be more accurate.instead of 3, must be more accurate.

Nope, first is more accurate!Nope, first is more accurate!

Subtleties of RoundingSubtleties of Rounding

Suppose we insist on floating-point Suppose we insist on floating-point operations being properly rounded.operations being properly rounded.

What does properly rounded mean for What does properly rounded mean for 0.50.5

Typical rule, round up always if half wayTypical rule, round up always if half way Introduces BiasIntroduces BiasSome computations sensitive to this biasSome computations sensitive to this biasComputation of orbit of pluto Computation of orbit of pluto

significantly off because of this problemsignificantly off because of this problem

Moral of this StoryMoral of this Story

Floating-point is full of surprisesFloating-point is full of surprises If you base your expectations on real If you base your expectations on real

arithmetic, you will be surprisedarithmetic, you will be surprisedOn any given machine, floating-point On any given machine, floating-point

operations are well definedoperations are well definedBut may be more or less peculiarBut may be more or less peculiarBut the semantics will differ from But the semantics will differ from

machine to machinemachine to machine

What to do in High Level What to do in High Level LanguagesLanguages

We can punt. We just say that floating-We can punt. We just say that floating-point numbers are some approximation of point numbers are some approximation of real numbers, and that the results of real numbers, and that the results of floating-point operations are some floating-point operations are some approximation of the real results.approximation of the real results.

Nice and simple from a language definition Nice and simple from a language definition point of viewpoint of view

Fortran and C historically did thisFortran and C historically did this Not so simple for a poor application Not so simple for a poor application

programmerprogrammer

Doing a Bit BetterDoing a Bit Better

Parametrize the machine model of Parametrize the machine model of floating-point. What exponent range floating-point. What exponent range does it have, what precision of the does it have, what precision of the mantissa.mantissa.

Define fpt model in terms of these Define fpt model in terms of these parameters.parameters.

Insist on results being accurate where Insist on results being accurate where possible, or one of two end points it not.possible, or one of two end points it not.

This is the approach of Ada This is the approach of Ada

Doing Quite a Bit BetterDoing Quite a Bit Better

What if all machines had exactly the What if all machines had exactly the same floating-point model?same floating-point model?

IEEE floating-point heads in that IEEE floating-point heads in that directiondirection

Precisely defines two floating-point Precisely defines two floating-point formats (32-bit and 64-bit) and formats (32-bit and 64-bit) and precisely defines operations on them.precisely defines operations on them.

More on IEEEMore on IEEE

We could define our language to require We could define our language to require IEEE semantics for floating-point.IEEE semantics for floating-point.

But what if the machine does not But what if the machine does not efficiently implement IEEEefficiently implement IEEE

For example, x86 implements the two For example, x86 implements the two formats, but all registers have an 80-bit formats, but all registers have an 80-bit format, so you get extra precisionformat, so you get extra precision

Which sounds good, but is as we have Which sounds good, but is as we have seem a possible reason for suprising seem a possible reason for suprising behavior.behavior.

IEEE and High Level IEEE and High Level LanguagesLanguages

Java and Python both expect/require Java and Python both expect/require IEEE semantics for arithmetic.IEEE semantics for arithmetic.

Java wants high efficiency, which Java wants high efficiency, which causes a clash if the machine does causes a clash if the machine does not support IEEE in the “right” way. not support IEEE in the “right” way.

Java is potentially inefficient on x86 Java is potentially inefficient on x86 machines. Solution: cheat machines. Solution: cheat

Python requires IEEE too, but does Python requires IEEE too, but does not care so much about efficiency.not care so much about efficiency.