What's going on with the precision calculation

Last week I posted a question on how to round in Matlab to match a simple calculator but now I think Matlab is getting the calculation all wrong. For example, when calculating:

p15 = 0.8683*1.15;

round(p15,5) = 0.99854

I believe the answer should be 0.99855 and here is why. If I take 10% of 0.8683 (.08683) and 5% of 0.8683 (or 1/2 of 0.08683) = 0.043415 and then sum p15 + 10% + 5% (0.8683 +.08683 + .043415), then the answer is 0.998545.

0.998545 rounds to 0.99855 at 5 decimal places.

Why is Matlab calculating something different than 0.998545???

Answer 1

[John D'Errico]

No. MATLAB is not getting it all wrong. The problem lies in your understanding of floating point arithmetic, and how numbers are stored.

p15 = 0.8683*1.15;

format long g
p15
p15 =
                0.998545

So, it LOOKS like the result is what you would expect. Is it? No. The issue is that floating point numbers are stored NOT in decimal form, but in binary. So, for example, 0.1 is NOT exactly representable in binary in a finite number of binary bits, just like you cannot exactly represent the fraction ⅓ in a finite number of decimal digits. As well, 0.998545 is also not exactly representable in binary.

So, the decimal representation of the number that is stored in binary for p15 is actually:

0.99854499999999990489385481851059012115001678466796875

This is the number as it is actually stored. Well, actually, it is stored in binary. But when you convert it into decimal form, you get that long mess of a decimal number.

So, how is the number 0.998545 stored as a sequence of binary bits? First, here is the repeating bit string that would represent 0.998545:

0.1111111110100000101001010010011010010101100101011111111011011010011001100001001010000011100100000100...

However, as stored in MATLAB, the binary bit used would look more like this:

0.1111111110100000101001010010011010010101100101100

As you can see, the two numbers become subtly different when you go past 50 binary bits or so.

Now, when you convert to decimal and round that result to 5 DECIMAL digits, you get 0.99854. The round goes down, because that is in fact how you should round the complete number shown above.

Welcome to the world of floating point arithmetic. As it turns out, no matter what base you store your numbers in, there will be problems like this that arise. (I believe that some calculators use a decimal form to store numbers, but there too you can contrive simple enough problems that may appear paradoxical if you do not understand how those numbers are stored and used.)

[Walter Roberson]

For example, to 5 decimal places, 1/3 = 0.33333 so 2 * (1/3) = 0.66666 so 3 * (1/3) = 0.99999, so to 5 decimal places 1 = 0.99999

So maybe the problem was 5 decimal places wasn't enough. Go to 20 decimal places perhaps? 1/3 = 0.33333333333333333333, so 2 * (1/3) = 0.66666666666666666666, so 3 * (1/3) = 0.99999999999999999999, so to 20 decimal places, 1 = 0.99999999999999999999 . Clearly that is still a problem, perhaps we should have used 50 decimal places? No, obviously that would just result in 0. followed by fifty 9's, and that still isn't equal to 1. Use any finite number of decimal places and you still get the result that two distinguishable numbers logically "equal" the same number. Fun fact: 0. followed by an infinite number of decimal 9's is mathematically equal to 1 . The problem is not "binary", the problem is not "decimal", the problem is "finite".