Numerical Comparison

[Bobo]

Why does this return false?

(1.1 - 1.0) == 0.1

The wonderful world of single precision floating point calculations.
You can read on Wikipedia about how 32bit computers represent floating point numbers internally to get an idea why 1.1-1.0 is not really precisely 0.1.

Here is an example:

a = 0.1
0.1
b = 1.1-1.0 --this causes an imprecision, the value in b is not perfectly 0.1
0.1
a==b
false
a-b
0.0
formattedPrint a --formatted print reveals the two values are not exactly the same
"1e+008"
formattedPrint b
"1.0000002e+008"

a = 0.1 * 10^9 --if you multiply by extremely large numbers, it becomes visible too
1e+008
b = (1.1 - 1.0)*10^9
1e+008
a==b
false
a-b
-24.0

close_enough a b 10 --this method should be used in such cases
true

a as string == b as string --or you can compare their printed representation
true


c = 0.1 --when both specified explicitly, no problem
0.1
d = 0.1
0.1
c==d
true

[JHaywood]

Thanks for the explanation. That makes sense, and I figured it was something going on internally like that.

"close_enough" seems like the best way to check for this, but I've never really understood how it works. So I'll probably use the string comparison method.

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.