## Sunday, 1 January 2012

### My computer sucks at maths.

Computers are excellent calculators.. Not.

Now I'm pretty much used to dealing with floating point precision errors, but this one is just silly..

A fun example trying to calculate a ResMed pressure from a stored integer value, converted normally.

470.0 * 2.0 / 100.0 = 9.39999962
470.0 * 0.02 = 9.39999962
470.0 / 50.0 = 9.39999962

An even simpler example:
float var=9.4; // ie, the real answer, which even my foggy brain can work out..

the result that actually gets stored? 9.39999962!

Gah!! A standard floating point variable can not hold the number 9.4. It must be a dastardly evil unholy number.

Using 64bit double precision gives the answer for all of the above as 9.4000000000000004, also wrong.

Rounding to two decimal places is enough to get it to display right, and of course it won't really effect calculations (except the particular one I was working on that exposed it), but there is a heck of a lot to be said for using integer maths..

Did I mention I'm starting to hate computers.. ;)

Some more info as to why this happens for anyone who's interested:
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

#### 1 comment:

1. Using a binary system to approximate a continuous valued thing is indeed a PITA. In this situation, consider the simple if (x=y)... often does not work as you expect. I also once a flaw in the MSFT math libraries where if you calculated something once, you'd get one answer, then if you executed the same operation again, you'd get a different answer, but would stay the second answer again and again for subsequent calls. Since we do a lot of math, this is a challenge for us.