Unofficial archive. This is a read-only, community-run archive of the Apple Support Community. It is not affiliated with or endorsed by Apple Inc.

Roundoff error in iPhone Calculator app when using x^2 function

I was helping my granddaughter with calculus homework. She evaluated 40*t - 16*t^2 at t=2.5 using the calculator on her iPhone (despite me asking her to use fractions for simplicity and accuracy):

40*2.5 - 16*2.5^2

To my surprise, there was a roundoff error as a result: 2e-32 instead of zero.

I realize that subtraction of nearly-equal numbers is the best way to get round-off errors, but I'm surprised that the answer was not 0.


Using the following on the same calculator, gives the proper answer:

40*2.5 - 16*2.5*2.5


Is this a problem with the x^2 function being used by Apple being different precision than the rest of the numerics?


Whatever the answer to the above, I don't think it's normal in programming languages, since the usual knock against x^2 is that it is slower than x*x, not that it's somehow less precise.


FWIW, all of the following yield 0 as the result: the calculator app on macOS, whether in reverse polish mode or not, the freeware calculator app on my iPhone (cmpxRPN), Stata (a statistics package) returns 0 whether using quad precision or floating-point (24-bit) precision.


Thanks for any insight.

0 replies

There are no replies.