PDA

View Full Version : Float and Double ranges


Neuro69
05-28-2008, 12:22 PM
I'm trying to write a script that converts UTM coordinates to geographic lat/long coordinates. The formula contains sections where small numbers (stored as double) are raised to the power of 5 or 6. These end up getting truncated to zero, even though Windows Calculater show results in the range ~ 2e-10. Just testing input directly in the Listener, it appears that anything smaller than 1d-6 is truncated to zero. I've tried formattedPrint as well, to see if it was only the display, but it makes no difference. Reading about double precision floats, I'm a bit confused about the different notations used, but if I understand correctly, double values should have a lower range limit around 2.225d-308? Am I missing something obvious, or are there issues with the maxscript number implementation?

erilaz
05-29-2008, 01:40 AM
Can you show an example of a number that is getting truncated in code?

Neuro69
05-29-2008, 05:47 AM
Well, in my code I have a variable "D" that contains the following value:

0.0243115d0

If you then do D*D*D*D*D*D, the result is 0.0d0

Neuro69
05-30-2008, 10:48 AM
Bump. Anyone? How could I circumvent the issue? Thinking about scaling all values by 1000 or so, but the formulas are kind of complex, so I'm a bit sceptical.

CGTalk Moderation
05-30-2008, 10:48 AM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.