View Full Version : Clamp precision

 fghajhe06 June 2012, 10:42 PMIs there a good way to clip off / remove values beyond a certain decimal place for attributes, vertex colors, or vertex positions? eg: a value .987, clip to the tenths place so .9 or clip to hundreths place so .056 becomes .05 is there a efficient way to do this to all components in whole objects?
nookie
06 June 2012, 07:18 AM
with python you can use the %.PUT_SOME_NUMBER padding, like

print ('%.2f' % 234.423424234344)
will give you 234.42 back

Nyro
06 June 2012, 08:49 AM
It can be done with a simple mathematical 'trick':

Multiply the floating point number by one hundred (or a thousand, ten thousand, etc.), remove the decimal part via a floor or ceil function, then divide again by one hundred (or a thousand, ten thousand, etc.).

Example (pseudocode):

float \$f = 3.132;
// Result: 3.132

floor(\$f * 100);
// Result: 313

\$f = \$f / 100;
// Result 3.13

By multiplying with a hundred, I kept the first two decimal values. Choose any multiple of ten depending on how much decimal precision you need after "clamping".

int \$precision = 3;
int \$multiplier = `pow 10 \$precision`;

float \$input = 3.13244

int \$tempMult = floor(\$input * \$multiplier);

float \$result = \$tempMult / \$multiplier;

// Result: 3.132

NaughtyNathan
06 June 2012, 09:02 AM
The python % feature is a string formatting function so is little or no use for modifying actual values. The OP was asking for a way to apply quantizing to mesh components.

I can't think of a particularly efficient existing way to do this, but I reckon it would be easy* enough to write a node to do this if you wanted a dynamic solution...

You'll just have to loop through each component and apply a rounding like Nyro suggests if you're using MEL (and/or use the trunc command), but of couse if you use python you have great rounding functionality built-in already without resorting to maths! ;)
round(3.14159, 3)
*easy if you have a basic working knowledge of the API :(

nookie
06 June 2012, 09:41 AM
yeah what I meant was to use that in a loop to set the values back :)
Is there any problem in using the % instead of the round function?
I use the % quite a lot (for populate the GUI with values for example) but I have ever wondered if it was the "correct" way :)

fghajhe
06 June 2012, 01:41 PM
Thanks for the replies. I am currently using a loop method which is pretty fast. Just wanted to make sure I hadn't overlooked a feature in Maya. I know in Maya you can change the precision value but its only for visual representation in the channelbox, not the actual value. I am not quite ready to learn the api yet but that would be a great skill to learn.

fghajhe
07 July 2012, 02:48 PM
I just got around to working on this again.. I am having a huge issue where the number I am querying and then multiplying is not correct. See example code. Much apreciated if anyone could say why this is happening.

polyColorPerVertex -a 0.645;
/// give a value of 0.645 to a vertex
float \$Vcolor[] = `polyColorPerVertex -q -a`;
//store the queried value in a float.
// Result: 0.645 //
float \$multiply = \$Vcolor[0] * 100;
// Result: 64.499998 //
////What the --- this should be 64.5 !!! ///

/// if I just do ///
float \$multiply = \$Vcolor[0] * 10;
//I get ///
// Result: 6.45 // but if I multiply this by 10 again the number is off

// If I do this//
float \$test = .645;
// Result: 0.645 //
float \$multiplytest = \$test * 100;
// Result: 64.5 // I get the correct number

NaughtyNathan
07 July 2012, 03:52 PM
welcome to computer mathematical precision! :D

fghajhe
07 July 2012, 04:10 PM
Gah! One thing I don't get is why the number is correct for float but not float[] ? I tried storing the float[0] back into a float but that doesnt work either. Why does my code at the bottom return the correct number but not the top?

polyColorPerVertex -a 0.645;
/// give a value of 0.645 to a vertex
float \$Vcolor[] = `polyColorPerVertex -q -a`;
//store the queried value in a float.
// Result: 0.645 //
float \$multiply = \$Vcolor[0];
float \$dddmultiply = \$multiply * 100;
// Result: 64.499998 //

This works though

// If I do this//
float \$test = .645;
// Result: 0.645 //
float \$multiplytest = \$test * 100;
// Result: 64.5 // I get the correct number

NaughtyNathan
07 July 2012, 04:34 PM
try this and see what you get:
float \$n = 0.64999999999;
basically, how Maya stores and prints vaues is different.

in your first case you are not storing an explicit value, you are setting an attribute to a value which means you are leaving Mayas internal comutations to store, retrieve and handle the value(s). When you query it Maya is simply printing the rounded value (as in the example above) even though the real value Maya has stored is probably nearer to 0.64999998 (due to the nature of float precision in computing). The reason Maya probably prints this as 0.645 when asked is probably just becasue it is aware of the precision errors.

Maya only hints at the real value of this stored float when you multiply it by bigger numbers.

in the latter case you are explicitely forcing a float to .645, and as 0.645 is not that precice a value it's probably storing it fine exactly as .6450000
(this could even be simply due to the fact that the MEL "float" variable type is actually a double, which is more precise)

fghajhe
07 July 2012, 04:43 PM
Thanks for your explanation. I didn't know the values that got printed out were not accurate.
This is also a way to show the internal value:

float \$Vcolor[] = `polyColorPerVertex -q -a \$vert`;
// Result: 0.88 //
string \$multiply = \$Vcolor[0];
// Result: 0.8799999952 //

So to solve this for me I think if I could somehow store the printed value as a float I would be all good.

float \$Vcolor[] = `polyColorPerVertex -q -a \$vert`;
// Result: 0.88 //
float \$SimpleVal[]= `print \$Vcolor[0]`
//just returns nothing eg:
Result: //

Is there a way to store the printed out value as a float?
Edit: Maya help says print is not queryable :(

Edit I found these awesome threads and they solve my problems. I will try to learn some python when I have time too: