PDA

View Full Version : Slow custom file read.


StephG
05-02-2006, 06:31 PM
The studio I'm in uses a custom file format for animation. It has a keyframe mode and mocap mode. Keyframe stores object, attribute, all the tangent data and infinity as well as keyframe and values. Mocap is just the object, attribute, frame number and value.

The In-house system is used because it's object name specific and allows us to remap characters and animation developed at different times with different naming conventions over a very lengthy production time, and this is one of those things that helps rectify past mistakes.

Keyframe works fairly efficiently, but in mocap scenes where there's 2000 frames and keys on everything on every frame, the load time is horific. Like 45 minutes. Also, it balloons the memory usage from say 600mb to 1.3 gigs. The machine has 2gb, and it doesn't appear to be going into swap.

I've tried buffering the entire file in an array, like this (I've used both fread and fgetline):
$fileId = `fopen $animFileName "r"`;
string $nextLine = `fread $fileId`;
$nextLine = `substitute "\n" $nextLine ""`;
int $fileArrayCount = 0;
while (size($nextLine) > 0)
{
$fileArray[$fileArrayCount] = $nextLine;
$nextLine = `fread $fileId`;
$nextLine = `substitute "\n" $nextLine ""`;
print ("FILE ARRAY " + $fileArrayCount + " " + $fileArray[$fileArrayCount] + "\n");
$fileArrayCount++;
}
fclose $fileId;


The printing doesn't occur until after the file is completely read, evidently.

fread seems to use less memory than fgetline. A lot less. Removing $nextLine = `substitute "\n" $nextLine ""`; also seems to speed things up a little.

Is there some faster way to read a file into memory and draw data out of it? Also it doesn't give the memory back when it's done, even after clearing the array. Changing the file format significantly isn't really an option.

Any suggestions or pointers to tutorials would be greatly appreciated. Thanks.

Buexe
05-02-2006, 07:18 PM
I would say that you are printing while you are reading the file, since the print command is within the while loop. This costs time, even if only little, it sums up. From my experience with custom animation textfile import/export I would say that the reading and writing actually only takes little time. What makes it really slow are the tons of setKeyframe commands. Are you really sure that the reading is so slow? Maybe the substitute command makes it slow?
But this task sounds more like a job for the API to me if you want it "really" fast.
my 2 cents...

StephG
05-02-2006, 08:46 PM
Yeah pulling the substitute command out speeds it up a little. Not that much though.

I've done the read with the print command in and out. It waits until the file is completely read to execute the print command, oddly enough.

I also don't run it with the print command in there. That was just a test to see if it read the data a line or character at a time, and would then allow execution of it on read or not. It doesn't execute things until it's done reading.

There's some pretty odd stuff about the whole file read process. The entire global process is essentially run out of site, repeated commands and all, until it's good and ready to start outputting. I'm trying to figure out a way to get everything to read into the array, and then separately run everything from the array in another global process. So far, I'm still trying to get that to work.

I don't understand why, if it only takes a few seconds to read a 10mb scene file, it can't load a 5mb text file somehow just as fast, and then let you do whatever you want with it in memory. To essentially treat it as one big string. There has to be a way to do that, since Maya must be doing that with its own scene files.

Buexe
05-02-2006, 09:14 PM
Neither do I understand why Maya/Mel is slow when it has to execute a lot of stuff on the "read" side. I mean when you save a scene in ascii format it is MEL form start to end. And even complex Maya scenes are loaded within a few seconds. In my experience it really has something to do with the MEL interpreter, because if you do the same operations you do in MEL with the API there is a tremendous performance boost like by a factor of 100 or so, depending on the operations. But if MEL is still the only option, maybe a batch process of those animation files will help, so that nobody has to sit there for 45 minutes and wait for the next file to convert?
Sorry, that I can not be of more help.
good luck though
buexe
PS: I like the parent option on your rig demo(head/hip/world). Did you use parentConstraints? Can you also dynamically constraint them to other body parts?

StephG
05-02-2006, 09:34 PM
I'm using parent constraints. I've experimented with using parent constraints for the entire hierarchy so the whole thing could be rearranged. Still untested in production though. And it only works so long as the character stays only in Maya.

I tried dynamically rearranging the hierarchy with parent constraints, but very very bad things would happen when I scrubbed the timeline (using scriptJobs). Total "transporter accident".

Looking at the saved animation file, one thing I notice is that everything gets an 8 digit floating point number for a value. This is thoroughly unnecessary. Is there any way to decrease floating point precision to like 3 decimal places? That would save a huge amount on the file size.

You may well be right about using the API. Do you know any good learning references for the Maya API? There are plenty for MEL, but maybe this sort of application simply goes far beyond what MEL is good for, and it's time to learn the API.

Buexe
05-02-2006, 10:31 PM
Well, when it somes to learning API, this is pretty much it I guess:
www.davidgould.com
www.ewertb.com
www.robthebloke.org ( he has examples of importing and exporting stuff)

When it comes to roundoff floats I always use this one:

// Procedure Name: roundoff
//
// Author: Duncan Brinsmead
//
// Description:
// simple function to round float values to a particular decimal
//
// Usage: roundoff
//
// examples
//
// roundoff -1.119 2
// Result: -1.12
//
// roundoff 256.812 0
// Result: 257
//
// roundoff 128.1 -1
// Result: 130

global proc float roundoff( float $f, int $n )
{
// we divide if n < 0 to avoid numeric
// precision problems
if( $n > 0 )
{
float $roundScale = pow(10,$n);
if( $f > 0 )
return( ((float)(int)($f * $roundScale + 0.5)) /$roundScale );
else
return( ((float)(int)($f * $roundScale - 0.5)) /$roundScale );
}
else
{
float $roundScale = pow(10,-$n);
if( $f > 0 )
return( ((float)(int)($f/$roundScale + 0.5)) *$roundScale );
else
return( ((float)(int)($f/$roundScale - 0.5)) *$roundScale );
}


}

pretty slick, huh?
This parentConstraint experimental stuff sounds pretty wild, hopefully it will be production ready someday. Best of luck!
b

StephG
05-02-2006, 11:52 PM
Thanks for the round off proc. I'll try that and see if it helps.

Another pathology I've found is that my file is a mix of string and float data. So all the floats are actually strings too. Don't know if that's slowing things down or not.

Thanks for the API resource suggestions. I will probably order the Gould book when I get home. Still checking out the other stuff.

This is all pretty frustrating, since they want to build a pipeline around these separate animation files. Sort of my fault, since I came up with the format to start with. DOH!

I didn't anticipate the speed problems with the more dense mocap data. Unfortunately, what works on 100 frames of mocap data, doesn't work well in a big way with 2000 frames. Instead of being 20 times longer, it's more like 60+ times longer to load. Ouch!

goleafsgo
05-03-2006, 12:40 AM
Sorry, I haven't looked hard at your problem but are you reading a large file into a string array? So that each entry in the string array is one line from the file? If that's the case then it may be slowing down because of memory reallocation for the array. Try commenting out just the line where you assign it to a spot in the array and see how much faster that is.

StephG
05-03-2006, 03:08 AM
Sorry, I haven't looked hard at your problem but are you reading a large file into a string array? So that each entry in the string array is one line from the file? If that's the case then it may be slowing down because of memory reallocation for the array. Try commenting out just the line where you assign it to a spot in the array and see how much faster that is.

Now if I do that, how do I utilize the data?

The thing is, I'm reading the entire file into an array, and then executing all the setKeyframe commands from the array. That actually sped things up a little. Parsing the decimals to two places (in the file save) sped it up a lot more (but running the parser proc slowed the write speed of the file 50%, a fair tradeoff). Actually reading the file a line at a time and using that data through the keyframe creation process took the longest.

The two decimal precision file brought the load time down from 45 minutes to 14 minutes. However, if I delete the scene, load it again (without animation) and try to load the animation onto the character again, it takes 30 minutes. Gotta figure that out.

Is there some other way to read a line of the file, or maybe the whole file at once into memory? I'd really like to know what my options are. Or how about clearing the memory out once you're done (clearing the array doesn't seem to help)?

I'm really in unfamiliar territory here.

Buexe
05-03-2006, 07:20 AM
Maybe writing the whole animation as ready to use mel commands could speed it up, so that you don`t have to fiddle around with floats, strings and so on. The drawback of such a strategy could be that you can not easily have advanced import features like to import only on certain handles, attributes and/or frame times. BTW ther is a animImport/Export plug-in in the devkit examples. If you haven`t seen it you might want to check this out, it is pretty fast and hhas probably some useful functionality for you.

goleafsgo
05-03-2006, 10:20 AM
Now if I do that, how do I utilize the data?
No, I wasn't suggesting that that would fix your problem. :)
But...if commenting out that one line made a HUGE difference then I would say it was memory allocation that was slowing you down.

If you are in a loop assigning new values to the end of an array then internally Maya will have to allocate more memory and copy values around...which can be extremely slow. You can't preallocate memory from MEL but you can assign a value to a high index in the array and Maya fills in the gaps. So in some cases doing the loop in reverse, i.e. starting with the highest index so the memory gets allocated only once, will actually speed things up a lot.

I don't think that this will work in your case but I was just wondering.

Robert Bateman
05-03-2006, 02:34 PM
mel's not the best way to read data into maya, but what the hey.

first off, disable the undo queue during the file load. You probably don't need undo information to be generated (cos it'll use some memory up, and waste time. This is why the API is quicker for this stuff because mel is simply a journalling service for maya's undo queue).

// query undo state
$startState = `undoInfo -q -st`;

// disable the undo queue
undoInfo -state 0;

// reset to what the user had set originally
undoInfo -state $startState;


secondly, don't read the data into a string array, it'll kill your performance due to re-allocations. If you do need to do that (you really don't, honest) make sure you use clear on the array afterwards to free the memory. Just pump the keyframe data into the keyframe command directly rather than holding onto data.

thirdly, don't print the data into the script editor! for a start, printing to screen takes a LONG time, secondly every single line of the script editor history will be buffered (which eats a lot of ram). If you want a lame debug log, write the data to a file (make sure you call fflush on it periodically - maya seems to do some horrible internal buffering when writing files, and that kills performance to a grim extent unless you flush the output file every hundred lines or so).

It's also so worth trying to stop the update of the DG (I had to do this ages ago from the API, but i can't for the life of me remember how i did it). This should stop the viewport updating during the script, and therefore saves a great deal of time. I have a feeling i cached the nodeState attributes of all transform nodes, and then set it to something temporarily to prevent them being updated. In theory though, this may not be a problem if you are using the keyframe command (at least i don't think it would cause a problem).

If the performance is still slow, use the API and a custom MPxFileTranslator!

StephG
05-03-2006, 06:50 PM
Thanks Robert. You "The Bloke" ;)

Pre-empting "undo" and disabling the print commands brought the load time down to a much more reasonable 2 minutes and 12 seconds. Amazingly, it was the print command doing the most damage.

This also solved my other problem: running the script a second time would take twice as long.

I ordered Gould's book last night, so hopefully I'll be able to learn how to use the API for things like this and make it even more efficient. But 2 minutes is within my window of acceptability for a 2000 frame mocap file with keys on every joint on every frame.

Thanks again.

CGTalk Moderation
05-03-2006, 06:50 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.