PDA

View Full Version : handling large amount of data with mel


nier
07-02-2007, 11:56 AM
Hello!

I was wondering how do you guys manage to handle lange amount of data with MEL... Imagine you have to store vertices data of several shapes on stage, as well as their names, volumes, surface area, etc... this could be easily done by using relational databases, or it could work with nested arrays... as far as I know, there are no such things in MEL (the closest you get is matrices, which are a downer for not being dynamically resizable).

You can always resort to writing data to external files, but how slow/fast that approach is?

When I first faced this problem, I resorted to Python and I am quite happy with it.

Any insights?

Daniel

Robken
07-02-2007, 02:22 PM
MEL is usually not your best choice for handling large amounts of data. it is still an interpreted language and therefore runs a lot slower than "real" programming languages. For doing stuff like that, you should dive into the C++ API

trancor
07-02-2007, 03:23 PM
Yeah, mel does suck for speed, but that is what you get when you sacrifice baseline coding for commands.


I make arrays, multiple arrays, one for each attribute I want to keep track of, but I make sure every object's attributes are set up at the same number element in the array.

$objects[13]
$position[13]
$scale[13]
$visibility[13]

all to effect object 13 in my overall objects array

nier
07-02-2007, 10:25 PM
Python's nested arrays are much easier to do that... imagine having to have an array for x values, another for y values, and another for z values for each vertex... in python you just have to make a nested list and voila....

trancor
07-02-2007, 11:12 PM
You can do vector arrays -

vector $vectorArray[];
$vectorArray[0]=<<3,5,2>>;
$vectorArray[1]=<<5,7,3>>;

but to read a x y z

vector $singleElement=$vectorArray[0];

print ($singleElement.x);

This will pring 3, the x value of the first element in $vectorArray.

AmyScott-Murray
07-03-2007, 11:26 AM
I've made it work using string arrays - for each object make a string with all the values you want seperated by spaces, and when you want to query them do a stringToStringArray on the array index for the object you want. It works fine (for lots of data... ) and lets you store more than 3 values for each object, but it is really, really slow.

I think I might try rewriting my script using Trancor's method (the first one he mentioned), although it'll get interesting where some objects have different amounts of data to store than others, and they all need to be cycled through in order...

trancor
07-03-2007, 02:18 PM
well if they have different ammounts information make an array with like null array elements. make every element 0 and the ones with a value will be like 30.534 or what ever and just do a check -

if ($surfArea[4]!=0){
print $surfArea[4];
}



sorry for the bad example, but it will sufice.

Robert Bateman
07-03-2007, 03:13 PM
I was wondering how do you guys manage to handle lange amount of data with MEL...

easy. Don't do it.

Imagine you have to store vertices data of several shapes on stage, as well as their names, volumes, surface area, etc...

use the DG!! It can quite efficiently store point data for you with their names, volumes, surface area's etc. That's actually what it's there for (this sounds like blendshapes to me). Turn the visibility off or make things intermediate objects if you don't want to render them.

this could be easily done by using relational databases

or, the DG which is what it is designed to do (it is a relational DB of sorts, but with a few additional features for geometry computation such as lazy evaluation).

or it could work with nested arrays... as far as I know, there are no such things in MEL (the closest you get is matrices, which are a downer for not being dynamically resizable).

However, if you consider that the DG is a very large DB for you to use which mel allows you to modify fully, then you do have these things available in mel. It's just that the language features in mel are not designed to do this. The DG however is. Use the DG to store data andmel to manipulae that data.

You can always resort to writing data to external files, but how slow/fast that approach is?

It's a very bad approach. Use the DG or a PDC hack in the worst case scenario.

AmyScott-Murray
07-03-2007, 08:33 PM
Nier - sorry to be hijacking your thread, this is all stuff that's exactly relevant to what I'm trying to write atm.

Trancor - I think I see how I would apply your example to what I'm doing. At the moment I've got about 12 different places in the script which either read or write to a string array (looped across between 10 - 200 times depending how large the scene is), if I replaced all these with arrays in the way you used them, do you think the script would run noticeably faster? At the moment it takes a good 3 minutes or so even with much less starting data than it would ideally need to use.

Robert - I take it you're referring to the dependency graph? Is it possible to use it in the way you're describing only knowing mel, or is it time I bit the bullet and learned some c++? If you have any pointers to books or anything that describe using the DG like this, I'd be very grateful.

nier
07-03-2007, 08:49 PM
Hey Amy, go ahead, I raised this topic exactly to generate new discussion, as I had already solved my issue by using Python...

The idea with the DG seems quite interesting though, I'll take a look at that....

trancor
07-03-2007, 08:49 PM
amy, best way to get a script running faster is to have things figure out once and then use them in the future. I don't know the exact nature of your script, but if there is a way that you can run a script to figure out the arrays without needing to rewrite things, it would be faster, sure.

You can also look up command times online, I'm sure google will have something out there for it. For example, Switch commands run faster then For commands if used in the same manor. Sure you save a few milli seconds but if it takes 3 minutes for the script to run, that time mounts up.

AmyScott-Murray
07-03-2007, 09:47 PM
Thanks Nier : )

Trancor, I'll look up the command times and see if I can shave a bit off the time that way. As for rewriting things... the script is a bit longwinded, but I don't think it ever figures out the same data twice. What I'm trying to do is let the user convert a bunch of intersecting nurbs curves into a poly object that follows the same topology. If you're interested to see, here's how I went about it. I expect there's already a script out there somewhere which does this, but I'd pretty much gone ahead and scripted it anyway because I need the practice (very beginner at mel here). But I'd be really interested to see if anyone has better ideas about the approach I took, as well as the actual way it was scripted.

Check over a set of nurbs curves drawn on the grid, or live object. For each curve, store the parameter values of where it intersects another.

For each intersection, work out the xyz coordinates from the original curve name and the parameter value.

For each curve, define poly edges as being pairs of intersections that are 'next to' each other on the curve.

Make and store 'edge pairs' as being two edges that share one intersection.

Find and store the intersection numbers of 3-sided faces by comparing edge pairs with single edges, to find matching intersections.

Find and store the int. numbers of 4 sided faces by matching edge pairs.

Find and store for 5 sided faces again.

Draw each polygon using createFacet and the coordinates previously stored for each intersection.

(and I'll add a bit at the end to combine all the poly objects and merge the vertices.)


Is this really inefficient? Or is it just inevitable that something this convoluted will take a long time to execute?

Thank you very much for advice and opinions so far. : )

trancor
07-03-2007, 10:51 PM
well, again not knowing exctly how it's writen, would finding the 4 sided and n-sided polys be needed? Can't just tiangulate and go from there with just 3 sided?


I might have skipped or misunderstood something for how the script works.

Which probably wouldn't help much anyway, what I'm thinking is, does your script query each edge every time it checks for 3 sided then again for 4 sided and again for n-sided?

Robert Bateman
07-04-2007, 01:23 PM
Any data entry in mel will be a pain (especially creating new polys). It's probably easier to just write out an obj file from mel and use the obj importer to generate that data (though still a bit un-pleasant). Generally any thing requiring to create poly data should probably be done in the API.

For the script method above, make sure you turn off the undo queue!!! If you don't, it'll get crippled fairly quickly. You might also want to disable construction history when you are doing that, since it'll generate an awful lot of nodes. and don't go printing stuff into the script editor for debugging purposes - write a debug log instead.

trancor
07-04-2007, 04:25 PM
OH GOD YEAH, printing stuff in the script editor is the slowest bloody thing in the world. NEVER DO IT IN A FOR LOOP. I accidently left a print command in a script when I put it into a for loops once, printed out 140,000 values and it took my computer like an hour to proccess, the reason I didn't crash out maya, because I didn't save my script....


If you are going to print anything, always use limits on the printing like -

if (($x % 10)==0){
print $value;
}

This will print the value every time $x is evenly devisible by 10. If anyone doesn't know % is the remainder of. $x/10 style

AmyScott-Murray
07-04-2007, 09:13 PM
No, it doesn't check again each time - it kind of branches out.

Thanks very much for the advice everyone, I'll be following that up as far as I can. API looks like it's probably the way forward, it's time I started learning to do it.

Robert Bateman
07-05-2007, 09:53 AM
.... the reason I didn't crash out maya, because I didn't save my script....

print screen
scan via OCR software :p

trancor
07-06-2007, 05:47 PM
naw, I got the script up to like 100 lines .. wouldn't be able to screen print because it was off the bottom of the script editor... yeah still didn't save, I don't know why I do that, mainly with mel I don't bother saving it, everything else I'll hit ctrl s and be real fast about it, but mel, aw well.

Not to mention most of my scripts are like one time uses and I never bother saving them to begin with. .... stupid of me, I know

CGTalk Moderation
07-06-2007, 05:47 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.