PDA

View Full Version : Rendering Performance improvement info


osxrules
08-01-2008, 07:17 PM
Some of you will already know this but for those of you with multi-core machines and needing to squeeze the best performance out of it, this may help.

Install Qmaster from the Shake installer disc
In system prefs, choose Qmaster and configure the following:

http://img243.imageshack.us/img243/3777/picture6ky4.png

QuickCluster with services
check rendering with share and managed (Compressor is for Final Cut stuff or general encoding tasks so you can turn that off if you don't use it)
Hit options to set the number of instances - generally choose the number of cores in your machine.
In advanced, choose show Qmaster in menu bar
It doesn't seem to like odd characters in your computer name like if you had your computer name set to Jim's Mac Pro, it doesn't seem to like the apostrophe - you can rename this in your system prefs > sharing panel by just editing the text field at the top and hit return
Finally, hit start sharing in the Qmaster system prefs.

The menu bar icon looks like this on the left:

http://img156.imageshack.us/img156/2029/picture7jn6.png

Now to enable support in Shake, go into a .h file in your ~/nreal/include/startup - just put one there if you don't have a preferences file and inside it, type:

sys.useRenderQueue = "Qmaster";

save the file.

Relaunch Shake to read the startup preference and open a project or create a test project with a fileout node. You should see a new option at the bottom of the render fileout nodes dialog called render queue. Open this up and you will see a set of options. Check the useQueue option and click Batch Monitor to see your render queue - it doesn't open automatically.

http://img520.imageshack.us/img520/3306/picture8by8.png

(^ looks like CGTalk's background is the same color as Shake)

Now if you set your frame range and hit render, you should see a process added to your batch monitor and it will have split your render into multiple segments and they will start to render in parallel.

I wish I'd known this before I started rendering a 65,000 frame film and left it overnight to see it had stuck at 10,000. Also, don't ever render using a movie format when you do this, PNG image sequences are much more reliable. They are pretty hard to manage in the filesystem though and much more trouble putting them into Final Cut.

Anyway, Qmaster rendering benefits even dual core machines. I was able to reduce render times by 20% on a dual core machine. Quad-core and octo-core should see the biggest improvement.

It seems odd that there is an improvement because you often see Shake using all the CPUs anyway but I guess fewer CPUs working on more images in parallel is faster than all CPUs working on one frame at a time - you can experiment with the number of qmaster instances to see what gives you the best performance. You probably won't benefit much going above the number of cores you've got but sometimes even say 5 instances on a 4-core can improve things.

Oh yeah and if you do use Compressor, you can increase the instances too but the options can be greyed out. To do this, type in a terminal:

qmasterprefs -list

This will show you current services. Then you can use:

qmasterprefs -service "Compressor Processing" on instances 4 autorestart off unmanaged on

and it will set the number of instances that way.

stuckfootage
08-12-2008, 08:06 PM
Thanks for the guide, osxrules!

I set up a Quickcluster with managed services on my Macbook Pro,
and submitted a Shake script to the Qmaster application.
(using renderQueue in Shake looked OK, but produced no output).

So the file was rendered, with a faster render time,
but there's no audio. There's an audio track on the Quicktime movie,
but it's silent. I'm doing animation codec with audio.
Doing a regular render without Qmaster
included the audio no problem.

Does that sound familiar to anyone?
Thanks for your time.

Les

osxrules
08-13-2008, 10:19 PM
I don't normally render clips with audio so I didn't notice that but it seems that it doesn't include the audio doing a Qmaster render. What's odd is if you remove the .mov extension and render it, it finishes up with a random chunk of the movie including audio. This however does not generate the intermediate .iff sequence.

It must be that when it recognizes the output format as a movie type, it renders to .iff so that it can split the frames up better and then it merges the frames into a quicktime clip but it thinks that it can't split and join the audio segments so just ignores them when all it really needs to do is add the flattened track right at the end once the video is finished.

You can do this manually - in the audio panel, specify the same frame range as the movie, create a mixdown, open it in quicktime and do a copy paste, then save as self contained. It should really do that step by itself though.

Out of interest, what amount of speed improvement are you seeing on average?

stuckfootage
08-14-2008, 01:01 AM
So it looks like I'll have to add an extra step pasting the audio back in.
Oh well, it goes fast.

Out of interest, what amount of speed improvement are you seeing on average?

I just tried it here at work on a 2x 2.66 GHz Dual Core Xeon,
running MacOS 10.5.3, Quicktime 7.4.1, Qmaster & Batch Monitor 3.0.1,
maxThread = 4 and Qmaster instances = 4, rendering 100 frames
of 720p animation codec.

Compared to rendering without Qmaster,
dropping a script into Qmaster and submitting was 25% faster,
and using renderQueue from within Shake was only 10% faster.
Dropping two scripts into Qmaster and submitting both was 5% slower.

When I get home I'll do some more experiments on my MacBook Pro.

Les

stuckfootage
08-14-2008, 06:15 PM
Back home, Qmaster has stopped running on my PB.
I get the dreaded "qmasterd not running" error message
when I try to start services.

A lot of Googling and experimenting and I've learned that
this is a common problem with no particular solution.

Sigh....

osxrules
08-14-2008, 10:55 PM
Back home, Qmaster has stopped running on my PB.
I get the dreaded "qmasterd not running" error message
when I try to start services.

Quit system prefs, open a terminal and type:

sudo /Library/StartupItems/Qmaster/Qmaster start

Then try to start sharing.

Interesting benchmarks you got on the quad. I've read about command-line rendering in Shake being faster than through the interface so dropping the script straight into Qmaster must be similar. I never got command-line rendering to work properly - I couldn't seem to set motion blur parameters but I see that they can be set in the Qmaster app. I'll have to try dropping scripts onto it directly.

stuckfootage
08-15-2008, 10:48 AM
God, this Qmaster thing is messed up!! It took me quite a while to get
it working again. Your tip didn't quite do it, but we are up against
a big nasty bug here ... the forums are full of complaints and voodoo.

Here's what worked for me:
1. Reinstall Compressor and Qmaster from FCP install disk, using Pacifist.

2. modify /Library/Application Support/Apple Qmaster/qmasterconfig
so the computer knows where qmasterd is:
QMASTERPATH="/usr/sbin/qmasterd"

3. Reinstall AppleQmasterNode.mpkg from FCP install disk.

4. Set up the Quickcluster according to your instructions above.

The exciting news is, after hours of Googling forums,
rebooting, etc., the render time is about the same,
maybe 5-10% faster on my short clip. So I guess
one really does need more cores.

Thanks for giving the tips on getting started.
Next step: an army of MacMinis!

Les

Kai01W
08-15-2008, 11:43 AM
But I thought on macs everything is totally easy??? :-)

scnr. please press the "useful post" button

-k

osxrules
08-16-2008, 07:06 PM
The exciting news is, after hours of Googling forums,
rebooting, etc., the render time is about the same,
maybe 5-10% faster on my short clip. So I guess
one really does need more cores.

Yeah, it only really helps significantly if you have enough cores and if the render sequence is long. If you render a long sequence to a movie format, it'll take longer as it has to merge the .iff files into the movie file (uses twice the space too).

Thanks for giving the tips on getting started.
Next step: an army of MacMinis!

The only trouble with Mac Minis is:

4 x 1.83GHz Mini = $2396
1 x quad 2.8GHz Mac Pro = $2299

So that's 8 x 1.83GHz Core 2 Duo vs 4 x 2.8GHz Xeon. I'd go with the Mac Pro even though you'll have less overall Ram - processing will be about even. You save the hassle networking and powering so many machines.

Virginia Tech are building a new cluster with the 8-core version:

http://arstechnica.com/journals/apple.ars/2008/07/24/virginia-tech-building-supercomputer-out-of-324-mac-pros

But I thought on macs everything is totally easy??? :-)

Nah, just easier.

In this case, it's still easy, it's just easily broken too. ;)

Qmaster is old and Apple have two packages that do roughly the same thing. Xgrid is the other one and is used more often in distributed rendering. You can set it up to use in 10.4 client as a controller apparently:

http://www.macgeekery.com/gspot/2006-06/setting_up_an_xgrid_controller_in_tiger

That will probably be more troublesome trying to get it to work properly with Shake though.

I probably should have mentioned this before but I just remembered about it. There is a utility for Shake to do this same sort of thing:

http://www.highend3d.com/shake/downloads/tools/render_managers/Mcore--Multicore-Rendering-software-for-Shake-5203.html

I haven't tried it myself though so I don't know how good it is - it doesn't support quicktime fileouts. Might be easier to set up though. There's a video of it working here:

http://hassanvfx.blogspot.com/2008/04/mcore.html

I don't know if it will restart after timeouts and it looks like it will simply start multiple processes. Qmaster makes smaller chunks so that there's always CPU being used.

CGTalk Moderation
08-16-2008, 07:06 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.