PDA

View Full Version : Setting up a rendering farm


TCLee
03-20-2003, 01:39 PM
Heya,

We're looking into setting up a render farm and was wondering if anyone can help me with some info, and also to point me to some places where I can find out more.

We're hoping to run Softimage 3D/XSI and 3DS Max. Starting out with possibly 3 CPUs with additions in the future.

Currently, I'm leaning towards setting up a rack with just motherboards/processors with a switcher for the monitor/keyboard/etc. But what about hard drives? Power supplies?

Also, my colleague seems to think that if we put a fast, high-end host machine with loads of RAM to delegate the work out to the other CPUs, we can get away with only about 256Mb RAM on each of the CPUs. I thought it would be the other way around as the CPUs would be doing the work.

Also, if we were to do it this way (1 host, 2 slaves) would the host be fine farming/administering files for Max, XSI and Soft3D simultaneously?

Also, if we were going with that set-up, which would be the bottleneck? Which would be the most important part of the chain?

Do I get a RAID or disk array and use that as a common drive which is accessed by the farm?

Sorry if all this sounds like really stupid questions but I really have no idea as most of the time, the 'render farms' that are used here are just normal pcs that are put aside just for rendering.

Hope you guys can help!!:p

Thanks!!

GregHess
03-20-2003, 03:58 PM
Currently, I'm leaning towards setting up a rack with just motherboards/processors with a switcher for the monitor/keyboard/etc. But what about hard drives? Power supplies?

Each machine has to have its own system. There are powersupplies which allow for multiple motherboards to be run off of them, as well as dual motherboard cases...but this isn't advised. Each machine will most likely be fully intact (mboard, cpu, cooling, hd, memory, onboard video, psu), in order for this to work. I think you might be able to boot the machines off the host, but I don't know if max's netrender can deal without having a local temp file on the machine thats actually rendering.

Also, my colleague seems to think that if we put a fast, high-end host machine with loads of RAM to delegate the work out to the other CPUs, we can get away with only about 256Mb RAM on each of the CPUs.

You colleagues are incorrect. Its the speed of the individual machines that are doing the rendering that determine the speed of each frame. Depending on the type of work your doing, a gig or two of ram could be required for each node. The same requirement for rendering as if you were using just a single workstation.

Also, if we were to do it this way (1 host, 2 slaves) would the host be fine farming/administering files for Max, XSI and Soft3D simultaneously?

I'm pretty sure you won't be able to have a machine render in max, xsi, and soft simultanously. Now distributing requests for one machine to render in max, and another to render in xsi, that would be fine. I wouldn't expect anymachine to be able to do a double app render with anytype of decent performance, unless it was a quad with very specific priorities and affinities setup.

Have you looked in DR (distributed rendering) for 3dsmax? Vray is one app that allows for DR, and allows multiple machines to work on the same frame simultanously.

Also, if we were going with that set-up, which would be the bottleneck? Which would be the most important part of the chain?

The bottleneck would most likely be the speed of the systems. You can set up a 100mbit network very cheaply, and even a 1000mbit network isn't too expensive. (Just the initial switch). At that speed, the hd's would actually begin effecting how much data was sent back and forth.

Do I get a RAID or disk array and use that as a common drive which is accessed by the farm?

Whats the budget for this? Have you considered having a professional company setup an array for you? Boxxtech or racksavers.com.

elvis
03-20-2003, 09:55 PM
greg's bang on right. each machine on the render farm will be rendering a frame/scene just like a single machine. you wouldn't put 256MB of ram in your main workstation, would you? that would just be silly.

our renderfarm here consists of dual athlon MP 2000+ chips in asus A7M266-D motherboards, each with 1GB of (non-ECC) ram. we felt the ECC ram wasn't worth the extra cost for render nodes, as if a single unit failed the rest of the farm would catch up, and a reboot would set it right again. (remembering also that prices in australia are roughly 30% more expensive then what you pay in the US). we use a 100mbit/s network for the farm, which is slow but does the job. (the whole idea was to reduce costs as much as possible). per node this setup is very cheap, and performs quite well even under stressful conditions (eg the day before a competition deadline :)).

we also have a render "server" (running the serversvc application for 3dsmax) which is a single processor PIII-866 with 512MB of ram. it really does nothing more than dish out render jobs, so the specs don't have to be that high.

as for monitoring all of them, each one runs Tight VNC (http://www.tightvnc.com) which means no monitors/keyboards/mice for any of them bar the initial setup.

TCLee
03-21-2003, 01:12 PM
Greg and Elvis, thanks for the replies.

You colleagues are incorrect. Its the speed of the individual machines that are doing the rendering that determine the speed of each frame. Depending on the type of work your doing, a gig or two of ram could be required for each node. The same requirement for rendering as if you were using just a single workstation.

greg's bang on right. each machine on the render farm will be rendering a frame/scene just like a single machine. you wouldn't put 256MB of ram in your main workstation, would you? that would just be silly.

Thanks for settling the argument on the specs on host/slaves. I couldn't really back my point fully about the slaves needing more RAM than the host as I am using Soft while the guy suggesting otherwise was using Max (not too sure if Max worked differently).

Whats the budget for this? Have you considered having a professional company setup an array for you? Boxxtech or racksavers.com.

I think I should explain the situation further. I'm located in Malaysia, so I can't get access to Boxxtech or Racksavers. My best bet would be Dell (I don't think they do proper renderfarm setups), IBM or HP etc (the big names). But I can't really do that because it would be tooo costly. We're mainly a post pro set-up with our primary focus being on our Flame, Inferno, Infinity and Avid suites. So unfortunately for me, 3D is taken as a support tool. So, I can't justify the cost associated with pro help in setting this thing up. And unless I can set this farm up, I can't get in bigger jobs to ask for a sizable budget. A Catch-22 on my part.

Sorry if that was a bit too much unrelated info. :shrug:

So, is the best (and safest) bet be to set up at least 2 dual Athlons with 768mb RAM, with individual 7,200RPM HDDs(power etc), a lower specced host (is 128mb RAM enough?) and link them all on a 100mbit network. And get a switcher so that only 1 keyboard/mouse/monitor is needed. Then either use VNC (I have to read up on it) or rely on Max's Backburner and Soft/XSI's distributed rendering capabilities.

Is that all too messy, underpowered (for starters tho)? Also, I only thought of the disk array because we have one lying around. Was hoping to modify it so that it could be used as a shared disk for 2 nodes or something.

Any further advice would be appreciated.

Thanks again!!

GregHess
03-21-2003, 01:23 PM
a lower specced host (is 128mb RAM enough?)

Since the machine will probably be running win2k or XP, I'd keep the min amount of ram to 256, with a 512 recommendation.

Realistically 256 megs of ram should be unbelievably inexpensive. I don't know how markup's are over there, but you can sometimes pick up 256 megs of DDR for 30-40 USD here.

As for raid, I'm not too sure if it would benefit you greatly. You could setup a Raid 1 for protection against possible HD failure, but then of course, you could use the money spent on raid on a DVD-/+RW drive and just ghost the system weekly to DVD.

TCLee
03-21-2003, 04:00 PM
Thanks again, Greg.

512mb it is then.

And I guess the raid's out then.

Thanks.

elvis
03-22-2003, 02:44 AM
malaysia can be VERY cheap for hardware if you shop in the right places. of course you don't want to get caught short with a dodgey dealer who can't offer after-sales support/replacement of bits that go bad. when buying in bulk always make sure you can replace parts that go bang.

i'd have to agree with greg on the ram requirements. don't skimp on this, as it will limit you when you start throwing big scenes at the farm (and that is what a farm is for, after all).

again, i wholly recommend dual athlon boxes with 1GB of ram. you don't need fast ram (PC2100 is plenty), and you can cut back on hard disk and video card cost. a cheap AGP/PCI card is all that's needed for a display (VNC requires a working display, so you can't go without video card if that's your choice). the hard disk will basically just host your OS and application installs, so a basic 7200RPM 20GB drive is more than ample. we use seagate barracude IV's in our farms.

as everything is networked, raid is pretty much a moot point. leave this for your file servers that hold your precious data. temporary farm units are no hassle if they blow up or crash. as greg mentioned a quick re-image will set you straight, and the frame or part-frame that dies will be handled by another machine.

just remember that the four most important things for rendering a scene is the CPU grunt to do teh maths, enough physical ram to hold your OS needs, as well as your entire scene and textures (plus whatever other maps/lights/etc are being generated), a motherboard that won't flake out on you and power to drive the lot. keep these four things high, and you can happily scrimp on the rest to save a lot of cash per node.

we build our nodes here for around AU$2600 a box (US$1500 or so). for the cost of a high-end dell workstation we can buy around 4-5 render nodes, which is great value for money.

TCLee
03-23-2003, 03:36 PM
Thanks Elvis,

We'll be looking around for pricing now. One of the guys threw in the idea of buying Shuttles but I'm a little skeptical about going that way, but we'll have to see.

Thanks again!

elvis
03-23-2003, 09:28 PM
shuttles are great for space saving purposes, and the newer nvidia based nforce2 boards will support athlonXP chips right up to the up-coming 400MHz FSB chips arriving 3rd quarter some time.

the only bummer is these are single-processor units, and you may find it more cost effective to use fewer dual-cpu units instead. make sure you do a very in-depth price to performance analysis before shelling out big bucks for multiple nodes.

oh, and don't let your tech guys suck you into the "shuttles look cooler" thing. yes they are very cool, but this is a business we're talking about. :)

TCLee
03-24-2003, 10:49 AM
Guess what?

One of the tech guys just told me that management was prepared to spend apprx US500 on each machine, with a maximum of 3 units.

This is just totally ridiculous . . . heh. Anyways, will be trying to push for at least 2.5x that (which isn't much to talk about either . . . )

Accountants and marketing . . . :annoyed:

GregHess
03-24-2003, 02:13 PM
For 500 USD....

They'd have to be inexpensive athlon units, in crappy beige cases. No way you can get rack capable units for that price.

TCLee
03-24-2003, 02:59 PM
I'm also looking into re-using some old Compaq desktop cases that are lying around. Save a little bit more to pump into RAM or something . . .

dvornik
03-24-2003, 09:18 PM
Originally posted by GregHess
I wouldn't expect anymachine to be able to do a double app render with anytype of decent performance, unless it was a quad with very specific priorities and affinities setup.

So you don't think it's realistic to setup a farm that would do it with dual-processor machines? Is there a way to set priorities and affinities on dual processors?

elvis
03-24-2003, 09:46 PM
Originally posted by TCLee
Guess what?

One of the tech guys just told me that management was prepared to spend apprx US500 on each machine, with a maximum of 3 units.

Accountants and marketing . . . :annoyed:
ha! your management there sounds just as cluey as ours here. it's a shame the people who run the place have got no idea.

and then next week you'll see your lead accountant spend $500 on lunch with some clients because "that's how real money is made". pffft...

GregHess
03-25-2003, 02:28 AM
Originally posted by dvornik
So you don't think it's realistic to setup a farm that would do it with dual-processor machines? Is there a way to set priorities and affinities on dual processors?

You can set affinity and priority into an executable file. So you'd probably be able to assign one processor to each 3d app...but then you'd still be dividing ram between the two renders...not really the most effective of situations, trying to render with two apps on the same machine. Makes alot of sacrifices. I guess if your really caught for money/space you could do it.

CGTalk Moderation
01-14-2006, 04:00 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.