Best Renderfarm for 10k budget

[JDex]

Does your 10k include Mental Ray licenses? If so that's not much of a farm.

Mental ray is capable in some respects of utilizing the GPU of a video card, but you really have to have the near-top of the line Quadro's, which would eat your renderfarm budget.

I would be looking at the 7104 series and maybe down the road you can get BOXX to upgrade them to dualcore Opterons. I don't think you're going to be able to much of a farm for under 10k unless they are custom builds.

[i601254]

If you want to consider all your options check out Tantus Computers. (http://www.tantuscomputers.com)
Give me a call and we'll answer any questions you have and give you some options to consider.

Steve
Tantus Computers
(877) 826-8871 ext. 611

[thesuit]

Does your 10k include Mental Ray licenses? If so that's not much of a farm.

Mental ray is capable in some respects of utilizing the GPU of a video card, but you really have to have the near-top of the line Quadro's, which would eat your renderfarm budget.

I would be looking at the 7104 series and maybe down the road you can get BOXX to upgrade them to dualcore Opterons. I don't think you're going to be able to much of a farm for under 10k unless they are custom builds.

The 10k does not include Mental Ray licenses... I will purchase this separately.
I'm actually looking for a small solution for the visualization department in my studio. Those renders are really climbing over them and they need a 6 - 8 processor machine to downlet some of that rendertime. Maybe just 3U but that would be enough for now.

[maninflash]

My suggestion would be to get three dual xeons, MentalRay takes advantage of xeon's Hyper thread and you could render on 12 threads instead of one. Also, mentalray would be faster on CPUs with more cache, so consider getting a procesor with 1MB L2 or L3 cache, You dont need to build complete systems for this, just get three dual racks, connect them together, you could then do a network batch render from a server.

also, consider getting a SCSI 15k hard drive for mentalray to write the final rendered images, since the drive has its own processor, it takes the load off of your CPU.

Mentalray needs RAM only when there are scenes with heavy GI or final Gather or Ray Tracing, if this is your case, consider getting 1GB per CPU, preferably two 512MB dimms to take advantage of motherboard's NUMA. If youre not going to render that much GI, 512MB per CPU would be enough.

The place I work at has a small renderfarm with 24 xeons on 12 rackmounts on SuperMicro motherboards and chasis, and it works without issues, on animation scenes when the farm is not busy the render comes out almost real time, for FX and GI, it takes more but is still very fast.

For motherboard, look at:
http://www.supermicro.com/products/motherboard/Xeon/E7500/P4DPi-G2.cfm

for chasis, look at:
http://www.supermicro.com/products/chassis/2U/822/SC822R-400RC.cfm



hope that helps :)

[Sieb]

Talk to ED (on these forums), hes a BOXX rep.

http://boxxtech.com/products/rendernodes.asp

[Ed Caracappa]

I'd go dual Opteron with 2GB of memory an 80GB hard drive and a micro CD. The Opteron 248s are at a great price right now and you can get three nodes for under $10K. Call me and I'll be happy to help you out.

Ed

[mlmiller1983]

My suggestion would be to get three dual xeons, MentalRay takes advantage of xeon's Hyper thread and you could render on 12 threads instead of one. Also, mentalray would be faster on CPUs with more cache, so consider getting a procesor with 1MB L2 or L3 cache, You dont need to build complete systems for this, just get three dual racks, connect them together, you could then do a network batch render from a server.

also, consider getting a SCSI 15k hard drive for mentalray to write the final rendered images, since the drive has its own processor, it takes the load off of your CPU.

Mentalray needs RAM only when there are scenes with heavy GI or final Gather or Ray Tracing, if this is your case, consider getting 1GB per CPU, preferably two 512MB dimms to take advantage of motherboard's NUMA. If youre not going to render that much GI, 512MB per CPU would be enough.

The place I work at has a small renderfarm with 24 xeons on 12 rackmounts on SuperMicro motherboards and chasis, and it works without issues, on animation scenes when the farm is not busy the render comes out almost real time, for FX and GI, it takes more but is still very fast.

For motherboard, look at:
http://www.supermicro.com/products/motherboard/Xeon/E7500/P4DPi-G2.cfm

for chasis, look at:
http://www.supermicro.com/products/chassis/2U/822/SC822R-400RC.cfm



hope that helps :)

I considered making my renderfarm(2-3 PCs) using the Xeon Noconas since they had HT. Just curious but what is the speed of your renderfarm Xeons. My main PC has Dual 3.2GHz Xeons and I was going to build 2-3 Dual 3.0GHz Xeons for a renderman.

[maninflash]

I considered making my renderfarm(2-3 PCs) using the Xeon Noconas since they had HT. Just curious but what is the speed of your renderfarm Xeons. My main PC has Dual 3.2GHz Xeons and I was going to build 2-3 Dual 3.0GHz Xeons for a renderman.

They used to be 2.8, after several complaints from the FX and lighting people, the boss decided to upgrade to 3.2 with 1MB L3 cache, and there's been no complaints ever since :) we use mentalray for the bulk of our work and the HT makes it really fast, I don't know about other renderers.

[Thakandar]

Totally insane suggestion:

20 Mac minis @ 499$ = approx 10 000 $ :bounce: :bounce: :bounce: :bounce: :bounce:

And before you ask, I have NO idea what I'm talking about :thumbsup:

[mlmiller1983]

They used to be 2.8, after several complaints from the FX and lighting people, the boss decided to upgrade to 3.2 with 1MB L3 cache, and there's been no complaints ever since :) we use mentalray for the bulk of our work and the HT makes it really fast, I don't know about other renderers.

If they make a mental ray render for Lightwave 3D. I have Maya 6.5 so I got the satillite render for up to 8 CPUs( as long as HT doesn't make mental ray count 4 CPUs per machine).

1MB L3 cache? I didn't know they made Xeon with L3 cache. The ones I have are 1MB L2 Cache and 2MB L2 cache.

[i601254]

My suggestion would be to get three dual xeons, MentalRay takes advantage of xeon's Hyper thread and you could render on 12 threads instead of one. Also, mentalray would be faster on CPUs with more cache, so consider getting a procesor with 1MB L2 or L3 cache, You dont need to build complete systems for this, just get three dual racks, connect them together, you could then do a network batch render from a server.

also, consider getting a SCSI 15k hard drive for mentalray to write the final rendered images, since the drive has its own processor, it takes the load off of your CPU.

Mentalray needs RAM only when there are scenes with heavy GI or final Gather or Ray Tracing, if this is your case, consider getting 1GB per CPU, preferably two 512MB dimms to take advantage of motherboard's NUMA. If youre not going to render that much GI, 512MB per CPU would be enough.

The place I work at has a small renderfarm with 24 xeons on 12 rackmounts on SuperMicro motherboards and chasis, and it works without issues, on animation scenes when the farm is not busy the render comes out almost real time, for FX and GI, it takes more but is still very fast.

For motherboard, look at:
http://www.supermicro.com/products/motherboard/Xeon/E7500/P4DPi-G2.cfm

for chasis, look at:
http://www.supermicro.com/products/chassis/2U/822/SC822R-400RC.cfm



hope that helps :)


Don't immediately assume that bigger is better. There's a lot more to rendering than raw numbers. For instance, Xeon's may have a large cache but Opteron's have lower latency and onboard memory controller. Additionally, while there may be more threads with HT, Opteron's allow data to travel both directions simultaneously. Plus Mental Ray is optimized for 64-bit multi-threaded paralellized processing :eek:. For the record, I think that Xeon's are awesome as well. I just don't think that you can generalize based on the numbers alone.

[mlmiller1983]

Don't immediately assume that bigger is better. There's a lot more to rendering than raw numbers. For instance, Xeon's may have a large cache but Opteron's have lower latency and onboard memory controller. Additionally, while there may be more threads with HT, Opteron's allow data to travel both directions simultaneously. Plus Mental Ray is optimized for 64-bit multi-threaded paralellized processing :eek:. For the record, I think that Xeon's are awesome as well. I just don't think that you can generalize based on the numbers alone.

One thing I don't like about most Xeon boards now is that they use DDR2(expensive and not as good as DDR) but thankyou asus for making the NCCH-DL. Only thing though it can only use 4GB of RAM. At least it gives you option to use ECC or Non-ECC memory.

[Magallanes]

Hummm...

Theorically:


To have 1 pc is equal to "x" time-render. For example 1 hour-render = 1 hour.
To have 2 pc is "equal" to "x/2" time-render (is a bit more slow, cause network/distribution task). 30 minutes
To have 3 pc is equal to "x/3" time-render ( 20 minutes.)

The myth say that a 4000mhz pc is the double of fast of a 2000mhz, also say that a dual cpu run twice as fast that a single cpu. They are completelly wrong.


Now, a 2000mhz pc give "x" time render. (1 hour)
a 3000mhz pc give "x/1.40" time render (42 minutes) 1.4 is a estimation.
a "4000mhz" pc give only a "x/1.70" time render (35.3 minutes) and even can be more slow.
a Dual 2000mhz pc give only a "x/1.80" time render (33.3 minutes).

So, buying 2 cheap 2ghz pc give more power that a dual 2ghz or a expensive 4ghz and 2 cheap 2ghz pc can be (in some cases) more cheap that a dual or a 4ghz pc.

So, the best choice is to buy the pc with the best ration in price/power (with 1gigs ram), not to buy the power powerful pc.

The dissavantage in to have a lot of pc for the renderfarm is:
1 ) Noise.
2 ) Power (cost $$$ each pc on)
3 ) Space
4 ) Licenses

So, if you think in some number of pc limit (for example 10), the divide the funds/the number of pc that you want, for 10pc, you need to spend 1k for each one, so you can find a good and decent one for 1k. But if you want 20pc then you renderfarm may be more powerful that 10pc, even when each is only a "cheap" $500 pc.

[mlmiller1983]

Are we taking 10K for the ideal company renderfarm or for the at home ideal render farm. My (at home) render farm will pretty much be two dual xeon pcs with 2GB of DDR400 each and thats it. Any other suggestions.

[i601254]

Hummm...

Theorically:


To have 1 pc is equal to "x" time-render. For example 1 hour-render = 1 hour.
To have 2 pc is "equal" to "x/2" time-render (is a bit more slow, cause network/distribution task). 30 minutes
To have 3 pc is equal to "x/3" time-render ( 20 minutes.)

The myth say that a 4000mhz pc is the double of fast of a 2000mhz, also say that a dual cpu run twice as fast that a single cpu. They are completelly wrong.


Now, a 2000mhz pc give "x" time render. (1 hour)
a 3000mhz pc give "x/1.40" time render (42 minutes) 1.4 is a estimation.
a "4000mhz" pc give only a "x/1.70" time render (35.3 minutes) and even can be more slow.
a Dual 2000mhz pc give only a "x/1.80" time render (33.3 minutes).

So, buying 2 cheap 2ghz pc give more power that a dual 2ghz or a expensive 4ghz and 2 cheap 2ghz pc can be (in some cases) more cheap that a dual or a 4ghz pc.

So, the best choice is to buy the pc with the best ration in price/power (with 1gigs ram), not to buy the power powerful pc.

The dissavantage in to have a lot of pc for the renderfarm is:
1 ) Noise.
2 ) Power (cost $$$ each pc on)
3 ) Space
4 ) Licenses

So, if you think in some number of pc limit (for example 10), the divide the funds/the number of pc that you want, for 10pc, you need to spend 1k for each one, so you can find a good and decent one for 1k. But if you want 20pc then you renderfarm may be more powerful that 10pc, even when each is only a "cheap" $500 pc.

While I do admire your analytical and scientific approach, you don't account for other factors such as system bus, # of threads, AMD's hypertransport (simultaneous two-way thread execution), software optimization, software memory management, latency, is the rendering software CPU or memory dependent, etc. No disrespect to your opinion but I think determining needs based on CPU frequency and cost isn't necessarily too accurate.

[ewltra]

Hypertransport has nothing to do with thread execution, its simply a interconnect system.

[Lorecanth]

Actually just got a great deal from Dell when we built a small renderfarm.
We got 8 rackmount machines for 1300 a piece. Specs as follows.

Dual Xeon 2.8 GHZ
2GB RAM
2 GigE connections
16x cdrom
40 GB SATA HD

I'm all for Opterons, but you can find some rediculous deals if you're willing to look at other options. Of course something to think about though is that mental ray is a grand a cpu. So it all comes down to the final price vs performance.

[maninflash]

1MB L3 cache? I didn't know they made Xeon with L3 cache. The ones I have are 1MB L2 Cache and 2MB L2 cache.

sure they do, it's for poor people who can't afford L2 ;)


Plus Mental Ray is optimized for 64-bit multi-threaded paralellized processing

I thought only the standalone version of Mentalray works in 64-bit mode, the version that comes with Maya (which most of us here use) is all on 32-bit.

Actually just got a great deal from Dell when we built a small renderfarm.
We got 8 rackmount machines for 1300 a piece. Specs as follows.

Dual Xeon 2.8 GHZ
2GB RAM
2 GigE connections
16x cdrom
40 GB SATA HD

I'm all for Opterons, but you can find some rediculous deals if you're willing to look at other options. Of course something to think about though is that mental ray is a grand a cpu. So it all comes down to the final price vs performance.

That's a very good deal, if the original poster is still with us, I suggest you give your money to dell and get five rackmounts, about 7k, and you'll end up with a ten cpu farm. mentalray couldn't ask for more unless you decided to make a feature.

[mlmiller1983]

sure they do, it's for poor people who can't afford L2 ;)
I thought only the standalone version of Mentalray works in 64-bit mode, the version that comes with Maya (which most of us here use) is all on 32-bit.

Did some checking they do have L3 cache Xeons. I have Noconas. Love Mental Ray 3.4 with Maya 6.5 Unlimited. I get 8 CPUs to render with. :D

[Magallanes]

While I do admire your analytical and scientific approach, you don't account for other factors such as system bus, # of threads, AMD's hypertransport (simultaneous two-way thread execution), software optimization, software memory management, latency, is the rendering software CPU or memory dependent, etc. No disrespect to your opinion but I think determining needs based on CPU frequency and cost isn't necessarily too accurate.


Ok, you are right, also usually more "modern" cpu came with good memory, disk and such.

Anyways:

http://www.zoorender.com/html/benchmark_mental.htm

If a dual xeon 3000mhz cost $1.3k (with simmilar datas to the benchmark), the render (in this test) will take 45 seconds (aprox.). You can buy a p4 3000mhz HT for the half of the price (ibm or another retailers) of a dual xeon and the render take 76 minutes the render. If you buy two p4ht ($1300) then the same render will take 40 seconds = better that the same solution with dual xeon.

The result can depend in the render (in the scene, in the type of render, GI an such) but it can give a idea about the power vs the cost.

[MadMax]

I'd go dual Opteron with 2GB of memory an 80GB hard drive and a micro CD. The Opteron 248s are at a great price right now and you can get three nodes for under $10K. Call me and I'll be happy to help you out.

Ed


Hey Ed,

I have no doubt you guys are already a step or two ahead of the pack, but are you considering that 8 way modular iWill board with 8 x 8xx series dual cores for a render node?

[ihavenofish]

actually, to add to that last question. im more interested in the upcoming (i hope soon) tyan 8 way board that has a pci express 16x slot :)

[i601254]

Hypertransport has nothing to do with thread execution, its simply a interconnect system.

Right you are. I stand corrected. HT wouldn't benefit AMD anyway since they are already a much better designed chip and will benefit more from dual core.

[i601254]

I thought only the standalone version of Mentalray works in 64-bit mode, the version that comes with Maya (which most of us here use) is all on 32-bit.


Yes, but the original question was posted by someone using the standalone version.

[Ed Caracappa]

madmax,

yes, multi socket (more than 2) rendernode is in the cards.

Ed

[MadMax]

madmax,

yes, multi socket (more than 2) rendernode is in the cards.

Ed


somehow I just KNEW you were going to say that. :)

excellent!

[twizler191]

I found this in another forum somewhere

http://www.artvps.com/products.ihtml?page=pureoverview

a card with 8 proccessors for 2,000 dollars. 2 of them should get the job done, since it supports dual like nvidia sli. That's 16 proccessors. Only $4,000 total.

[ihavenofish]

but then you are limited to the use of their renderer, which may or may not have the features you need. its also not a whole heck of alot faster than a dual cpu machine using a normal renderer from what ive seen. not to mention you still need a fairly expensive machine to stick the card in.

solutions like gelato that use generic video cards are much more versatile since "you have to buy a video card anyway". unless someone makes a general purpose render card that any third party renderer can run on.

later

[JDex]

That makes me wonder if there could be a solution that would allow for a daughterboard with 2 dc-opterons and 2 memory slots that you could plug into a PCI-E slot on a dual dc-opteron mobo...

That would be a killer addon card for a renderbox that doesn't need a good GFXcard.

[ihavenofish]

i dont know about opterons but there are all sorts of cards of that nature. the problem is (well, for lets say the tyan k8we) that you can have 2 16x cards... so if you stuff 2 cpu's in each, you get 4 processor (8 cores). thats in addition to the 4 you have already. 12 total. still less than a true 8 way, and "probably" much less efficient, and more expensive.

opterons are also not going to be the most efficient processor if all you are doing is rendering. this is where a streaming "cell" type cpu might be nice. 4 cells and 4 gb ram on one board. yum. the issue then is that people need to port the software. bah. its all trade offs in this :) i think nvidia seems to have the best idea for now... use hardware that you already have (a quadro 4000 by numbers alone should crush 2 puny opterons) and send parts of the process that make sense to it. we'll have to see how much a video card can be leveraged, with pci-e the small amount of onboard ram may not be such a huge limitation, as it would effectively be used like cache.

we shall see.

[novadude]

That makes me wonder if there could be a solution that would allow for a daughterboard with 2 dc-opterons and 2 memory slots that you could plug into a PCI-E slot on a dual dc-opteron mobo...

That would be a killer addon card for a renderbox that doesn't need a good GFXcard.

The Sun v40z uses a daughterboard to do just that, and it uses its own connection so it doesn't rely on the systems bandwidth that would be better used for something else like a RAID array.

[MadMax]

That makes me wonder if there could be a solution that would allow for a daughterboard with 2 dc-opterons and 2 memory slots that you could plug into a PCI-E slot on a dual dc-opteron mobo...

That would be a killer addon card for a renderbox that doesn't need a good GFXcard.

That is what the iWill board I mentioned does.

They have a system with cards, 2 Opteron sockets per card. Install 4 cards with dual cores, and you can have up to 16 cores total.

[ihavenofish]

O_o

i know there is one 8 way like that, but i thought the iwill was 2 quad boards linked with HT and then a io daughter board... unless you mean a different board? the upcoming tyan 8 way is built the same way except the io in on the base board.

having 4 dual cpu cards is supposedly less efficient (something about the number of hops to the furthest cpu). i dont know if it really makes a huge difference.

later

[MadMax]

Is it? Okay, I'll take your word for it. I only glanced over the article on the board and saw the illustration of it. I didn't pay that close attention to it. Only that it was a card based system and 8 dual cores.

having 4 dual cpu cards is supposedly less efficient (something about the number of hops to the furthest cpu). i dont know if it really makes a huge difference.

That would really depend on how the board was setup. There is no reason that 4, 2 way cards would be any less efficient than 2 4 way cards. It would really depend on how the main board for it it setup.

Either way, Tyan and iWill make prety solid gear. doesn't matter how you get 16 cores, the fact is it does and that's all I really care about.

[ihavenofish]

http://www.iwillusa.com/product_2.asp?p_id=90&sp=Y

thats the current iwill. it doesnt seem to be available outside of a full system kit.

http://www.v-t.jp/english/products/hpc/opteron/server/sv8000_top_main_e.html

thats the card based one... not sure who fabs the mobo.

http://www.xbitlabs.com/articles/editorial/display/cebit2005-7_6.html

and thats the one that will own them all :)
haha
/me drools

oh, and yes i suppose you could make the card config work, but the routing becomes really complex, and from what i read they dont do it. then again, the iwill is being bitched about because it uses a ladder config and that its not as efficient as it could be. the tyan from the diagrams is a crossbar, which is supposed to be ideal for an opteron 8 way, but who knows till someone tests them all. rendering is not nearly as bound by bus latency as other apps (which is why it usually gets a nice boost from multiprocessors)

[Lorecanth]

Ok, you are right, also usually more "modern" cpu came with good memory, disk and such.

Anyways:

http://www.zoorender.com/html/benchmark_mental.htm

If a dual xeon 3000mhz cost $1.3k (with simmilar datas to the benchmark), the render (in this test) will take 45 seconds (aprox.). You can buy a p4 3000mhz HT for the half of the price (ibm or another retailers) of a dual xeon and the render take 76 minutes the render. If you buy two p4ht ($1300) then the same render will take 40 seconds = better that the same solution with dual xeon.

The result can depend in the render (in the scene, in the type of render, GI an such) but it can give a idea about the power vs the cost.

When looking at single proc machines, though you fail to take into account other costs. IE electricity, heat generated (and needed air conditioning), and space in freezer. Its not all about the final performance, its about the overall most bang for your buck, and when it comes down to it the cpu is only one part of the equaton.

[hugodog]

So is it safe to say that it comes down to either: ?

1. Get the most dense most powerful n-way systems and probably very expensive

2. Get the cheapest mainboard with rather powerful but not so expensive single CPU and go for quantity.

I think both have their own valid pro and cons.

[Lorecanth]

So is it safe to say that it comes down to either: ?

1. Get the most dense most powerful n-way systems and probably very expensive

2. Get the cheapest mainboard with rather powerful but not so expensive single CPU and go for quantity.

I think both have their own valid pro and cons.

Right, so it ends up being terribly dependent on your enviorment situation. Server air conditioned space being a premium here in Vegas it makes a lot more sense to go with 8 dual proc machines with 1 400w PSU a piece than 16 single procs with 200 W PSU's. It's all about finding that most fine balance, and due to the fact that technology has changed so much in the past month, what was the "right" solution last month can be drastically different from this month.

[ihavenofish]

"....8 dual proc machines with 1 400w PSU a piece"


ahh, if only. 89 watts x 8 plus the rest. that 8 way needs 1000w to be safe. the 16 way (dual cores) will need 1500w, especially if you use the pci express slot.

so just to fill you in on the math. thats 1.5kw per hour to run. so if you let it render 24 hours a day, 7 days a week. at 15 cents per kwh for electricity it will cost $162 per month just to have running. nevermind cooling and such. brutal :) but yes, 16 single cpu machines will probably use more. and of course there are opteron EE chips, that use 35w instead of 89w.

anyway..

[novadude]

"....8 dual proc machines with 1 400w PSU a piece"


ahh, if only. 89 watts x 8 plus the rest. that 8 way needs 1000w to be safe. the 16 way (dual cores) will need 1500w, especially if you use the pci express slot.

so just to fill you in on the math. thats 1.5kw per hour to run. so if you let it render 24 hours a day, 7 days a week. at 15 cents per kwh for electricity it will cost $162 per month just to have running. nevermind cooling and such. brutal :) but yes, 16 single cpu machines will probably use more. and of course there are opteron EE chips, that use 35w instead of 89w.

anyway..

He is talking about running eight individual dual CPU boxes, not a eight way system.

[ihavenofish]

oh please... you cant expect me to read EVERY word in someones post.

:x

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.