Investing in a small C4D Render farm

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

REPLY TO THREAD
 
Thread Tools Search this Thread Display Modes
  08 August 2017
Investing in a small C4D Render farm

For a number of years I've wanted to build a small farm, or a 'Render garden' as my girlfriend prefers to call it.
My estimate for how many cores I'd ideally like to have goes up as processors get more sophisticated, but right now stands at 144.
That's 2 machines - Both with 4 x 14 core Xeon E7 processors.
And while its frowned upon by some, those Xeons V2's that originally sold for 5k, can be picked up for between $250-$500 a piece on EBay,
Simply marked down in price or the forbidden ES chips 'engineering samples'..

So a farm with a fair amount of horsepower is relatively affordable. Around 12k, as a high estimate.
Not a huge amount of cash. The highest of the high end Mac Pro's used to sell for about this much.

The only thing stopping me and where I have a question. Is in the imminent release of the AMD Naples server CPU's.
Those could pose a real threat to Xeon's. Although you'd be hard pressed to get them at a discount when they first come out.

So a couple of questions.

In my position, would you wait for pricing on the AMD cpu's and possibly build a couple of quad socket machines with AMD tech.

And more importantly, I know the C4D R18 update had support for AMD's new single socket line up.
Would C4D R17 support them too? As I haven't upgraded to R18 at home.

And a small FYI. When operational, it'll be my intention to let others use the farm for a very low daily rate. Maybe $100 a day.
With an el-cheapo site that provides manual access via TeamViewer w/ free ftp access. I have Paxio here, with 1gb up/down. 
144 core doesn't represent a tremendous amount of horse power, but could be useful for quick turnaround renders on a budget.
__________________
۩PRIST


Last edited by ThePriest : 08 August 2017 at 05:26 PM.
 
  08 August 2017
Before setting this up make sure to do the full calculation of cost, especially CB points/power consumption. Quad Socket systems are loud and power hungry, be prepared to have them in the basement or any other well inuslated room. At render time the power consumption can be >2KW, which will all be turned into heat, so factor in aircondition as well. Another thing to consider is single core speed, check what kind of scenes you have. Simulation, MoGraph and animation heavy scenes benefit a lot from high single core speeds, since the preparation before rendering is faster. I would only consider a huge multicore setup like yours if the scenes render at least a couple of minutes per frame. If you were to render scenes that take a few seconds per frame the setup will cvery likely be inefficient.
__________________
- www.bonkers.de -
The views expressed on this post are my personal opinions and do not represent the views of my employer.
 
  08 August 2017
Originally Posted by ThePriest: And a small FYI. When operational, it'll be my intention to let others use the farm for a very low daily rate. Maybe $50 a day.
With an el-cheapo site that provides manual access via TeamViewer w/ free ftp access. I have Paxio here, with 1gb up/down. 
144 core doesn't represent a tremendous amount of horse power, but could be useful for quick turnaround renders on a budget.
I wouldn't invest 12k right now. The Price war between amd and Intel just began. And unfortunately I Think you will need extra licensing if you want to give your farm for rent.
But. I also would love to have one or two of these machines
 
  08 August 2017
Yea I agree with Bjorn on this. I used to run a 12 machine setup with Xeon 5670's overclocked to 3.8Ghz. Problems I ran into were:

1. Overheating issues in the small room they were running in. I had the window open always. In wintertime this was no problem, but they overheated quite a bit in summertime. I didn't ran an AC so I had to ventilate the room pretty heavily and down-clock the CPU's. 

2. Rebooting problems. If there was a crash I sometimes had to manually reset the BIOS or it had to have a keyboard connected. These small issues took up more time than I wanted.
 
3. Power usage when idle. The machines used up to 60 watt's a piece when running idle. This can be quite a lot which meant I turned the machines off when not rendering. Booting them back up could be a pain because the motherboards I used needed a keyboard to be plugged in. 

4. Teamrender used to suck. It sucked hard and I did not manage to get it running perfect ever. I eventually sold my machines to friends because I never managed to get the pipeline running the way I wanted to. I had to manually intervene all the time. Even so much so that I ended up manually inputting the frame ranges in the cinema editor on each individual machine. Running the farm was a full time job and basically killed my love for the craft completely. 

I think the E5 platform is great since it will not have a lot of the problems I encountered. I hope it runs well for you!
 
  08 August 2017
I assume you've outsourced to farms before - have you calculated an average per job farm cost x average number of jobs per year?  I wonder where 12K would land in that equation.  You could probably get pretty far on that without the maintenance and heat/cooling costs, but it lacks the geekiness which might be what you are after .
 
  08 August 2017
A 4 socket machine is not that efficient for rendering tasks, those machine are conceived mostly for virtualization. Even a 2 socket system suffer a significant loss in performance(i.e. a single 2699v4 reach 3200CB points, 2x2699V4 reach only 5600, things do not scale linearly and the more socket you add the worst it will be). Unlike OEM version ES CPU should be avoided, there's a reason why they are sold for that cheap, there's often something that doesn't work smoothly. For about 5000$ you can build a dual E5 2696V4(OEM version of the 2699), you can get two system like this within your budget for a total of ~11.000CB points, alternatively you can build 7x Threadripper nodes for a total of ~21.000points.
__________________
www.3drenderandbeyond.com
www.3dtutorialandbeyond.com
www.facebook.com/3drenderandbeyond
 
  08 August 2017
It was my understanding that clock speed doesn't make that much difference these days, or at least  It's not the only determining factor of CPU power.
Especially when comparing older Xeon's to newer ones. I mean, of course a 2.8ghz V4 Xeon is much faster than a 1.8ghz V4 Xeon, but it's not a concern that these chips aren't pushing beyond 3ghz.

Take for example my old workstation, it has  2x8 core Xeons with older 'Sandy Bridge' CPU's. They're the fastest ones available in that v1 segment, clocking in at 3.16ghz I think. I also have a 24 core Xeon with the V4 chips, that clock at 2.4ghz and ramp up to about 2.8ghz. The newer machine suffers no slowdowns and is markedly improved over the v1 chip. I don't have benchmarks to offer, but could easily run some.

I understand that there may be some loss on a quad socket board, but the goal was to save money, by purchasing less components overall. .
Buying a bunch of dual socket machines, would mean installing 128gb of EEC ram in each machine, more motherboards, and would require more C4D licenses. I've even priced up going the dual socket route and it starts to look way more expensive.

Also, I wasn't aware that there were additional licensing requirements for Cinema 4D, when running a farm.
I was prepared to buy the two additional licenses and additional copies of Octane, XP etc. But that is a surprise.

In the long run, I need more horse power. I'm a creative director working at an agency, with no access or no plans for them to invest in a farm.
It was my hope that the company where I work, would be one of my primary customers. Where I can save the project managers a ton of money with my extra machines.

That will remain my plan, what I need is a common sense approach to seeing it through..

Thanks for the advice.
__________________
۩PRIST


Last edited by ThePriest : 08 August 2017 at 06:12 PM.
 
  08 August 2017
I got a couple of old workstations vom ebay for my farm. incl. ram and so on. they are really cheap this days. if you have lets say 4 boxes you can scatter them all over the place to avoid thermal problems (btw. this is my mayor problem, too(having a much smaler farm (4 dual xeons)))
 
  08 August 2017
Originally Posted by Ealexander: I assume you've outsourced to farms before - have you calculated an average per job farm cost x average number of jobs per year?  I wonder where 12K would land in that equation.  You could probably get pretty far on that without the maintenance and heat/cooling costs, but it lacks the geekiness which might be what you are after .

A recent job I did cost nearly 10k on Rebus. Granted that wasn't an out of pocket expense for me, but it was expensive. And that's my rationale when it comes to ownership: That I don't necessarily need a farm's insanely fast turnaround speed, but I could use 10-20x more power than I have now. On both the GPU and CPU side.

Originally Posted by Srek: I would only consider a huge multicore setup like
yours if the scenes render at least a couple of minutes per frame

I work primarily in VRay, where it's not unusual for even a highly optimized scene to take 30 minutes or more.
On my work workstation and the 2 I have at home, I might get back 3 seconds of footage per day. 

If wanted 10 seconds of footage back from rebusfarm that takes 30 minutes per frame to render, you're looking at $1049.51  (from their calculator).
As it stands on a current VR job that I'm tackling, I have 3600 frames to render at 4k at around 1hr per frame. Their estimate for this is $25188.20 USD
That would pay for my farm, C4D licenses and I'd still have spare change.

It would take 1 day on Rebus. That's the luxury. I'd be rendering for 15 days straight on a home farm. Or billing my client $1500
__________________
۩PRIST


Last edited by ThePriest : 08 August 2017 at 06:46 PM.
 
  08 August 2017
25188.20 $ for 3600 1h frames? that's insane. out of curiosity i just ran a quick calculation on pixelplow, if you set the power slider very low you can get that rendered under 1k... you'd probably have to wait a day or two and not have everything rendered within an hour. but even with the power slider at maximum speed and also vray checkbox checked it's still under 10k. maybe that's something worth looking at? i recently rendered a 10 sec animation there taking around 20 min per frame for something like 30$ if i remember correctly... and it didn't take more than 2-3 hours... if you have the power slider all the way down it really makes a huge difference, only a fraction of the full speed cost. and if the farm load is very low at the time you submit your job it's still reasonably fast. 
__________________
________________________________
my stuff on vimeo
 
  08 August 2017
I wish we had an easy way to communicate rendering needs other than time - something akin to processor-hours, but in a way that accounts for different processor speeds, as a scene  could take 50 processor hours on one machine but only 20 on another.

If Cinebench measures the processing speed of a machine, these units would represent how much total computing power was needed to render out a particular frame at a given quality - something that would be spit out as a statistic at the end of a render that would allow us to better compare apples to apples. 

I know... just what we need... another benchmark. 
 
  08 August 2017
honestly, external rendering got so affordable, all this year projects  c4d and vray  just sent it externaly, made sense as price/return/time saving option

On the other end you have Octane, which I never managed to make it look photoreal (good for mograph and abstract renders though), and redshift which comes without a library of shaders, the #good looking naked renderer that you need to create whatever you need, which is not production streamlined (I wish it was).

Although I have a small garden of rendering beasts, whenever is crunch time, i just pass it on externaly, the project has plenty of versions/revisions and passes to be rendered, one thing to take off my table
__________________
My link
 
  08 August 2017
i would go with an even smaller farm... maybe two additional nodes and zync...
https://www.zyncrender.com/maxon/
__________________
ArtStation

Last edited by oglu : 08 August 2017 at 08:07 AM.
 
  08 August 2017
Originally Posted by LukeLetellier: I wish we had an easy way to communicate rendering needs other than time - something akin to processor-hours, but in a way that accounts for different processor speeds, as a scene  could take 50 processor hours on one machine but only 20 on another.

If Cinebench measures the processing speed of a machine, these units would represent how much total computing power was needed to render out a particular frame at a given quality - something that would be spit out as a statistic at the end of a render that would allow us to better compare apples to apples. 

I know... just what we need... another benchmark. 


How would you calculate that? Processors may have different command sets. Applications may or may not use those new command sets. Certain commands may execute slower or faster on different processors. Certain commands may be parallelized on one processor but not on the other. Lookahead and predictive branching may be different. Communication between cores or between processors on a multi CPU board may be different. Cache sizes are different. Cache strategies may be different too. RAM access may be faster or slower or better parallelized or... etc.

All of this is the reason why processors got any faster without having to resort to faster clock speeds.

Now what measurement would this "total computing power" be? There is no overall computing power. The speed of a processor may excel in one area and suck in another. And the render time of a certain frame would depend on whether this frame will require the processor to work in the excelling or the sucky area. Which again depends on what plugins you use, what render engine, what xpresso commands, whether you do more IK or more Noise or more displacement or more texture layering.

Ultimately, the time a frame needs for rendering can only be reasonably compared between machines by... rendering the damn thing. In other words, you would have to benchmark your preferred scene setup (in various situations) and just run the render (in a controlled environment). Cinebench does that for a fixed scene so people don't need to exchange test scenes (assuming that all the differences between command execution times will even out statistically to generate an overall estimation), but if you are buying hardware for five-digit sums - and have the opportunity to do so - it might be best to do your own benchmarks on all the available machines for a comparison.
 
  08 August 2017
Originally Posted by oglu: i would go with an even smaller farm... maybe two additional nodes and zync...
https://www.zyncrender.com/maxon/

I've spoken to Zync several times about supporting VRrayForC4D, they say "Soon", but they never say when.
__________________
۩PRIST

 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright ©2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 01:02 PM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.