Renderfarm in the cloud


#1

I’m looking at some renderfarm options and came across this:

http://forums.cgsociety.org/showthread.php?t=999606

For 4$/hour I can have a “26 EC2 Compute Units and 68.4 GB of memory” do the hard work.
(A EC2 unit a ~1Ghz 2007 Xeon model equivalent)

Does any one have any experience with this? any pitfalls or other problems?


#2

I´ve been working on a Cloud-Based-Renderfarm based upon Amazon´s Services for around a year now. It will be part of my Master´s Thesis and we are planning to release part of the work in future. I´m also working on a solution to integrate the cloud-based farm into an existing farm, as if it where right next to the other slaves in your chamber :wink:
But that part is still in experimental stage and so i can´t tell much about that…

We have it as far, as it can be used in heavier production-work (we got around 1000 GHz up´n´running for out latest project)

It works…but i had to put quite some work into it…you have to find ways to accelerate the start-up, automate as much as you can, because otherwise it could become tedious to maintain the whole thing. Speed in control is vital, because you pay for every started hour.

There are also some things, you have to work out like data-access, preparation of project-data to be easily transfered into the cloud and stuff like that. And then there is also backburner, which can have its quirks here and there :wink:


#3

I’ve been reading up a little on Amazon’s EC2 service… interesting stuff!

So if I’m not mistaken I can create a windows based in-the-cloud-computer, install 3dsmax+plugins etc on it via remote desktop, setup a VPN connection make it part of my LAN.

Then I can use backburner or Deadline to submit a job as normal?

If I understand correctly I only pay for the virtual computer when it’s on per hour and the traffic they produce. So I can just bootup as many as I need and it won’t cost me a thing when there on not being used…


#4

Funny you mention Amazone’S EC2 - a transformator was hit by a thunder strike and in the Us the service is offline or so ?

http://www.neowin.net/news/amazon-ec2-useurope-experiencing-widespread-outages

last big outage was in April or so … not too long ago…


#5

Alright, I’m going to spend the rest of the day trying to make this work… as a tutorial/future reference I’ll post my steps…

I’ve jumped the hoops of the EC2 signon process (see stratusfarm’s docs for a detailed how-to)

I’ve created a new instance ‘windows server 2008 64bit’. by following these steps, make sure take the windows route: http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/

I’ve created the instance as a simple 4core/7Gb machine to do the installations and such, I can increase the power later on. It has booted up and I will get my windows admin password within a few minutes orso. Now the money is draining @ $0.48 per hour until I shot it down.

I retreived the password, now I can use remote desktop software to take control of it… I’ll be back after I installed a max rendernode on it


#6

I’ve also considered using Amazon-EC2 for rendering, although I’m still a bit unsure about costs…

Stratusfarm.com says:
High-CPU Extra-Large: $2.20 per hour/machine
20 EC2 Compute Units and 7 GB of memory

According to Amazon, 1 compute unit equals roughly 1 GHz… so 20 GHz will cost $2.20 per hour. This means 1 GHzh costs $0.11 whereas on Rebusfarm 1 GHzh costs a maximum of $0.1 (or even less, if you render larger animations). Also, Startusfarm doesn’t support any plugins (besides V-Ray), which requires more work when preparing your scenes to be renderable.

So right now, I don’t see an advantage over using a renderfarm.


#7

Via windows ‘remote desktop connection’ I got 3dsmax2012-64bit+SP+hf2 installed

No problems there actually, I downloaded max from my subscription login at a whooping 3.5MB/s directly to the it’s desktop, installed without a hitch.

Next step is to get the EC@ unit into my LAN so I can use the normal backburner or DeadLine tools to render.

I’ve tried the stratusfarm plugin and that isn’t going to fly for heavy productions. You can submit a job and hope it starts… zero control except a cancel button which shuts down the whole renderfarm.

So I getting it into the LAN via a VPN is up next…


#8

I’ve abandoned statusfarm, setting up my own ‘hardware’ for this now.

The advantage is you get full control over the computer, can install plugins, scripts etc you can literally be on it’s desktop and check the rendering. Also you can keep using backburner or any other tool to submit jobs.

REBUS is faster but has a differnt pricing schema, hard to compare.

home:
~10ECU workstation
100 frame @ 5 minutes/frame -> 8.3 hours
cost: ‘0’

REBUS:
100 frame : 5min/frame -> 0.01 hour
cost: 14 euro

EC2:
20ECU node
100 frames @ 2.5 minutes/frame -> 4.15 hours
cost: 4.15 * $1.16= 3.37 euro

($2.2 is stratus price, $1.16 is directly from amazon) I can start up 3 or 4 units, and I pay the same, only it’s done faster.
4 units running 1 hour is the same price as 1 unit running for 4 hours.

It’s the flexibility, control and the use of backburner/deadline that make it interesting for me.


#9

Though the Stratusfarm page says there’s a limit of 20 nodes being started on the Amazon Cloud… is that true?

Edit:
Regarding the lower costs you got by setting up your own instances, I think Stratus Farm isn’t actually free (I thought it was), but they actually have a higher GHz/h price to finance the service… at least that’s what I think.

[b]“you will need to subscribe to the Stratus Farm product”

[/b]And I guess that product costs money. If you take this into consideration, cloud rendering on EC2 might just as well be cheaper than a renderfarm.


#10

This seems very interesting. So stratusfarm increases the price? And from what I gather, when you sign up to the Amazon service, and you can access a Windows install from it and then you can install 3ds Max and render off from there?


#11

Yep I’ve got max installed via remote desktop. I’m having some issues with VPN but that because I broke a SSTP thing on my vista workstation, normally I would be up and running by now. I’ have to do soem real work now so I hope to spend a few hours on it tommorow again… I’ll keep posting my progess here :slight_smile:

For now I’ve installed everything on a simple virtual computer which cost $0.48 per hour to run, with a few mouse clicks it turn into a 20Ghz 8 core monster for 1.16$/h… not bad. I can spawn multiple instances if eeded… The goal is to create a LAN in the cloud with a flexible amount of nodes and bridge that to my local lan with a single bridging-VPN tunnel… resulting a single LAN that backburner can do it’s thing on.

edit: Yeah stratus increase the price ot cover their service i guess. But their service doesn’t include a VPN so you have to submit your work trough a plugin and have no management or feedback at all like you would with backburner.

this is what Amazon charges: http://aws.amazon.com/ec2/pricing/
stratus price: http://www.stratusfarm.com/


#12

You can get more but you’ll have to make a special request to amazon to get that limit up. 20 nodes is the max any ‘normal’ user can run at the same time.


#13

As i said, i ventured into all this for quite some time and found it to be a pretty workable solution for rendering, when done the right way.

You should keep a few things in mind though:

Amazon does not only charge the usage-time, but also the transfer-volume…allthough this is almost a neglectable amount of money. Just to give you an idea of the price-range: For our last project, we had around 70 Nodes up for around 2-3 days of renderwork. The project itself was around 4-5 GB with all the XRefs and stuff…the renderoutput was much more, as we rendered Floating-Point RPFs and several passes per scene. With all that in mind, we paid only around 10$ for the I/O-Transfers in and out of the cloud. That compared to the full renderprice of the project wasn´t even noticable :wink:

Stick with EBS-Backed instances. You have the choice to not do so, but i would definitely recommend you to take the EBS ones. The first advantage is, that you can run the windows server 2008 version only with EBS-Backed instances and i found win server 2k8 much easier to manage. The second…and way more important advantage is, that you can handle the creation and maintenance of the AMIs (amazon machine images) much easier when on EBS. You have to do some tricks though to get an EBS-booted machine to also be able to access all of its volatile, non-persistent storage. I can´t tell you why, but that is only possible, if you boot the image up by API-calls rather then the web-interface.

Try to stay in one availability-zone! You have to pay for internodal transfers from one zone to another. You dont have to pay much, but it isn´t necessary to pay at all, if you organize what you do before you do it. Amazon considers EU-West1a and EU-West1b also as different availability zones.

I found it easier - especially in the beginning - to create one AMI for the backburner manager-node, that is located in the cloud and an AMI for the render-nodes. The advantage of the manager in the cloud is, that it has a much better bandwidth. When rendering with backburner, all the rendernodes get their jobs from the manager and keep those as a local copy. That can sum up to quite a few hundred MB per Job, that have to be transfered from manager to rendernode. Doing that from home via VPN is almost ridiculous, because the usual uploadbandwidth for consumer-class-connections is around 1-7 MBit/s.
When you start to get bigger projects with the need for more rendernodes, this comes into account even more, as there might be times, when a couple of nodes want their job data simultaneously. You can´t handle that from home, but a manager-node inside the cloud can easily do that.
In our last project, we had to use a 4core, 7Gig Node as manager, because the network-traffic was really punishing that little thing. When there are around 20 nodes, that want to access data simultaneously and that data is around 800MB for each node, the gigabit-interface on the manager-node is really starting to hurt :wink:
It´s better to let the amazon-infrastructure handle this kind of stuff…

I´ll stop here, because i could possible write a lot more, but i am not sure, if you are interested in it and it would take some time.
As i mentioned initially - i am writing my masters thesis about cloud-based rendering and all it´s quirks at the moment, so i did quite a lot of tests for sure and would share that knowledge…if you don´t put it into a master thesis about that exact topic and publish that as yours :wink:


#14

Hi Guys + Companies,

Becareful ! Don´t use cloud Rendering for your commercial Projects. The Data are not save and you don´t know, where are the Data is at the Internet/Cloud. I had many talks with a old friend Fefe from Chaos Computer Club, and he say: " Don´t do it! ".

Better and save is the Rebusfarm or other Renderfarm, that not use a cloud.

Cloud Rendering is ok for Students or Hobby Projects.

mfg
hot chip


#15

Yeah sure…you mean the guys from Rebus farm, who look into your running projects while you render them there and run their own 3d company at the same time? Why should they have any less interest in my data then amazon?

Sorry for that straight-up answer but i honestly don´t think, that my data is either completely save on a commercial farm-provider or on a cloud…but on the amazon-cloud i have the utmost control about my stuff and can handle it all for myself.

Sure…there is some “minor” risk, that amazon could take my data and keep it in a backup or whatever…but so what?! What are they supposed to do with that data? Sell them to my competitor? Really? I better put my foil-hat back on then, so the aliens cannot steal my thoughts.

Sorry, if the last part was a bit harsh. I got an IT-background and can understand the fears of the current situation, when it comes to moving everything into the cloud and all that. But you also have to judge carefully from case to case, whether to fear or to use the opportunities.

I wouldn´t host my corporate-sensitive data on cloudspaces, as i can´t guarantee, that there wouldn´t be a case, where someone could have a look into them. But i do put my render-stuff up there with no problem at all, because i am sure, that this is data, that is hard to be used the wrong way.
You would first have to figure out, what kind of data it is, how to process it, what conditions (plugins, scripts, shaders and stuff) are necessary to make it work and then use it. Why would you do that?


#16

Hi Decarus,

Rebusfarm is absolutly save. I had a NATO - Project rendern on Rebusfarm. Andre (the boss of Rebusfarm) don´t use your Data for himself. He is a correct Guy.

You have no controll about your Data in a Amazon-Cloud, that is a fact.

For example, if you create a animation of a world news BMW car, can´t you do that in a cloud. A Hacker from a other company, can hack the cloud and copy your new Data. That was the last project from your client.

mfg
hot chip


#17

As i said…i understand your point of view, but mine is slightly different. Nothing digital is save…why should this hacker not try and hack the pc´s at my place or the farm where i render?

I don´t trust Rebus or any of the other renderfarms any more then i would trust amazon.
For the scenario, that you described, the said hacker would have to know quite a lot of things to do, what you suggest:

  1. He has to know, that i got the contract
  2. He has to know, that i am rendering said project at the moment on amazon
  3. He has to know the availability zone i am rendering in
  4. He has to know quite a lot about my amazon-account and the underlying structure of amazon to locate my stuff there (that is secured in quite a few ways for sure, as amazon is running its complete own business on the very same infrastructure)
  5. He has to find a way to get my data from there…ideally the whole EBS-Volume, where my working data is stored
  6. He has to find a way to get the encrypted data to work. Because you have complete controll over the virtual machine in the amazon cloud, you can also make use of TrueCrypt and stuff to realtime-encrypt all your data there. So if there is the case, that someone fiends traces of your data, that you left there, he only finds AES-encrypted containers. If he is able to crack them, he really deserves the price, because he would be one of the first guys on earth to do so.

I find this to be quite a lot of premises to pose a real threat, that would keep me from rendering my stuff there.

Wouldn´t it be easier for the thief to simply break into my office at midnight and rob my computers with the data on? To find my adress he only has to use google and it´s way easier to pick a lock or throw in a window, than to crack a 256Bit-encrypted password…

You rendered a highly critical problem on Rebus. How did you ensure, that your data was save there all the time? How did you ensure the save traffic from your pc to their farm? Was your workstation connected to the internet while you worked on that project?
Sure…with data inside the cloud comes additional risks, but i wouldn´t go as far and say, that all the projects inside the cloud are directly to be assumed as leaked and stolen. There are quite a lot of risks in all the other areas, where you work with your data.

And again: I don´t want to convince you to any cause. I can fully understand, when you say, that you don´t like all that cloud-stuff for your projects and all that. This is perfectly fine for me. But i don´t like it, when somebody acts like telling a higher truth about things, that are only based on his own presumptions and feelings


#18

Hi Decarus,

i know the good reasons for cloud Rendering, and Fefe need a long time to convince me. :slight_smile: I think it is playing with fire.


#19

OT: Judging by laws of probability it’s actually way more likely that certain (in this case 3d) data is stolen from a dedicated 3d service than from a service that hosts lots of different data which of < 1% is 3d data… until the whole industry starts to use amazon, of course.
Honestly, anything can happen to anything, anytime and anywhere.

The thread is very interesting, please keep us informed how it all goes.


#20

Out of curiosity, and because it could be quite useful for an upcoming project, would you be able to run fumefx simulations on the Amazon cloud? Then render them on the cloud too, so you don’t have to deal with trying to download hundred of gigabytes of cache files?