PDA

View Full Version : Backburner 2011 (issues)


cesboa
04-16-2010, 08:55 AM
First of, 2011 finally works on OS X, the 2010.2 version had some major issues and I never got that in to production. I only have a few potholes I'm hoping to clear out before switching our small farm to backburner.

1. The test setup I have made is of 4 machines (MacPro running Maya 2011 on Slow Leopard). All 4 machines are in the same group and renders beautifully (so far). The manager is running on the machine "MacPro01" and no other machines are running the manager service. All but ONE of the machines can submit jobs, but the machine "MacPro04" can't.
The error - "Cannot create job: No route to host."
"MacPro04" renders fine with the other machines as a part of a group, but the machine it self can't submit. I have no firewall or strange host workarounds on it. I had a trial version of smoke installed, maybe there are a conflict there.
Anyone has a solution to what could course this machine to refuse to submit?

2. Is there a way to force stop rendering?
An example:
A job with a "Task Size" of 10 and 1000 frames with a render time on 10 minutes pr. frame. When deleting or suspending a job every machine has to run a full "Task Size" cycle to stop, it even appears idle while still batching away until it has finished it's "Task Size". On this particular job it could result in hours of wasted cpu and a slow render on the next submission until full Task Size" is done.
"Work arounds":
- Manually login on to every machine and force quit the Maya Batch. (tedious and dangerous, like what if I crashed the wrong job).
- Make smaller Task Size (Works great but can result in longer render times, runups, network load etc.)
Is there a way to make backburner Force Quit it's current Maya Batch threads while deleting a job.

Does anyone have any solutions to my backburner problems

Anything else?

cesboa
04-16-2010, 04:20 PM
As usual I have to reply to my self ;)

The "Cannot create job: No route to host." issue was on a deeper level that just an backburner issue. After digging around route tables and network stacks for an hour I found out that it was VMWare Fusions network interfaces that messed up the backburner submitter. :banghead: Vmware surly messes up the route table!!

So after killing the interfaces on the OS then backburner submits fine.

I don't know if this problem is with backburner, vmware or os x.
UPDATE: I talked to a friend of mine who is an engineer at Ericsson making big router systems (and is a network developer on OpenBSD) - After looking into the backburner/vmware problem he says that it's backburner who are doing a poor job and it's not VMWares fault...

So on to question "2". How to kill a job from backburner that also kills the batch render?!

Kinematics
04-18-2010, 12:20 PM
I wont be helping here at all but I too have a question. I am using BackBurner 2008.1.

Are you implying that when you issue out a job and suspend the task...the computers still continue rendering until all frames in that single task has been completed?

I've had problems where a job submitted will take forever to render and then when I check out the file system, backburner was re-rendering frames over itself.

I think the cause was that the guys in my office work by day and then I have backburner launch by night and then stop in the morning again. The start stop seems to f* it all up. is this common?

onetime
04-20-2010, 08:09 AM
@ cesboa

1. I had that same "No route to host" issue with our Macs running 10.6 and 2011. I cleared it up by removing 'localhost' from the manager.host, and entering only the info for the machine running the manager service. What we were doing wrong before was simply adding the manager service info below 'localhost'. You probably already tried that or resolved this issue, but hope that helps. If not, all I could think of would be to make sure you can ping the machine in question. Also, does it show up in your servers list from the manager? Can you send it jobs? (I read your other comment. It seems you fixed it already!)

2. As for forcing rendering to stopů I found that if you log in as backburner/backburner, you have a few additional options for deleting/archiving jobs. (I haven't given any other account admin rights successfully) In my small tests, it works so-so, but I'm not sure how long will take to cancel the server's MayaBatch process if the frame will take 10 minutes or so. Maybe it's after the frame finishes, but it's after the task finishes. I found that since the MayaBatch process runs as root, it's a pain to force quit the process. So after re-reading your comment, I don't think that info helps because it looks like you've already tested it. I also get the idle MayaBatch process on occasion.


So, now my problemů Here's a snippet from the backburnerServer.log:

2010/04/20 09:42:18 INF Receiving new job from 172.20.96.194
2010/04/20 09:42:18 INF Job 'bbTest093619-100420' received and ready
2010/04/20 09:42:18 INF Launching 'Command Line Tool'
2010/04/20 09:42:18 INF New task assigned: 1
2010/04/20 09:42:20 INF [ServerMessage]ProcessStarted processId=14622
2010/04/20 09:42:20 INF #??#
2010/04/20 09:42:20 INF Starting "/Applications/Autodesk/maya2011/Maya.app/Contents/bin/maya"
2010/04/20 09:42:20 INF #??#
2010/04/20 09:42:20 INF File read in 0 seconds.
2010/04/20 09:42:20 INF Result: /Volumes/Renderfarm/Renderpool/mrSatTest/01_Szenen/bbTest.mb
2010/04/20 09:42:20 INF Error: The renderer mentalRay is not registered yet.
2010/04/20 09:42:20 INF Error: Scene /Volumes/Renderfarm/Renderpool/mrSatTest/01_Szenen/bbTest.mb failed to render.
2010/04/20 09:42:20 INF #??#
2010/04/20 09:42:20 INF // Maya exited with status 210
2010/04/20 09:42:20 ERR Task error: Cmdjob adapter: Process exit abnormally,Exit code is -45

2010/04/20 09:42:20 ERR Cmdjob adapter: Process exit abnormally,Exit code is -45

2010/04/20 09:42:20 INF Application is down

As you can see, 'The renderer mentalRay is not registered yet'. I'm pretty sure it's not a license issue, but a plug-in issue, since we've activated. Maya Software renders fine. I can render the same job fine in mentalRay via command line. I've tried deleting preferences and also setting Mayatomr to autoload. Might this be an issue of the MayaBatch process running as root? Oddly enough, it work on my laptop. Any suggestions would be appreciated. Thanks.

onetime
04-20-2010, 12:03 PM
Please disregard my previous post. Maybe there was a problem with my test scene that caused mentalRay to not register, because I tried a fresh scene and it works.

The main problem I have now is that there is no way to force renders to stop from the Web Monitor before the task has been completed. So for now we will use a task size of 1, but I've filed a bug report with ADSK. Hopefully someone reads it.

cesboa
04-20-2010, 12:38 PM
@onetime: I second that!! It must be a bug :)
I have made a script that kills MayaBatch via SSH, but it very ugly way to do it... And I have to exchange SSH keys on every machine in the farm because MayaBatch runs as root via backburner...
It is possible to make backburner kill the MayaBatch it self by sending the same command via cmdjob - but it's of no use because it's getting back in line behind the already running MayaBatch :banghead:

So Onetime, we just have to wait ;)

onetime
04-20-2010, 12:59 PM
Yes, I considered an SSH shell script to force quit the process as well, but what a pain! Rebooting the machine is a little easier, but still inconvenient.

And I'm still having the problem with 'The renderer mentalRay is not registered yet' error. It seems that Backburner will cause a server to error out even if the error can be ignored. It seems this can be resolved by setting the verbosity to 0... another workaround.

I'm pretty disappointed to find that there is no frame verification method in sight.

Also, all my machines need to be logged in and the network volume mounted. Is there a simple way around that?

onetime
04-20-2010, 07:50 PM
So tonight we decided to test out Backburner with a heavy scene with lots of depth of field on about 4 machines. Each frame takes over an hour to render. Well, I just checked the progress and got a fun surprise that you should be aware of. Especially Kinematics, since I think this is what was causing his frames to rerender for no reason.

Basically, the default Task Timeout is 60 minutes (shown in the Advanced Info tab in the Job Details). But if a frame takes more than 60 minutes, Backburner is courteous enough to assume that the task is dead and sends another task without killing the first task that's still in process. This means that for each frame that takes longer than 60 minutes, a duplicate process is started, cutting the CPU time in half and rendering a duplicate frame PER SERVER. But guess what? Another 60 minutes goes by and you have THREE duplicate processes per server... all rendering the same frame! What a waste of CPU time. Sweet! :banghead:

Now for the good news... Entering this bit of code into the "Use Custom Command" box, after you have clicked "Populate Command", will not only increase the Task Timeout to 6 hours, but will ALSO run the MayaBatch process with user rights! It's not as good as being able to kill the process from the Web Monitor, but at least admin rights aren't required.
-userRights -timeout:360



Edit: Anyone who can find the correct config file to add the above code to the default CMDJOB template gets bonus points!

cesboa
04-20-2010, 09:05 PM
@onetime: HAHAHAHA, that is a priceless find!! :) Has AutoDesk actually tried this messy pice of software before release...

Are there anyone out there who uses backburner with Maya on OSX/Linux in real production with success?

I'm going to wrap my SSH kill MayaBatch script with a GUI from Maya. It would not be generic/clean enduser pipeline script - but It's should be easy to implement.

Kinematics
04-22-2010, 03:39 AM
Thanks onetime! That is really golden information. But does that mean if a frame takes longer the 60min. That would cause all these problems? Wouldn't that be really likely to happen and shouldn't that not have been programed into backburner. Thats freaking weird.

How is Backburner 2011 compared to Backburner 2008? I need to call autodesk and check if I can get an upgrade on Backburner but not upgrade to Maya2011. I dont trust it.

cgbeige
04-22-2010, 05:54 AM
Backburner doesn't seem to like V-Ray so I'm back to Deadline and it works fine with 2011 x64 on OS X.

eikonoklastes
04-22-2010, 06:03 AM
Where I can find a complete list of all Maya-specific Backburner custom commands?

Things like -mr:rt are priceless but seem to be completely undocumented.

cesboa
04-22-2010, 08:15 AM
Check the "Render" parameters. On OS X, Mental Ray:
/Applications/Autodesk/maya2011/Maya.app/Contents/bin/Render -r mr -help

Maya Software:
./Render -r sw -help

etc.

It's a render "flag" not backburner specific stuff :)

onetime
04-22-2010, 10:11 AM
Last night we tried setting the TaskTimeout to -timeout:1000 , or over 16 hours, but we still got duplicate processes starting after about an hour. I'm not sure why, but the problem was isolated to a job that was assigned to more than one server. Oddly, a seperate job (same scene) that was assigned to a single server did respect the -timeout:1000 setting, and did not time out all night for 3-hour long frames.

I tried setting the -timeout to '0' for a small test scene, hoping it would indicate 'infinite', but it does seem to read as zero, so a new process is sent every minute or so. I want to disable it... It seems downright stupid that it's enabled by default, not to mention as a rather low value of 60 minutes. I've submitted several bug reports/feature requests and I recommend you all do the same.

356800 Backburner Web Monitor is unable to quit server tasks in process.
356874 Backburner Manager sends a duplicate task if the first task 'times out'.
356875 Add a functioning -userRights checkbox to the Create a Backburner Job dialog
356876 Kill renders processing on servers from Backburner Web Monitor
356877 Backburner Frame Verification <-- This would be awesome
356883 Create Backburner Job - Checkbox/Field that permits # of threads

Another thing if you're using the -userRights tag. All servers need to be logged in with the same user name as the machine who creates the job. For example, if user ABC submits a job with -userRights, and user XYZ is logged on to another server, the job will try to execute as ABC and will not have permissions, at least in our case. We're trying to arrange that all our servers have a default user account with network volume access.

So far, in the 2 nights that we've used backburner, we have reverted to rendering via command line with a batch shell script. 0/2... I hope we can get to the bottom of this. I've heard deadline is good, but that's more time and money. (I know you get the first 2 free.)

onetime
04-22-2010, 11:01 AM
Just a clarification on the Render parameters...

In the Create Backburner Job dialog, if you set the Renderer to mental ray, you can use '-rt 16' etcetera in the Additional Options field. If you set the Renderer to 'from scene file', it seems that CMDJOB will not recognize that, and you should use the '-mr:rt 16' syntax instead. It seems that Backburner doesn't always respect the Renderer that you set in the scene file if 'from the scene file' is selected, so I just select mental ray, but this may have happened because I did not refresh the Populate Command field (you need to do that almost every time you change or resubmit.).

You can find a full list of CMDJOB parameters in the Backburner user guide under Advanced Operations, or (in OS X) enter /usr/discreet/backburner/cmdjob into your terminal.

cesboa
05-03-2010, 09:44 PM
Backburner doesn't seem to like V-Ray so I'm back to Deadline and it works fine with 2011 x64 on OS X.

Hi. Is Deadline what you would recommend for a 4-6 Mac Pro OS X setup? Running Maya 2011, Mental Ray, RMfM (Maya 2010), After Effects and Nuke? Or is there anything else out there I should look at? So far Backburner has been a bag of hurt and we are currently using an qmaster pipeline for network rendering - and it's also a bag, but a smaller one. :)

cgbeige
05-04-2010, 04:49 AM
ya, Deadline is great. The interface is crusty as hell since it's mono based but it works. Give it a try with two slaves and if you need more, buy the licenses. It supports all those programs and way more.

CGTalk Moderation
05-04-2010, 04:49 AM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.