View Full Version : Use Background in Renderman for Maya?

07 July 2009, 12:50 PM
So, noob question for me:

I've been messing around with the eval version of Renderman for Maya in the hopes that it'll be the magic bullet I need to quickly get nice hair/fur rendered for some projects I'm currently working on.

I've managed to get *awesome* looking hair (using Maya hair curves and some patience), and my initial workflow thought was to use the hair & shadow matte as a render pass via use background (which is now supposedly supported), and keep various other assets as mental ray passes (because they're already built, but also because of heavy raytracing).

The RGB channel comes out great with use background, but I'm getting some super weird artifacts that look like hard clipped self-shadowing in my alpha... cast shadows is flipped off in the body geometry, no reflection or anything active, but stuff like the armpits and side of the nose are showing really chunky shadow clip areas incorrectly.

So I guess, a few questions:

1. Is this an artifact of using deep shadows? Are there workarounds?

2. Is there another technique that I can use within RFM to make a shadowcatcher/holdout matte? I've read that you can append a ribbox or something, but the manual for the eval version is kinda lacking in much practical detail.

3. I've looked all over for framebuffer settings in RFM to get it rendering to 4x32 (or even 4x16), but I can't find any control over the framebuffer to save my life.

Thanks in advance,


07 July 2009, 04:04 PM

Generally if you want a pass in renderman, you use an AOV. There is a shadow aov, give that a crack.

There is lots of talk on the pixar forum about bugs in the useBackground shader, so i would avoid it if posible.


07 July 2009, 05:13 PM
Thanks for the advice... I'll do some digging & give it a shot. I've been able to cheat my shadows via light linking... they're still casting artifacts, but not as many/big.

Not sure I can access AOVs without the fullblown Renderman Studio, though... the only reference I see to setting them up details working through Slim, at any rate (I'm using the basic Renderman for Maya v3 that does pretty much everything through hypershade: no Slim/Alfred/it).

Any other ways you can think of achieving that with vanilla Maya nodes? Can you think of any renderman .sl's offhand that can handle it? I've figured out how to compile and use custom shaders now, and that's most likely the best solution. Not sure I've got time to figure out how to code it myself on deadline, though.

I must say that even with the super sparse documentation, I'm impressed with how smoothly integrated the renderer is. Stuff's quite intuitive, which is coming as a shock to my system after all this time in mental ray.

I'm sure once I dig deeply I'll find something akin to "connect a rayswitch to a scalar node, then into a vector node, then pass the alpha out through an sss_normal node and into your bump slot", but I haven't found anything approaching that level of insanity just yet :)


07 July 2009, 06:32 PM
Well, think I'm halfway there now...

I've figured out how to attach a RIB Box in RFM 3.0 to make a clean knockout matte (but without shadows):

-Create a new shader (blinn, lambert, whatever)
-Assign it to your surfaces
-Fire up hypershade and navigate to the new shading group
-Choose Attributes->RenderMan->Add RIB Box

Matte 1 the box


I found another example for a TCL box, but with my newly googled knowledge, I'm not sure where I need to be picking up $elementtype from... this appears to be a rayswitcher (yeah, joke's on me from my griping above).

Note: not working at the moment....

if { $elementtype == "shadow" } {
set ribout "Opacity 0 0 0"
} else {
set ribout "Matte 1"

It appears to recognize the command, but it's not particularly happy that $elementtype is empty, and I'm doubtful that it'll premultiply the alpha properly if I get it working.

But, I'm at least headed in the right direction.


07 July 2009, 11:21 PM
I'm also having massive problems with the useBackground shader, many other people have encountered inconsistencies as well.

I have provided as much info as I possibly can on the issue, so hopefully it will get resolved soon.

BTW, you should be able to output AOV's with the RFM eval with no problems.

If you happen to find a way to seamlessly use RFM for a footage workflow I would really appreciate it.

07 July 2009, 07:27 AM
Haven't been able to sort it out completely, but I found a good enough workaround for the time being.

I'm using a straight up matte ribbox, then rendering an occlusion pass to cheat the hair shadows in MR. Not exactly ideal, but the alternative is to render everything in a single pass and learn how to tune RFM's sss on deadline which I'm not thinking is the greatest idea.


I tried a couple of things tonight with no joy. I tried picking apart usebackground.h for a technique I could make into a separate shader or ribbox. I tried piping the alpha out of a use background through luminance/clamp nodes then onto a surface shader... the clamping softened the artifact edges at least, but the artifacts are still there. It's almost like there's a depth limit to deep shadows, kind of like raytracing depth. After a few hours I made the executive decision to just make the purchase and brute force a solution instead of losing more time in R&D.

Hopefully Pixar'll release an update soonish.


07 July 2009, 11:02 AM
Although it will probably never happen, if someone wants to write a list of things for an ultimate renderman useBackground-esc shader, i'll happily whip one up in my spare time.

We use a pretty customized workflow at work, so i never really have to deal with maya's standard shaders, which seems like a good idea from what ive read :)


07 July 2009, 06:34 PM
That would be great nick!

In reality, the default Maya usebackground shader works great as is, the problem is that RM translates it differently, and as far as I'm concerned, it can't be used for a proper footage workflow.

Take a look at this scene and you'll see what I mean.

07 July 2009, 07:51 PM
I'll second that. It'd be amazing if you could whip up an .sl that could handle it (for maximum flexibility).

I guess for a use background type shader, I just think straight-up utility instead of anything fancy (as long as it works). Simpler is better when you're doing layer overrides and such.


- Works with shadows (toggleable)
- Works with reflections (toggleable)
- Accepts shadow map (including deep shadow), raytraced and GI/AO shadowing.
- Would be great to be able to split the Env GI/AO/conventional shadowing into discrete outputs via maya methods (ie layer overrides -- I'm not a fan of Maya's pass workflow) Maybe just a checkbox toggle to turn each on/off?

Am I missing anything? Nothing really fancy here besides separating out the shadow types (if that's possible).


I was pretty blown away how good I was able to make fur/hair look knowing almost nothing going in. 4 minute rendertimes with insane long hair counts at 720p don't suck, either. I was looking at 35-90 minutes a frame depending on the technique between MR/Maya hair and Shave's various methods for this particular project, and they didn't look anywhere near as good.

Going to dive headfirst into learning to write shaders & get my hands dirty with Renderman in earnest over next week... I'll be dragging a book or two to Siggraph with me. Any suggestions?


07 July 2009, 10:43 AM
I can promise nothing, but this doesnt sound to hard. Will post back, at some point :)
Just as a sugguestion in the mean, you can get aovs for:
- Shadows (should pass raytraced, depth maped, and deep shadows all to this)
- amb Occlusion
- Indirect (color bleeding)

So, with some fancy compositing, you should be able to achive what ever it is you need to do?


Re: renderman books:


Advanced Renderman

also Fundza is the BEST online resource for all that is renderman. Downlaod cutter from there too, has loads of basic shaders, good intro to building them.

08 August 2009, 12:15 AM
So, I've dug into RSL a bit since returning from Siggraph, and I'm starting to realize that creating a Use Background type shader as an .SL shader declaration might be more difficult than I'd assumed.... mostly because the PRMan workflow is just different. I think they're just assuming you'd output AOVs and just handle everything in comp.

I've figured out how to capture shadow data, even add an AO pass within a shader with its own custom multiplier etc etc, but outputting a decent alpha channel seems to be quite a trick, while still matting out objects (but with alpha 0) behind the surface with the shader attached.

It seems that setting Oi affects the overall transparency of the object, letting objects behind through, and you can't use the alpha keyword directly within a shader (like the 3DL/BMRT examples do) or the shaders won't compile. And I really need to be able to set the opacity to 1 over the entire object (to matte out stuff behind it), but control the alpha output on a per-point basis.

It looks suspiciously like 3DL is actually writing out AOV's then projecting them back onto the object from looking at their source code.

Anyhow, I'm hammering away at the problem still as an excuse to dig deeper into RSL, but I haven't made very good progress yet. If anyone's got suggestions as to an approach to take, I'm all ears.

Oh, and thanks ndeboar for the Advanced Renderman suggestion. Picked it up from the Sigg bookshop and struggled valiantly to stay awake reading all that math without a compiler to play with at hand. It's helping tremendously since I got back to the studio though. :cool:


08 August 2009, 10:50 PM
Just a quick bump to see if anyone's found a decent solution to this.

I've pretty much got a handle on AOV's now but creating a separate holdout group and chopping up alphas on 5 billion AOVs after doing subtract math is just hacky & silly (then you gotta invert everything). There's got to be a better way, or maybe I'm just approaching this wrong.

Scene: HDRI environment, sun keylight, sphere, poly ground plane.

I want to comp my sphere on the background with direct and indirect shadows on the groundplane, and it not have to be (quite literally) 50,000 noodles in nuke or shake.

The only way I can figure out how to do it now is to either add an ID pass and chop everything up with alphas (problematic due to the ground being in the diffuse AOV and matte edges won't be 100% clean), or run a second alpha pass with a separate object set and no ground... but that still leaves me alpha-chopping up about 15 aov's just to make a simple comp, and to do it right it'll have me calculating my AO/raytraced shadows twice (with and without the floor for a perfect premulted matte).

There's gotta be a more elegant solution to this. Am I missing something? Ideally it'd be a workflow for plain-old RFM or just an slo shader, but I'm the proud owner of RMS and RPS here now, so slim or RIB hacks are ok too.


08 August 2009, 09:23 AM
Ok, so I've been banging my head at this and came up with an easy hack that will help folks with footage/photo background workflows get what they need easier at least. I'd still like a full-on useBackground shader that scales shadow opacity/recognizes all shading types/doesn't suck, but this should work without many hoops or additional render time.

I'm going into pretty much step-by-step detail here as just about everything that can be googled regarding this issue is out of date -- variable names, syntax, where RIBBoxes are inserted, etc. This is has been tested on Renderman for Maya 3.0.1 / Renderman Studio 2.0.1.

There might be a better/easier/less stupid way to do it, but I'm still a noob at this, so go easy :shrug:


This is assuming that you're going to be comping over photo/footage. You can project your footage/background as a texture onto your matte objects so that you can get quasi-accurate reflections, color bleeding etc etc, and it works in both Renderman for Maya and Studio (no slim needed). Works kinda like mip_matteshadow that way.

Once you get the idea of how it works, you can change out conditions based on other passes etc.

1. Make your matte background any shader type you like as long as it is capable of receiving diffuse illumination and/or reflections as needed. White lamberts or blinns are ideal for this. Yeah, you lose a little bit of the texture due to diffuse/reflection, but it's good enough in most cases.

2. As mentioned, you can project your footage onto your matte objects if you've modeled a set or floor for shadows/reflections/holdout stuff or whatever. If that's the case, you can leave "visible in relections/refractions" on on the shape node. If you're comping within an HDRI environment sphere, turn reflection/refraction visibility off on your matte objects.

3. Go to your render globals, on the passes tab under final defaults add all the outputs you need. Shadows are included in everything with the keywords "Shadow" or "Occlusion" ... colorbleeding in keyword "Environment" and "Occlusion"... or just activate all the AOV passes like I do. They don't take any extra time and they come in super handy in comp.

4. In the uppermost Pass menu of your render globals, do Create Pass->Reference Render. Once you do that, right click in the camera slot and add your shot cam. Make sure resolution is set the same as your render globals, and that your outputs (rgba, exr etc) are set there as well.

5. Dig and find the shading group for your matte shader and objects you've assigned it to. Go to Attributes->Renderman->Add Rib Box.

6. Change the RIBBox to TCL, and enter the following (you may have to paste it from notepad or textedit etc if the box is feeling flaky):

[if {$PASSCLASS=="ReferenceRender"} {return "Matte 1"} else {return "Matte 0"}]

Fire off a render, then check to see if it worked. If it didn't, check the shading group again and enter the command with backslashes before the square braces like:

\[if {$PASSCLASS=="ReferenceRender"} {return "Matte 1"} else {return "Matte 0"}\]

Near as I can tell, it's really picky about being escaped some of the time and flakes out others and just empties the box.

If it worked, you'll have:

- A beauty render (including your background image, WITH NO ALPHA MASK)
- In the (project)/renderman/(scene)/data/(frame) folder, a reference render with a perfect alpha to use
- Shadows & reflections for everything in AOV passes.. you should be able to extract your direct and occlusion shadowing, reflections, etc using the alpha (or inverse alpha) in there
- No additional render time and minimal setup if Maya's in a good mood.

If it didn't work, check the prior steps and keep trying to get the box to accept the TCL command. It's quite picky. And random.


There appears to be a way to swap out shading groups with an adaptation control (which I haven't gotten to work yet -- zero documentation) as well as a way to automate this via MEL pre- and post-shape scripts, but I'm still getting my bearings. So much of this stuff isn't covered in the docs, and most of the forum post style info out there is woefully out of date so it's slow going. Lots of this stuff got migrated from Slim to Maya recently and works a bit differently.

Hope it helps someone,


CGTalk Moderation
08 August 2009, 09:23 AM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.