using writeToDepthBuffer/writeToColorBuffer nodes with alpha transparency

#1

Hey guys!

So, I’m having some issues with the writeToDepthBuffer nodes. I’m using them to store a custom float value in my images (rendering to EXR on mental ray for maya 2013, comping in Nuke 6.something).

Basically, it works great, apart from objects with textured alpha transparency. With standard depth passes you have various options for dealing with transparency:

  1. setting the camera’s depth type to ‘closest visible depth’, enabling ‘transparency based depth’ and setting a threshold
  2. setting the shading engine’s ‘transparency threshold’ and ticking ‘use shading engine threshold’ on the renderPass
  3. setting the ‘transparency threshold’ directly on the render pass

These all work great on regular depth passes…but none of those seem to have an effect on custom depth passes! 1 doesn’t seem to do anything, and 2 & 3 aren’t even available as options for Custom Depth, Label or Vector renderPasses. Surely there’s a way of doing this?

Here’s an demo of the difference:

Any help would be appreciated, I’ve asked around quite a bit and nobody seems to know!

#2

There is an Alpha input in the writetoColorbuffer but Im not sure if using alpha with depth renders will ever work as you expect. The grey levels created by the alpha will confuse the Z depth value. People even render un-antialiased depth images so that the antialiasing doesn’t provide a invalid depth value.

Maybe you could consider using deep image compositing, where each pixel contains multiple color and alpha values which define the color and depth. This was only available with mantra and renderman but

Here are a couple of links to explain how to do it in maya and nuke

with mentalray but not a true deep image
http://lesterbanks.com/2014/06/mental-ray-deep-image-data/

with vray with a special vray plugin
http://therenderblog.com/deep-compositing-with-vray/

#3

Hi Willanie,

Thanks so much for the prompt reply! I’ve had a post up on a different forum (which shall remain nameless) since 2013 and no replies, and then you got back to me as soon as I posted here
nice!

There is an Alpha input in the writetoColorbuffer

Yes! Nice point, I’d assumed that would just write an alpha value in the same way the RGB values just write, but it turns out that it does affect what gets written. Which is awesome! There isn’t the same option on writeToDepthBuffer, but hey, I’ll just use the color one and only use one channel :slight_smile:

but Im not sure if using alpha with depth renders will ever work as you expect. The grey levels created by the alpha will confuse the Z depth value.

Yup, you’re quite right, but actually I can get the same functionality as the built in ‘transparency threshold’ function by putting a condition node on the alpha channel before it goes into the writeToColorBuffer. It seems like it’s adding rather than overwriting, but managed to get round that by premultiplying by the thresholded alpha channel. So
cautiously optimistic!

People even render un-antialiased depth images so that the antialiasing doesn’t provide a invalid depth value.

yup, that’s exactly what I’m doing actually, just rendering out everything without antialiasing at double resolution, then scaling it down after all the post is done. Simple, brute force - only downside is I had to buy a 3 terabyte hard drive because the files are massive :thumbsup:

Maybe you could consider using deep image compositing, where each pixel contains multiple color and alpha values which define the color and depth. This was only available with mantra and renderman but
Here are a couple of links to explain how to do it in maya and nuke
with mentalray but not a true deep image
http://lesterbanks.com/2014/06/ment...eep-image-data/
with vray with a special vray plugin
http://therenderblog.com/deep-compositing-with-vray/

Yeah, can’t wait for deep compositing to be the norm, z-depth is a silly way of doing these things :slight_smile: Unfortunately it’s not quite there yet - I’m committed to using mental ray on this one, and although the lesterbanks video has some interesting techniques, without a valid depth channel for the 0% alpha areas it’s still going to have the same problems I had before. But no worries - just gonna use writeToColorBuffer with alpha channel and see how it goes

Thanks again! :cool:

d

#4

Good news everyone! Finally figured it out. What a pain.

So, without further ado, here is “How to set up label, depth, or other custom passes to work with transparency in Maya with Mental Ray”. Which doesn’t seem to exist anywhere on the internet :slight_smile:

So, first off, label & depth passes ain’t gonna work with any kind of interpolation, so you’re gonna have to turn it off everywhere. I’m just rendering everything out at 2x resolution and scaling it down later after post, works fine and really simple to set up. If you have the disk space.

The system I’ve got set up works great for materials with some 100% solid areas and some 0% solid areas, but isn’t going to work with anything that’s partially transparent, or has coloured transparency, and obviously doesn’t take other transparent effects like reflection and specular highlights into account. I’m just using lamberts here so it’s no problem for me, but YMMV innit. You may be able to fudge something to work around these issues, let us know if you do!

Right, so, here’s a basic outline.

[ol]
[li]Set your renderer to mental ray, make sure you set the frame buffer to 4x32bit, set antialiasing to “fixed” and “0” and disable “interpolate samples”. And filetype to OpenEXR if u want it work :slight_smile:
[/li][li]Add a customColor render pass
[/li][li]Look at whatever is going into the transparency input on your material (lambert3 in the pic). In my example case it was just a bulge texture. Right, so we’re going to need to force the alpha to be exactly 1 or 0 and nothing in between (ie, we’re creating a transparency threshold manually). We do that by inserting a condition node (transparencyThresholdCondition in the example above) between the two nodes, connect the input to “First Term”, set “Second term” to 0.5 (or whatever you want the threshold to be) and operation to “Less Or Equal”, and then connect the output back to the shader. Ok. That’s our transparency threshold.
[/li][li]Right, now we make a writeToColorBuffer node (customLabel_writeToColorBuffer2 in the example). Make sure it’s set up as in the example - “Evaluation Mode” to “Always” and “Write Operation” to “Add”, and don’t forget to hook it up to the correct pass (select the “custom Color Pass” in the drop down).
[/li][li]Now, the ridiculous step that was eluding me for ages, because AUTODESK DIDN’T MENTION IT IN THE MANUAL thanks guyz :slight_smile: That “Write Operation” you just set? Ain’t gonna do anything until you tick “Use Transparency” in the render pass node’s settings. Doh!
[/li][li]Make a singleSwitch or tripleSwitch node (sorry, forgot to show it in the example) and connect up all the objects, and set the inSingles to whatever you want the labels to be.
[/li][li]Ok, we’re getting there. So, normally we’d just connect the switch node’s output to the color attribute of the writeToColorBuffer node, but because we have to deal with transparency, and its set to add, that’s gonna add up where it shouldn’t. So, to counteract that, we’re gonna essentially “premultiply” the values, just as you would in 2D compositing if you wanted to do an ‘OVER’ compositing operation. So, make a multiplyDivide node (multiplyDivide2 in the example) and connect the shader’s outMatteOpacityR to input 1, and the switch nodes output to input 2. Yeah! However, because of this, we’re gonna need a separate writeToColorBuffer + multiply + condition setup for each shader that uses this…doh! Script it yeah?
[/li][li]Right, to actually make this work, you seem to need to connect the writeToColorBuffer directly to the shader network (I thought the connection via the multiply node would be enough, but it doesn’t seem to be, maybe somebody could explain that?). Anyway, just slap the outMatteOpacityR to the Alpha input on the writeToColorBuffer, and we’re all good.
[/li][/ol]

Hopefully that’s enough to give you a basic idea of the process I used. Obviously I skipped the basics, hope I didn’t miss anything important. You probably need a pretty good understanding of maya to get it running…I was definitely struggling a bit there! But the end result works - you can see it in Nuke at the bottom of the example, working great :slight_smile:

Seriously, why is that so frickin’ fiddly to do? I just wanted labels for compositing! It almost makes me wish I was a 2D animator. Almost :stuck_out_tongue: