avoiding fringe when rendering for composition with global illumination in mental ray

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

THREAD CLOSED
 
Thread Tools Search this Thread Display Modes
  08 August 2012
avoiding fringe when rendering for composition with global illumination in mental ray

Hi, I tried searching for this, but it's kind of hard to explain in a searchable way, so hopefully I didn't miss something obvious...

I have a character. I want to render him, and then composite him into a picture in photoshop/illustrator later, so I need him to have an alpha mask in the rendered image. But, I am using mental ray with global illumination and use the camera's Environment Background color for tuning the lighting. I'd like a relatively bright background color for rendering because it's used for the GI and it looks better, but I'd like to composite the character over a dark background.

The problem is when rendering at high resolutions (this is for print), the edge pixels of the character have a fringe where there's alpha for anti-aliasing partial coverage. This fringe looks like mental ray is blending to the camera's background color, which makes sense when a pixel is partially on the character and partially on the background, but since the alpha is non-0 there, it looks wrong against a dark background.

So, it seems like there are a few options I can think of:

1. set the camera environment background color to match the composition background to get rid of the fringe, and try to recreate the nice look with lights and big diffuse reflectors, but this is not easy

2. put a dark shape behind the character so the lighting environment stays the same but the fringe goes away since edge pixels will blend to the dark shape's color, but then how to I get the right mask? I can't render twice easily at full resolution given my time constraints
2a. well, I might be able to use Geometry Matte preset since it's fast if I can figure out how to get it to go through mental ray, since it appears to use the maya software renderer and so the subd surfaces don't match up, but I can't figure this out (discussed here: http://forums.cgsociety.org/showpos...616&postcount=2)

3. some other idea I haven't thought of?

Thanks for any help you can give me,
Chris
 
  08 August 2012
If you want to have different backgrounds for different rays (eye, reflection, fg ecc ecc) you can use the rayswitch shader.
Here a few link to have an idea about it
http://www.pixelcg.com/blog/?p=665
http://www.puppet.tfdv.com/tutorials/ray_type_optimization_e.shtml

but googling gives you a lot of results
That is a production shader that is hidden by default, you can create it by command line or just unhide them (google is a good friend also there)

A part from that, it really seems like a premultiply/ unpremultiply issue
 
  08 August 2012
You can try to get an ID pass, clean and smooth.
 
  08 August 2012
Originally Posted by nookie: If you want to have different backgrounds for different rays (eye, reflection, fg ecc ecc) you can use the rayswitch shader.

But it looks like you can't have "nothing" in one of the rayswitch slots, letting the rays pass through. It just renders black in the example video if nothing's hooked up?

Originally Posted by nookie: A part from that, it really seems like a premultiply/ unpremultiply issue

Yeah, that's what I thought, but I'm not sure how to fix it at that level.

Originally Posted by Cuni: You can try to get an ID pass, clean and smooth.

Will that be anti-aliased, and done through the same subd tesselation as mentalray? How do you do that?

Thanks,
Chris
 
  08 August 2012
if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!
__________________
mathiasmarkovits[dot]com
 
  08 August 2012
Originally Posted by chrishecker:
Will that be anti-aliased, and done through the same subd tesselation as mentalray? How do you do that?

Thanks,
Chris


May sound silly, but the fastest way is to save another scene and render just a 8bit tif without light, without raytrace nor GI or FG ... and use a surface shader.
 
  08 August 2012
Originally Posted by m0z: if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!


What m0z said.
 
  08 August 2012
Originally Posted by m0z: if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!


This is definitely the best option, but Adobe products (ala After Effects) handle alphas in very unintuitve ways. if youre in Nuke it all makes sense.
 
  08 August 2012
If this is a premultiplied problem in photoshop, maybe you can use this:

Layers > Matting (at the very bottom) > Remove Black Matte or White Matte, or Defringe.
 
  08 August 2012
If you want it to pass through when using a ray switch you can use "transmat".
 
  08 August 2012
Thanks everybody, here are multiple replies!

Originally Posted by chafouin: If this is a premultiplied problem in photoshop, maybe you can use this:
Layers > Matting (at the very bottom) > Remove Black Matte or White Matte, or Defringe.

There are problems with these...the first two only work for black or white backgrounds, and defringe eats a pixel and has alpha threshholds in its algorithm so it can screw up stuff like translucent glass. I'd much rather do this correctly on the render side if possible.

Originally Posted by m0z: if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...
Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!

I found this setting, but apparently it only works for black camera environment backgrounds. There's still a fringe if you set the camera environment color.

I found a thing in the maya docs about how compositing only really works right if you have a black background, and I'm definitely finding that to be the case, having tried a ton of different variations.

This leaves me with the problem of how to reproduce the affects that the camera environment background color has with normal lights. It has a significant but subtle impact when you're doing GI and FG, so I'm not sure how to reproduce that but still keep the background color black. Does anybody know a trick for doing this?

Assuming that's hard, I'm thinking a workaround might be to keep that camera color non-black for the GI effects, but then put a black quad behind the scene (far enough away that it doesn't impact lighting or cast shadows), and then try to use Cuni's trick to get a reasonably fast alpha mask for the scene. This would have the advantage of not actually having alpha set whereever the rays hit the background, which is currently the case, so if there's a reflection/refraction to the background it's got alpha right now.

Originally Posted by tobbew: If you want it to pass through when using a ray switch you can use "transmat".

Hmm, can you have it pass through for the alpha, but not the color? Or, wait, I'd want it to to be black for direct ray colors, transparent for alpha, not block lights, but have an effect on GI/FG.

Nothing is every simple, sigh.

Chris
 
  08 August 2012
Originally Posted by chrishecker: Nothing is every simple, sigh.

Okay, famous last words, but I think I have cobbled together a rendering algorithm to solve this:

1. use the camera environment color and GI/FG all you want
2. put a black surface shader quad behind the scene so the entire background is black, mark it non-casting, and I suppose you could get really fancy and use rayswitch on it if you want
3. turn off premultiply (actually, this doesn't matter now that I think about it, since there's no alpha in the image), render normally, your alpha is all 1 and useless because of the quad, but you have a black background
4. use Cuni's trick of 8bit tiff, no raytrace effects (GI/FG), select everything and put a white surface shader on it, delete the black quad
5. render this scene, the rgb will be aliased, but the alpha channel will be the right mask
6. in photoshop, take the alpha from the mask render as your selection, lift the background of the color render to a layer, invert the selection, delete to clear the background, and poof, you have something that composites perfectly against all colors, yet has the nice GI warmth
7. you can multiply the mask in photoshop if you want the resulting image to have premultiplied alpha if you're going to use a non-broken program for compositing

This is pretty clunky, but it seems to work.

Hmm, wait, one issue...any glass held in front of the background won't have the correct alpha to show the composited background color...ugh. Maybe mip_rayswitch on the background quad could fix this, and then you multiply the masks...

Chris
 
  08 August 2012
That sounds pretty heinous. . .

m0z and chafouin are basically correct, your image is not being un-multiplied in post by Photoshop.

Nuke does this automatically and basically correctly.

This isn't really a mental ray or rendering problem. It's a Photoshop problem.

I would avoid using "premultiply off" since some shaders rely on this for correct results. I would also render to 32-bit float (16-half) when possible and onto black (rayswitches are a good idea here) If you are rendering to 8-bit directly you need to bake in your colorspace in the render to avoid more trouble later when correcting it (especially doing color correction on a premultiplied image which is a no-no and enhances fringing)
__________________
My opinions are always my own...and maybe a friend's, but never my employer's.
 
  08 August 2012
Originally Posted by Bitter: This isn't really a mental ray or rendering problem. It's a Photoshop problem.

As much as I love to bash photoshop, and it's almost always well-deserved, I actually don't think so in this case. The problem is maya/mentalray isn't handling eye rays that go off into infinity correctly for compositing, so you're forced to have a black there so their math comes out right (or muck with mip_rayswitch, maybe, haven't tried to fix it that way). The docs basically say this, subtly. They could have handled this case correctly, but didn't.

Originally Posted by Bitter: I would avoid using "premultiply off" since some shaders rely on this for correct results.

Does the premultiply setting affect shaders at all? It's in the framebuffer settings, so I figured it was just a final multiply before writing the pixel, or not. I can easily keep it on for the beauty pass, though, like I said, since there's no alpha there anyway.

Still, I wish I could figure out how to deal with transparent objects against the black background. Hrm.

The other alternative is to figure out how to have a black camera environment background color, but still get the nice and easy warmth to the scene with GI/FG without a zillion lights.

Chris
 
  08 August 2012
Quote: As much as I love to bash photoshop, and it's almost always well-deserved, I actually don't think so in this case. The problem is maya/mentalray isn't handling eye rays that go off into infinity correctly for compositing, so you're forced to have a black there so their math comes out right (or muck with mip_rayswitch, maybe, haven't tried to fix it that way). The docs basically say this, subtly. They could have handled this case correctly, but didn't.


Whatever reaches infinity calls the environment. If your environment has a color, then it gets filtered with the pixel. That's correct. All renderers do this.

So yes, this case is handled "correctly".

Quote: Does the premultiply setting affect shaders at all?


They can, especially for some effects like volumes. In fact, we are looking to remove the option entirely as it should not be used.

Correct compositing methods will handle these issues correctly.

Also, are you multiplying in post? This is also incorrect. Your passes/effects should all be additive or you risk increasing the fringe/darkening because the post procedure cannot replicate the math during render when the samples are merged to a pixel.
__________________
My opinions are always my own...and maybe a friend's, but never my employer's.
 
Thread Closed share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright ©2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 03:52 PM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.