CGTalk > Software Specific Forums > Autodesk Maya > Maya Rendering
Login register
Thread Closed share thread « Previous Thread | Next Thread »
 
Thread Tools Search this Thread Display Modes
Old 08-09-2012, 01:36 AM   #1
chrishecker
New Member
Chris Hecker
Technical Director
definition six, inc.
Oakland, CA, USA
 
Join Date: Feb 2005
Posts: 21
avoiding fringe when rendering for composition with global illumination in mental ray

Hi, I tried searching for this, but it's kind of hard to explain in a searchable way, so hopefully I didn't miss something obvious...

I have a character. I want to render him, and then composite him into a picture in photoshop/illustrator later, so I need him to have an alpha mask in the rendered image. But, I am using mental ray with global illumination and use the camera's Environment Background color for tuning the lighting. I'd like a relatively bright background color for rendering because it's used for the GI and it looks better, but I'd like to composite the character over a dark background.

The problem is when rendering at high resolutions (this is for print), the edge pixels of the character have a fringe where there's alpha for anti-aliasing partial coverage. This fringe looks like mental ray is blending to the camera's background color, which makes sense when a pixel is partially on the character and partially on the background, but since the alpha is non-0 there, it looks wrong against a dark background.

So, it seems like there are a few options I can think of:

1. set the camera environment background color to match the composition background to get rid of the fringe, and try to recreate the nice look with lights and big diffuse reflectors, but this is not easy

2. put a dark shape behind the character so the lighting environment stays the same but the fringe goes away since edge pixels will blend to the dark shape's color, but then how to I get the right mask? I can't render twice easily at full resolution given my time constraints
2a. well, I might be able to use Geometry Matte preset since it's fast if I can figure out how to get it to go through mental ray, since it appears to use the maya software renderer and so the subd surfaces don't match up, but I can't figure this out (discussed here: http://forums.cgsociety.org/showpos...616&postcount=2)

3. some other idea I haven't thought of?

Thanks for any help you can give me,
Chris
 
Old 08-09-2012, 08:11 AM   #2
nookie
Know-it-All
 
nookie's Avatar
portfolio
Luca Fiorentini
Lighting Artist
Lucasfilm Animation
Singapore, Singapore
 
Join Date: Jan 2004
Posts: 367
Send a message via MSN to nookie
If you want to have different backgrounds for different rays (eye, reflection, fg ecc ecc) you can use the rayswitch shader.
Here a few link to have an idea about it
Code:
http://www.pixelcg.com/blog/?p=665 http://www.puppet.tfdv.com/tutorials/ray_type_optimization_e.shtml

but googling gives you a lot of results
That is a production shader that is hidden by default, you can create it by command line or just unhide them (google is a good friend also there)

A part from that, it really seems like a premultiply/ unpremultiply issue
 
Old 08-09-2012, 05:17 PM   #3
Cuni
Frequenter
portfolio
Lluis Barcelo
Eternal Learner
Mexico
 
Join Date: Sep 2006
Posts: 239
You can try to get an ID pass, clean and smooth.
 
Old 08-09-2012, 07:24 PM   #4
chrishecker
New Member
Chris Hecker
Technical Director
definition six, inc.
Oakland, CA, USA
 
Join Date: Feb 2005
Posts: 21
Quote:
Originally Posted by nookie
If you want to have different backgrounds for different rays (eye, reflection, fg ecc ecc) you can use the rayswitch shader.

But it looks like you can't have "nothing" in one of the rayswitch slots, letting the rays pass through. It just renders black in the example video if nothing's hooked up?

Quote:
Originally Posted by nookie
A part from that, it really seems like a premultiply/ unpremultiply issue

Yeah, that's what I thought, but I'm not sure how to fix it at that level.

Quote:
Originally Posted by Cuni
You can try to get an ID pass, clean and smooth.

Will that be anti-aliased, and done through the same subd tesselation as mentalray? How do you do that?

Thanks,
Chris
 
Old 08-09-2012, 08:54 PM   #5
m0z
SOuP cook
 
m0z's Avatar
portfolio
Mathias Markovits
CG Allrounder
ViennaPaint
Austria
 
Join Date: Apr 2010
Posts: 413
if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!
__________________
mathiasmarkovits[dot]com
 
Old 08-09-2012, 09:30 PM   #6
Cuni
Frequenter
portfolio
Lluis Barcelo
Eternal Learner
Mexico
 
Join Date: Sep 2006
Posts: 239
Quote:
Originally Posted by chrishecker

Will that be anti-aliased, and done through the same subd tesselation as mentalray? How do you do that?

Thanks,
Chris


May sound silly, but the fastest way is to save another scene and render just a 8bit tif without light, without raytrace nor GI or FG ... and use a surface shader.
 
Old 08-10-2012, 04:26 AM   #7
pap87
Lord of the posts
 
pap87's Avatar
portfolio
John Papafotiou
Lighting TD
Wellington, New Zealand
 
Join Date: Dec 2005
Posts: 1,441
Quote:
Originally Posted by m0z
if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!


What m0z said.
 
Old 08-12-2012, 01:08 AM   #8
JasonA
Expert
 
JasonA's Avatar
portfolio
Jason Adank
3D Artist
Portland, USA
 
Join Date: May 2002
Posts: 2,331
Quote:
Originally Posted by m0z
if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...

Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!


This is definitely the best option, but Adobe products (ala After Effects) handle alphas in very unintuitve ways. if youre in Nuke it all makes sense.
 
Old 08-12-2012, 08:33 AM   #9
chafouin
Frequenter
 
chafouin's Avatar
portfolio
Fabien Christin
France
 
Join Date: Dec 2009
Posts: 239
If this is a premultiplied problem in photoshop, maybe you can use this:

Layers > Matting (at the very bottom) > Remove Black Matte or White Matte, or Defringe.
 
Old 08-13-2012, 07:15 PM   #10
tobbew
Know-it-All
Torbjörn
Sweden
 
Join Date: Apr 2005
Posts: 434
If you want it to pass through when using a ray switch you can use "transmat".
 
Old 08-15-2012, 11:43 PM   #11
chrishecker
New Member
Chris Hecker
Technical Director
definition six, inc.
Oakland, CA, USA
 
Join Date: Feb 2005
Posts: 21
Thanks everybody, here are multiple replies!

Quote:
Originally Posted by chafouin
If this is a premultiplied problem in photoshop, maybe you can use this:
Layers > Matting (at the very bottom) > Remove Black Matte or White Matte, or Defringe.

There are problems with these...the first two only work for black or white backgrounds, and defringe eats a pixel and has alpha threshholds in its algorithm so it can screw up stuff like translucent glass. I'd much rather do this correctly on the render side if possible.

Quote:
Originally Posted by m0z
if you're in MentalRay and rendering just for print, turn off Premultiply @ Render Settings -> Quality -> Framebuffer...
Then just premultiply the Beauty pass with the Alpha in Photoshop. Clean borders!

I found this setting, but apparently it only works for black camera environment backgrounds. There's still a fringe if you set the camera environment color.

I found a thing in the maya docs about how compositing only really works right if you have a black background, and I'm definitely finding that to be the case, having tried a ton of different variations.

This leaves me with the problem of how to reproduce the affects that the camera environment background color has with normal lights. It has a significant but subtle impact when you're doing GI and FG, so I'm not sure how to reproduce that but still keep the background color black. Does anybody know a trick for doing this?

Assuming that's hard, I'm thinking a workaround might be to keep that camera color non-black for the GI effects, but then put a black quad behind the scene (far enough away that it doesn't impact lighting or cast shadows), and then try to use Cuni's trick to get a reasonably fast alpha mask for the scene. This would have the advantage of not actually having alpha set whereever the rays hit the background, which is currently the case, so if there's a reflection/refraction to the background it's got alpha right now.

Quote:
Originally Posted by tobbew
If you want it to pass through when using a ray switch you can use "transmat".

Hmm, can you have it pass through for the alpha, but not the color? Or, wait, I'd want it to to be black for direct ray colors, transparent for alpha, not block lights, but have an effect on GI/FG.

Nothing is every simple, sigh.

Chris
 
Old 08-16-2012, 12:45 AM   #12
chrishecker
New Member
Chris Hecker
Technical Director
definition six, inc.
Oakland, CA, USA
 
Join Date: Feb 2005
Posts: 21
Quote:
Originally Posted by chrishecker
Nothing is every simple, sigh.

Okay, famous last words, but I think I have cobbled together a rendering algorithm to solve this:

1. use the camera environment color and GI/FG all you want
2. put a black surface shader quad behind the scene so the entire background is black, mark it non-casting, and I suppose you could get really fancy and use rayswitch on it if you want
3. turn off premultiply (actually, this doesn't matter now that I think about it, since there's no alpha in the image), render normally, your alpha is all 1 and useless because of the quad, but you have a black background
4. use Cuni's trick of 8bit tiff, no raytrace effects (GI/FG), select everything and put a white surface shader on it, delete the black quad
5. render this scene, the rgb will be aliased, but the alpha channel will be the right mask
6. in photoshop, take the alpha from the mask render as your selection, lift the background of the color render to a layer, invert the selection, delete to clear the background, and poof, you have something that composites perfectly against all colors, yet has the nice GI warmth
7. you can multiply the mask in photoshop if you want the resulting image to have premultiplied alpha if you're going to use a non-broken program for compositing

This is pretty clunky, but it seems to work.

Hmm, wait, one issue...any glass held in front of the background won't have the correct alpha to show the composited background color...ugh. Maybe mip_rayswitch on the background quad could fix this, and then you multiply the masks...

Chris
 
Old 08-16-2012, 08:04 PM   #13
Bitter
Also Highly Corrosive
 
Bitter's Avatar
David Fer Real
Fixer of things
USA
 
Join Date: Apr 2003
Posts: 1,720
That sounds pretty heinous. . .

m0z and chafouin are basically correct, your image is not being un-multiplied in post by Photoshop.

Nuke does this automatically and basically correctly.

This isn't really a mental ray or rendering problem. It's a Photoshop problem.

I would avoid using "premultiply off" since some shaders rely on this for correct results. I would also render to 32-bit float (16-half) when possible and onto black (rayswitches are a good idea here) If you are rendering to 8-bit directly you need to bake in your colorspace in the render to avoid more trouble later when correcting it (especially doing color correction on a premultiplied image which is a no-no and enhances fringing)
__________________
My opinions are always my own...and maybe a friend's, but never my employer's.
 
Old 08-16-2012, 08:24 PM   #14
chrishecker
New Member
Chris Hecker
Technical Director
definition six, inc.
Oakland, CA, USA
 
Join Date: Feb 2005
Posts: 21
Quote:
Originally Posted by Bitter
This isn't really a mental ray or rendering problem. It's a Photoshop problem.

As much as I love to bash photoshop, and it's almost always well-deserved, I actually don't think so in this case. The problem is maya/mentalray isn't handling eye rays that go off into infinity correctly for compositing, so you're forced to have a black there so their math comes out right (or muck with mip_rayswitch, maybe, haven't tried to fix it that way). The docs basically say this, subtly. They could have handled this case correctly, but didn't.

Quote:
Originally Posted by Bitter
I would avoid using "premultiply off" since some shaders rely on this for correct results.

Does the premultiply setting affect shaders at all? It's in the framebuffer settings, so I figured it was just a final multiply before writing the pixel, or not. I can easily keep it on for the beauty pass, though, like I said, since there's no alpha there anyway.

Still, I wish I could figure out how to deal with transparent objects against the black background. Hrm.

The other alternative is to figure out how to have a black camera environment background color, but still get the nice and easy warmth to the scene with GI/FG without a zillion lights.

Chris
 
Old 08-16-2012, 08:33 PM   #15
Bitter
Also Highly Corrosive
 
Bitter's Avatar
David Fer Real
Fixer of things
USA
 
Join Date: Apr 2003
Posts: 1,720
Quote:
As much as I love to bash photoshop, and it's almost always well-deserved, I actually don't think so in this case. The problem is maya/mentalray isn't handling eye rays that go off into infinity correctly for compositing, so you're forced to have a black there so their math comes out right (or muck with mip_rayswitch, maybe, haven't tried to fix it that way). The docs basically say this, subtly. They could have handled this case correctly, but didn't.


Whatever reaches infinity calls the environment. If your environment has a color, then it gets filtered with the pixel. That's correct. All renderers do this.

So yes, this case is handled "correctly".

Quote:
Does the premultiply setting affect shaders at all?


They can, especially for some effects like volumes. In fact, we are looking to remove the option entirely as it should not be used.

Correct compositing methods will handle these issues correctly.

Also, are you multiplying in post? This is also incorrect. Your passes/effects should all be additive or you risk increasing the fringe/darkening because the post procedure cannot replicate the math during render when the samples are merged to a pixel.
__________________
My opinions are always my own...and maybe a friend's, but never my employer's.
 
Thread Closed share thread


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright ©2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 06:39 PM.


Powered by vBulletin
Copyright ©2000 - 2016, Jelsoft Enterprises Ltd.