masking problem


#1

Hi everyone,

I’m a beginner with Nuke and I wonder if someone could help me.

I’d like to render my 3d scene in one go in maya/Arnold and do some colour fine tuning in Nuke with some masks embedded in a EXR format.
I did some test with some simple 3D primitives and I have some issues on the edges of the colour graded objects.

It looks like my masks are not large enough. I’ve already tried to unpremultiply an area with a mask and merge it over the background but I still have the problem.

Cheers


#2

How are you creating the masks?

Many people use the Object id or material id aov’s but they are very poor for masks as they are not anti-aliased. I would suggest writing regular alpha masks or rgb masks.


#3

The antialiased edges are the usual problem for mask-based corrections indeed.

This happens, because as the color of the corrected object mixes with the surrounding on the edge, the mask does exactly the same. And some peripheral pixels, containing a mix of the object’s color are masked only partially. So, with strong corrections, it gets obvious, that some remains of the initial color stay under-corrected.

With unpremultiplication, you can completely solve this issue on the edges, adjacent to the transparent background. But with the edges between different objects, it’s more complex and the solution depends on the particular situation…


#4

Thanks a lot for your reply guys. much appreciated.

@beaker, I thought the same about the AOVs but I have the same issue when I render the object alone with an alpha.

@GringoFX, Ok it makes sense now. So what is the common solution to minimise the problem? what about brightening the alpha so the antialiasing become more solid.

cheers


#5

For the edges between the object and the transparent background, mere unpremultiplication is enough. This way you completely restore the color of the object in the antialiased area and the mask (indeed, you need to unpremultiply both the mask and the object’s RGB).

For the edges between the objects, you can do a dirty trick, expanding the mask with one of the Erode filters (maybe, the erode itself can be masked with the masks of the adjacent objects).

To make a really clean solution, you would need a pass with all the objects, excluding the corrected one. Then you could subtract the surrounding objects' colors from the edge of the corrected one and divide the object's RGB by the mask (in Nuke's terminology "unpremultication" means in fact division by the Alpha, so in this case, the division with the mask is a similar operation).
As you've divided the RGB by the mask, you can apply the correction, then multiply the RGB with the mask and merge the corrected object into the pass with the others.

If the colors of the surrounding objects are quite even (without too many details), you can getaway with expanding the surrounding color to extract it from the corrected object’s edges (and vice versa), divide the object’s RGB by the mask, perform the correction, then merge back into the source image using the mask.

This looks like a lot of text and may be hard to follow, so feel free to ask something in particular or upload the image with the mask, so that I made an example setup.


#6

Thanks a lot for your help !

I think I understand the different techniques.

I’ll have a try but I’d like to avoid rendering a pass per object if possible.


#7

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.