Unwanted halo when using alpha

[Creamdog]

On second thought, it looks almost as if the semi-transparant parts glow up. So not the 100% visible or invisible parts, just the half-transparent parts of the same texture.

[Ls3D]

Your post is a bit confusing and with no example image I can only throw you a guess or opinion as to whats is happening.

It sounds like there is a light value at the edges of your alpha map which is probably the result of anti-aliasing against a lighter background at some point in your workflow. (photoshop?) Take a close look at your alpha map edges.

In some cases you can disable max renderings from anti-aliasing against the background, but I don't think this is your issue.

Ther are many things that could be going on, so post a render for better help if you can.

BTW - Nice physical models on your site. What an iddy bitty world.

-S

[Rens]

Sounds like you used a pre-multiplied alpha instead of a straight alpha. Pre-multiplied alphas have colour information from the background multipled with the transparent parts, so when you have a pre-multiplied alpha created on a light background and want to composite it on a black background then you'll see the light background in the transparent parts.

To fix this you should use another method (straight/non-pre-multiplied) of saving your alpha, if possible. What might also help is to save the alpha using a black background instead of a light background.

Good luck,

- Rens

P.S. Is that a "Gamma roerhoutje" your scupture is leaning on? :D

[Creamdog]

Sorry about the vagueness of my question, I am kind of an unmatched champion in posting vague stuff. I'll try to be a bit more accurate. First of, let me post a pic of the problem:



http://www.creamdog.net/SurPlusImg/palace_renderAlphaproblemsA01.jpg

I hope this explained most of it. What I am trying to do (inspired by some extras on the Mulan 2-disc dvd) is to make a 2,5 D representation of a painted background for an animation. It's just a test by the way, nothing specific. I painted the overall compostion and then saved several different pics for the different layers, cut out those parts and refined it. So, there are seperate images, with seperate alpha channels, for the tree, the rocks, the palace etc. I roughly modelled the geometry and mapped the textures on. It looks good in my viewport, but once rendered, I get unwanted lighter parts. Compare the clouds and edge of the tree with the painting, viewport and render. I assign the material, put the map in the diffuse slot, drag and drop it on the opacity-slot, say "mono channel output --> Alpha", put on 100 % self-illumination (since the lighting is painted in) and that's it. Does this make it any clearer?

Thanks for the compliments guys. I love modeling with clay, have some cool projects on the shelf this moment. And yes... that is a "Gamma roerhoutje", good you noticed. I always like to put something in it that prooves it's handmade. I once made a maquette of building, textured and painted it with a lot of detail. Then I cut a hole in the base, so that, when you turned it over, you could see it was built of pizza-boxes. Don't know why, but I like that. Anyway, hope you guys can help me out.

[Ls3D]

First off the mask you posted has a range of values above black, I can even see brush work at several low shades of grey, the source of your edge problem.

Seems like between Rens & my feedback you have a few things to try.. and the details (method) you layed out are right on (from memory). So if straight alpha does not fix it..

Could it be that your monitor is on the dark side and you just are not seeing subtle problems with your alpha channel? The lighter (and linear) band is particularly suspicious. Have a look with the eye dropper/info in photoshop. Another thing to try is to separate the alpha as a new map, and setup a version of the material using that map.

Also of note, just like vectors (.ai) rasterized into photoshop produce nice clean edges, vectors in most 3d render engines produce crisper edges that mapped opacity, even when oversampling. In other words consider exporting paths and creating billboard cutouts of your layers. You can just load selection and convert to path with .5 pxl radius to get a shape out of photoshop for example. This opens up possililties for volumetrics too!

Lets see what else, O - make sure you have black in diffuse and specular values of that shader also..

-S

[Ls3D]

Just doing some midnight compositing here & when I output premultiplied alpha from Max7 (scanline) and import it into AE 6.5 pro I can set it to straight and get a perfect comp, but if I output from Max as straight (ie without pre-multiplied) I get the halo no matter how I interpret the footage.

Of course I am rendering with a matte shadow material and affecting the alpha with shadows so I can composite in AE with shadows, so perhaps that is part of the equation?

Just of note.

-S

[LoneRobot]

Ls3D,

Ive had the same issue with rendering non-premultiplied out from MAX7. Its confused me because, like you i got the halo on the non premultiplied footage. I thought with an non PM image, you were more or less getting a clip mask of the object. The only way it would work is with a premultiplied alpha render, then it seemed to work out the alpha correctly. I thought it was something to do with the scanline antialiasing, since it looked like the alpha didnt have any AA, but then the halo was created when the RGB render applied the AA settings. (in my case an Area at 2.5) the difference between the alpha and the blurry RGB channel was the 1-2 pixel white area.
So i just did a pre-multiplied render in the end. It was the same in Combustion and AE, so i assumed it must be the order in which max generates the channels - ie the alpha before the AA. confused? i was. I've always done it without PM before for compositing - i know this because our Matrox Digsuite needs PM'ed TGAS to perform realtime overlays and that was the only time i enabled it for renders. :shrug:

[Ls3D]

Hey Pudg, good to know.

Someone asked about straight alpha a while back and I made a gay joke. 16th year in 3D & I'm still learning AND still an A-hole.

Having shadow data in the alpha is working out nicely. Now if I can just figure out why my RPF camera data in AE is wonky (that is a technical term) all of the sudden. No on second thought SLEEP...

-S

[Creamdog]

Lets see what else, O - make sure you have black in diffuse and specular values of that shader also..

Thanks guys, that solved my problem. It took me some time to get behind my pc again, hence the late thank you. You helped me out a lot!

[Ian Jones]

I tried the process described as a solution and found that it does not completely remove the halo. In Max I made a bright green background, rendered both *.png and *.tga with premultiplied alpha channels and brought them into AfterFX 6. I interpreted the footage and selected 'straight - unmatted' which works better than 'premultiplied' as previously mentioned. I found that there was a faint bright green halo of the background colour I had selected in Max, so this proves it doesn't completely dissapear. Then I discovered a solution, effects > channels > remove color matting. You select the colour of the background the same as in Max and voila! it removes the fringing 'halo' completely! You get a clean alpha. I believe it works like 'unpremultiply' (Jon knolls famous 'unmult' plugin does this) or maybe it is using some sort of 'division' mode... I'm not sure but it appears to work really well. Does anyone know for sure if this is the correct compositing process?

[LoneRobot]

i'm sure that would work in many cases but you'd also remove any pixels that are coloured similar to the background colour that are the image. You're not using an alpha channel - its just using the pixel color as a trigger for the transparency. If you wanted to do that you could just render a 16 bit image and matte it like you would if you filmed someone on a green screen.

[stuh505]

Hmm.. I am getting the Halo as well.

To fix this you should use another method (straight/non-pre-multiplied) of saving your alpha, if possible. What might also help is to save the alpha using a black background instead of a light background.

I do not understand this statement, either.

My Alpha is made in Photoshop...I have a picture of a leaf, select around the leaf, and use that selection to make a new channel for the alpha. I then save it as a TGA.

Then I use this same TGA image for the diffuse color and for the opacity, and I get a halo that didn't exist before. Toggling pre-multiplied has NO effect for me.

Now that I think about it...I'm going to try something new when I get home, which is instead of using the 32-bit TGA image as an alpha, I'm going to make the alpha into a BMP and try just using that...which should force it to not use the "premultiplied" effect I think.

[Ian Jones]

i'm sure that would work in many cases but you'd also remove any pixels that are coloured similar to the background colour that are the image. You're not using an alpha channel - its just using the pixel color as a trigger for the transparency. If you wanted to do that you could just render a 16 bit image and matte it like you would if you filmed someone on a green screen.

I did another test including objects as the same colour as the background colour. I applied the 'remove color matting' and it still worked just fine! I was surprised as I had overlooked this last time I posted but it seems to actually still work.

I did another test, using 'premultiplied' option in AFX interpret footage and this time had good results. What I did differently this time was to select the colour of the background in the same dialog box and it removed the 'halo' without the need for the additional filter. Seems I overlooked this last time again... I've managed to confuse myself a bit now. I think the original suggestions actually do work as long as you select the appropriate background colour in the interpret footage dialog box.

[D-Wang]

I'm having the same problem in Max7... not trying to composite in any external app... im getting the "light fringe" around my alphas. Basically i have a scene of a bunch of planes with images mapped onto them that were created in photoshop. a lot of the images are text with a dark outer glow applied in photoshop. all around the edge of the outer glow is "the fringe". I've re-saved the image in black and white for a seperate opacity map and put that into the opacity slot on my max material. Is there a fail proof process to getting rid of the fringe within max?

[D-Wang]

Okay... after about 5 mins of testing...

- I took my photoshop file and saved each image (if its text it has to be merged with an empty layer to rasterize the layer effects) as its own .psd with 2 layers. the top being the merged text and the bottom being filled with the same color as the outer glow.

- in Max, i used the .psd as the diffuse and opacity map. the opacity map as a copy with Alpha on in the mono channel output, Alpha as Grey in the RGB channel output, Pre-Multiplied Alpha OFF and Image Alpha selected as the Alpha source

Render and no more fringes :P

I guess you could use one psd with all your layers and select the layer you want for each map too.

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.