Black fringe outline in composites

#1

I can’t figure out why a simple composite in Combustion 4 results in horrible black fringe when using premultipy. The same files composite flawlessly in After Effects.

Images rendered in Maya 7 as .iff over a black background with alpha. I’ve even tried converting these to .tif files but get the same results.

Is Combustion doing something different in their compositing math? Combustion has been spec’d as our compositor for the 3d department and I can’t always be running over to a separate Mac for an After Effects comp.

Examples follow below…

#2

Somebody else here posted about having this problem too, some time ago. He did not really find a solution.

All I can say is that it still works fine for me here (same “path” maya renders into Combustion). Maybe contact discreet…aehh… autodesk…

-k

#3

You check all the settings premultiply ? I find that you can try use a compound alpha in screen (divide) mode

#4

This is a rendered element? Make sure to uncheck “Pre-multiply” on the render. I’ve found that trying to “undo” pre-multiply inside of combustion is essentially worthless. The un-multiplied render will look kind of funny, your diffuse layer will be pixely and smudgy. But then layer on the alpha and it’ll look a million times better than trying to bake in the effects of alpha on black straight into your diffuse.

#5

There is no “pre multipy” feature in a Maya render.

*EDIT I spoke too soon. There is a pre multiply check box in the Mental Ray Framebuffer->Primary Framebuffer tab

I’m simply rendering an element/layer that includes the exact alpha channel output in the file.

The thing that confounds me is that After Effects handles this type of comp with absolutely no problems or artifacts. All I have to do is make sure to pre-multiply black when importing the footage in AF.

Sadly, this same workflow does not work in Combustion.

#6

I would be interested in you talking to Discreet about that. Try submitting it as a bug report. I’ve always found it weird that the checkbox for pre-multiply does absolutely nothing. I always assumed I didn’t know how to use it properly.

I’m a little glad to see I’m not alone in the world in this. That being said, I think in general you’ll be better off if you don’t render with pre-multiply. I haven’t ever done any comparative quality tests, but my gut tells me you’re getting more information with it off.

#7

try curves in keyer

#8

Try switchig the layer’s transfer mode to ‘full additive mix’ and it will render the same as your after effects comp. Check also the 'Premultiply with:" option so that you set your color to pure black. Check your preference setting 'coz that might contribute problem as well.

In the help docs, it’s clearly stated how to deal with images with alpha channels so check that out for more infos.

I’ve been compositing on these two softwares and I think they work the same way (unless you have different settings). Another thing–I was able to reproduce that problematic comp you had with AE just by changing few settings.

Know your software–its not a bug.

#9

Just to make sure I wasn’t going insane I loaded up a quick test file from Photoshop. It worked without a hitch. I know when using Maya I’ve had a great deal of trouble if I render out pre-multiplied, I would say check your settings in Maya, perhaps thats where I’ve been getting trouble all this time.

Also, perhaps you could post the RGB and alpha file so that we can check to see if it’s a localized issue on your computer.

#10

I don’t think this is a bug. You should not need to set a certain blend mode, if the premultiply option is turned on in the footage controls. Others said that turning this on or off would not change anything.

However I just did a test again, with a render from maya, and for me it works fine and switching the premultiply box does what it’s suppose to do (black fringe / no black fringe)

I’d definately try that old flame trick Jugeras suggested with the blending curves in the keyer.

-k

edit: sorry for the typo, I meant to say I do think this must be a bug!

#11

wireFrame, it IS a bug - with TIFF files.

I have talked to Discreet/Autodesk about this and there’s definitely a bug with TIFF files generated by Maya. As these files work flawlessly with both After Effects and Shake, it’s a Combustion problem. (Also, did you not fully read or see my first images with “Additive Mix” also causing the problem?)

Listen guys, this is Compositing 101 here - a freaking simple OVER composite.

After Effects (with the obvious selection of Pre Multiply over the background color, black in this case) and Shake (in fact, I don’t have to set ANYTHING in Shake, I just create a multiple layer node, hook 'em up in proper order and it just works) beat out Combustion here with the desired file formats I want to be working in.

#12

hmm. all the work I have done in combustion was .dpx cineon or .iff . So I am very supprised by all this. Then again not using Maya 7 …

Can I ask a couple of quick questions ?

You said initially that you got this problem with BOTH .iff and .tif right ?

Are using Combustion 4 or 4.01 or 4.02 ?

I am very keen to see if I can replicate this probelm as it may cause us problems when we upgrade to Maya v7.

#13

I feel sorry for those who composite using tiff.

Ok, there’s some special treatment with comping tiff images. However, I can still get rid of the black fringes even on tiffs. I don’t know how you did it but I would like to ask if you comped it (tiff) straight on a blank bg (flat color) or you placed another layer as bg undereath your first layer (which it supposed to have)–this makes the diference. The setting “fully additive mix” will work.

I have to admit that AE handled the comp correctly but C4 may have a different approach to it the same way as shake has it’s own (maths). Yes it’s a simple compositing but we don’t render cgi as tiff sequences. We stopped doing this like 8 years ago when discreet suggected us to render in targa 'coz tiff is buggy (and still is). It’s only recent that I learned that tiff format now is capable in storing alphas (before it can’t).

Bottomline–we are compositors. We deal with all sorts of problems and the very least is to get stuck with a buggy format (which fortunately has workarounds). Why render in tiff when you can render in targa?

I don’t know if one of you guys saw the “Techinicolor Thead” few years ago when some of the guys here tried using all the deskstop softwares available (including shake) to replicate the techncolor batch setup from fxguide.com. That you will understand how different softwares are.

The inferno we’re using (v5) isn’t capable of handling files (afaik) with their own internal alphas. The alpha has to be detached as a matte file. Now I’ve encountered this type of files a hundred times when some cg artist didn’t render it on a color close to the bg plate but on black; the very least was on a grey bg. If I comped it straight in inferno, I’ll get those black fringes like the ones we’re having right now. Those fringes will disappear if you change the layer’s blend mode provided you add the bg plate as a bg layer underneath it. If you comp it straight on the app’s default bg layer it won’t work.

Curves in keyer is the last option but the matte tends to shrink using this approach.

#14

We initially used Tiffs on my last project. Gave up on that pretty quick. Amongst other things, we would have the alpha channel flipped horizontally from the RGB. I would agree it’s just a bad format to use. Maya 7 doesn’t like PNGs, so I would use Targas like wireframe recommended. Also targa offers an extremely diverse color depth selection. Great for floating point and gray scale images.

#15

That, I can agree with. I very much prefer TGA, however many apps do not offer RLE compression on TGAs although the format supports it.

Huh? You’re sure? Should it not technically be the same as setting your surface to “add” (flame/inferno)?
Advantage is you can use tranparency.

-k

#16

I’m sure of that 'coz if you use luminance curves in keyer, you’re now dealing with front and back matte of the composite–that’s why you see “front and back” in the parameters. You adjust the luminance of the back matte to lose the fringes–but it has some adverse effects (imo).

Technically we should just be dealing with the color image and its alpha–no keying involved (it already has a matte/alpha) and choose the best settings to lose the fringes.

I’ve also consulted this before (3 years ago) in the flame discussion group on how to deal with renders like this. It should be composited as “Add” blending mode in flame. Now translate that in C4–I find “fully additive mix” mode works best. Just don’t forget the background layer 'coz it’s part of the equation or it won’t work.

Just remember the compositing formula:

Comp = (Fg x A) + ((1 - A) x Bg)

#17

“Over” formula. Why the name “Fully Additive Mix”? The Curves (In Discreet keyer or in CC Curves) it almost doesn´t have effect in hard matte ( come by 3D packpages)

#18

I think you’re wrong. There are various “compositing formula”. Yours is expecting unmultiplied images. With premultiplied images, the 3d package already applied the first part of the formula (Fg x A)!
So it would multiply twice and thereby darken the edges -> black fringes.
If in the keyer you pull the front curve all the way up, your front matte gets all white or 1. If then it does Fg x AFg [size=2]it does no harm cause multiplying something with 1…
So adjusting curves should work exactly like setting your surface to “add”. However if you simply set to “add” in flame you cannot control tranparency anymore.

-k

[/size]

#19

Well then show me your other formulas. If those aren’t derivative of: Comp = (Fg x A) + ((1 - A) x Bg) you’re gravely mistaken! Premultiplied or not it all boil down to this basic formula.

Did I mention multiply that twice? in Comp = (Fg x A) + ((1 - A) x Bg) you already have the premultiplied image (Fg x A) so you’re math logic is wrong.

Okay, you’re somewhat correct in this one but why add another operator when you could just switch to “add” mode? It’s an additional node for the computer to process.[/size]

We’re discussing C4 here. If you adjust the curves in C4’s keyer you’ll lose control of the transparency the same way.

C4 and flame’s curves works differently. RGB mode works well (for premultiplied images) in C4’s keyer while in flame it’s LUMinance (front + matte).

Going back to the original problem it’s just a matter of corectly setting your layer parameters. The principle is that premultiplied cg images like that should work fine in composites with no additional operators.

In the end there are several ways to composite and work around problems but again know the basic rule:

Comp = (Fg x A) + ((1 - A) x Bg):rolleyes:

#20

Not at all! I simply stated that if you feed a premultiplied image into your formula (which cannot know if it is premultiplied or not) you’ll end up with black fringes cause the 3d package already performed Fg x A and if your compositing software does it too it was done twice, hence the black borders. So for premultiplied images the corect formula should simply be Fg + ((1-A) x Bg)

I was refering to your statement that adjusting the keyer curves gives a different result to changing to “add” (in flame) which it does not.

No. They work exactly the same way. The only difference is that its not so easy in C4 to specify the alpha of a layer as the key in. Thats why it was only meant as a last resort trick if all the other things did not work.

Well, the point is that if in flame you set your layer to add you have no transparency, in C4 transparency behaves rather weird in fully additive mix. In C4 you should not need to use fully additive mix, cause there is the nice little premultiplied button in the footage controls that should eliminate the need for any workarounds and any transparency issues, however it did not seem to work for the original poster (apparently a tiff problem), whereas for me it worked just fine.

-k