How to fix white fringe from sharpening in green screen footage


TL;DR: It appears I somehow missed or overlooked a sharpening setting when I shot green screen footage for a school project, giving all of the foreground subjects whitish fringes around the outside. Reshooting isn’t really an option, so I’m looking for any advice as to how I would lessen the influence of the artifact during keying.

Hi everyone,

I’m new to this whole forum thing so I apologize if I posted this in the wrong section. Just a warning, I realize this post is long, but I wanted to make sure I explained my problem thoroughly.

I’ve been working on an extensive school project that involves NUKE (10.5v1). Long story short, I want to see how to Keylight, Primatte, and Ultimatte nodes perform in comparison to each other with common keying obstacles such as hair detail, motion blur, and spill. Besides comparing them subjectively by just looking at the keys, I wrote some scripts to provide objective results as well, so it’s my hope to not tamper with the footage in any way besides using the keyers and a basic Denoise node.

However, when I looked at the footage, I noticed there was a light gray/white fringe artifact around the foreground subjects in every single shot. The green screen may have been a little hot, but it was lit pretty evenly and there were no rim lights or back lights. This doesn’t bother me too much for the spill scenes since I could just choke the matte, but it is a problem for the motion blur and especially the hair detail scenes.

After a lot of research, the only things I could find related to this are that sometimes a white fringe occurs with DV footage or from internal camera sharpening settings. I shot the scenes at 1920 × 1080 and I’m almost certain there were no sharpening filters toggled, although it’s become more apparent that’s what had happened based on examples I found in online. The weird thing is that I’ve encountered the same issue in footage that my professor shot for my compositing class as well as in other footage that I was asked to key for a museum piece, so part of me thought it was something else. It’s not in everything piece I’ve footage I’ve brought into NUKE, but it usually appears if the green screen is on the bright side or if there is a lot of contrast between the background and foreground.

Here are some of the steps I tried to fix it. I’ve only tried using Keylight because I know it does well with fine detail. I’m sure Primatte and Ultimatte might give me better control, especially since Primatte is a chroma keyer instead of a color difference keyer, yet I am the most familiar with using Keylight.

• I looked at the color channels in Python to see if NUKE was the culprit by some sort of weird pre-processing. I’m only an amateur compositor so I questioned everything even if it doesn’t make sense to those who are more experienced, but it turns out I still saw the same fringe.

• The only solution I found online was to change the colorspace to YCbCr, blur the chrominance channels slightly, and return to RGB. This, however, expanded the fringe, but an inverse of a blur might help. I’m unsure if there’s a way to do that in NUKE, and even though I haven’t tried it, I don’t think eroding the channels would have the same effect. I don’t care about retaining the exact same color information as long as I can mask the original footage with the improved matte.

• Since the fringe is light gray, I figured that if I desaturated the red and blue channels, I could blend it in with the green screen while still keeping the background distinct enough from the foreground. For some reason when I did a basic key, I got the same exact result as before I did anything. I’m no expert in Keylight, but I figured there should have been at least a slight improvement. When I switch to the Screen Matte view, the matte is near solid white a few pixels PAST the fringe and doesn’t start to decrease in opacity until a few pixels later. This still baffles me.

• I also tried pulling a luminance key in Keyer on just the green channel, but again gave me the same issue as in the previous bullet. I feel like my thought process makes sense but it doesn’t provide me with any more of a tight matte.

• Last, I tried using the HSVTool to replace the color of the fringe with that of the green screen, but not only did it not replace it with the correct color, it changed the color of the whole green screen and affected the very fine hair detail. I’ve never used this node before so I plan on learning how to use it while I wait for a response.

Thank you for taking the time to read this. I’m aware I can’t reverse the problem since it was baked in by the camera, but any insight as to how I can possibly remedy it would be much appreciated! Reshooting isn’t really an option so I’m trying my best to work with what I have.