I've quite a bit of material to be bluescreened, and some of those a bit trickier than others. Now, on the more difficult ones, I tried to make a rotoshape mask that would cut out the actors from the background when possible. Sure enough this works, then attaching this node to Primatte. Funny things start to happen, first I get something rectangular outlining the mask, then if I connect the mask to primatte as well, I get some artifacts along the edges.
How can one fix this? I tried setting the soft edges and such on the mask, but I see no difference there.