Most of the times that I use a garbage matte before a keyer node, the result is an undesired premultiply problem.
Often, using the unpremult, dilate or edge blur nodes are not options to solve this issue.
So, I have to apply a new rotopaint node as a mask to solve this “error”.
For those who are interested in this issue, please open the nuke file attached to this message and contribute with some help.
All the best,