Face painting onto talking head 2d footage: Lens Studio or Spark FBX mesh export to C4d? Crazytalk? Other?


Goal. I have 2d talikng footage of a cubist head similar to this:


That is already comped together. I need to face paint it so the paint follows the distortions of the face as he talks. I have wasted days trying AE face tracking, Mocha, and others to push puppet pins, overlay with displacement maps, and other 2d tools. It’s not working.

Looking to get thsi done ASAP, but If I am forced to expand into a 3d workflow, I see two possible paths forward:

  1. Paint the already comped cubist face (alpha 2d footage): Comping it together required some liquid distortions in AE, so if I could just paint the finished comp as a single face, it would help hide those distortions where cheek skin doesn’t quite move correctly at junction with nose, etc.

  2. If I need to do this via facial tracking, and if those trackers just won’t work with the cubist face, then I suppose I could paint the front and side footage separately, then reassemble them into the cubist mashup after they are painted. (Not ideal as the liquid distortions required to do the comp would be more visible since I’m now stretching the paint.

Taking a look at FB (Spark AR) and Snapchat (Lens Studio) tools at the moment, and l’m not seeing any real ability to output high res footage. One of them mentioned that it can do FBX exports to C4d, though, making me wonder if I can minimize the learning tools by just using their face tracker, and doing the rest using tools I have at least some familiarity with.

I need to know if there are time wasting snags, pitfalls, or overall better approaches to be had, though.

Has anyone used any of the tools to do such face painting? Any insights into pipeline issues with any of the apps or render issues getting realistic looking output from c4d?

Am I barking up the wrong tree with the whole face mesh thing? Crazytalk? Something else entirely?

Really want to just pick a workflow today and crank this thing out asap. Open to anything that gets it done in next 48 hours.