CV-AR: Easy facial performance capture with iPhone X


#1

Hi all,

I was lucky enough to be a beta tester for a cool new plugin from Cineversity: CV-AR.

CV-AR has two components: an iOS app that works with the TrueDepth Camera of the iPhone X to do facial motion capture; and a C4D plugin that takes that data and makes it available within C4D. Donovan Keith has an intro tutorial up on Cineversity at:

https://www.cineversity.com/vidplaylist/cv-ar

The free iOS app is available at:

https://itunes.apple.com/ca/app/cv-ar/id1378696551?mt=8
The iOS capture tool leverages Apple’s ARkit to provide realtime capture of 51 facial movements, as well as eye orientation and head position.

The plugin is a blast to work with. The initial textured mask you get lets you know how well the performance was captured, but that’s just the beginning. You can then use the blendshape strength data streamed from the capture object to drive your own rigs. I think it has real utility for animators, both as a way to provide reference for hand-tuned animation, and as a way to speed a more automated animation workflow.

Here is a sample movie with the capture object mask on the left and my rig driven by that data on the right. At this point I hadn’t quite nailed some of the morph target sculpting, and the head movement needs adjustment. Let me know if you have any questions about how this works…

https://www.dropbox.com/s/aw5dev1iuw9ddnr/CV-AR_newBlendShapes2.mp4?dl=0

Props to Kent Barber for developing the plugin, and to Rick Barrett and the folks at Cineversity.


#2

Wow, very cool! But I only have an iPhone 7 :sad:


#3

Yes, Its too bad that it is limited to the iPhone X right now. As I understand it, the iPhone X has the requisite 3D sensor… I have a feeling that other iOS devices will be getting this capability in the future.


#4

Cineversity w/another surprise gem. Thanks Donavan…and team!

Alas…I too lack the requisite phone model.


#5

Yeah, that’s a virtual certainty, especially with the AR / Animoji announcements earlier this week at WWDC.


#6

Nice, I’m not interested for my self but I imagine that a few years from now we will be able to perform real time 3d scan of small object using our phone.


#7

Does it also require C4d Studio?


#8

This is the first time I’m considering an iPhone X… I’ll sleep on it :slight_smile:
Epic plugin!


#9

Me too : )

Depth sensing camera tech will have immense implications for VFX in general and it’s heading our way pretty quickly!


#10

As far as I can tell, no.

Caveat: I’m not really familiar with the stratification of C4D offerings. A quick look at the product comparison page shows that the Pose Morph is limited to studio, so the approach I took (driving a pose morph tag with data from the face capture) wouldn’t be possible, but other approaches (like Donovan’s) would be.


#11

Very cool plugin.

Too bad I just got a new iPhone 8. Didn’t care for the X, but this may have swayed my opinion.

Already have a feature request since I know I will be getting this awesome plugin at some point in the future.

A calculate button that summarizes the motion data supplied by Apple and displays static values near the per frame value for minimum, maximum, average, and neutral values. This would make adjusting your range mapper more simple when retargeting.


#12

That is a very good suggestion!


#13

Hi,

I’d been working on an app that was similar to instagram face filters that used an open source library openCV. I recently had to animate a little robot(only equivalent to head rotations) so quickly converted that to give me some mocap rotational data that I applied to the robot - a long winded way exporting a text file of rotations then importing it as csv into cinema. It worked ok, but could never get exact rotations because the openCV is a C library and I’ve only really used the language Swift. So I was hoping that advancements in apple’s ARKIT 2 would possibly bring face tracking to non iphone Xs(and therefore easier access to the data). From from I’ve seen though their face tracking is limited to that 1 device which is a severe limitation.(I don’t see point in creating any kind of app that works on only one device). Maybe I’m wrong, I’ve not tried the ios12 beta but nothing I’ve read so far indicates their face tracking works on other ios devices, which is a disappointment.

Trig

EDIT: just to show you the sort of thing possible here’s a link to the github where I started(I used the rotational data of the bounding cube only)

https://github.com/FaceAR/OpenFaceIOS

It manages to track eyes, nose, chin and mouth live(ish on my iphone 6s probably 15-30fps). So if I(or someone else) could understand the C code better could be made workable.


#14

I wish it was more broadly available too. It appears that to get the best recognition, the Truedepth camera is needed (people have shown that the Truedepth camera isn’t necessary for for ongoing face mapping, but without it the captures are more stuttery and laggy). There are rumors that the TrueDepth camera will be coming to other phones and perhaps iPads this fall.


#15

If you really need facial motion capture, open your email client and write to a producer of Stereo 3D cameras about it.

A technology giant like Apple cannot be depended upon to provide iPhone X functionality forever - they will pull that 3D camera off the market as soon as it doesn’t make any money.

It is the opposite with manufacturers of Stereo 3D vision cameras - they are desperately looking for attention-grabbing real world applications of their Stereo 3D product that they can show off.

Send them an email detailing your needs and what you would use the product for - for example, animating CG characters for Youtube shorts.

If you are dealing with a good company, and more than one person inquires about the same thing, chances are that the company’s management will be notified of the request.


#16

Amazing plugin! I borrowed a phone from my collegue, watched tutorial on Cineversity and 2 minutes later I had my face animated inside Cinema!
Congrats to all involved in the development!

Anders


#17

I just saw this yesterday. Gotta say, I’m beyond impressed. It makes justifying the insane price tag a little easier if you can use it for actual mo-cap work!


#18

This plugin can’t replace high-end professional systems, nor is it meant to. But it does inexpensively allow for surprisingly high-fidelity facial motion capture, and might find a spot in some professional workflows.

I think it is wise to be concerned about whether Apple continues to support this going forward, but I guess I am more hopeful than you. This relies on ARKit and the camera used in FaceID, which are two technologies Apple is betting heavily on.


#19

Every other manifacturer can not guarantee a functionality forever. Also it’s far more likely that manufacturers of stereo 3D product “desperately looking for attention” will fail faster than Apple if they struggle to sell a few pricey item while iPhone sell hundreds million of unit every years. On the other end with the iPhoneX you can use this now and for the foreseeable future, and they’ll probably extend the X sensors to the entire line very soon and for cheaper price. Honestly seems just another of your attempt to blame Apple products.


#20

So you would rather spend 1000 Dollars on a dinky phone with a pseudo-3D IR dot camera with a tiny range - manufactured in China for less than 400 Dollars - than spend 500 - 800 Dollars on a real dual-lens Stereo 3D camera that can scan in 3D for 20 feet or more?

You do realize that a real Stereo 3D camera can be used to scan 3D objects and locations with surface texture, may provide multi-person full body-and-face mocap at the same time when paired with the right algorithms, and will let you shoot green screen type work without needing an actual green screen in the background if the camera can spit out a Z-depth map?

Your argument confirms my long held view that Apple users are borderline technology illiterate.

You have the opportunity to support other more innovative companies that may give you a far more powerful 3D product that can do much more for the same price, but you feel compelled to throw your money at something inferior “because it has the Apple logo on it”.

Precisely this is the reason why almost everything Apple makes is extremely mediocre from a technology standpoint.

You see, Apple does not have to try very hard at all when tens of millions of Apple fanboys buy anything that has the Apple logo on it at any price Apple sets, regardless of whether it is good or not.

So if you want mediocrity - give your money to Apple like tens of millions of other naive customers. Apple are shameless. They’ll gladly take your money and sell you a very ordinary product.
[b]
But if you want something that gives you far more 3D content creation firepower than Apple’s overpriced crap, go and support a company that is actually trying harder than Apple.

Give your money to a company that is actually trying to be at the cutting edge of what is technically possible, not a profiteering mega-corporation with a fruit logo.[/b]