CV-AR: Easy facial performance capture with iPhone X

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

REPLY TO THREAD
 
Thread Tools Display Modes
  2 Weeks Ago
CV-AR: Easy facial performance capture with iPhone X

Hi all,

I was lucky enough to be a beta tester for a cool new plugin from Cineversity: CV-AR.

CV-AR has two components: an iOS app that works with the TrueDepth Camera of the iPhone X to do facial motion capture; and a C4D plugin that takes that data and makes it available within C4D. Donovan Keith has an intro tutorial up on Cineversity at:

https://www.cineversity.com/vidplaylist/cv-ar

The free iOS app is available at:

https://itunes.apple.com/ca/app/cv-ar/id1378696551?mt=8

The iOS capture tool leverages Apple's ARkit to provide realtime capture of 51 facial movements, as well as eye orientation and head position.

The plugin is a blast to work with. The initial textured mask you get lets you know how well the performance was captured, but that's just the beginning. You can then use the blendshape strength data streamed from the capture object to drive your own rigs. I think it has real utility for animators, both as a way to provide reference for hand-tuned animation, and as a way to speed a more automated animation workflow.

Here is a sample movie with the capture object mask on the left and my rig driven by that data on the right. At this point I hadn't quite nailed some of the morph target sculpting, and the head movement needs adjustment. Let me know if you have any questions about how this works...

https://www.dropbox.com/s/aw5dev1iu...hapes2.mp4?dl=0

Props to Kent Barber for developing the plugin, and to Rick Barrett and the folks at Cineversity.
 
  2 Weeks Ago
Wow, very cool! But I only have an iPhone 7
 
  2 Weeks Ago
Originally Posted by LonChaney: Wow, very cool! But I only have an iPhone 7
Yes, Its too bad that it is limited to the iPhone X right now. As I understand it, the iPhone X has the requisite 3D sensor... I have a feeling that other iOS devices will be getting this capability in the future.
 
  2 Weeks Ago
Cineversity w/another surprise gem. Thanks Donavan...and team!

Alas...I too lack the requisite phone model.
__________________
C4D R19 Studio, MODO 902, VRAY, Octane, Cycles. PC/Mac.
 
  2 Weeks Ago
Originally Posted by NWoolridge: Yes, Its too bad that it is limited to the iPhone X right now. As I understand it, the iPhone X has the requisite 3D sensor... I have a feeling that other iOS devices will be getting this capability in the future.
Yeah, that's a virtual certainty, especially with the AR / Animoji announcements earlier this week at WWDC.
__________________
Mac Pro 12 x 2.6 GHz 64GB Quadro K5000
OSX 10.10.4
MacBook Pro 4 x 2.3GHz 16GB GeForce GT 750M
OSX 10.10.5
C4D R18 Studio/CC/VizRT
Will's Works
 
  2 Weeks Ago
Nice, I'm not interested for my self but I imagine that a few years from now we will be able to perform real time 3d scan of small object using our phone.
__________________
www.3drenderandbeyond.com
www.3dtutorialandbeyond.com
www.facebook.com/3drenderandbeyond
 
  2 Weeks Ago
Does it also require C4d Studio?
__________________
"...if you have faith as small as a mustard seed... Nothing will be impossible for you."
 
  2 Weeks Ago
This is the first time I'm considering an iPhone X... I'll sleep on it :-)
Epic plugin!
__________________
Follow me on Twitter@nosemangr - Watch me on Noseman's Youtube Channel
 
  2 Weeks Ago
Me too : )

Depth sensing camera tech will have immense implications for VFX in general and it's heading our way pretty quickly!
 
  2 Weeks Ago
Originally Posted by mustardseed: Does it also require C4d Studio?
As far as I can tell, no.

Caveat: I'm not really familiar with the stratification of C4D offerings. A quick look at the product comparison page shows that the Pose Morph is limited to studio, so the approach I took (driving a pose morph tag with data from the face capture) wouldn't be possible, but other approaches (like Donovan's) would be.
 
  2 Weeks Ago
Very cool plugin.

Too bad I just got a new iPhone 8. Didn't care for the X, but this may have swayed my opinion.

Already have a feature request since I know I will be getting this awesome plugin at some point in the future.

A calculate button that summarizes the motion data supplied by Apple and displays static values near the per frame value for minimum, maximum, average, and neutral values. This would make adjusting your range mapper more simple when retargeting.
 
  2 Weeks Ago
Originally Posted by Jim-H: A calculate button that summarizes the motion data supplied by Apple and displays static values near the per frame value for minimum, maximum, average, and neutral values. This would make adjusting your range mapper more simple when retargeting.
That is a very good suggestion!
 
  2 Weeks Ago
Originally Posted by GruvDOne: Yeah, that's a virtual certainty, especially with the AR / Animoji announcements earlier this week at WWDC.
Hi,

I'd been working on an app that was similar to instagram face filters that used an open source library openCV. I recently had to animate a little robot(only equivalent to head rotations) so quickly converted that to give me some mocap rotational data that I applied to the robot - a long winded way exporting a text file of rotations then importing it as csv into cinema. It worked ok, but could never get exact rotations because the openCV is a C library and I've only really used the language Swift. So I was hoping that advancements in apple's ARKIT 2 would possibly bring face tracking to non iphone Xs(and therefore easier access to the data). From from I've seen though their face tracking is limited to that 1 device which is a severe limitation.(I don't see point in creating any kind of app that works on only one device). Maybe I'm wrong, I've not tried the ios12 beta but nothing I've read so far indicates their face tracking works on other ios devices, which is a disappointment.

Trig

EDIT: just to show you the sort of thing possible here's a link to the github where I started(I used the rotational data of the bounding cube only)

https://github.com/FaceAR/OpenFaceIOS

It manages to track eyes, nose, chin and mouth live(ish on my iphone 6s probably 15-30fps). So if I(or someone else) could understand the C code better could be made workable.

Last edited by Trig Fuller : 2 Weeks Ago at 03:38 PM.
 
  2 Weeks Ago
Originally Posted by Trig Fuller: Hi,

I'd been working on an app that was similar to instagram face filters that used an open source library openCV. I recently had to animate a little robot(only equivalent to head rotations) so quickly converted that to give me some mocap rotational data that I applied to the robot - a long winded way exporting a text file of rotations then importing it as csv into cinema. It worked ok, but could never get exact rotations because the openCV is a C library and I've only really used the language Swift. So I was hoping that advancements in apple's ARKIT 2 would possibly bring face tracking to non iphone Xs(and therefore easier access to the data). From from I've seen though their face tracking is limited to that 1 device which is a severe limitation.(I don't see point in creating any kind of app that works on only one device). Maybe I'm wrong, I've not tried the ios12 beta but nothing I've read so far indicates their face tracking works on other ios devices, which is a disappointment.

Trig

EDIT: just to show you the sort of thing possible here's a link to the github where I started(I used the rotational data of the bounding cube only)

https://github.com/FaceAR/OpenFaceIOS

It manages to track eyes, nose, chin and mouth live(ish on my iphone 6s probably 15-30fps). So if I(or someone else) could understand the C code better could be made workable.
I wish it was more broadly available too. It appears that to get the best recognition, the Truedepth camera is needed (people have shown that the Truedepth camera isn't necessary for for ongoing face mapping, but without it the captures are more stuttery and laggy). There are rumors that the TrueDepth camera will be coming to other phones and perhaps iPads this fall.
 
  2 Weeks Ago
If you really need facial motion capture, open your email client and write to a producer of Stereo 3D cameras about it.

A technology giant like Apple cannot be depended upon to provide iPhone X functionality forever - they will pull that 3D camera off the market as soon as it doesn't make any money.

It is the opposite with manufacturers of Stereo 3D vision cameras - they are desperately looking for attention-grabbing real world applications of their Stereo 3D product that they can show off.

Send them an email detailing your needs and what you would use the product for - for example, animating CG characters for Youtube shorts.

If you are dealing with a good company, and more than one person inquires about the same thing, chances are that the company's management will be notified of the request.
 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 06:13 PM.


Powered by vBulletin
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.