iPhone Facial Tracking Data


Has anyone explored this? I haven’t upgraded my iPhone in a number of years–and haven’t been very motivated to do so. But now the AR Kit intrigues me as well as the new camera. And perhaps most interesting for 3d artists is the ability to do facial tracking and export that into 3D rigging.

Here is an example of a guy doing it. (He’s a Maya user)


A C4D plugin to do that would be great indeed.


In 2015 Apple bought FACESHIFT, for their expertise in face recognition/tracking/scanning using commodity depth sensors.

It was a very sad move for us 3D artists, as faceshift was awsome, but comparing a niche facial mocap software to the multibillion dollar mobile market, there wasn’t even a fight between the two.

That’s the technology they’re now using in the new iphoneX and the truedepth camera.

So technically this doesn’t surprise me one bit that somebody is trying to reinvent faceshift using an iphone.

If they actually make a good app, that would even make me think about buying an iphone… I guess the next “entry level” iphone will also have a truedepth camera, and we’ll be able to buy it for “only” 500 o 600 euros/dollars…

This being said, there have been numerous other siggraph papers since then for extraordinary facial mocap solutions (with monocular or depth sensors), and none seem to have surfaced as commercial products yet.


Stereolabs and Intel make Stereo 3D cameras that can also be used for things like 3D face scanning, facial tracking and getting 3D models from the real world.

Instead of waiting for Apple to do something interesting with its little phone facecam, I’d lobby a software developer somewhere to create an app or plugin that supports Stereolabs and Intel cameras instead.

Apple’s solution may or may not go anywhere good, whereas Stereolabs and Intel cameras are already being used for robotic vision applications and similar in the real world.

Do you really want your face tracking locked into an overpriced iPhone of all devices?


The stereo labs camera is $450. I’m sure it offers some tracking an iPhone X can’t deliver, but either it doesn’t excel w/facial capture…or I haven’t seen good examples. Apple put a lot of hardware/software sensing tech together to pull off some nifty facial capture.

I’ve soured on Macs with their pricing, but I’m not down on iPhones. They have (by far) the fastest mobile processors, good cameras and lots of compelling tech.


Apple’s face camera just projects some light dots on your face and has very limited range. Its not much of a real 3D camera at all and literally just for very closeup use on a human face.

The Stereolabs camera on the other hand is a true Stereo 3D camera that can see and scan in 3D for up to 60 feet. And not just faces. Anything it sees.

This means that with the right software or C4D plugin, a Stereolabs camera will probably be able to do full face and body motion capture, and probably at the same time as well.

You could set it up on your workstation monitor, stand a few feet back from it and do a dance routine in front of it that gets transformed into Mocap data.

If you are looking for such a solution for C4D, find some other interested people and take a few minutes to email the people at Stereolabs and Intel to let them know such a function is desired.

They may already be rolling Mocap software based on their 3D cameras. But they may not be aware at all that C4D users, for example, are willing to pay for such a software.

Unlike Kinect and the Apple face camera, these new Stereo 3D cameras are designed for real robotic vision applications.

Mocap is one of those applications, and the software to make this happen may already be in development. They are just not showing it yet.

If Intel and Stereolabs are not willing to do it themselves, these cameras are open to all - a C4D plugin developer may be able to roll together the necessary C4D motion or 3D model capture plugin.


For full body mocap I think any technology that is optics-centric is limited and second-best. The Kinect, Stereolabs or iPhone are all uni-positional optical, mainly just leveraging the benefits of their stereo lenses.

By contrast, The Vive’s laser tracking system, where you have base stations opposing one another, combined with iKenema can do slick mocap. With something like xSense one can raise that bar.

The guy I linked to above is deliberately using two disparate systems: one for body capture and a different system for facial expressions. The amazing demo of live performance capture in Hellblade (https://www.youtube.com/watch?v=CmrXK4fNOEo&t=84s) also uses two systems. They used Viacom facial mocap gear for the face and Xsens for the body.

So getting back to the iPhone… I would appreciate the depth data and planar mapping that can be captured (pretty cool, you can see in links below: ) But facial capture is where it really shines and for me…it would get combined with my Vive tracking system.



Some systems create FBX which can be imported into c4d.

I agree that a plugin with config and recording right inside c4d would be awesome.


Not sure if the tech on facial recognition has been improved since this article:
Link: https://www.wired.com/story/can-apples-iphone-x-beat-facial-recognitions-bias-problem/


Truedepth camera seems like a structured IR scanner, just like the kinect or Carmine sensors. And it’s range is actually exactly made for selfie distance and facial ID application, unlike the Stereolabs’ Zed camera which is made to work at a minimum range or 50cm up to 20m and is more of a general purpose tool.

So despite the Zed solution being great and more versatile, the iphone camera + the team and tech behind it actually make it a better candidate for facial mocap.

Stereolabs efforts are more geared toward depth environment mapping and tracking for VR (= room scale) at the moment.

I don’t think that Apple gives a damn about our field or work, it’s way too small money for them. But maybe one of the older researcher on the project will give it a try…

Intel realsense cameras are great too, and much cheaper, and you have a variety of other solutions, including the old kinect or simple webcams and neural networks. But all in all, the problem is not the hardware, it’s the software…

Whichever the solution, there are ways to stream data to C4D just like for any other DCC package.