iphone X for facial capture and 3d camera controller

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

 
Thread Tools Search this Thread Display Modes
  04 April 2018
iphone X for facial capture and 3d camera controller

Does anyone know of any developments in using the iphone X for facial tracking / capture or as 3d camera controller? I recall there was an iphone camera controller for Max or Maya several years ago, but that didnt seem to get any further development. And im aware of the odd tests out there for facial capture, but doesnt seem like anyone is really developing this into a usable product.

So many different ways to use an Iphone X for VFX / animation. The above are two that would be immediately useful for me.

Come on developers! create something already

 
  04 April 2018
The apple 3D face camera works by projecting little infrared dots towards a face, and looking through the phone's 2D camera to see where the dots end up on a face.

Facial motion capture at short ranges *might* be possible to *some* extent with *limited* fidelity and accuracy. The camera is not explicitly designed for motion capture, more for distinguishing one 3D face from another.

Using this camera to capture 3D space/objects at longer distances probably is probably not feasible - this thing really is just made for short-distance face-recognition.

If iPhone X could really scan objects in true 3D at long distances, the phone would be 1K to 1.5K more expensive than it is currently. Nobody will sell you a 3D scanner cheaply.


So what you want *may* be possible to some extent. The question is whether it will give you usable mocap data - the camera is not designed for face mocap right now.
 
  04 April 2018
So it does work as long as your face is close enough. That's good.

Try writing to the guy who did the 2nd video with the eye-tracking added in.

Maybe he'll release his Maya bridge commercially.
 
  04 April 2018
There's enough communication with the guy thorugh out his youtube video / comments. He has no interest in sharing or distributing the tool. He's focused purely on the game he's developing with the tool. He also notes that it's heavily tied to his setup.
 
  04 April 2018
If he can do this, it suggests that the iPhone-X lets software developers read the distance-values of the infrared dots projected in some way.

Basically, the iPhone can spit out a z-depth image or z-depth pointcloud captured by the IR dots, and he's deforming his face meshes with the distance readings in that z-depth image.

He's probably averaging many different dots together into one XYZ coordinate, then mapping that coordinate onto a specific vertex in his face mesh, so the vertex moves appropriately.

Anybody offering a face-capture technology or plugin of any kind for Maya and Max should be able to do exactly what he is doing.

He probably doesn't want to share his work because he's building a game around it, and doesn't want other developers to copy his approach.

As I said, any little company that has done face-capture in any way should be able to do what this guy is doing.
 
  04 April 2018
Btw, just found this markerless face-mocap software that works with 2D webcams:

https://mocap.reallusion.com/iClone...ap/default.html

Might that do it for you?
 
  04 April 2018
Check out Bellus3D.http://www.bellus3d.com/faceapp/ You canapply for the beta there. No motion capture, just face scanning. So far you can't export any 3D data but since its a beta you could just request the feature.
 
  04 April 2018
Originally Posted by skeebertus: The apple 3D face camera works by projecting little infrared dots towards a face, and looking through the phone's 2D camera to see where the dots end up on a face.

Facial motion capture at short ranges *might* be possible to *some* extent with *limited* fidelity and accuracy. The camera is not explicitly designed for motion capture, more for distinguishing one 3D face from another.

Using this camera to capture 3D space/objects at longer distances probably is probably not feasible - this thing really is just made for short-distance face-recognition.

If iPhone X could really scan objects in true 3D at long distances, the phone would be 1K to 1.5K more expensive than it is currently. Nobody will sell you a 3D scanner cheaply.


So what you want *may* be possible to some extent. The question is whether it will give you usable mocap data - the camera is not designed for face mocap right now.

Dude...it's built from Faceshift software integrated with a 3D sensor...it's a whole lot more than a maybe for facial MoCap capability. I only hope Apple lets the original Faceshift devs release/re-release the pro app in some form. There are a couple of people experimenting already. Otoy was experimenting with it using Light field rendering, so it wouldn't surprise me if a third party releases tools to transfer the capture data.

https://www.youtube.com/watch?v=hw7kPyVmUHQ


https://www.youtube.com/watch?v=w047Dbo-fGQ


https://www.youtube.com/watch?v=i51CizUXd7A

Last edited by Imhotep397 : 04 April 2018 at 04:23 PM.
 
  04 April 2018
The only thing is, before Faceshift was acquired by Apple it was delivering better/more nuanced results with fewer blend shapes needed than iClone.
 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 09:30 AM.


Powered by vBulletin
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.