Uncanny valley crossed [Meet Emily]

[Phrenzy84]

not yet. Im afraid.

The worst thing about the uncanny valley is you get closer and closer to crossing it and thats when it stands out the most.

Could this character be used for background work, reactions included. Yeah definitely but when it comes to passing one human for another one, its still "off".

Eye capturing is really good, alot of shading work needed i feel.

Again, i wish i was more articulate in topics like these, all i can say is its getting closer and closer but still no cigar.

(In my opinion of course)

[Jozvex]

Yeah certainly *very* good but the mouth and eyes are still a little awkward in parts, the mouth particularly I think.

In a game though... amazing!

PS. I want to see a HD version!

[P_T]

Yeah certainly *very* good but the mouth and eyes are still a little awkward in parts, the mouth particularly I think.I agree with the mouth part but then again it could also be caused by "bad acting" on the performer's part, where the performer's expression doesn't really match the tone of voice.

I don't think you can comment on the shading considering the size and quality of the video.

[duke]

Yeah the mouth looks like the dentists anaesthetics are only just starting to wear off.

[SheepFactory]

Kinda hard to tell if the uncanney valley is crossed from a low res heavily compressed video. But I guess that is one way to cross the valley! Throw some blur on that vid and nobody will be the wiser :D

[Tryn]

She could actually pass for a celebrity with too much botox. I'm impressed...and a little creeped out. Its almost....uncanny.:D


Sorry...

[MOSS]

I think Image Metrics has some of the best facial mocap I have seen. I know people in the industry will be able to spot flaws, and that will always be the case. However, the real test is if the general public can be fooled. I believe that clip, at that resolution, would fool most people into believing she was real. Any higher resolution clips of her?

[LucentDreams]

I think this fell flat into the uncanny valley, the Eyes are just to creepy and bizarre. Its when that hyper real behaviour makes things not feel right and thats exactly what you have here.

[MrPositive]

Two things that pop into my head here.
1) You cannot state in your article that you've crossed the uncanny valley and then show a shoddy pixelated video of someone talking. We are talking about the Holy Grail of CG here, at least do it right. Which brings me to number 2) To pass the Holy Grail of CG you cannot tell people you are doing it. A video needs to be posted and people then question whether it's real or not and uncertainty arises. That is the day the uncanny valley will be surpassed.

Saying all that, even though I can't see much, it's not bad but it's still in the valley.

[bobtronic]

I saw this at Siggraph in HD. When I saw it first I was like, hmm, is this CG or bad acting. They also showed the original performance and it was not much different.

cheers,
Matthias

[ivanisavich]

Saw this vid on digg earlier today...

At first, I thought the entire thing was CG (body and everything) so I was blown away...then I realized it was just the face so immediately things started popping out at me (issues with the mouth, mainly)

But honestly, it's pretty freaking awesome. Image Metrics seems to be a step above the game on this uncanncy valley stuff...can't wait to see their further improvements!

[R10k]

I agree completely, MrPositive.

[gabe28]

I don't believe in the "uncanny valley" as it is usually described. My alternate theory is that the animation must match the level of the modeling/materials. For instance, if you rotoscope/motioncapture a person's movement to apply on a really cartoony character it's not going to look good. So, the problem is, it's easier to make a really good, convincing model of a human than to animate it. Too many small subtle things that are really hard and tedious to get right. It's almost impossible to use standard keyframe animation to get that level of animated realism so you have to use increasingly sophisticated forms of performance capture... if you ask me, it'd be easier to just use live action for those types of things.

Anyway, decent video but not there yet.

[Solothores]

bit better resolution (http://www.awntv.com/videos/image-metrics-emily-project)
It's pretty awkward to see her express herself "quite" naturally, while the eyes remain just... death. Makes me shiver and think about watching a sociopath, that mimics a role, before she turns out this big knife...

[monstroCity]

Saw this in HD over the weekend at USC's Institute for Creative Technologies. The digital Emily was immediately distinguishable from the real actress. Don't get me wrong; the work they've done is impressive, but touting that they've 'crossed the valley' is just empty hype.

[Solothores]

Actor's page, with some more details (http://www.emilyobrien.net/projects/image_metrics.php)

[Ruramuq]

the video looks very very good

the phrase Uncanny Valley is convenient to promote this kind of technology, but this all is about the amount of detail in the process of capturing the motion. So there's is no uncanny valley barrier. the complex the character the more detail it requires or it will look weird.

And the word 'Animation', well you cannot animate something that is already fully animated, by capturing(analyzing) it's motion. this is more like a projection than actual animation

[xcvbnm]

Two things that pop into my head here.
1) You cannot state in your article that you've crossed the uncanny valley and then show a shoddy pixelated video of someone talking. We are talking about the Holy Grail of CG here, at least do it right. Which brings me to number 2) To pass the Holy Grail of CG you cannot tell people you are doing it. A video needs to be posted and people then question whether it's real or not and uncertainty arises. That is the day the uncanny valley will be surpassed.

Saying all that, even though I can't see much, it's not bad but it's still in the valley.

Well, the claim is by the company (usual marketing via news media) and I'm sure there must be many such claims by others. I must agree that it comes close. At least it doesn't look like a cheap horror movie clip. :D

The problem is that when you declare its CG, or when you say tell me if its real or CG, people intentionally watch for deficiencies and the CG stuff pops out. But if you casually show it to someone whos not in this profession he/she would not notice much. (invisible CG, works best for non-character stuff.)

[Pixelbear]

Yeah. Put that in tv ad and nobody would know the difference really. Obviously if you know that it's cg you'll find something to tell that "oh yeah that's not right". Atleast it blows Beowulf out of the water.

[MrPositive]

The problem is that when you declare its CG, or when you say tell me if its real or CG, people intentionally watch for deficiencies and the CG stuff pops out. But if you casually show it to someone whos not in this profession he/she would not notice much. (invisible CG, works best for non-character stuff.)

You just reiterated my point. They should have placed it in a fake promo or something, posted it on some site and see if anybody says anything. Or do one of those clever hoaxes, like can you guys on cgtalk tell me why my video I have here is not importing correctly into After Effects or something clever, not mentioning the CG involved. See if anybody catches it. There is no doubt that people think they can see anything, thus, you mustn't mention anything about CG or uncanny valley when testing. Regardless, after seeing the higher res video it is pretty damn close but that last 2% is a doozy.
P.S. Plus to be legit, I'm saying right here it should be a complete CG head and not just the front face plate. I personally think we should be fooled on all fronts. Shrug.

[blank]

Now if the video didn't start with " what your about to see is 3D" that may of helped as bit and i may of just focus on the fact I found her slightly annoying more than her being CG.

Although am i right in thinking it is only her face thats CG and the rest of the room and actor are real?

[R10k]

That's correct, blank.

I think the surrounding reality helps the CG elements appear more real. However, I don't think too many people would be fooled by that example in a real-life, just as I'm not fooled by fake voiceovers in dental and skincare commercials.

[Solothores]

So put up a screen masked as mirror and confront people with a realtime rendered cg reflection of themselves. If they don't notice the difference, I guess this would settle all arguments for once and all and people could start to praise themselves for having finally made it. (Not that this is going happen soon) ;)

Aside from that, featuring something in a short promo is not the same as watching a movie, that might unfold as a two hour character drama, in which you have to sympathize with a protagonist that actually interacts with a visible env. and goes through the whole spectrum of emotional state and thus undertakes a quarter million frames to prove himself to be real and alive. Only if people don't realize him or she being cg after seeing this (while not being told ofc), the industry can start talking about the big breakthroughs, at least that is my opinion.

BTW:
This cg demonstrations should really start having their cg creations interacting with real people, while showing them both. As the contrast between having real and unreal in the same scene, makes it even harder to maintain the illusion.

Cheers
S.

[fgdf]

Nice shading and lighting but otherwise she looks kinda ...high

[Poisen]

I agree with Sheepfactory

Add a bunch of blur, compress the hell out of the video and fly right on over that valley!


:surprised

[popol]

uncanny valley crossed ?

Bah Show me a hd video of the same face with a full cg body in motion with fat, hair , tongue.... movement and we should see that the uncanny valley is not crossed at all.

[yenvalmar]

Add a bunch of blur, compress the hell out of the video and fly right on over that valley!
works for your bigfoot and UFO hoaxes too :)

i also think going to black and white would help..

[xen_ninja]

I personally thought that Davy Jones crossed it in Dead Man's Chest. So did tha King Kong fella......I mean how unreal/uncanny can you get ?

As for the video ,I'd be more impressed if they used a totally synthetic face. As in the face cannot be an existing topology.

[iainbanks]

More about this on FX guide in hi-res, A discussion with the guys who made it aswell as some hires footage of the animation

http://media.fxguide.com/fxguidetv/fxguidetv-ep030.mov

[popol]

I personally thought that Davy Jones crossed it in Dead Man's Chest

davy jones is fantastic! but he's not fully human, thats why we can easylly accept him .

[Trenox]

uncanny valley crossed ? more like smeared.. They need to put out some HD clips if they are gonna prove anything with this..

It all looks nice and all, but as a 3dartist im a little concerned that this is more or less "generated" and not "hand-created", which ofc can be a great benefit for production, but not so great for the 3dartists. oh well thats a whole other discussion i guess :)

[xen_ninja]

davy jones is fantastic! but he's not fully human, thats why we can easylly accept him .

True ....I was just implying that animators were behind this achievement. Just like Gollum who is almost human.

I know that technology such as Image Metrics has unwieldy amounts of data and must be quite impractical to animate/modify it. Still I am pretty sure an animation team with enough patience and time can take the data and make it believable.

[phyle]

Saw this vid on digg earlier today...

At first, I thought the entire thing was CG (body and everything) so I was blown away...then I realized it was just the face so immediately things started popping out at me (issues with the mouth, mainly)

But honestly, it's pretty freaking awesome. Image Metrics seems to be a step above the game on this uncanncy valley stuff...can't wait to see their further improvements!

Yeah I was a bit less impressed after i realised the cg was just the face. It says at the start "What you are about to see is a wholy computer generated animation" which I think is being a bit sneaky with words.

[P_T]

uncanny valley crossed ? more like smeared.. They need to put out some HD clips if they are gonna prove anything with this..

It all looks nice and all, but as a 3dartist im a little concerned that this is more or less "generated" and not "hand-created", which ofc can be a great benefit for production, but not so great for the 3dartists. oh well thats a whole other discussion i guess :)It's right there in post #30.

Don't worry, I don't think it's cost effective to animate real human from scratch so I don't think this technology has much impact, if any, on animators job.

[vfx]

Interesting how they state its completely CG when in actual fact the video is real, with a synthetic CG version of the face projected back onto the original actress in the scene later in post.

[blank]

Just looked at the actress and its actually appears to be a CG version of her face projected onto...

Yup you guessed it her face!!

now call me cynical but surely her real facial muscle movements are going to match how her cg face is meant to move helping to prevent any unnatural movements?

Now if this tech is really going to help bring back dead actors and make them believable are the stand in actors faces going to have to have a similar muscle structure to pull it off?

[ChewyPixels]

Getting closer, just not there yet.

[Trenox]

p_t - thnks for pointen that out :) that video answered a lot of questions. Quite impressive, but still very heavy to produce.

[MadeInUterus]

Here are my 2c

first when i first saw the video i thought, oh my... they've done a really great job with the lighting and textures, the way her hand moves is perfect... the only problem is her mouth(specially when in grin) and somewhat all of her face seems a bit off, other than that spetacular, Then it came to my knowledge, wait a minute... its only the face thats rebuild, why on earth did they say "all you'r about to see is fully computer generated" when its not?

so the pros at AMD say that the valley is not to be crossed over untill 2020, thats when we will get freaked out... never the less the work is good, too bad is most for promotion than an actually breaktrough in the industry.... don't get me wrong "pixel facial scanning" looks really promising... just wished they waited a little longer for the technology to mature...blank has a point... if muscles are so unique in an individual how can "pixel perfect matching" work? i've watched another sample video over the actress page (http://www.emilyobrien.net/) and it sure has its flaws.. as i said.. good technology.. too early to be shown as a product that crosses the valley

[szewei]

I met these guys at the Annecy animation film festival last June. Emily wasn't finished yet but she was pretty impressive already! ;)

I don't think the uncanny valley has been crossed yet though... Pretty much agree with what that guy says at the end of post#30 video: It's the best out there today... but there will be better! (btw: great video. very informative. thanks so much for posting!)

Imo one of the really cool things about this mocap process is that it doesn't require a lot of setups for the actual capture. The actor doesn't have to go through hours of preparation getting all those markers stuck onto his/her face as for optical mocap (not to mention how uncomfortable it is to have these things stuck to your eyelids!). Image metrics showed us a demo of a guy running around with a mocap suit (traking of body movements) but then he just had a small webcam attached in front of his face to track his facial expressions. I definitely think "markerless" is where mocap is heading... at least for facial...

[Venkman]

I was pretty impressed with the footage, but at a loss as to why it crosses the uncanny valley when it is only the face that is being replaced. That's really odd!

At first I thought it was damn good, but it's the smiles and teeth that really give it away to me. It can't be simple to try to get that done.

It is still really, really impressive work, though. If I had done that, I'd be marching around with a party hat on tootin my own horn. :bowdown:

[vlad74]

Kinda hard to tell if the uncanney valley is crossed from a low res heavily compressed video. But I guess that is one way to cross the valley! Throw some blur on that vid and nobody will be the wiser :D


Hahah. Exactly.

[mosconariz]

beowulf looked a lot better than these... It needed tweaking and bla... yeah, whatever, it looked a lot better, and it was full CG, not just a weird face... interesting tech

[szewei]

beowulf looked a lot better than these...

Ouh no do not agree! Saw beowulf just a few days ago and at that conference on mocap in Annecy (they presented at the same time as Image metrics)... definitely not as good imo!

Actually the difference was stunning to me! Although it's not perfect I find it's the first time realistic facial animation feels like it's almost there... (as opposed to smack at the bottom of the valley... hehehe)

[Nathan]

they're also simply recreating the exact same footage. same face, same angle, same exact performance frame for frame, basically recreating all the pixels on screen 1 for 1. at that point it should look 100% perfect, but it doesn't. it's still kinda creepy, especially the eyes, the corners of the mouth, and the lips interacting with the teeth underneath.
and in the video on her personal site, they show a little of the original footage at the end and you can see a definite difference in the shading, the shape of the eyes, etc.

this would be more impressive if it were from a completely different angle, or better yet, capturing her performance and applying it to a totally different fully cg character.
actually, in the same video on Emily's site, the video shows exactly that, and it falls smack dab in the middle of the valley.

they also conveniently chose a young female with a very smooth face, plenty of makeup and very flat lighting. much easier to hide imperfections that way. show us someone older and wrinkly with very prominent facial features in dramatic lighting.

they do seem to be doing some good work, but it's way too bold a claim to take them seriously. meh

[MrPositive]

http://media.fxguide.com/fxguidetv/fxguidetv-ep030.mov

Ok, watched this version and I must say it's pretty damn good, definitely an improvement (although considering I'm not sure how this was composited back onto the face I'm not sure how much of an improvement). However, the whole thing comes off misleading, beginning with the opening claim that the entire scene is computer generated and ending with themselves proclaiming they crossed the uncanny valley and they've done the 'best' animated cg human to date (shouldn't we be the judge of that?). I still think they made a huge mistake telling everyone up front it's a cg face....oh well.

[Michael5188]

Can't anyone on the team see that she blinks like a animatronic at disney land? I know nothing about face scanning, so I can't really comment, I can say that I still believe in having a keyframer go through and touch up.

[szewei]

Hmm just saw this:
http://www.image-metrics.com/videos/231
It's their commercial demo but that warrior guy looks pretty cool! :thumbsup:

[circusboy]

True ....I was just implying that animators were behind this achievement. Just like Gollum who is almost human.


King Kong, Davey Jones and Gollum are all stylized humanoid. And as brilliant an acheivement as each are - if any of those production teams tackles an realistic digital human actor there will be much more work to make that absolutly convincing.
And when its so easy to use a real actor-is there any motivation to doing that?

[agreenster]

I dont think it's out of the valley yet, but it's definitely on the *better than in the middle of the valley* side

[Jon A. Bell]

Oddly enough, I think it would've helped to have used an actress who wasn't already "model beautiful" -- she already looks somewhat synthetic.

Hire an actor/actress who has some character in his/her face -- I'd like to see a completely oridinary-looking person done in CG. That, I think, would be a better test.

(FYI, I know a guy whose 30-year-old Panamanian girlfriend is so stupefyingly beautiful that she doesn't look real, even in person -- she's so perfect-looking that she resembles a CG character in real life! As beautiful as she is, she's exactly the person I wouldn't use as a model for a realistic CG person.)

However, I do think the animation is pretty amazing overall -- keep trying, guys.

-- Jon

[bentllama]

as artists, we need to stop focusing on crossing that Valley just to say we did, and revert our focus to amazing storytelling, pathos and emotive entertainment - like the good ole days.

seems like Walt and the boys were on to something...

[R10k]

Yes, but Walt didn't understand the true power of this technology. Imagine the amazing storytelling, pathos and emotive entertainment through bold attempts to stitch Mickey Mouse's face onto Snow White.

[P_T]

I think that whole "entirely computer generated" was just the marketing people at work or simply ignorance on the journalist part.

Like that guy said in fxguide video, if people think this was live action, they wouldn't be scrutinising every little details for flaws and most likely would really think that this is live action despite some claiming the ability to spot a stretch pore texture in that video (an exaggeration!!).

Watch the HD live plate in that fxguide TV and you'll see some of the flaws pointed out here aren't actually flaws. At least to me they're not, but then again I don't have the eyesight capable of spotting the penis on a fly.

[EvilGnome]

Even in the compressed version the eyes were dead and things in the face looked weird.

Ty watching it without sound.

I was more impressed with the subtle body motion until I discovered it wasn't part of the demo!

[DuttyFoot]

not out of the valley at all, cg written all over it even if it wasn't posted here. the eyes and mouth are very strange looking.

[trthing]

eyes... mouth... *THUD* middle of the valley

[Shletten]

Somewhat I was more impressed by Mova demonstrations and Pendulum Studios' alter ego old man.

The real emily has different teeth that aren't always sticked, has much more deformation and is a little less skin blurry. She looks best on the Specular pass, you can see all the wrinkles.

[factotum3d]

The performance capture is some of the best I've seen, especially given the way it is captured (i.e not spending hours in making up or putting on fancy markers). Yes, the lighting and shading is a bit 'off', but I think the Image Metrics guys would be the first to concede they are more technically than artistically inclined (I think I read somewhere this was a completely in house test).

I think it would be really interesting to see what the likes of ILM, Weta, D2 et al could do on the lighting/shading side of things to take this test up a level.

Maybe that valley would'nt look so uncrossable after all...

[DrFx]

The point of the video is the animation, but most people complain about the shaders.
I agree it's not really over the valley yet, but make the face less red, the eyes and teeth less white and more SSS and get the deformations, bumps and wrinkles better and it could be quite good.
IMHO this clip is most impressive.

[ezdz]

Overall it's pretty impressive, but I think they needed to work on the composite a bit more (less red in the face) to really sell it.

[withanar]

Image Metrics have been very successful at marketing themselves, and in an industry full of hype and hyperbole they are merely playing to their audience, which is not really CG production specialists but rather producers and project managers.

If your producer watched that video, would she think the results were better than anything she's seen your studio produce? If yes, then she is interested in their product because it is beyond the scope of her current resources. If no, she may still be interested in their product for the purpose of quality to cost comparison. Either way, mission accomplished.

Let's hope however, our producers are asking the right questions before pulling the trigger on anything. For example, how long did they spend on this one shot? Is that a typical length of time for their client-driven work? It must also be considered that this shot is, for all intents and purposes, a roto of an already existing shot. There is a perfect reference source for both shape and lighting. While there is definitely a huge need for this type of roto work, it is a different animal from the process of capturing from face A and converting the performance to face B, which seems to be more of what we're expecting to see from a facial animation service.

The stark differences in challenge and process between roto matchmoving and facial performance transfer thus require one to ask, is this really an appropriate test bed to claim victory over the unaccany valley?

[arctor]

to be fair...it was the original poster of this thread that claimed "Uncanny valley crossed"

this article:
http://technology.timesonline.co.uk/tol/news/tech_and_web/article4557935.ece
it's the author of the article that mentions the Uncanny Valley

this link:
http://www.awntv.com/videos/image-metrics-emily-project
doesn't mention it at all

and the FXGuide piece:
http://www.fxguide.com/qt/415/fxguide-from-siggraph-fake-or-real
doesn't mention it at all....

the Image Metrics process is pretty simple...
and it's all dependent on the rig...if you build a rig that is capable of the appropriate range of expression and capture the performance you want, Image Metrics can generate animation that will drive your rig with a level of detail that is really impressive.

the 'Emily' clip is, from an animation standpoint, is just about the best I've seen...most of the criticisms, however valid, have been about the tracking and shading...and the manner in which clip is presented puts the viewer in an inappropriate mind set right off the bat.
This demo, at least from my point of view (as someone who is working with IM right now) is a very impressive demo of the sort of 'base line' results that IM can deliver, a digital recreation of a human facial performance...where it goes from there is up to the client...take the actor's performance and put it on an 'aged' version of her face, or on a different face etc.
I can't talk about what we're doing, but I can say that the results we are getting are pretty remarkable, and we're doing some fairly extreme stuff.
The real question here is about scale and cost effectiveness...
can an animator, given a rig that is capable of the appropriate range of expression, produce a performance that is as good or better than what IM has demonstrated with 'Emily'?
the answer is, of course, yes.
but can it be done is the same amount of time? = $
can it be revised as quickly? = $
can the animation be re-targeted to another character?

[DimeS]

Saw this clip this morning on a breakfast news tv show. The two hosts showed the clip then went into the old 'is this the end of traditional actors?' bit. They were under the impression that the entire thing was CG, one of them even noted how realistic the leather chair was!

IM is getting a lot of publicity out of this, but maybe not for the right reasons.

[Michael5188]

It is too bad some people are being mislead, or misinterpreting the tech, but you need hype like this to get funding and further the R&R.

I mean look at the "mocapped" Gollum.

[arctor]

It is too bad some people are being mislead, or misinterpreting the tech, but you need hype like this to get funding and further the R&R.

I mean look at the "mocapped" Gollum.

it's unfortunate that many things like this in CG are not presented as well as they could be...and even worse for those not in the industry - it's left up to media people (who NEVER have half a clue what they're talking about) to interperate it for them...

with the first frame of "Emily" I know exactly what I was looking at...that's not the case with most people...

[Boone]

I would say "it gets the job done" but they need a HD version to finally convince me. At the current resolution she looks real enough to convince me - had I not known about it.

I think her eyes when she says "Hmmm...." that look to the right are a bit of a give-away but other than that I would say she is real.

Makes you wonder what is 10 years down the road...

[Pedrotheartist]

but can it be done is the same amount of time? = $
can it be revised as quickly? = $
can the animation be re-targeted to another character?

That's exactly it, it comes to spread sheets.

It's a very good demo for what it is, sure there are some issues but it would be interesting to know how much it cost. It's great to see this area being pushed.

[Antropus]

The complete video shows the real actress at the end of the flick. And guess what? The CG model looks just like her! (not like "she doesn't exist" or " is "fully animated"). Only the face was replaced... by her own face!
It's cool to see a digital double of her anyways. Nice stuff.

[kap13]

It looks weird, the eyes look scary.

[Lunatique]

I think whether the valley needs to gets crossed depends on the application. What's more important to me is how expressive the acting/animation is, not the shaders or modeling. I think for many applications such as storytelling in video games or animation, what they've done is already good enough to solicit empathy in the characters, and that's the most important aspect of storytelling. But if for the purpose of fooling people into thinking they are watching live action, then until I see more examples I won't be able to say for sure. Right now other than the shader, I think it's damn close and the best of its kind in terms of expressiveness out there.

[sfox8]

To be honest, I think if this is not the closest animation to crossing the valley that I've seen, they I don't know what is. Definitely the best I've seen and despite any of the criticisms made against it, you have to give IM credit.

Although I do agree with MrPositive. That was the -exact- first thing that I thought as soon as I saw those words pop up on screen. Instead of letting people know it was CG right off the bat, they should have just posted the video and see peoples' reactions. Then a little later down the road they can say "oh by the way, that video was CG." I've already shown the video to some friends and family and not one of them picked up that it was CG until I told them. They all asked me why I was making them watch an interview about a company they've never even heard of. People who work with cg every day are bound to pick out flaws that a normal person in the audience would not recognize.

Anyway, I honestly see hardly any differences between the CG and the real actress. There are some subtleties not captured very well like the eye movement, and probably the wrinkles around the mouth (which are more visible when they show the real actress at the end). But when you look at the HD video, it somehow seems more believable. Maybe it's just me.

[OlegAlexander]

Hello everyone! I am the Director and Rigger of the 'Emily' project. I've read all the comments so far and most of them are fair.

The TimesOnline video: :cry:

It's a real shame that, for many of you, your first exposure to the 'Emily' project was through the TimesOnline video.

1. We did not write these captions! (e.g. "What you are about to see is a wholly computer-generated animation.") Our intention was exactly, as many of you have pointed out, to reveal that Emily's face was CG after the viewer has watched the whole "interview". The original, full-length version is here:

http://www.awntv.com/videos/image-metrics-emily-project

2. The TimesOnline video is extremely compressed--the original is in HD. We'll be sure to post the HD version soon. (Or at least a half-res version. The HD version will be a huge file to download.)

Ok. Got that out of the way.

The facial animation:

As arctor eloquently pointed out, the whole point of the demo was to show off Image Metrics' facial animation technology. From a facial animation perspective, we've nailed Emily's performance perfectly. All of the subtleties, nuances, and idiosyncrasies of Emily's face are in the animation. At times, Emily's personality really shines through. (This is best appreciated when looking at the HD version...)

The facial animation was actually the easy part for us, since that is what Image Metrics excels at. 90 seconds of Emily facial animation took only 2 man weeks to complete. (I need to clarify here that the 'Emily' project falls within our 'film-quality' standards--highest quality. If this were a game project, the animation turnaround time would be quicker.)

Here are some other statistics:

The entire project, including scanning Emily's face, modeling, rigging, animation, matchmoving, texturing, rendering, compositing, and editing, took approximately 4 months total.

We had 8 artists working part time. Most of the time was spent in the rigging and rendering stages.

The project was done in Maya. The rendering was done in Mental Ray. Compositing was done in Vision. The eyelashes were done in Shave.

The rig consisted of 75 blendshapes based on the FACS. We also had 30 animated displacement maps driven by the blendshapes.

Here are some issues we ran into:

Getting the eyes right was 90% of the battle. I concur with many of you that we still didn't get the eyes quite right. But believe me, we tried our best! The facial scan did not provide good data for the top eyelids because the eyelashes created dirt in these areas. So we had to remodel the eyes from photographs multiple times until we were "happy". Some of the eye movement is still very stiff on certain frames. I have no one but myself to blame for this, since I was the rigger. :p

The matchmoving of the face was also a huge challenge because there were no tracking markers on her face. We did ultimately solve this problem with some custom code. However, the results were more consistent than they were accurate. Therefore in some frames she just looks weird because the facial matchmoving is slightly off. (Next time we do something like this, we'll make sure to put a few head-tracking markers.)

How I really feel about the 'Emily' project:

Animation-wise, we've nailed it, as usual. Rigging-wise, we've nailed it, but not for the eyes. Rendering-wise, we pushed Mental Ray to the limit.

The 'Emily' project is just the next step in the journey across the uncanny valley. Jedi-Juice said it best: "Getting closer, just not there yet."

[Boone]

Seeing the HD version I am now convinced this is a success. Sneak a few shots into a film and I would be surprised if anyone would notice.

Oleg, is this the same technique used in Heavenly Sword for the PS3?

[Shletten]

I haven't been fair in my criticism, it may not look as impressive as some other ¨motion capture¨ performances but since it's keyframed and looks nearly as good as motion capture, I have to congratulate Image Metrics for their job on Emily. I really like it, no doubt.

Next time I would prefer an older actor with more complicated skin :P This girl is strange, she's already too CG. ;)

[OlegAlexander]

Oleg, is this the same technique used in Heavenly Sword for the PS3?

No. I believe Heavenly Sword used traditional facial mocap. Image Metrics' animation technology is based on computer vision.

[Spin99]

Wow that was nice to watch.
Reminded me of SoftImage's first release of Face Robot.
I wonder if they're playing catching up now.

I agree that the eyes mostly give it away. Also the lack of facial muscles definition.
Emily would look phenomenal on a video game though.

I'd say Ilana ( http://www.image-metrics.com ) is a bit further behind.
With her the teeth are a dead giveaway.
But wow definitely "Uncanny Valley" ;)

[MisterE]

Truly an amazing job by the team at Image Metrics that put this together. While I agree that it isn't quite 100% believable in terms of passing the "uncanny valley" (I'm getting sick of seeing this phrase now..), I think the important thing to take away is that this was a very unique, learning experiment. Obviously, from the sounds of it, Image Metrics knows what they need to fix to improve and will, hopefully, make their next assignment even better and more photo real.

At the same time, other companies and artists may be inspired by the techniques seen here and try to create their own, more fitting animation. This will eventually lead us closer to photo real, virtual characters, satisfying not only those obsessed with the "uncanny valley" :argh: but help bring us, the audience, closer to identifying with characters in both games and movies. I, for one, am all for it.

Oleg, if you don't mind being our go-to QA guy, I was wondering, from a curious modeler's standpoint, what program did you use to retopo the hi-res scan data? Was this done with Maya? In house tools? Or are you guys using TopoGun?

[OlegAlexander]

What program did you use to retopo the hi-res scan data? Was this done with Maya? In house tools? Or are you guys using TopoGun?

Thanks for the kind words! We did learn a ton from doing this project. Now we know exactly what to watch out for the next time around.

We started with the ZBrush retop tool and finished the mouth bag and eye interiors in Maya. We used oaRelaxVerts.mel (http://www.highend3d.com/maya/downloads/mel_scripts/modeling/poly_tools/oaRelaxVerts-4499.html)on the cheeks and forehead. The base mesh was 4000 polys.

We also brought in a makeup artist to do a dental cast of Emily's teeth. Then we scanned the dental cast. The teeth scan was retoped using NURBS curves in Maya (Spline cage > NURBS surfaces > Polys = Old School :cool: ). We then used xyShrinkWrap.mel (http://www.highend3d.com/maya/downloads/mel_scripts/modeling/misc/xyShrinkWrap-1721.html) to wrap the geometry around the teeth scan. The upper and lower teeth meshes were 10,000 polys each.

Finally, we used oaSmoothCompensate.mel (http://www.highend3d.com/maya/downloads/mel_scripts/modeling/poly_tools/oaSmoothCompensate-5273.html) on all the geo.

[MisterE]

Hey Oleg, thanks for the insightful post!

So you guys had a dental cast of Emily's teeth, too, huh? Seems kinda hardcore at first, but obviously, a very good idea when you think about it; and, almost a necessity to keep the photo realism top notch.

Some of the eye movement is still very stiff on certain frames. I have no one but myself to blame for this, since I was the rigger.

Also, don't knock yourself too hard about the rigging. From seeing the "Making of" video, it looks like you did a very nice and intuitive job setting everything up for the animators. I'm not sure if this will help or inspire you in the future, but here's another great facial animator/rigger, of whose work was just recently posted here on CGTalk:
Liam Kemp (http://www.liamkemp.com/thenormals_animationTest.htm)

Thanks again for taking the time to answer our questions! :)

[ienrdna]

I have no idea how animation works, but hoe does blend shapes and capture data fit together.

[OlegAlexander]

The "Making of Emily" video is up!

http://www.image-metrics.com/node/361

[SuperRune]

I am surprised no-one has mentioned Universal Capture yet? Isn't this pretty much the same as the techniques mentioned on these websites?

http://www.fxguide.com/article333.html
http://www.virtualcinematography.org/
http://www.virtualcinematography.org/publications/acrobat/UCap-s2003.pdf

(http://www.virtualcinematography.org/)

[robodesign]

The "Making of Emily" video is up!

http://www.image-metrics.com/node/361

With all due respect, but I only hear the music. The voice volume is too low. Something bad happend when you guys mixed the voices with the music :).

Anyways, congrats.

[anobrin]

Uncanny valley.....Not crossed:cool:

[OlegAlexander]

I am surprised no-one has mentioned Universal Capture yet? Isn't this pretty much the same as the techniques mentioned on these websites?

http://www.fxguide.com/article333.html
http://www.virtualcinematography.org/
http://www.virtualcinematography.org/publications/acrobat/UCap-s2003.pdf

(http://www.virtualcinematography.org/)

No. Universal Capture is an awesome system, but its limiation is the use of predetermined "clips". These animated clips are perspective-projected onto the model as textures. While the Universal Capture system allows for huge flexibility in the rendering stage, it allows for very little flexibility in terms of being able to change the facial animation. In other words, you're stuck with the animation in the clips you've captured.

On the other hand, the Emily rig is a traditional animation rig. It's very flexible and the animation could be modified at any moment. The Emily rig is actually much closer to the Shrek rig than it is to the Mr. Smith rig!

There is a final important difference between Universal Capture and Image Metrics. With Universal Capture, the performance capture process itself is cumbersome: It requires that the actor's face is mocaped AND captured on video from 3 to 5 different angles simultaneously. With Image Metrics, there is usually only one video camera and no facial mocap is necessary, since it is the video that is driving the rig.

Emily was a slightly special case because we were going for a one to one. So Emily O'Brien's performance was captured on one day and Emily O'Brien's face was scanned at ICT on a completely diffent day. We then used the scans to build the rig and used the video to animate the rig. But in theory, Emily's performance could be used to drive ANY traditional facial rig. Even Shrek! :)

[circusboy]

No. Universal Capture is an awesome system, but its limiation is the use of predetermined "clips". These animated clips are perspective-projected onto the model as textures. While the Universal Capture system allows for huge flexibility in the rendering stage, it allows for very little flexibility in terms of being able to change the facial animation. In other words, you're stuck with the animation in the clips you've captured.


Although one could argue that Universal Capture more closely captures an actor's performance (if he happens to do a great one-which is what actors are suppose to do) *as it happens*. And as he wanted it.

Then if the director wants a retake Universal Capture maybe the best, fastest way to 're-capture' the acting performance and integrity with the new request.

Meanwhile an Image Metric's style system not only tweaks but then also re-interprets the actor's performance by skilled animators BUT who are not 'Hugo Weaving'. So you could say the performance is more diluted than the Universal Capture system. Because no matter how good 'actors' animators are suppose to be-they never will be Hugo Weaving so that nuance will forever be 'off' or 'different'.

Not saying one is better than the other-but there almost could be a fork in the road here.
Generic performance and capture performance. But which will be easier to get a 'best actor'
Oscar for? :hmm:

[ice-boy]

isnt james cameron using this for hes new movie AVATAR?
if he is then you guys are awsome. Cameron always uses the best. he is always making something new and groundbreaking.

if you analize something for more minutes you will notice that it is CGI. but if this video was playing on the street i think that 8 of 10 people would think that it is real.

i am not an expert. so as a guy who likes watching movies and as a guy who likes to watch CGI in movies i think that this is 10/10.

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.