3d animation triggered live on tv

[AutodeskLab]

textures should be rendered and calculated before tv show.

[bsteagal]

Selpakius

You can do this with the voice device and character face, but they have to be done with a character that actually has the morphs (or blendshapes) built into the character and exported to Motionbuilder. It is not perfect, but it will try to lipsynch as best it can, with relatively good results. Your performer would need to have a wireless mic that is attached to the pc running mb, and the voice device has to be run from that audio input

You can PM me for more details if you want more information,

hope this helps

[Selpakius]

Hi AutodeskLab, do you mean "baked textures and lighting"? Thanks

[Selpakius]

Hi bsteagal, after some research came across a bunch of facial motion capture systems. Thank you, I am aware of Motion Builder and may even try "face robot" in XSI for that part, but I guess my knowledge is weak on the capturing system part. Have you heard about phasespace? Have you used their products? or is there any other system that you can recommend?

p.s:I was going to PM you but then I thought it may be a better idea to keep the conversation here so that others can benefit as well, if thats ok? :)

[bsteagal]

Hi Selpakius!

Perfectly ok.. My intention was to describe the process if you were interested...

Now, I apologize beforehand because this might be a long post...:blush::beer:

I've actually done some live performance with mocap, and if money and space were no object, I'd probably go with a Phasespace, but to do this you are talking some serious money, since you need to have more than your standard setup to do this (back in 2008 I got a quote from them to buy a system for 2 person capture, and I was looking at 85k, and this was ONLY for body capture, did not include face or fingers). Prices have probably dropped some since then, but I'd guesstimate still about 100k or more, and you need space to set it up, probably around 20x20 or so

There are a few other systems out there, but most do not combine both types of capture and only 2 (phasespace and vicon) include finger capture in there too

I use an Optitrack system for body capture and motionbuilder for the voice capture (using a wireless mike that runs the voice device, allowing my character to talk) and then use other devices such as a joystick and keyboard (run by an assistant) to do some amount of finger animation and some expressions. The only drawback here is space, you need at least a 20x20 space to mount the equipment to get a 10x10 capture volume. This means that unless you have a separate stage to run your performer, it is hard to do live performance, but doable nonetheless. Optitrack does have a facial mocap solution, but you cannot do both simultaneously. The only way I think you could do this, is by having two performers run one character in mb.. (one system for body, another for face, and both streaming to mb on 1 character). This would cost about 20k or so for two separate systems, plus the cost of mb. (just a side note.. having two operators for one character is used quite frequently.. Jim Henson Studios, for example, for their Sid the Science kid program, use a mocap actor and a puppeteer to run the face for each character.. the show is recorded with live performance, then the rest is added in post)

There is another route, and one that I'm hoping to have the funds soon to try, is by using something like the Animazoo Gypsy 7 for body capture (about 9k with mb plugin), together with mb's voice device.. the advantage here is that it can be used in a very tight space, and the system has two joysticks as part of the suit where you can build in triggers to do a certain amount of hand and facial animation.

There is one other facial system that just came out, although right now it is strictly post, and that is Maskarad by Di-o-matic. I've been doing tests by attaching a camera to a head harness and capturing video at the same time that I'm doing my body capture with some success. This is the first release of the software, so there are a few things yet they need to work in, but still pretty good and if they ever make it realtime... you could attach it to any mocap system and do both (price for Maskarad is 1500)

I am a distributor for both Naturalpoint and Di-O-Matic and know the distributors for Animazoo if you'd like more info..

phew!:cool: hope I didn't bore ya..

[Selpakius]

bsteagal, thank you very much for your time on writing back such great info!

I checked all the systems you mentioned. It looks like they have advantages and disadvantages in different areas.
The system I need needs to do the facial and body capture simultaneously, with facial being more important then the body, body needs to be solid too of course, with fingers.

From your experience, how was the performance(quality) of the characters on screen when you the character was controlled by two people? Do you have any link to an example?

Would it be correct to say that there would always be a need for an operator? How do eyes work for instance? they cant be rigged, so I am assuming thats definetely an operators job? or maybe be an automated rig? what about blinking?

Curious about whether capturing the facial animation with a mic gives better results then using markers on the face?

Space wont be an issue, money is partially, however if the right system is in front of us, our perspective may change as well.

Thanks again!

[bsteagal]

Selpakius:

As far as 2 people running one rig, the only example that I have is Sid the Science kid, but I'm not sure what mocap system they are using (probably a Vicon), and their facial animation rig is a proprietary finger puppeteering rig.. here's a video showing it

http://www.creatureshop.com/sid.php

go to the "making of"

Now, when I said that Naturalpoint's system could do this, I was speaking theoretically, I haven't heard anyone trying it yet... but there's no reason why it couldn't be done, it is just a matter of having 2 systems stream to MB.. the quality is there too.. a lot depends on your rig. Here's a tutorial on one way of doing facial mocap.. on a bones based rig:

http://www.naturalpoint.com/optitrack/products/motion-capture/tutorials/expression/

(click on the second tutorial)

As far as eyes go, you can constrain these to a joystick or mouse, and the blinking you can build an automatic relational constraint to do this (which will blink randomly) or you can constrain it again to a button on a joystick. (it is very simple..) This is why the animazoo gypsy could be very intersting, because you have a whole series of buttons built in on the two joysticks that are built into the suit. (the suit is tethered, so it does limit you somewhat, via USB) I've used a wireless mouse to do some of this, but you are restrained on the distance...

A mocap system would give you better results than a mic, as far as I'm concerned, because the voice device cannot map emotions, only phonemes, but in a pinch, works very well, especially if you have someone who is a good puppeteer.

Now, if you really need finger animation, you can always try one of the different gloves, but I've talked to some that have used them and they consider them a bear to configure correctly with other systems. Phasespace would be the way to go there, since they have an all integrated solution.. but here's a link so you can look at different types of gloves (they run anywhere from 5k to about 15k.. each glove!, so that should give you an idea... grin)

http://www.inition.co.uk/inition/products.php?CatID_=12

It all depends on what you are planning on doing.. Most of what I do are very cartoony characters, so I don't really need a lot of flexibility...

hope this helps!

Bryan

[Selpakius]

Bryan, I am so lucky that you are on this thread, you are really helpful!

Jim Henson' show is outstanding. It has pretty much everything one would dream of for realtime mo-cap, but it is probably millions of dollars worth investment in that room. Very impressive.

On the other hand, like you said, I see the possibilities of how much of it can be done with Optitrack or other similar systems. Thanks for the link to the tutorials. Yes rigging is extremely important for having a good look/deformations on the character.

That being said, the system I am after will be used (for now) with cartoony characters. But different projects might bring different needs, so thats why I am trying to figure out something that can transfer the most of the actor's performance. when using markers( or LEDs) on the face, 6 seems to be the number used in most cases, however I think thats a quiet low number for facial deformations. Do you think that the systems you've mentioned can handle more markers on the face? And also remember that it will be 2 persons acting at the same time.

Say the body is captured with optitrack, is there a better option for the facial captures that can be used simultaneously? Not sure if MB can read multiple inputs(systems at the sam time)?

Thanks again,

Serkan

[bsteagal]

Serkan

You're welcome, glad to be of help!

Optitrack's expression handles much more than 6 markers, in the tutorial that I linked, I believe there's 41 markers and has different configurations depending on the software being used (facerobot, mb, etc), so that is not a problem. Now, Motionbuilder natively, via Actor face, uses 17 markers. These are then tied in to the morphs or blendshapes mapped in Character face... that is why if you are going to do this, make sure that you have the morphs that correspond to the ones requested in Character face. Better option in my opinion would be to rig your character face like the one in the tutorial, which should allow you even more control than MB native Actor face..

Now, MB has no problems receiving inputs from several sources, that is more dependent on the specs of the box than the software itself. Optitrack systems stream to mb via IP address. So you would have, let's say, two laptops... one running a 12 or more camera body mocap session while the other runs a 6 or 7 camera face mocap session... these stream via IP to 1 MB box, where you then map the inputs to your character (via Actor and Actor face or via a contraint to the bone rig face... you can download the sample from the video to play around with if you want). I can find out for you if you want if anyone has done this....

Now, there are other considerations to think about with MB though, the lighting and shading is not as sophisticated as with maya or max, so it is relatively simplistic (it is real-time rendering, remember...) I use mb 7.5, I think the newer versions of motionbuilder have better lighting and shading, but I'm not sure (AutodeskLab.. can you provide some input here?) and your textures have to be low-res, otherwise you can choke it really quick.. Jim Henson's show, although it is recorded with live actors, is NOT live broadcast, it goes thru a final stage where animation is refined, textures and lighting is applied, etc, etc. If your characters are cartoony and relatively simple, this is not so much of an issue.

One other thing, this is fine and dandy if you only have one character... The optitrack system can only capture officially up to 2 actors, so if you needed to have 2, you would require 3 setups, 1 for the body of the two actors and 2 separate face systems... so it would get relatively complicated... if you need more than that.. then you have to look at something else..

If you think your needs would balloon out to several actors at a time, go with the Phasespace..

Now I have a question for you... how LIVE does the system really have to be? if it is a show that is recorded, let's say in the morning to be shown at night, I would feel a lot more comfortable with any of these solutions, because you would always have time to refine the animation and if there are glitches, you can correct those. (and trust me.. you WILL have them)

Every mocap system has its defects.. Optical systems suffer from occlusion (markers being hidden) except for Phasespace, in which the markers are active LED markers that communicate with the system.. so they are never hidden. Inertial systems (like the animazoo suits) don't do a good job of locating themselves in a capture volume, and have to depend on methods such as sonar (with Animazoo) or software to correct this (and suffer a little more of issues like feet sliding, etc)

phew! now that really dragged on.. sorry about that!:rolleyes: I can go on and on if not careful.. *grin*

[bsteagal]

oh, one other thing. Animazoo has a product that is somewhat what you are looking for .. which is used live, and they create the characters specifically for use with their setup.. called animalive

http://www.animalive.com/

they use in this system the voice device, and they don't bother with animating the hands too much.. At least it can give you an idea of what is possible going this route... now you can do this with other systems too... like the optitrack, or moven, or any others..

later!

bryan

[Selpakius]

Hi Bryan, thanks for all the info as always !

Animalive looks promising,however I will probably go with one of the other systems we've been talking about here.

The show will be as real-time as it gets ! :) Meaning, there wont be any time for doing renders on a farm or anything. Everything will have to run realtime while the show is running.

Very good question tho. I spent sometime this weekend making researches on (say the motion part is running ) how to get the best look (render) of the character for the broadcast. Since it is all live, I won't have time to render anything or do any kind of post process. But I need the system that can take the animation, say from motion builder, and send it to the switch(I think this is what they call it in broadcasting, the device to comp the 3d render onto the real footage) in the fastest and the best looking way.

I was thinking maybe a system that is close to a game console could be used? but I thought about this mainly because of my lack of knowledge of what systems are available for that purpose in the industry nowadays. Simply put, trying to figure out how to real-time render this 3d character with real-time shadows. Any ideas on this?

I know you mentioned MB7.5 (or maya/max) however, maybe I am missing the point here, do you mean applying realtime shaders to the character and somehow sending (exporting?linking?sharing?) the viewport play to the client system that will comp it with real footage?

I think I am a bit confused about how to output the realtime render from motion builder to broadcast..

Thanks man,

Serkan

[bsteagal]

Hi Serkan

Ok, I don't know that much about broadcast systems either, as I've never had to do something like that, perhaps you could ask the TV station that you might be doing this for? Or if anyone else out there reading this could pipe in, that would be greatly appreciated. Since I do most of my work like this for kids, they don't care really what is in the background, and I've never done this overlaid on live footage..

Your system running MB (whether it be 7.5 or 2011) will be doing all of the rendering on the fly, which is why you need a system with good specs for this. I would probably suggest something with a very powerful 3d workstation class card, quadro fx or firepro. When you import your character into MB, you import it via fbx with all and textures (depending on the resolution of these textures, you might have to bring them down to use them effectively). In MB, you can add lights, shaders (for transparency, etc). But everything you add takes resources...

What I normally do, is setup multiple monitors and output the MB viewport to just one of these (which is the one usually where the kids are looking at). If your card has output to S-Video, that could be used too... Now, I don't know if this would work or not.. but you could setup a camera in MB, set the background to saturated green/blue and have the broadcast switcher key this color out (like they do with weather reports on the news)... The station could also add a "mask" so that just what is in a certain area of your output will be displayed...but again, I've never done this before either...

I'm including a link to a snapshot of a character viewed thru a camera setup in mb.. with background and floor set to blue for keying, This character only has basic shaders setup and has very simple textures.. but can give you an idea of what it looks like, the camera is set to antialias and render special effects live..

http://www.kkstudio.us/cc/screenshot.jpg

I would experiment and research what the different shaders in MB do and what type of lighting effects it can give you...

cheers!

Bryan

[Selpakius]

Hi Bryan,

How you been? Sorry, I couldnt get back to you for some time, I had so much other stuff at work that I had to put this project a side for a few days, then a small long weekend escape, and here I am again!

Coloring the Background in MB for keying is genious! Thanks.

I was able to reach the guys who worked on this spot sometime ago:
http://www.metamotion.com/spotlight/semerad_tv.html
they've apparently done the same thing as you mentioned. I think it looks good, video is compressed and all but I think that part is solved.

I finally had a chance to see phasespace in action, in person. Very impressive. However just like any other system it has some limitations. I think the bigegst problem with it is the fingers, they said it may be too risky for a live broadcast, because of occlusion problems. I might have to look for a different solution for that. Maybe using a joystick for gestures, or combining a different system. Any ideas?

Thanks a bunch as always,

Serkan

[bsteagal]

Hi Serkan!

I'll be honest with you, in my opinion, using any optical system in a live broadcast like that is risky.. because of that one issue (occlusion) , this doesn't mean that you couldn't be successful at it, only that it is risky.....(if you have enough cameras, and you calibrate really well, there should be any reason why you cannot do it..) that is why my suggestion was to use the Gypsy 7 or the more expensive Moven or Animazoo igs-190.. which are inertial based and don't suffer from occlusions, and Animazoo can help you both with designing the character and building in triggers do do things like basic hand motions. But then again, they don't do facial animation, so you are stuck either way... no easy solution here really..

For the fingers.. there are also gloves, like the ones I pointed out before..(5dt, etc, that website http://www.inition.co.uk/inition/products.php?CatID_=12 has a lot of info) but can be hard to incorporate to the overall solution.

So let me run thru this really quick

1.-Naturalpoint systems do not do fingers, and would need to have 2 separate setups to do face and body... risky and untested so far..
2.- Phasespace can do all, but as they mentioned, risky for the fingers
3.- Inertial solutions don't do facial, you would have to build in triggers (which is very doable, but wooden, not very realistic), these systems don't suffer from occlusions... (as the others do) so data would be more stable on the body. Would have to build in triggers for simple hand motions
4.- VR gloves are a bear to incorporate with other systems

I know you want to get something you can use all around, but in this case, I'd gravitate more towards the inertial system, since you can eliminate some of the risks inherent in optical systems, but will have to live with facial and finger animation that is not as realistic as you'd want.. Do you have a character already designed for it? I can get with animazoo to see what they think of your project and what they could do..

so.. Phasespace or Animazoo? depends on the budget and what you want to live with...if wooden facial animation is ok.. go with the animazoo...if you don't mind riskiness with occlusion, and budget is no issue.. go with the phasespace.. each have their strengths for something like this...

Bryan

[bsteagal]

Hi Serkan

Thought you might want to see this..

http://features.cgsociety.org/story_custom.php?story_id=6096

cheers!

[Selpakius]

Thanks!

Serkan

[Selpakius]

Thanks also wrapping it all up!

Even tho hands are super important, I had an idea for making things easier maybe. Which is basically having the character hold a tablet device(like an ipad) and use it during the 2 minutes of his presence. I thought this way, we can lock one of the hands to even a single pose for holding the tablet device, and the other hand can be animated(triggered) between a few poses to make it work with the tablet.
Cause I think the priority will be the facial animation.
Also, when I was at this one studio earlier this week where they use phasespace, they said that I would need a good 2 rings of cameras for capturing the facial animation without any glitch. Of course, more cameras means more precision, but I was wondering how many cams you guys use for facial capture in general?

On the other hand, inertial systems are really cool! cause you can take them anywhere without the need of cameras and stuff. I really wish there was an inertial facial system out there that can do both! damn! heh

[bsteagal]

Hi Serkan

For the naturalpoint system, we are looking at 6-7 cams, but you can use more. You are limited on the space though, as it is designed to have the actor relatively static (like sitting down) to do the motion.

How soon are you planning on implementing this?

[Selpakius]

Yes, I figured that its more like a desktop system for now.

I will give the project a green light if I can solve this in the next week or two.

[Selpakius]

Besides the fingers and facial, I guess I would need a system for each actor if I were to go with animazoo or any other inertial system..

[DSW]

There is also Zign Track 2 for facial animation capture:

http://www.zigncreations.com/zigntrack2.php

And the 5DT data glove for hand/finger animation:

http://www.5dt.com/products/pdataglove5u.html

We'll be checking out Maskarad one of these days - just like it since it does not require any special setup like little balls on the face. :)

[bsteagal]

@ Selpakius

Yep, you would need one for each actor.. and, if you were going to have them interact directly, you would probably want to get their solution for positioning the characters in a given volume (sort of a sonar) which is the problem with these types of systems (one is no issue.. two, you get into problems) and especially if you are doing it live. When suggesting this system, I wasn't thinking more than one actor... If you don't have them interact, you could probably get away without it...

@DSW - I have Maskarad, and I'm a reseller for them if you need any info.. works good, however, as it is the first version out, there are things that will be coming in the next few months.. first after this version is the ability to blink, then following eyeball movement.. I don't have an ETA yet though, except for that they are working on this and will be coming out shortly...

[Selpakius]

Maskarad looks very promising! However the fact that you cant move away from the desk/cameras is a problem for what I am after for.

Have you seen this?
http://www.image-metrics.com/full_performance_capture.html

I dont think that it can do the facials at realtime tho, I sent an email to them, will share the info with you.

[DSW]

@DSW - I have Maskarad, and I'm a reseller for them if you need any info.. works good, however, as it is the first version out, there are things that will be coming in the next few months.. first after this version is the ability to blink, then following eyeball movement.. I don't have an ETA yet though, except for that they are working on this and will be coming out shortly...
This is good to know - especially the extra functionality of eye-blink and movement. We most likely won't have the time to test this out for a few months, but I seriously like the fact that it is markerless. We have voice talent from around and outside the country and using a recording of their performance to create facial mocap would be far easier than trying to recreate it using a markered system - and it would save considerable amount of time!

When we're ready to begin our testing, I'll let you know. We have to create a system that will easily allow us to integrate it with MotionBuilder and Lightwave. We've used Optitrack (8 cameras) with MotionBuilder and LW and it's worked perfectly - though facial animation is quite a bit more detailed. It should be interesting. :)

[bsteagal]

@Selpakius

Actually... You can use Maskarad with a head-mounted camera trained on your actor.. doesn't have to be in a booth or in a specific recording space. Pair this with the ability of the Optitrack system to capture video with the "scene AV camera" that is synched to your mocap take...

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.