A question on rotoscoping. How do the pros do it?

[DLangley]

Check out this site, by Scott Squires. There is even a video podcast in two parts that takes you through roto.

http://effectscorner.blogspot.com/

[visualchaosfx]

Hey thanks for the link. There's no way of saving the videos to the hard drive is there?

[evanfotis]

It is always better rotoscoping in AE 'cause as you preview, you can fix the jaggies and such.
Animating masks with as few keyframes as possible and or colour/luma keys.
You can also try Silhouette Roto (http://www.silhouettefx.com/roto/)plug/stand alone.

[joviphang]

Pros did what most of us did, muilt masking, paint them in white matte and similar back one frame and forward one frame to check. Is a long process, but is worth it.

But what pros in the film industries are getting, are different from us. They got really high resolution, 32-128 bit for compositing. Infos from broadcast Asia, Discreet was giving a talk

When we do mulit masking in a 8-16bit footage is a bit hard to see which portion is what we want to mask, but 32 bit and above could clearly see those portions. So that why u got moving edges, even u do it cautionly. But is alwalys good to know how is done.

correct me if I am wrong, I m also learning.:)

[visualchaosfx]

When we do mulit masking in a 8-16bit footage is a bit hard to see which portion is what we want to mask, but 32 bit and above could clearly see those portions. So that why u got moving edges, even u do it cautionly. But is alwalys good to know how is done.

I know what you mean. I was shooting my stuff with an analog Sony handycam. I was having a hard time distinguishing the edge of myself and the background that I needed to take out. I just bought me a DV camera from Walmart yesterday and shot a test footage to roto and I'm getting much better results. The edge of my mask isn't dancing alot when I play it back now. Plus I read some tips on how to move the points in groups rather than 1 point at a time. My footage is only 9 seconds long but my goal is to composite myself onto another enviornment which I'm going to design. Actually, last night it took me 2 and half hours to animate a mask around myself. But I'm still in the tweaking phase so hopefully I'll have the masking all done maybe tonite or tomorrow and then I can start designing my enviornment for compositing.

[visualchaosfx]

Here is the test shot I'm working on. Its not done yet but I wanted to post it on here and see what ya'll thought. The background image is not what I'm going to use in the final version. And I still have some more rotoing and tweaking to do.

Roto Test (http://www.wsfilms.net/roto.wmv)

[DLangley]

If you have iTunes and sign up for the podcast (its free) it will download it to your drive in the iTunes folder.

[visualchaosfx]

WOW I didn't know that they did rotoscoping in the movie Dragonheart with Dennis Quaid. I thought that when Dennis interacted with the dragon, he was filmed infront of a bluescreen. But he wasn't. Thats friggin awesome!! Now I'm gonna go rent the movie again.

[joviphang]

take a look at "waking life" too, sometime different with rotoscope!

[beenyweenies]

Here is the test shot I'm working on. Its not done yet but I wanted to post it on here and see what ya'll thought. The background image is not what I'm going to use in the final version. And I still have some more rotoing and tweaking to do.

Roto Test (http://www.wsfilms.net/roto.wmv)

just watched your roto test.. Not bad! There seems to be some floating around your ear and glasses, but here's a good tip for roto work that will fix this problem AND make your life much easier all around. Don't make one big roto mask for your whole roto job. Make one mask for the main head, one for each ear, one for your neck, one for your trunk, upper and lower arms etc. Sounds complicated and time consuming I know. But truth is once you build these masks the truly hard part, animating all those points, gets much easier and delivers much better results.
The idea is that the shape of your ear (or neck, etc.) won't change much, it's just moving. So if you have a mask just for your ear, you can animate the position of the entire ear mask (with a few minor point tweaks if needed) rather than nudging points around all afternoon. This technique actually leads to far fewer points needed to form a really accurate mask. Your ear isn't very complex, it's the areas where it attaches to your head that require extra points and are tough to smoothly animate. With a separate mask, this issue is ancient history. The corners around your ears, neck etc. will look very clean and natural.
One last point - using one large roto mask you can and will end up with dozens of points that are orphaned (the object is no longer visible but the points on your mask still exist) yet those points still need to be animated for every frame, which leads to a lot of matte "chatter" and a wobbly, hard to control mess. Using the ear example, when you turn your head your "more complex" ear is replaced by your "smooth" cheek but you will still have all the points that formed your ear to wrangle with. Multiply that by all your body parts, and you are looking at quite a mess.

The time it takes to make all these masks is not much more than making one big mask, and the results are guaranteed better. This is usually how it's done at ILM, The Orphanage and most other VFX houses.

For the newbies (you already did this in your roto test) make sure to add some feather to your roto masks (1-2px is usually enough), using an amount that isn't excessive but matches the natural feathering present along the edges of the element you are rotoscoping. Mask feathering is almost always overdone and a dead giveaway of a faked shot. Go easy!!

If you're really into roto, check out Pinnacle Commotion (http://www.123dv.com/catalog/pinnacle_commotion_pro_4_3942610.htm).

Good luck!

[visualchaosfx]

just watched your roto test.. Not bad! There seems to be some floating around your ear and glasses, but here's a good tip for roto work that will fix this problem AND make your life much easier all around. Don't make one big roto mask for your whole roto job. Make one mask for the main head, one for each ear, one for your neck, one for your trunk, upper and lower arms etc. Sounds complicated and time consuming I know. But truth is once you build these masks the truly hard part, animating all those points, gets much easier and delivers much better results.
The idea is that the shape of your ear (or neck, etc.) won't change much, it's just moving. So if you have a mask just for your ear, you can animate the position of the entire ear mask (with a few minor point tweaks if needed) rather than nudging points around all afternoon. This technique actually leads to far fewer points needed to form a really accurate mask. Your ear isn't very complex, it's the areas where it attaches to your head that require extra points and are tough to smoothly animate. With a separate mask, this issue is ancient history. The corners around your ears, neck etc. will look very clean and natural.
One last point - using one large roto mask you can and will end up with dozens of points that are orphaned (the object is no longer visible but the points on your mask still exist) yet those points still need to be animated for every frame, which leads to a lot of matte "chatter" and a wobbly, hard to control mess. Using the ear example, when you turn your head your "more complex" ear is replaced by your "smooth" cheek but you will still have all the points that formed your ear to wrangle with. Multiply that by all your body parts, and you are looking at quite a mess.

The time it takes to make all these masks is not much more than making one big mask, and the results are guaranteed better. This is usually how it's done at ILM, The Orphanage and most other VFX houses.

For the newbies (you already did this in your roto test) make sure to add some feather to your roto masks (1-2px is usually enough), using an amount that isn't excessive but matches the natural feathering present along the edges of the element you are rotoscoping. Mask feathering is almost always overdone and a dead giveaway of a faked shot. Go easy!!

If you're really into roto, check out Pinnacle Commotion (http://www.123dv.com/catalog/pinnacle_commotion_pro_4_3942610.htm).

Good luck!

Hey thanks for the tip. In this roto test, what I did was once I animated the mask, I replaced the background with a green screen by making a new solid. And then I made a new composition and dragged the first composition and keyed out the green with an edge feather of 0.5. I then used matte choker and adjusted the Choke 1 and grey level values to whatever looks the best. It seems to work better doing it this way rather than adjusting the feather on the mask itself.

In my next roto test project I will mask off each individual body part rather than creating one big mask. And thank you for the link.

I'm also wondering, I'm using After Effects 6.5. Are the masking tools so much better in After Effects 7 or are they basically the same?

Added: Rather than starting a new thread - I am basically done with the rotoing part and I am beginning to composite myself onto another background plate. The roto clip was shot outside on a cloudy day and I want to composite myself into an interior background. How do I go about color matching it so I look like I belong into the interior scene? I've been playing around with some of the tools in AE but I have a feeling I'm not using the right one. Here is link to what I'm talking about:

Color Matching Images (http://rdelavegan.addiction-host.com/nic.html)

A user from FXhome.com used Composite Lab Pro to do the following images of integrating himself into a scene with Nicole Kidman. And the color matching looks superb. This is what I want to do in After Effects.

[beenyweenies]

In this roto test, what I did was once I animated the mask, I replaced the background with a green screen by making a new solid. And then I made a new composition and dragged the first composition and keyed out the green with an edge feather of 0.5. I then used matte choker and adjusted the Choke 1 and grey level values to whatever looks the best. It seems to work better doing it this way rather than adjusting the feather on the mask itself.

Just out of curiosity, why would you go to the trouble of roto, only to then go in and do a mock greenscreen setup? If this was all for the edge feather, I assure you this method will not get a cleaner edge than feathered masks. If your mask feather wasn't giving the look you wanted, you could always omit the greenscreen step, and just matte choke the precomped roto. But even this should be totally unnecessary if you set up your mask properly and everything is working properly.

I'm also wondering, I'm using After Effects 6.5. Are the masking tools so much better in After Effects 7 or are they basically the same?
The masking tools are basically the same. No serious need to upgrade if roto and masking is your primary interest.

Added: Rather than starting a new thread - I am basically done with the rotoing part and I am beginning to composite myself onto another background plate. The roto clip was shot outside on a cloudy day and I want to composite myself into an interior background. How do I go about color matching it so I look like I belong into the interior scene? I've been playing around with some of the tools in AE but I have a feeling I'm not using the right one. Here is link to what I'm talking about:
It's complicated, but here's the breakdown of how we did it at The Orphanage, if you care to follow it step by step:

1. Drop the rotoscoped footage of you into a comp with the background plate

2. Add an adjustment layer on top of this stack

3. Apply a seperate "Levels" effect to each of your three layers

4. Use the levels call on your background plate to get it looking exactly how you want it (lock it down!). Then, use the levels control on the foreground layer and do your best to get it matched in using the RGB mode of the Levels control. Because your FG was shot outside on a cloudy day, you will probably need to boost the input whites (upper-right slider) and bring in your gamma a bit (pull the middle slider to the right).

5. In the "Levels" effect that is applied to the adjustment layer, crank the input white slider (the upper-right slider) all the way to the left so your image (both foreground and background) are blown out but there is still some detail.

6. Then, in the comp viewer window, use the channels toggle to show only the red channel. When viewing the individual channels like this (and with the levels cranked) you should see the tonal differences between your FG and BG elements pretty clearly.

7. At this stage you are adjusting the individual color channels to match the plates up. To do so, you will use the Levels applied to the foreground layer, but YOU MUST set it to the color channel you are viewing. For example, if you set the comp viewer to show your red channel, open up the foreground Levels control and set it to the red channel as well. Adjust the sliders to match your FG and BG. It is best to try and stick with the upper set of controls (the input black, gamma and input white sliders) since they degrade the image least, but using the lower deck of controls (ouput black and white) is ok. remember, right now you are adjusting the input/output blacks and whites for the RED CHANNEL ONLY!

8. Once you have them somewhat matched up, change the color channel to green, change the levels control to view the green channel, and repeat the above step. Do so for the blue channel as well.

9. once you are done, hide your adjustment layer and set the comp viewer to show all RGB channels. It should match in pretty well. If not, you may need to further refine your color correction, and possibly you used too much of the output black and white controls.

Anyway, this is the professional way of getting a good color correction since it matches the individual color channels, not just the RGB composite which can trick your eye. Using the above method you are viewing greyscale values which the eye is much more sensitive to.

[visualchaosfx]

Just out of curiosity, why would you go to the trouble of roto, only to then go in and do a mock greenscreen setup? If this was all for the edge feather, I assure you this method will not get a cleaner edge than feathered masks. If your mask feather wasn't giving the look you wanted, you could always omit the greenscreen step, and just matte choke the precomped roto. But even this should be totally unnecessary if you set up your mask properly and everything is working properly.

Kind of backwards type method isn't it? Anyways, after posting my procedure on how I did things I went back and took the green out and went through a readjusted the mask and so now I'm getting better edges.


The masking tools are basically the same. No serious need to upgrade if roto and masking is your primary interest.

I figured that. I think the only things they added in there was the keylight plugin and a timewarp feature.


It's complicated, but here's the breakdown of how we did it at The Orphanage, if you care to follow it step by step:

1. Drop the rotoscoped footage of you into a comp with the background plate

2. Add an adjustment layer on top of this stack

3. Apply a seperate "Levels" effect to each of your three layers

4. Use the levels call on your background plate to get it looking exactly how you want it (lock it down!). Then, use the levels control on the foreground layer and do your best to get it matched in using the RGB mode of the Levels control. Because your FG was shot outside on a cloudy day, you will probably need to boost the input whites (upper-right slider) and bring in your gamma a bit (pull the middle slider to the right).

5. In the "Levels" effect that is applied to the adjustment layer, crank the input white slider (the upper-right slider) all the way to the left so your image (both foreground and background) are blown out but there is still some detail.

6. Then, in the comp viewer window, use the channels toggle to show only the red channel. When viewing the individual channels like this (and with the levels cranked) you should see the tonal differences between your FG and BG elements pretty clearly.

7. At this stage you are adjusting the individual color channels to match the plates up. To do so, you will use the Levels applied to the foreground layer, but YOU MUST set it to the color channel you are viewing. For example, if you set the comp viewer to show your red channel, open up the foreground Levels control and set it to the red channel as well. Adjust the sliders to match your FG and BG. It is best to try and stick with the upper set of controls (the input black, gamma and input white sliders) since they degrade the image least, but using the lower deck of controls (ouput black and white) is ok. remember, right now you are adjusting the input/output blacks and whites for the RED CHANNEL ONLY!

8. Once you have them somewhat matched up, change the color channel to green, change the levels control to view the green channel, and repeat the above step. Do so for the blue channel as well.

9. once you are done, hide your adjustment layer and set the comp viewer to show all RGB channels. It should match in pretty well. If not, you may need to further refine your color correction, and possibly you used too much of the output black and white controls.

Anyway, this is the professional way of getting a good color correction since it matches the individual color channels, not just the RGB composite which can trick your eye. Using the above method you are viewing greyscale values which the eye is much more sensitive to.

Thanks for the breakdown on this. I've been trying to do it right for the last 2 and half hours. This will help me. Thanks again.

[visualchaosfx]

Ok here is an image of the color matching test. The background is not the final location. This was just an experiment. I apologize for the loss in image quality. The original footage and composition looks better than this. Tell me what you think:

http://www.wsfilms.net/colormatching

[beenyweenies]

Ok here is an image of the color matching test. The background is not the final location. This was just an experiment. I apologize for the loss in image quality. The original footage and composition looks better than this. Tell me what you think:



For a simple match job - meaning you didn't shoot the foreground specifically to match the lighting angles etc. this looks pretty damned good.

My only feedback would be this. When you color correct using the technique I layed out above, it is a great start but mostly applies to matching things near the subject. Since your pyramids are in the distance they are getting milky, a little washed out and somewhat blueish from the amount of atmospheric haze between them and the camera. Because of this, your foreground element should match color-wise, but the density of colors and the black levels could probably come up a bit. I personally would apply a Hue/Saturation to your foreground and bump it up just slightly, and, using the Levels on your foreground, bump up the input blacks a touch (upper-left slider) and reduce the amount of blue just slightly. The blueish tone of the pyramids is simply atmosphere and haze that your FG wouldn't have as much of.

But other than that it looks really good, I'm sure your final version with the right BG will be awesome.

[visualchaosfx]

For a simple match job - meaning you didn't shoot the foreground specifically to match the lighting angles etc. this looks pretty damned good.

My only feedback would be this. When you color correct using the technique I layed out above, it is a great start but mostly applies to matching things near the subject. Since your pyramids are in the distance they are getting milky, a little washed out and somewhat blueish from the amount of atmospheric haze between them and the camera. Because of this, your foreground element should match color-wise, but the density of colors and the black levels could probably come up a bit. I personally would apply a Hue/Saturation to your foreground and bump it up just slightly, and, using the Levels on your foreground, bump up the input blacks a touch (upper-left slider) and reduce the amount of blue just slightly. The blueish tone of the pyramids is simply atmosphere and haze that your FG wouldn't have as much of.

But other than that it looks really good, I'm sure your final version with the right BG will be awesome.

When I did the matching job, I actually used the photo filter in After Effects. I didn't even know it was in there until I looked at the help file. I just used the color picker and selected the sand to the left of me as the basis of the color matching and made some adjustments. Turned out rather well. I actually restarted the roto project because I wanted to try to roto parts of me instead of the whole thing. It actually works better although if I had a higher quality DV camera I think I wouldn't have any problems with the head area particularly the ears. But once I get everything else rotoscoped I'm gonna go back and fix that. The thing I like about rotoscoping, as opposed to bluec sreening is the fact that I can film just about anywhere and when I make my 3D backdrops, I can match the lighting to the 3D scene as opposed to setting up lights to get the blue screen properly lit and to properly light the talent and yadee yadee yadah. Downside, is that rotoing takes too damn long but alas, patience is a virtue. Thanks for your tips man.:thumbsup:

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.