Intergrating 3D into real life footage

[rendermania]

matchmoving, compositing, colorgrading and grain, basically. There's no magic to it, but it takes some skill and lots of post tweaking to match the lighting, look and motion of CG content to that of a film or video backplate.

[basilisk]

I am currently trying to develop a low budget way of doing this. Currently working on a music video - you can see the first half here:
quicktime (28mb) (http://basilisk.co.uk/biomechanical/first2minV2sor3.mov)

Going for a very stylised look which simplifies things, but some of the rules are the same. You want the footage and CG elements to occupy the same world, even if it is not a photorealistic one.
I agree with Rendermania on the fundamentals. I find the key is to shoot your footage to keep the problems to a minimum. You can shoot on a tripod, and then add movement in post. Close up shots allow you to have a blurred indistinct background. Even handheld shots can be matched by estimate in some cases. After Effects is your friend. 2D motion tracking is not rocket science and can give a fairly tight match between cg elements and video footage for zoom shots. Render out lots of elements in C4D with alpha channels then layer them in After Effects. You can use AE 3d compostions to create camera moves. Don't let the camera stay on anything too long, and you don't notice the fakery so much.
Footage shot on Canon XL2, in a barn. Color Finesse plugin in AE allows you to tweak you color and levels with great precision to match CG and footage. Adding grain is a good idea, but quite time consuming, I haven't done it in this case. If I was going for a more naturalistic look, and had a higher budget, I would probably do it.

James Newman
basilisk.co.uk

[thedoc]

Okay...
is there a way of doing it in Cinema? a More technical answer is needed if you can... thanks

[KBOC]

There's only one part of this I don't get... let me explain what I believe I do, and you can correct me if I'm assuming wrong:

I get that precision of camera placement in relation to scene elements and their scale is necessary...

The part I don't get is this: how do you know what scale to make the scene in relation to your camera? Or does it matter?

For instance, say I've got a camera placed in the middle of a street. 45 degrees to the right and fifty feet away is a building that is twenty feet tall and another on the left exactly the same. The camera uses a 35mm lense (or it's equivelent on whatever format film)

The camera lens is parallell to the ground and stands five feet off the ground...

Okay, so all the elements are measured and placed into the 3D Scene in order to render 3d animated action that'll go into the scene... accounting for camera angles and rotation, etc...

But here's the part I don't understand: What is the actual scale of the camera I'm using in 3D? I'm pretty sure Ineed to know this in order to scale the scene appropriately.

[basti]

hi
have you tried scaling your camera in c4d (or any other 3d application)? it won't change anything in your cameraview
only filmback and focal length are important to match up a scene

and matching up a scene means to find out the position and rotation of the camera (but not the scale)

basti :)

[thedoc]

how does "match moving" come into it? come to think of it... what IS match moving?

[martinp]

Matchmoving is the general term for working out (from a series of 2d stills/frames of a film) where the camera was and where it was pointing at any instant of the film. Once you've done that, you can start to build proxy 3d objects in your package that will let you work out where your 3d components should move and how their shadows should fall on existing footage.


In your case, it's just a single frame.

Martin

[martinp]

Should have said in my previous post. There's a very readable introduction to all this in


Matchmoving: The invisible art of camera tracking by Tim Dobbert


For a scene like yours, he works through an example in the first chapter.

Basically:

Put the image in the background
Make a camera with the right film back etc
Build a rough proxy for your first building (guessing the height, distance if necessary)
position camer and building accordingly
rotate camera to get one corner to line up with the image
Create a null and snap it to the corner from the last step
put the camera under the null
Rotate (around Y) the null to bring the rest of the builing into rough alignment
Scale the building up, across, down etc to get a better match (watch out you don't grow it into the ground i.e. check your scaling point)
Now play with camera focal length/distance until happy......this may take a while :-)

Martin

[protocultura]

I think for a more simple aproach. Why you don't try boujou or some camera tracking software. I made some time ago a little animation with 3D objects into a real video, and boujou solve the integration problem. boujou (and other tracking camera softs) reads the 2d still images from your video and decode the camera position, angle, pan, tilt, etc. making a virtual reproduction of the camera and placing reference null points. Then is only matter of export this model into cinema.
Cheers.

[basti]

I think for a more simple aproach. Why you don't try boujou or some camera tracking software. I made some time ago a little animation with 3D objects into a real video, and boujou solve the integration problem. boujou (and other tracking camera softs) reads the 2d still images from your video and decode the camera position, angle, pan, tilt, etc. making a virtual reproduction of the camera and placing reference null points. Then is only matter of export this model into cinema.
Cheers.

hi
a cameratracking software will not work with a single image. all the algorithms need parallaxe (which means at least two images taken from different positions). although it is possible in some matchmove tools to define some perspective lines as eg x and y axis to calibrate your camera for a still frame. but it's not working very accurate and sometimes not working at all. and i guess spending about 5000$ for a matchmove application just to match a still frame is a bit too much :) these are tool to match sequences which is close to impossible to do manually (frame by frame).
trying out mm packages is pretty much useless because you won't find a demo that allows exporting.

also check out this thread.. some nice explanations in it, too
camera tracking (http://www.cgtalk.com/showthread.php?t=242847)

basti :)

[martinp]

This is certainly the way to go if you have motion involved. Programs like Boujou and Matchmover rely on the parallax differences between all the elements of the individual frames to do the calculations in the first place.Working with a single still image isn't necessarily going to be any easier in these programs.

[martinp]

Beat me to it :-)

[protocultura]

Sorry for my bad english. I'm lost in translation :scream: . Where you read "2d still images from your video", please read "the frames from your video".
Cheers

[pnoland]

If you get the time visit www.pixelcorps.com (http://www.pixelcorps.com) and view the free tutorials. You'll have to give some info like name and email but they will not use it for spam purposes. Alex Lindsay made some great tutorials showing the pipeline of basic VFX from Modo, to Cinema 4d, to Motion Builder, back to Cinema 4D, to Shake for the final output. This will give you a basic idea of putting characters into live environments but mainly this shows for a still background but it's idea is the same for video. Good match moving is really important in the production of VFX to get the most realistic look so you can hardly tell that the animated object(s) aren't in fact 3d. The main match moving applications out there like Match Mover Pro and Boujou are pretty expensive but you can find some alternatives. Iccarus was a freeware (for personal use) program that was great but it's extremely hard to find now. PFHoe is another great one based on Iccarus that's really inexpensive and some people loving it from what I understand. It can be found here: http://www.thepixelfarm.co.uk/contentfull.aspx?content=pfhoe.htm I tried the demo for it and it seems like a very easy to understand application. This is one area I'm trying to begin getting into and I'm considering using this package for my camera tracking but I'm looking for some good resources for video plates since I do not have a DV camera yet...funds will not allow it yet.

[KBOC]

hi
have you tried scaling your camera in c4d (or any other 3d application)? it won't change anything in your cameraview
only filmback and focal length are important to match up a scene

and matching up a scene means to find out the position and rotation of the camera (but not the scale)

basti :)

Yes, I understand this. I'm sorry if I wasn't clear. I wasn't talking about scaling the camera, but scaling the scene in relation to the camera...

Can I depend on the unit measurements in a 3D ap for this purpose if I set them to represent feet or inches, for instance? Or do I have to compensate by adding a multiplier?

[Primitiv]

There is a plug-in called PhotoMatch that you can find here: http://www.vreel-3d.de/plugins/PhotoMatch/links.html

I am not sure how it can help with a moving camera but it make the integration of 3D objects in real life scenes pretty easy.

[Simon Wicker]

Yes, I understand this. I'm sorry if I wasn't clear. I wasn't talking about scaling the camera, but scaling the scene in relation to the camera...

you only have to worry about the relationships for everything to work, cinema doesn't care about the units. if your object is 25 units away and 12 units tall this could be mm, cm, miles or whatever. looking through the camera you will always see the same thing. you can test this out yourself. place a camera at world zero looking straight down the z axis. make a cube 10x10x10 that is 100 units away. make another cube 100x100x100 1000 units away. looking in wireframe mode the cubes are co-incident.

during the matchmove process if you wish you can define a scale for your scene - normally by measuring a realworld object in your matched scene and then telling the app how big this object is supposed to be (in this way a matchmove unit is scaled to the measure you wish to use when you transfer across to your 3d app).

cheers, simon w.

[thedoc]

Okay...

I have'nt FULLY explored Cinema but i thought that maybe the 'background" object was used for intergration. So matchmoving softeware analyzes your footage? How does cinema comeintothe pipeline?

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.