The effect I want is depth of field in an image that blurs both the foreground and background, and only the object(s) in between are in focus. This is not an unusual effect, but seems to be almost impossible to achieve due to the way after effects (and I'm sure a few other programs) blurs objects based off a depth map.
I'm rendering using V-Ray in 3dsmax 9, compositing in After Effects CS4.
Before I get into specifics, I know that there are easier ways to get the effect I want, but they aren't exactly possible or practical in my current scene. One would be rendering out each element (or layer of elements) separately according to depth, then composite them on top of each other and blur relatively. In fact I'm using this idea for another shot, but it doesn't work here, because the object itself is a single mesh (output from realflow). Another idea would be to use clipping planes to render out selected portions of depth, but that would show definite edges between the different render planes, unless I did that about 50 times, which is a bit obscene.
Of course I could just skip the whole post process blurring, and just do the darn thing right in 3dsmax, but that would take a long time to render (seriously... like 3 weeks straight rendering), which I don't have, so I really don't want to do that.
So I'm left with trying to do this in post using a depth map, which produces an effect where the blurring ends at the exact point the depth map tells it to. Which would be great for objects that get blurrier as they move away from the camera, but not so much for the opposite. I have some examples to illustrate the issue.
For these examples I exaggerated the blurring in order to show the effect better. In the Composite images, the blurring effect should make it look like that object is on top of everything else, but instead, it looks like a blurred cutout. I know why it does this, but I want to know how I can get around it (if possible).
Here's another example to describe what is happening, and what I want:
I've been trying to figure out how major studios produce foreground blur in depth of field so well. I find it hard to believe that they render those depth of field effects in the render since that would increase render times significantly, but I also can't see a way to do it properly using a depth map. What trick am I missing (if there even is a trick to it)?