AO is a shader in mental ray so you have to set it up as a material.
In your material editor select an empty slot then scroll down to the “Mental Ray Connection” group usually located at the bottom. In the “Basic Shaders” section of that group unlock “Surface”. Click the button to select a material for it and in the Material/Maps Browser select “Ambient/Reflective Occlusion Base” it’ll be listed as one of the mental ray shaders with a little yellow box next to it at the top of the list.
The basic settings for it are pretty simple. Samples are the same as they are in anything, how many samples max takes to compute the result. The higher the number the smoother the output but the longer the render. The bright and dark colors are pretty self explanitory, in this case white being the “lit” areas and black being the “shadowed” areas. Spread is how blurred out from the object the result is. A high number will give a really spread out soft looking effect and a low number with make a smaller but more crisp shadow. The last one you’ll worry about for what you’re trying to do is the Max distance. That refers to how far out from a surface a ray is cast before it returns as if it hasn’t been occluded. If you leave it at 0 it’ll continue on forever until it gets occluded by something in your scene. If you set it to 100 or something like that it’ll only continue on until it’s traveled 100 units then it’ll come back and say it hasn’t hit anything. Like most mental ray shaders you can apply a shader to any of these values right below the parameters area.
The next step is to open up the Render Scene dialog and go to the “Processing” tab. Here you’ll find a field called “Material Override”. Check the enable box and drag an instance of your AO material into the material slot. Now when you render it’ll replace every material in your scene with the AO material you created.
One last tip. Since the AO material doesn’t use lights to get it’s result I’d suggest turning off all the lights in your scene when you do the AO pass so that max doesn’t waste time calculating something that isn’t going to affect your render.
Personally I like doing my occlusion as a seperate pass as I find I have much more control over the final look of the thing in a compositing software like photoshop or after effects.
I hope that helps.