Ok, had the problem in our art department myself. Imho we are talking about at LEAST 2 problems here:
1st people who simply dont have enough memory for their scenes or too large scenes
2nd people who have plenty of memory and still cant render especially high res images
we are of the second category here. Mental ray usually fails with an error message about not beeing able to create a temp file caled “fb000.something” because it does not have enough memory. This alone is strange because creating a file handle needs very little memory and mental ray tells us that it used only about halve of our system memory. Our guess here is that this is a so called “memory mapped file”, a file that is mirrored in the main memory. So what actually happens is that mental ray tries to allocate a big chunk of memory in one go and fails. The bigger your screen resolution is the bigger is the chunk of memory mental ray needs, in our case about 100MB and thats the problem. Every application in 32Bit Windows gets max 2 or 3 GB of memory. mental ray is not a separate application but runs within max and uses the memory Max got assigned. Max itself is a memory hog and uses some 100 MB itself. Before you render max has to load a scene, create a internal representation and display it on screen. So it does already do a lot of memory operations which fragment the memory. Mental ray has to live with that fragmented memory and although everything works well as long as it allocates small chunks of memory at a time, it might fail when it tries to allocate a 100MB big chunk simply because the memory is already too fragmented.
Solution? The best solution might be to render it standalone so mental ray gets a fresh and clean memory space.
If you insist rendering from within max, start rendering directly after you start max, switch to software Z Buffer (Opengl seems to fragment memory more, thanks RickStefani, worked for me), try the the 3GB swith so the fragmentation is slower, and if tall this does fail, think about how else you can reduce the memory usage of max itself. One of our artists had to disable the splitting of his viewport and display everything as box. After a Max restart his scene rendered. Sounds like voodoo, doesnt it?
hope it helps
Jan Jordan