View Full Version : Mental Ray Fixes: Mental Ray Runs Out of Memory

10 October 2009, 04:25 AM
this is a crossposting from my blog (

I am by no means an expert, but I hope this helps. If anyone DOES happen to be an expert, please let me know where I’m wrong, and please offer whatever advice you have.

Also, I'm certain that many others have tackled this issue, so if you know of any good sites, threads, or blogs, please post them.

PROBLEM: Mental Ray Runs Out of Memory

If you’ve been using Mental Ray for any length of time, you’ll probably be familiar with this message: “Got 8 satellite CPUs… Rendering with Mental Ray… Mental Ray is out of memory… Mental Ray is out of memory… Cannot allocate 223000 bytes… Go eat bleach and die, because Mental Ray hates you”

Well, maybe not the bit about bleach. But the Mental Ray Memory Grinder is a true annoyance. The problem is flushing. No, really, that’s the term that programmers use, and the analogy is pretty apt. The way that Mental Ray is set up by default is like having a toilet that doesn’t flush. The poo (in this case your scene and textures (no offense)) just keeps piling up until the smell can kill household pets. Then Mental Ray commits seppuku and you’re left with a cryptic message and an excuse for your boss/professor/client that just won’t fly.

Mental Ray’s render process has several steps:
• Load Raw Scene Data
• Load Geometry Data
• Tessellate Geometry
• Occlude Geometry BSP tree
• Load and Apply Textures
• Output Render Data

Each of these processes takes up a certain chunk of RAM on top of what Maya, Mental Ray, and all the other programs currently running require. If it gets past the amount that XP or Vista can handle, then Mental Ray takes a nosedive. By default, all of those processes are non-flushable, which means that Mental Ray can’t flush the memory toilet before pinching off another loaf of RAM. Alright, the metaphor is getting old, I know. In any case, to fix this, you just need to make sure that Maya knows how and when you want to flush certain kinds of data.

SOLUTION: The Mental Ray Settings Dance of Doom

STEP ONE: Close other open programs!

It has recently come to my attention that there are idiots in this world trying to run multiple copies of Maya while they’re rendering. This is not good. Close it. Also, close down ALL OTHER PROGRAMS, especially that RAM whore known as Mozilla. Please. Just do it.

STEP TWO: Batch Render

Batch rendering instead of regular rendering skips the onscreen output step — this can be a double-edged sword, because it allows you to continue working in the background. If you attempt to do something memory-intensive, while you are rendering, you can sabotoge your own attempt to make the render happen.


BSP stands for Binary Space Partition, which is all I know about it in a technical sense. It has something to do with Maya’s raytracing algorithms and the way that the scene is broken up to render. In order to make this process flushable, you have to set your preferences for BSP. In Maya 2008 and 2009, this option is hidden away in a node called miDefaultOptions. In order to get to it, click in the script editor at the bottom of the window and type this in, with this capitalization. ( sure to capitalize the D and O)

Hit CTRL-A to open the attribute editor, and you will get a new window with options you can’t find in the regular render globals window.

The option we are looking for is under the Mental Ray Globals tab.

Set the Acceleration Method to Large BSP. This setting is specifically for large scenes, and will break up the render into smaller BSP chunks, enabling flushing for the BSP step of the process. Don’t ask me why, because I really don’t know. In 2008 SP2 and above, you can also use BSP2, which is the default for Maya 2010. Large BSP will slow down your render a bit, but it might be worth it. ( Acceleration to Large BSP or BSP2)

STEP FOUR: create .MAP files

If you still can’t render after the BSP tree is set, convert your texture files to .MAP . This file format is memory-mappable, which means that Mental Ray can cut it up and use only bits and pieces in RAM as they’re needed. Also, it can flush those parts as soon as they’re done being used.

To convert your textures to .MAP, you need a batch file called “convertfile.bat” You can find one version on my website here:, but there are many versions available online. To use it, just copy the file to the directory where you store your textures, and you can click and drag up to ten files at a time onto the batch file. It will automatically copy and convert your textures into .MAP using Maya’s command-line process called imf_copy. If you’re working with a ridiculous number of texture files, and don’t want to re-import all the .map files, you can rename them to be .jpg, .tga, or whatever. This tricks Maya into loading the right file, but still uses the .map. Unfortunately, this means you can’t open the file in photoshop while it’s still named incorrectly.

Using MAP files will make your render take longer. However, this is often preferable to not being able to render at all.

STEP FIVE: Instance your geometry

If your file has a large number of repeated geometry, such as a warehouse with stacks of the exact same crate — instance your geometry instead of creating straight duplication. To do this, click on Duplicate Special instead of Duplicate ( Ctrl - Shift - D instead of Ctrl - D ) and choose “instance” in the dialog box. Instanced geometry is flushable.

STEP SIX: Set your Memory Limit

If, after all this, you still can’t get the file to render, you can manually reduce the amount of memory that Mental Ray is allowed to use. This seems to be especially useful in Vista, because the OS never seems to realize just how much RAM it actually has or doesn't have.

In the render tab, go to the render menu. Next to the options labeled “Render Current Frame” and “Batch Render” there are little boxes. In the menu that comes up, uncheck the “auto memory limit” and click “calculate” Then, ignore this number and input something lower. 1024 or 1512 seem to work pretty well for me on a machine with 3 gigs of RAM, running Vista. ( Box!) ( versions of Maya don't have the "calculate" button)

This number sets how much memory Mental Ray should take up before initiating the flushing process. If it’s set too high, you still won’t flush before you crash. If you’re working on a particularly low-memory machine, this might be your first step.

STEP SEVEN: Render from the Command Line

When you install Maya, it edits your environment variables, and installs its own set of command-line instructions you can run from DOS. It also edits your command shell structure.

So, provided you have Maya set up properly, you can create a text document known as a batch file to render from the command line. Open Notepad, and write this:render -r mr yourscene.mb;

then, save the text document as a .bat. Make sure you choose as the file type “All Files”. Put this file in the same directory as the scene you want to render, and simply double-click. Your file should batch render to wherever you set the render path, without ever opening Maya — leaving you with at least a few hundred megabytes of RAM to play with. There are many tutorials out there about setting up command-line rendering.

STEP EIGHT: Consult Google

There are a number of other ways you can optimize rendering with Mental Ray. I’ll probably be posting more tips as I collect them, but for now, try looking up the following things: baking occlusion maps, baking final gather calculation, portal lights for final gather, and setting up render tiles. I’m also hoping to start playing with BackBurner for Maya, see what that old Max standby can do for us. Good luck!

10 October 2009, 03:05 PM
You've got a lot of great info there. Couple things to make a note about though:

1. The acceleration tab that you got to through miDefaultOptions actually is found in the render settings window. In Maya 2009 and 2010 you can find it in the Quality tab, scroll down to Raytracing, and then down to Acceleration.

2. The triangle count can be controlled much better by understanding the geometry in your scene. I like to create display layers or quick select sets based on polygon, NURBS, SubD (including hitting "3" on polygon meshes), and displacement surfaces. Anything that goes through additional tessellation at render time like NURBS, SubD, and displacement should have the Mental Ray approximation editor applied to the surfaces to help control how they are triangulated. The presets can get you started in the right direction - the main thing is understanding what sliders the presets unlock. I personally like using any preset that unlocks the distance attribute. This sets a distance tolerance, initially based in Maya units, for all triangles. So any new triangles created will have to be smaller than whatever value you put. The other thing that can help out is the "view dependent" checkbox. This changes the value from Maya units to pixels. This simple change basically gives your geometry an LOD effect, where triangles are reduced when farther away from the camera and raised when closer to the camera.

3. It's also worth looking into the tiling settings contained in the Render Single Frame and Batch Render options. This defaults to auto-tiling, like the memory limit. If frames are crashing, try unchecking that and setting to something smaller like 16. This creates smaller tiles of 16x16 pixels. Between this and BSP you can really help minimize how much is stored in RAM at one time.

Hope this helps. :thumbsup:

10 October 2009, 08:50 PM
Thanks for the info!

1) ah HA! that's where it's hiding! edit will be on the way.

2) I had not realized that the subdivision approx. nodes could be used on SubD and Nurbs surfaces -- I rather assumed it was simply a "subdivide on render" for polygonal meshes only. Do you happen to know if the approx. node works on instanced geometry?

3) meant to include render tiling, ran out of time. edit will be on the way. There seems to be a magical number when setting your bucket size, I've found that sometimes in Vista setting the tiles LARGER actually makes the render more stable.

Thanks again!

10 October 2009, 09:06 PM
I've never tried it with instances, but I'm pretty sure you can. Just do some tests and apply it to the original mesh and see if the instances innately pick it up. If not, you may have to apply it to the actual instance.

CGTalk Moderation
10 October 2009, 09:06 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.