After more searching I found a site that does plugins for forests and 3DS Max. I’m posting the link because they have galleries where renders where done in mental ray:
Vray with 500 billion polys and the mental ray picture has many millions of polys too, so you should be able to do this.
Edit: Just see this in their known problems list, some good tips you might find useful too.
To get acceptable rendering times in large scenes, it’s necessary to use BSP2 as raytrace method. It is available only in Max 2009 or greater. For previous versions, or small-to-medium size scenes, you can use BSP instead.
Always disable “Scanline” checkbox
BSP2 is not fully stable yet. It uses a dynamic memory management engine, able to flush memory when necessary, but sadly, under heavy load it crashes sometimes. From our tests results, stability depends of the scene complexity, including number of trees and triangles. We have found problems in scenes using above 100.000 trees of 500.000 triangles each one (that really is a lot). These are some problems that you can find:
[li] Max 2009: Fatal errors after completing a render as “accessing unknown tag” or “accessing null tag”. You can minimize this error increasing “Memory Limit” parameter at “Rendering Setup->Translator Options”. The default value is 650Mb. Try using different values above 1500Mb. (of course, only if you have enough free memory available).[/li][li] Max 2009: Fatal error cancelling a render when Final Gather is being processed. Previous fix may be useful here as well.[/li][li] Max 2009: Other errors while rendering.[/li][li] Max 2010: Crashing error if material editor is opened after completing a render. This issue is already reported to Autodesk.[/li][/ul]You must understand that these errors are not caused by Forest, and we cannot fix them. In most of situations, Mental Ray is stable because common scenes don’t include hundreds of thousands of high poly meshes as Forest is able to handle. These problems are known by Autodesk and Mental Images (Mental Ray creators), because latest versions (as Max 2010 or Maya 2009 ) are more stable, although there are bugs to fix yet.
- Rendering times with more than 500.000 trees grows exponentially. This seems to be the limit of Mental Ray handling large number of instances.
And remember that Forest allows you to combine 3D trees with billboards to reduce memory footprint.
If you really need to use millions of complex trees in your scenes, we suggest you switch to VRay. This renderer is much more stable and fast, and using it together Forest is an amazing experience.
I get sometimes the same errors in Maya (where it talks about accessing unknown tag)