I got similar problems with Rino 4.0 + V-Ray.
There are some things that can help:
-
mesh the models, usind detailed options for foreground, and less detailed for far away (medium/low detailed mesh can take much less memory than nurbs surface)
-
use blocks (something like instances)
-
use less resolution in textures
-
run tasks menager and see how much memory rhino takes when it crashes, and what is it doing at the time
under 32 bit systems it can only use 1.7Gb of ram until it crashes, but switching to a 64 bit system should change it to 3.7 Gb (rhino still is a 32bit program, but will work and handle bigger scenes)
hope it solves your problem!