View Full Version : Mental Ray/ Hardware Normal mapping problem (maya 8)

07-18-2007, 05:04 PM
I seem to have gotten myself into a rather difficult problem with maya and mental ray. I am currently working on a high poly game character using several normal maps. I Have been successful in generating and applying normal maps that look great with maya hardware render. The issue comes into play when I am trying to render the model with mental ray. It seems to flip the normals that are stacked. It seems like there must be a checkbox somewhere that allows it to "fake" the normals like hardware does.

If it's not clear yet these might help

Here's the hardware render


Here's the mentalray render


Some other various renders


I've read that the uv's contain a normal information that has to be flipped as well, but I don't know if maya can do that.

Flipping the normals of the problem polys creates an ugly seam.
Any help is greatly appreciated.

07-19-2007, 01:13 PM
Unfortunately this is probably not solvable. I have no Maya experience, but it looks like mental ray is not using the tangents & bitangents that are stored in the vertices of your mesh. Every renderer needs to use this data when rendering tangent-space normal maps, otherwise you'll see incorrect lighting wherever opposite UV edges aren't parallel in UV space.

I started a section about this on the CGWiki, but haven't had time to refine it. Might help you as-is though.

07-19-2007, 01:35 PM
That seems quite dissapointing because my maya Hardware Render doesn't work with transparancy and to my knowledge maya software can't render normal maps. I've gotten some info from other various source and plan on testing them out today. One included applying a different material with the green channel flipped, another seems to be more about the baking process. If they don't work I guess I'll rerender in 3ds Max. Thanks for the info.

07-19-2007, 01:45 PM
What tool did you use to bake the normalmap? That matters a great deal, for example 3ds Max's scanline renderer only lights properly when the tangents/bitangents are created by baking the normalmap with Max's own Render To Texture.

I'm curious if Maya supports FX or HLSL shaders in its hardware renderer... you should be able to get transparency working by using a different shader.

07-19-2007, 09:02 PM
For the previous examples I baked my normal map in maya 8's Transfer maps feature. I also did a test today using the 3ds Max scanline and creating bitangents. It renders fine in scanline but the bitangent are either not saved between the objs or mental ray just doesnt handle them. I have been using some directx 9 fx shaders for Max 8 and they are working fine.
On the issue of hardware rendering I'm not sure if its a shader problem. I've read some info regaurding my graphics card and its inability to render transparency in hardware mode. Again, I'm not an expert on the subject of fx or hlsl shaders. All I know is that Maya seems to be really lacking when it comes to rendering realtime game models.
Thanks for the help and if anyone has any additional info or workarounds I'd love to hear them.

CGTalk Moderation
07-19-2007, 09:02 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.