(simple test scene, mental ray, no reflections, no indirect lighting, just maya blinns with some specular, directional/point light, raytraced shadows)
When i render diffuseMaterialColor pass of objects that use alpha transparency textures (hair cards, fur etc.) AND objects that are assigned a plain color shader (say blinn) that is transparent, in both cases the transparency is ignored in this pass and areas that should be transparent show the material's color too as if there was no transparency on the shader itself or coming from the texture.
All other passes (directIrradiance, diffuse, beautys etc.) work properly when i check "attenuate according to transparency" in their nodes.
But if i turn the Attenuation on for the diffuseMaterialColor, then the parts that are supposed to be fully transparent in it change from the solid material color to black, instead of disappearing like it is with all the other passes.
With Attenuation still on, if i turn the shader's transparency down, to say 50percent it wont be completely black in the pass anymore, but darkened a bit. if transp is zero, it has its normal color again same as if Attenuation was off. So could it be doing the opposite thing? adding the color on top of itself instead of removing it while "attenuating according to transparency"? Maybe Bug/MR Compatibility issue with "old" maya shaders? (also alpha is still solid white on the render, even in the black places)
ill be needing diffuseMaterialColor for many objects and scenes with alot of different textures using alpha trans in all my work, so Iam not interested in the typical maya workaround suggestions like doing multiple render layers / assigning surface shader materials / creating custom color buffers and passes / rendering transparency passes (that wouldnt work anyways if i need whole scene as passes without separating each object's passes, as it would make a hole through everything.)
i also tried playing with mia_x_passes which makes the transparency sort of work in the pass (still weird edge but much better) if i plug my texture alpha into its cutout opacity but this material isnt suitable for my work. I do low poly stuff and need to be able to separate speculars from reflections etc. Also mia doesnt seem to work with ambient light which i will need as well.
So knowing that diffuseMaterialColor sort of works with mia material and the problem is not caused by a texture or lighting, this must be just some compatibility issue between the maya shaders (blinn, lambert etc) and the mental ray's diffuseMaterialColor pass output module.
I tried fiddling around with every single setting in whole maya that was remotely relevant to rendering but no success. i have no shader coding knowledge so i dont know where to look to try and fix this manually outside maya's graphical interface.
Any help or tips greatly appreciated, ive been trying to fix this for several days now...