PDA

View Full Version : Best way to render Maya fur in Mental Ray


UniqueCharacter
07-29-2011, 02:16 AM
Hello,

I'm working on a werewolf-like character and I have the model, rig, and textures mostly done. I've saved figuring out the fur for last and so far my attempts have been quite irritating. I have NO money so I know lots of people will say "use shave and a haircut," but I don't have that option at this point. Perhaps later on.

From what I've read on other sites (tutorials and forums), is that it is best to set your fur to render as a volume and to not use raytracing. That's fine and good, but it doesn't look very nice when I try it. I tried using depth map shadows instead of raytrace shadows, and also had to set my shadow type to segments because of the volume fur. It renders about twice as fast as raytrace and looks about 2% as good. When I turn the camera at different angles, the shadows have lots of artifacts (even giant squares). I tried bumping up the filter size on my depth map shadows - but that just slowed down the render to a point that I should just use raytrace. I found one good tutorial online about this, but it doesn't say what light settings or render settings to use.

I'd appreciate your help and thank you for your time.

kiryha
07-29-2011, 12:07 PM
You should render fur as hair primitives with rasterizer, not volume, than you will get predictable result.
And it would be more easy for everybody if you will post links on "good tutorials" that you talk about, cos i cant remember any advise to use volume with MR, for example.

murralls
07-29-2011, 12:37 PM
Follow the link below to find what you are looking for

http://kiryha.blogspot.com/2010/06/rendering-fur-with-mental-ray.html

Libor
07-29-2011, 02:50 PM
search maya / rendering forums for "hair fur rasterizer settings" ...try some of them or all...this has been discussed here a lot

L.

UniqueCharacter
07-29-2011, 07:26 PM
Kirya: Here is the link to the tutorial I tried following that went over using volume:

http://www.djx.com.au/blog/2008/06/29/tips-for-rendering-maya-fur-with-mentalray/

While I'm sure it wasn't intentional, requiring me to post the links I talk about sounds like you're calling me a liar or something. Your tutorial is nice, I will go through it after this and update on what I came up with.

murralls: Thanks, I found Kirya's tutorial a while ago, but skimmed through it until I saw "shave and a haircut," but didn't realize he shows the maya alternative too.

Libor: I usually hate using CGtalk's search function because it *usually* brings up a bunch of nonsense and isn't very good and finding anything. Your search terms proved to be successful and I'll go through the threads. I really don't like the attitude lots of people have on CGtalk (not you or anyone in this thread) have about people asking repetitive questions. I rarely post in the general CGtalk forum because of the bad attitudes. These are problems people run into when they are learning 3D stuff. I'm not experienced with Maya fur and it's rendering, so I will need some help via a good tutorial and be able to ask questions. If it's that much of a problem CGtalk needs to sticky these topics so that people like me don't bug the regulars.

Thank you all for your individual help and I appreciate you taking the time to help me. I will update accordingly.

kiryha
07-29-2011, 08:24 PM
Sorry, mate, i didn`t want to say anythig rubbish to you:)

The dJx tutor, that you talking about is qute old, and he has a new one: http://www.djx.com.au/blog/2009/04/14/p_hairtk-p_hairtk_shadow-p_shader_replacer/

For maya and mental ray there is the best way to deal with fur for now, and you should have no doubts about it. In my article you can find values for shadows to start from. And if you dont need passes- render just beauty, its even easyer.
And there is no difference how u create fur- with shave, or maya fur- for mentalray is the same.
If you will have some problems with memory, quality, speed or something else- just ask. With pictures:)

UniqueCharacter
07-29-2011, 09:18 PM
Thank you kiryha, I'm working off the tutorial right now. Maya keeps telling me :

" rasterizer with auto volume may force ray tracing to be enabled during rendering"

I tried googling it, but it just comes up as a batch rendering issue someone had, which doesn't help.
I'll ignore it for now and keep working on the tutorial.

UniqueCharacter
07-29-2011, 10:38 PM
Okay so after following both tutorials, I got the "puppet" technique setup and working somewhat.

I now have some questions for you all since I know nothing about puppet's shaders and setup except what I've learned today.

1: Do I still need to attach shadow attributes to the lights in the scene? If not then does that mean the HairTK_shadow node controls the drop shadows of the fur?

2A: In the DJX blog he uses raytrace still to render, does that mean the setup cancels out any raytrace calculations on the fur? Or do I still need to use rasterizer shadows in my render settings?

2B: If I use rasterizer settings, what is the best way to setup my lights? Do I use depth map shadows? Or do I use raytrace?


Thank you again for helping me.

Edit: added screenshot of render with final gather and raytrace shadows. Took 2:34 to render!

Edit 2: I went back and followed the "256:Mental ray Rendering..." tutorial and am not getting anywhere good. I tried setting up the light using puppet spot, and it renders bizzare if not quick. See attachment 2.

kiryha
07-30-2011, 10:05 AM
>Do I still need to attach shadow attributes to the lights in the scene?
Not sure what are you talk about. For every light you should create puppet light shader and plug it into light shader slot of light shape. If you dont need passes- you can skip this step.
If you will use transparancy in pHairTK shader you should create pHairTK_shadow with transparancy enabled and plug it to shadow shader of pHairTK Shading group.

>2A, 2B
I didn`t get anything, sounds like random mix of termins:)
In render globals set primary render to RASTERIZER.
For every light use depth maps shadows, switch shadow map format(in mental ray section) to Detail shadow map, turn on "use mr shadowmap ovverride" and setup shadow settings. Also, to get detail shadow working, in render globals set shadow method to segments.

>added screenshot of render with final gather and raytrace shadows
When i sad NEVER USE RAYTRACE i meant DO NOT USE raytrace shadows, FG and GI.
Thats why you get huge rendertime, and definatley you will get flickering in animation. No FG, mate, never.

Actualy, the basic steps is prety simple, seems you didn`t get an overal idea and have a bit mess with understandig of some terms. Focus on one tutorial, try to read everything carefull to get good understanding of base.
Do not mix "raytracing" as general term in CG with "raytracing" mode of primary render in render globals.

>" rasterizer with auto volume may force ray tracing to be enabled during rendering"
If you use shave and haircut: in shave render globals you should turn of "MR globals safe mode" and turn off autovolume in render globals. No shave- just turn off autovolume in render globals. Than you will be able to turn OFF "raytracing" under secondary effect.

Bitter
07-31-2011, 04:35 AM
More recent versions of Maya should allow you to render with auto volume off.

UniqueCharacter
08-01-2011, 06:17 PM
Thanks Blitter, I'm using Maya 2008 right now. Perhaps some of my problems are because of this, but I have no choice just yet.


Update:

Went back again through the tutorials and double checked that my setups are the same as they say. It does not render the same. The bizarre render looks the same as the one I posted earlier showing a slice down the middle of the face where the shadow gets bizarre. I have the fur working fine and showing the default HairTK colors and settings. I have my light setup as instructed in the "mental ray rendering fur with passes" tutorial.

Spotlight with SpotTK attached in mental ray settings. Using the default settings right now, just like the tutorial. I have the p_HairTK_Shadow node attached to the "shadow shader" attribute in the p_HairTK1SG mental ray custom shaders, just like the tutorial says. I have raytrace shadows unchecked in the main light settings and have setup the detail shadow + overrides as instructed. I used the defaul depth map shadow settings, but set the resolution to 1024, and put filter size to 2.

Both tutorials fail to tell me how to setup my render globals for this fur. I've tried final gather preset (like the djx tutorial says he used) with Raytrace unchecked under secondary effect and Rasterizer selected as primary renderer.

Thank you for your continued support and I hope to get this figured out soon.

UniqueCharacter
08-01-2011, 06:30 PM
Here is another render of the shadow problem.

SGIFreak
08-02-2011, 04:10 PM
Mentalray is good, but it can better.

Bitter
08-02-2011, 05:44 PM
mentalray is bad for very fine details.

Well that was helpful. . .

You can achieve the same level of detail using the Rasterizer as you would in Renderman. It's the same type of mechanism for anti-aliasing. You can increase your visibility samples without incurring a lot of extra shading calls. Where you will suffer is raytraced effects in a rasterizer are more expensive and have worse quality without more work.

Keep in mind that these small details will be lost when the object is in motion or blurred so don't stress over the details in that case.

You can keep the resolution of detail shadow maps somewhat lower than a regular shadow map because it also has a samples parameter to achieve better detail on edges without having to increase the map size.

You can use FG with hair without flicker but it will require you tune the FG solution like you would any other scene (or renderer with GI)

rasterizer with auto volume may force ray tracing to be enabled during rendering

Volume hair requires raytrace shadows in Maya 2008 (this changes in more recent versions I believe) because it is a volume that needs to trace the path of the shadow ray (segments) to render correctly. Detail shadow maps do not operate like that. Shadow maps are basically a post process slapped onto the geometry. This is why raytrace secondary effects are generated even when using the Rasterizer for volume fur.

Sounds like your problems are mostly shading related.

Hair is small and thin, you do not need a great deal of shading to reproduce it. So the rule of thumb when rendering fur is a high visibility samples (this is your anti-aliasing) and lower shading samples (the number of times your shader is called per pixel.)

Keep the hair counts as low as possible to achieve the look you want. Rasterization is a modern version of scanline and lots of geo will still slow it down even if raytraced effects are off.

kiryha
08-03-2011, 10:20 AM
can you post scene file here?

UniqueCharacter
08-03-2011, 10:32 PM
Well that was helpful. . .

You can achieve the same level of detail using the Rasterizer as you would in Renderman. It's the same type of mechanism for anti-aliasing. You can increase your visibility samples without incurring a lot of extra shading calls. Where you will suffer is raytraced effects in a rasterizer are more expensive and have worse quality without more work.

Keep in mind that these small details will be lost when the object is in motion or blurred so don't stress over the details in that case.

You can keep the resolution of detail shadow maps somewhat lower than a regular shadow map because it also has a samples parameter to achieve better detail on edges without having to increase the map size.

You can use FG with hair without flicker but it will require you tune the FG solution like you would any other scene (or renderer with GI)



Volume hair requires raytrace shadows in Maya 2008 (this changes in more recent versions I believe) because it is a volume that needs to trace the path of the shadow ray (segments) to render correctly. Detail shadow maps do not operate like that. Shadow maps are basically a post process slapped onto the geometry. This is why raytrace secondary effects are generated even when using the Rasterizer for volume fur.

Sounds like your problems are mostly shading related.

Hair is small and thin, you do not need a great deal of shading to reproduce it. So the rule of thumb when rendering fur is a high visibility samples (this is your anti-aliasing) and lower shading samples (the number of times your shader is called per pixel.)

Keep the hair counts as low as possible to achieve the look you want. Rasterization is a modern version of scanline and lots of geo will still slow it down even if raytraced effects are off.

Thank you very much Bitter for helping me on this. Those are good tips, and I didn't realize volume fur was forcing raytrace. I guess that using it was a bad idea. I'm kind of confused about what you wrote about a smaller map size - so when you use a 'detail' map with the samples tweaked it allows you to keep the finer details? Sounds pretty cool (or reads I guess). This is for a low-budget (no money is going to 3D) movie, so I don't even need final gather. The creature will be moving around in the dark through most shots, but there are some that will be of a fully lit character with little movement at full 1080HD, so I'm hoping to get a decent result without huge render times if possible.

Kirya: Thank you. I got contacted by DJX, and is kindly looking at myscene and will hopefully figure out all that I'm doing wrong. If not I will post the .mb here for you.

Bitter
08-05-2011, 01:01 AM
Under resolution in the mental ray shadow map rollout is another one for samples, this will help keep the edges of a the map more accurate without have to increase the resolution very high.

UniqueCharacter
08-22-2011, 11:07 PM
Thanks Bitter, that's pretty cool that it can do that.

JVG didn't email me back here and its been a few weeks. I've started a full time manufacturing job so I haven't touched the project in a couple weeks.

I'll go ahead and put the file on my webspace and see if you all can figure out what is happening. If it helps, I went back to a version before doing the puppet shaders and it appears to be doing the same bizarre blocky shadows when I try and setup the light with depthmaps + MR overide as suggested.

Here is the file (http://www.uniquecharacter.net/Dogman_GeometrywithFur_forJVGFX.zip) (800K)

Thanks again

Bitter
08-25-2011, 08:32 AM
The one posted seems to still have the Puppet shaders. Renders black for me. Is that the correct version?

UniqueCharacter
08-26-2011, 04:18 PM
Thanks for looking at it Bitter. I'm not sure why it's rendering black for you. I just checked it out and it still renders the same for me. I left the puppet shaders attached because I'm still using the spotTk on my light. There is a file in hypershade controlling the tip color of the original fur - but the hairTK overrides the maya fur color so it shouldn't be making your render black. I'm not sure what the problem is with the scene. Thanks for your continued help B.

kiryha
08-30-2011, 10:48 AM
You need p_spotTK on lights only if you output passes.
And you didn`t setup shadows in render globals, set shadow method to segment and format to detail.

Libor
08-30-2011, 02:55 PM
Jeremy:

Ok, I see. Let me clear the situation from me. Well my answer was more practical from my point of view / time - of course I could search for my last answer (time consuming) on the same topic by myself and paste it into here, but hey you should at least prove you are willing to find solution yourself when I know you will find an answer (deep) to your problem. No offense! It was maybe not as polite as it could - not any "please" bla bla bla (takes time to make nice sentences in english for non en speaker)

and second this topic is quite huge as you can see from answers from others...so again why to write again screens of text when everything was deeply discussed! Dont you think?

Ill be next time more polite so its obvious I really want to be helpful...my fault ;)

Peace!
L.

UniqueCharacter
09-02-2011, 03:57 AM
You need p_spotTK on lights only if you output passes.
And you didn`t setup shadows in render globals, set shadow method to segment and format to detail.

Thanks Kiryha. I thought it only had to be on segment for volume. I'll give it a try and let you know. Thanks for your help.

UniqueCharacter
09-02-2011, 03:59 AM
Jeremy:

Ok, I see. Let me clear the situation from me. Well my answer was more practical from my point of view / time - of course I could search for my last answer (time consuming) on the same topic by myself and paste it into here, but hey you should at least prove you are willing to find solution yourself when I know you will find an answer (deep) to your problem. No offense! It was maybe not as polite as it could - not any "please" bla bla bla (takes time to make nice sentences in english for non en speaker)

and second this topic is quite huge as you can see from answers from others...so again why to write again screens of text when everything was deeply discussed! Dont you think?

Ill be next time more polite so its obvious I really want to be helpful...my fault ;)

Peace!
L.

I think it's just a translation problem Libor, I'm sorry for being defensive. I'm just no rendering expert, and am not planning on becoming one. Just need some help on rendering this dang fur.

UniqueCharacter
09-09-2011, 08:11 PM
One final post and Admins/Moderators can lock this one up. Just want to post the answer for anyone having this problem in the future.

Very special thanks to JVGfx! After some difficulty exchanging emails and files (going from USA to Russia), he found out what was happening.

"The problem of black spots, dirty fur and shadows - you do not use special transition shader p_maya_shadingengine. New versions of Maya 2010 and higher, this problem has been solved."

I thank JVGfx, as well as everyone who was kind enough to offer their help.

CGTalk Moderation
09-09-2011, 08:11 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.