I guess your talking about this little fella:
This was done with the fur plug-in for maya. I’ll try and find the file so I can tell ya the settings but from what I remember I made the fur really short, made the base wide and top thin, this gave that triangular,clumpy look. I varied the colour at the base of the hair and the tip slightly to give it that cartoony/flat/cell shaded look. Then I just upped the scraggle settings to get that messy look. I think the image on the right is with a bit more clumping to get those big chunks of messy fur. Basically I played around with settings for a couple of days to get the look I was after.
The contour is done in photoshop. In maya 5 you can’t render fur with mental ray and therefore I couldn’t find a way to do it in maya. I would render out the scene/animation with the fur flat black and everything else flat white. In photoshop I would then use Filter->Stylize->Find Edges. This would give me the outline and then in Adobe After Effects I just add the contour images over top of the normal renders and set the layer blending mode to darken. It sounds a little complex and time consuming but if you use the ‘actions’ in photoshop it doesnt take too long.
Maya 6 on the other hand…
You can render fur with mentalray but i still didn’t find a way to make the contour work with it because the fur isnt geomtry and is rendered/calculated seperately. There is probably a way to do it but I didn’t really get into it. I was looking at ways to convert Fur to geometry but was unsuccesful but like I said, I only had a quick look at it.
Let me know how you get on. I can send you the scene files if you want and if i find them