View Full Version : How to implement a Displacement mask?

01-10-2004, 06:19 PM
Hey all,

I've recently been texturing a hill, using displacement maps to create extra details in the rock face. I'm using a procedural shader, and I've tweaked the scaling until I've ghot the right sort of striation in the rock. It looks great, all is well and dandy.
However, I want to be able to tell it where to displace and where not to displace. Basically, I don't want it to displace on flat surfaces, where there are supposed to be roads and houses. To this end, I've made a b&W TGA "displacement mask" that fits fine on the hill.
My basic thought was to multiply the displacement mask by the procedural, so that where the displacement mask is black (i.e. zero), the total displacement would amount to zero. But that isn't what's happened.
What's happened is that the procedural texture doesn't subdivide the same way it does when it is acting without the mask. In fact, it provides radically different results. I'll try and post some renders in a couple days (when I am back near an FTP server I have access to), but essentially, the unfettered procedural produces nice results, the fettered one makes a very CG-looking, unnatural result. You can actually see the pattern of the procedural. It looks awful.
Far as I can figure it out, it could be one of two things:

1) The scaling of the procedural has somehow been gimped by the adding of the displacement mask texture. It looks like it's been reset to a 1x1x1 scale, although its placement node reads as 15x5x15.
2) Mental Ray's subdivision of the shader has been gimped by the adding of the displacement mask texture. It will provide similar (although scaling appears off) results if I try to bake the procedural texture out to a file and render it that way.

I'm wondering what to do, how to work around this. I am working in Windows, Maya 5.0 Unlimited, rendering in Mental Ray. The only solution that is really coming to mind is somehow using vertex colors as a substitute for the displacement mask (baking the B&W mask onto the vertices as opposed to applying a file texture), but not really sure how to incorporate that into a shading network.

Thanks for reading this far, and any help would be greatly appreciated.

01-10-2004, 07:42 PM
Maybe create two materials (one without displacement, one with) and combine them into a mix material (I dunno what the terminology would be in maya) using your mask to tell which material goes where.

01-11-2004, 12:41 AM
Thanks for the suggestion. I tried it out, but to no avail.
Layer shaders and layer textures don't seem to work with this; Mental Ray either doesn't recognize the procedural once it goes through the layering or does the slightly screwy version it does with straight multiplication.
Looking at some of the docs, I am coming to believe that this is a function of mental ray's displacement tesselation. I think that by providing a image map that is 1024x1024, it's limiting how much mental ray will subdivide it. I tried manually bumping up the tesselation (using the approximation editor), but it crashed upon render. So right now, I am rendering it out again, having bumped the image map up to 4096x4096. Here's hoping...

[edit: corrected grammar]

01-11-2004, 02:21 AM
Well, the 4096x4096 provided no difference, so scratch that idea.
Here's a small picture of what I'm talking about. The left half has the correct displacement, except I can't control it to only work on incline places -- you'll notice it overruns the road I have in there. The right half has the correct mask applied, but the same procedural now produces that result, instead of the one on the left.
As this may be a more application-oriented question, perhaps the question should be moved to a different forum? I can repost if needed, but from what I've seen, reposts are frowned upon...

01-11-2004, 02:22 AM
Maybe I should just render out the road and whatnot in a different pass, composite it together later. But I would like to solve this problem. Any suggestions?

01-11-2004, 03:25 AM
Okay, figured it out. In the correct displacement pass, the displacement shader was using the outAlpha attribute of the procedural. This is the default connection between the two, which I didn't realize. Whenever I manipulated the procedural with the displacement mask, I was manipulating the color channels, unaware that what I really wanted was the Alpha.

... :blush:

So that solves that problem. However, why would Maya set it up so that instead of feeding the color attribute of a file -- i.e. what you can actually SEE -- it feeds the alpha, which, in many cases, bears no resemblance to the color attribute whatsoever? I understand that since displacement is a b&w attribute, it only needs one channel, but it still makes little sense to me why they would choose the alpha channel, which is reserved normally for transparency, so I thought. Is it standard practice in the industry to make a color image texture for an object, and toss in an alpha map when it needs a displacement map? That seems incredibly strange to me.

01-11-2004, 03:11 PM
Is it standard practice in the industry to make a color image texture for an object, and toss in an alpha map when it needs a displacement map? That seems incredibly strange to me.
Only place I've seen this combining of features in common use is in the real-time 3D field, where you usually want to make the best use of all available channels, due to constrained texture memory. Some games will use the R,G,B, and A channels each for separate effects, not just color.

But I can't see why Maya uses this...

Glad you figured it out!

CGTalk Moderation
01-17-2006, 03:00 AM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.