Imagine this. I am modelling in MAYA a cube and 2 duplicate cylinders. I rotate the cylinders to form a cross and form a union boolean object. Then I difference boolean the cylinder cross to the cube and join up the points. So now I've got a cube with 4 openings.
For UV layout, I used automatic layout, as this is a relatively simple object based on a cube, I get 6 nice faces and tunnel tops and bottoms and cylinder walls. Neat.
So I bring the object into mudbox and proceed to sculpt the cube. And then create a displacement map
lvl 0 smooth texture coordinates, 32 bit displacement map, edge bleeding 4 (default)
, and export the lvl 0 mesh from mudbox into an obj file.
I then proceed to import this new obj file back into maya and connect the displacement map to the shading group of the material used for the object. I apply mental ray subdiv approx using length/distance/angle mode set to length 0.1 view dependant, with min subdiv 2 max subdiv 5.
Now after all that, the overall shape of the cube was more or less what I did in mudbox, but there were some really bad artifacting puncture marks on the opening of the tunnels on the cube. The opening of the tunnel is also the border of the uv shell of the cube face. I took a look at the displacement map and I found that there were some black spots and hard white spots near the uv border of the opening in the cube face. These were the things that were causing the displacement punctures.
It seems that mudbox doesn't like to deal with uv borders that coincide with hard edge corners on the mesh.
I added edge loops along the cube face around the tunnel openings and loaded the cube into pelting tools and pulled the cube into 4 pieces that had uv borders on the new edge loops.
Then I tried the whole process again, and now, since the uv borders are on nice flat regions, there was no sign of artifacting, and the displacement map worked like a charm.
So the moral of the story is, always try to put your uv layouts in a way that the borders don't coincide with hard corners or edges. Mudbox doesn't like having to deal with seams that are on hard corners i think.
Hope this helps. And if anyone knows any other way to tackle this problem, please share your ideas :)