"Geometry Masking"?

#1

Is there a way to set up a mesh as a 3D ‘mask’; where in, the mesh would tell XSI to only render the background (transparent or color) in it’s place, with no geometry, lighting, or effects? If I had a background color, I could just color-match my masking-mesh and it would ‘disappear’, but I’m rendering to transparent PNGs for game assets; and transparent-matched object don’t mask anything…

To achieve the effect now, I’m using Photoshop. I create my ‘mask’ mesh and paint it a solid dark green(close to black to avoid visible artifacts, but not common in my scene). Then after rendering my scene, I can open it in PS and magic-wand the dark green patches out.

The Photoshop work around works, but with hundreds of small files, batch proccessing using the magic wand or ‘color range…’ isn’t perfect (leaving an outline or masking a bit too much). Better quality Photoshop routes simply can’t be done using PS’s batch feature, requiring the mind numbing task of manually ‘masking’ each file by hand.

Thanks in advance :slight_smile:

#2

I tried setting the ray depth setting to 0. It turned my transparent mesh into an eraser, but it still rendered black behind it, not transparency. With a black background it works, but not with transparent PNGs.

Besides, it would interfere with transparent objects created in the future.

Still hoping for a solution :slight_smile:

#3

I’m not sure this will work for but you could try "shadow " shader.

#4

Stick a constant shader with an alpha of 0, rgb 0, onto your object and it won’t show up in the render. Is that what you mean?

#5

Well, it seems like you found a solution as long as you can find out how to replace the default “background” space from black to transparent. Being a former 3Ds Max user, your situation could be avoided. In Max, when you chose .PNG as a file source, Mental Ray would render and use a transparent background to anti-alias against. I mentioned this too support a year back about having something like this in Softimage, but so far I haven’t found any solution to this. It would be great though, because it would relieve a lot of headaches…“black edges” when not rendering with a background/environment image. If you find a solution - please post, because I am also interested in this, as I will do the same.

#6

I never get any issues like this. Just render to a format supporting alpha and off you go.

What program are you getting black edges in? If you set up your passes with partitions and the like you shouldn’t have these issues.

#7

Thanks for the reply.

Here is a quick test. Make a sphere and give it a material - light in color - I used a constant white, and I also tried it with a lambert with a rim light to give it a little lighting on the edges. It is more obvious with the constant white though. I rendered it out as a .PNG with the alpha embedded. I opened the rendered image in Photoshop, and it shows it with the transparency…so far so good. Make a new Layer and paint this new layer white. Take the new layer and place it below the image layer. You will now see a thin black line around the object that you rendered. This has been an issue for me for sometime… 3Ds Max does this scenario perfectly - no issues. So, it led me to believe it might be the way Softimage handles its anti-aliasing with a non-existant/transparent backround. Then again…maybe I am doing something wrong.

#8

I see what you mean now. This issue bugged me a lot too. If you bring the same image into After Effects and then interpret the footage without the premultiplied by alpha setting (ie straight) then you get the same black edges as the Photoshop document.

As soon as you pre-multiply by alpha these things go away. I always comp in After Effects regardless of the project so I’m able to fix this really easily. Not sure of the method of sorting that in Photoshop… I’ll have a further fiddle with it and let you know if I come up with anything.

#9

This may help http://forums.cgsociety.org/showthread.php?t=496772

#10

That would be awesome if you or someone found a solution. I do a lot of things for the web so I don’t always have the ability to comp. Sorry Lakehaze, I didnt mean to have the subject change on your thread, I thought my original post would bring a solution to your issue too. I will keep searching to see if I find something for your issue too LakeHaze. I will also start a different thread to keep mine seperate from yours.

#11

In Pshop you can remove the premultiplied antialias edges with “layer > matting > remove white (or black) matte”.

#12

Thanks ajcgi and toonafish…I will look into that.

#13

max and xsi outputs same shit, really. If you see black matte when you render PNGs from xsi thats because XSI is default set to Premultiply with alpha, while MAX has that option turned off by default.

#14

Perhaps my question was too detailed… My thread seems to have been hi-jacked.

As of now, there are no shadows in my scene.

I can render PNGs with transparent backgrounds. Cutting out the background is not the issue, I’m trying to cut out geometry, ie: a mask.

example: A mask in Photoshop is a shape that defines what part of an image is not rendered. It doesn’t paint a giant black shape in the middle of the picture, it simply doesn’t render that part of the image.

I’m looking to create a mask using a mesh in my scene instead of a shape on a PS canvas.

Is it possible?

#15

For hold-outs plug rendertree->nodes->illumination->more/mental images/mip_matte_shadow
into your surface port.

#16

Edit: After actually reading your post I’ve realized that this is /not/ what you’re after… But it’s still relevant as far as geometry masking is concerned, so I’ll just leave this here.

I belive the BoolTrace shader would do what you want. It is included in Alan Jone’s BinaryIris shader pack: http://www.binaryiris.com/node/27

   It's essentially a shader-side boolean-operation:
   
   [img]http://www.mayulive.com/bool_1.png[/img]
   
   The Invert checkbox inverts the normals, not the order of the two meshes you define in the material for each mesh.
   You can do all sorts of fun things with it:
   
   [img]http://www.mayulive.com/bool_2.png[/img]
   [img]http://www.mayulive.com/bool_3.png[/img]
#17

Fantastic! It works perfectly, for now… I don’t know what will happen when I add shadows, if I add shadows…

Although I couldn’t find a mip_map_shadow, or a mental images folder (using 5.1), but good ole ‘shadow’ worked… which means…

Apologies neso. I’ve never used the shadow shader before and didn’t really know where it was. Shouldv’e followed your lead. Thanks for the help :slight_smile:

Fingers crossed that this doesn’t lead to complications down the road (shadows)… If they haven’t made one already, a ‘Mask’ shader would be quite useful. Perhaps it could be written.

Thanks again :))

#18

Though the easiest way und probably the also the fastest would be what ajcgi
already suggested. Just assign a Constant shader to your “blackhole” object and set the RGBA values of the Diffuse color to zero. You could also uncheck Reflection, Transparency and Incandescence (I am not sure though if this helps in speeding things up).
You have a prefect cutout and do not have to worry about shadows.

#19

Um, constant shaders render an opaque color. This is what I was doing as a work around. It certainly is not the easiest or the fastest considering that it requires post processing to remove that color and produce a somewhat inferior quality result. The shadow shader renders a completely transparent hole in the image (as of now without shadows in the scene), and the rendered PNG doesn’t require any more attention.

You keep the “black”, I’ll take the “hole”.

Thanks again to the forum for helping me out :slight_smile:

#20

Seriously, constant all the way… with alpha down you get a cutout, with alpha up you get a constant. How is this not easy?