PDA

View Full Version : post-lighting using bent normals (yet anouther bent normal thread)


ant-
12-29-2004, 08:07 PM
Using the Dirtmap shader in mentalRay for Maya it is quite easy to render out normal information as an RGB image, and up until now i have known no use whatsoever for this, but I had some spare time so i did some experimenting. I usually try and work stuff out for myself and then go looking for tutorials and information when I get stuck, but there doesn't seem to be that much information around on exactly this subject.

ok this is just experimenting so if i have completly missed the boat here please tell me

I saw this as an oportunity to use shake to do lighting as a post processing effect (yes i know there are the obvious limitations, but the benefits of lighting in near realtime are also obvious.)

I've been using renders from mental ray at float but these examples are simply jpgs

http://www.platformzero.com/forum/bentNormalRGBA.jpg

I am using this kind of image (excuse the dodgy model) in a tree like this

http://www.platformzero.com/forum/bentHeadTree.jpg

The Mult1 is where you choose a colour, that in this case controls where the light is coming from, it fairly closly matches the colour wheel, so if you move to the far right of the wheel you get a light scource that appears to be coming from the right, and the monochrome node is set to 1,1,1 and not the default. As you want all hues to have the same value.

Here is a tiny! quicktime example of how an animated light scource could be used

http://www.platformzero.com/forum/normalHead.mov

The problem is

a: I can only light an object from the top right or front with this tree. Now by simply inverting the final image you obviosly get the oposite so left, bottom, behind. But this seems like a bit of a hack, it also involves changing the brightness and contras alot to get a good result

b: I just dont work out the best way of applying this to a "colour only" render pass, I get odd results when i use lAdd nodes with more than one on top of each other, brightening one makes the other darker. (i thought lAdd only added)

I know what i am doing is not exactly rocket science but I still think i am probebly missing a stupidly simple solution.

Aneks
12-30-2004, 05:29 AM
Oh goody I'm sooooo glad you asked this as i am in the process of deciphering a methodology to do exactly what you are asking. First of have a look at this :

http://www.highend2d.com/files/dl.3d?group=shakemacros&file_loc=SexyLight_V1B-v1.0-.h&file_id=3041

This is a great macro for doing exactly what you are asking about.

Now onto the crust of the issue.

I have written a tutorial on how to composite lighting and ambient occlusion in shake using dirtmap. It is about 3mb and I am waiting to get Paul over at vfxtalk.com to host it for me ! If you PM me, and give me your email adress I will send it over to you ! Have a read and see what you think.

At the moment it does not use a bent normal pass. But that is the topic for my next tut !

BUT bent normals are a little more complex I'm still trying to get this straight myself ! What you are trying to do is sample an rgb value on the render of 3d model and then use the normal direction of this point to lookup the corresponding color value of your ibl environment and then use this as part of a colour correction which would then appear to increase the 'lighting' effect on the render of the model ! Complex HUH !

Adding directional lighting based on normal map as in 'sexy light' is much easier !

ant-
12-30-2004, 09:59 AM
thanks, the above link for me doesn't work, I think they have an unusual way of preventing hotlinking. instead I found it here, http://www.highend2d.com/files/?group=shakemacros&section=misc called sexy light, will have a play with it now.


its great you are loooking into using ibl, I had thought this must be possible, I just had no idea where to start.

Maxim
12-30-2004, 05:50 PM
hi,
i recently spend some days of thinking about lighting 3d-objects based on normal maps, too.
the reason why you only can light your object from the direction of right, top and front is that your lighting covers only half of your object. look at the red channel. there is no information about the left side of your object. same with green and blue.
i'm not a 3d expert but you will need this information to get a good result.

check this link: http://www.drone.org/tutorials/normal_maps.html
it helped me a lot understanding what i was looking after. there you have a really nice and useful image which covers 180degrees (from right to left, not only from right to center) of info instead of only 90degrees.

as i said, i'm not so much into 3d, but your sourceimages have to look like the ones from drone.org to light your object from every side.

also check: http://www.misterschroeder.de/downloads.html
the normal2light makro is based on the sexylight-makro but you have a slider for direction!
give it a try (your object needs an alpha!).

hopefully this was helpful!

ant-
12-30-2004, 06:47 PM
thankyou, actually the image i rendered does contain enough info for simulated 360 lighting, (though it barely works in 8 bytes, my float version is capable of some very nice backlighting now(fromt he left!) its simply a matter of inverting the information, for the oposite sides. if you look at the images from drone, they are the same red-+X, green-+Y, blue-+Z. I just wasn't sure how to implement the smooth transition between the + and -. (if that makes sence)

the sexyLight macro works ok, but I was suprised how heavey it was computationally, proxy was essential when using the manipulators in the viewport. Normal2light has a simpler approach that seems just as good, but is exceptionally fast.... I will have to rummage round this macro and work out how it works

its interesting to see the contrasting styles of macro making, i think a persons shake tree can tell you a lot about a person (see my one above simple and ineffective)

Maxim
12-31-2004, 12:52 PM
hmmm, floating point. nice to have.
the normal2light macro is designed to work with values between 0 to 1. but there should be no problem to compress your range from -1_1 to 0_1.

yes, my macro has less controllers than sexylight (x and y are combined in direction, z is angle), but you should come to similar results. somehow i'm more comfortable with vectors which uses direction(angle) and lenght rather than coordinates which uses x and y. don't know what this is telling about my personality...

anyway, the onscreen-controls in sexylight are really cool. the use of colorX (as well as all X-nodes in shake) might be the reason for heavy cpu-usage. but i don't want to blame anybody. reverse-engineering is always easier than inventing something!

ant-
12-31-2004, 01:20 PM
ah! you wrote the shake macro, you did a very nice job, it works with float perfectly from the start actually, heres the float file if you want to see how nicely it works.
http://www.platformzero.com/forum/bentNormalRGBfloat.zip
its about 4mb, rendered directly from mray in maya

SexyLight is brilliant, I find the controls really nice, but I can also get on with the directional sliders, i'll be honest I dont yet understand exactly whats going on in either script, I think sexy light has used a method that creates a slightly different type of that makes the surface seem different.

Maxim
12-31-2004, 01:44 PM
it works with float perfectly from the start actually really!? great, never tested.

normal2light is just splitting the sourceimage into the three channels/coordinates and mixes them "somehow" back together. i used sine/cosine-functions to blend them smoothly together.

Blur1
01-02-2005, 03:34 AM
normal2light is just splitting the sourceimage into the three channels/coordinates and mixes them "somehow" back together. i used sine/cosine-functions to blend them smoothly together.

I've found that the best way to check the accuracy of a relighting technique is to try it on a normal mapped sphere because shading imperfections are then apparent. Your sine/cosine blending seems to do the trick here! My source normal map was 8 bit so I did bump up to 16 bit before attaching your macro, in order to smooth out any banding (float was not needed in this case).

Blur1
01-02-2005, 04:18 AM
Back to the topic at hand, how to do post-lighting with a bent normals pass (as opposed to a "normal" normals pass). Basically the goal here is to replicate - in a compositing program - the bent normal lookup into a blurred environment map, that is generally achieved within Renderman.
Here is a great visual explanation of what is going on:

http://www.3delight.com/Zdoc/3delight_36.html#ANC60

This shows a comparison between an environment map lookup with conventional normals versus bent normals (with ambient occlusion pass thrown in to attenuate the result).

So there are big shops that do this in post instead of within Renderman. They render their bent normals pass and then use that to drive a displacement effect that warps the blurred environment map source around the rendered scene. Thus you get real-time image based lighting.
How do they do it? I don't know, but here is my take on the underlying theory:

Displacement in a compositing program is in x and y. Often you may feed the red channel of a displacement source into the xDisp and the green into yDisp. But bent normal passes have blue channels for Z, so where does that go? This necessitates a different approach:

If you take a sphere and place a normal shader on it, then bake the results to a map with a regular spherical unwrap, you will have a flat rgb co-ordinate system for every normal direction on the sphere's surface. Think of this as equivalent to a spherical environment map surrounding your scene.
If you then render a bent normals pass, this is a co-ordinate system that is designed so that corresponding rgb pixels know where to look into the environment map.

So what you need to do in your compositing program is first scale your unwrapped map to the output frame size (say HD res). Then say this rgb value in my unwrapped map is corresponding to this rgb value in my bent normals pass, what is the x and y offset in camera space? Note that camera space is 2D so you don't need to worry about calculating Z.
Once that offset is known, you can apply it to a blurred environment map that matches the unwrapped spherical normal map so that it offsets the corresponding pixel to the new location. Of course, you need to do that for every pixel in the image, then you have relit your 3D scene with just a "simple" warp!

So anyone who knows how to do this with expressions in Shake???

flowerdealer
01-03-2005, 08:57 PM
Hi, I wrote the Sexy Light macro and I've been trying to implement image based illumination in shake. However, the problem is that Shake does not do per pixel operations. So I'm stuck right now. Any other ideas on how to do it? Someone at work suggested doing it with lookup tables, but it would require an external program to generate them.

flowerdealer
01-03-2005, 09:08 PM
I wrote the Sexy Light Macro and I've been trying to implement image based lighting (enviroment mapping) within Shake. The main problem is that Shake doesn't do per pixel operations, so I'm stuck. Any ideas? Someone at work suggested using a lookup table file generated in an external program to do it.

Blur1
01-04-2005, 12:07 AM
Sexy Light is awesome! Thanks for contributing this to the community at large. I have tried implementing something similar in After Effects by reverse engineering your macro, but since AE doesn't do float, it won't work :(

Back to the idea of image based illumination, OK so Shake doesn't do per pixel operations. The closest thing to a per pixel operation maybe is when you feed a source image like a displacement map or optical flow data to a node as a data source. In my last post I mentioned an unwrapped spherical normal map as a co-ordinate source for an environment map, and a bent normal pass as being a "warped" version of this source. So another idea I had is that if you have a kind of displacement map that acts in x and y that shows the pixel shift between the two images, ie. the xy shift of corresponding rgb colour values, the resulting map could be applied to your source env map using a displacement effect.

Maybe this is convoluted, but it is an idea of a possible approach. A per pixel lighting effect would be more logical but Shake is a 2D program.
Another approach based around sexyLight would be to set up multiple light sources that were colourised from the source env map (like importance sampling) and the combined illumination would be equivalent to a light dome in a 3D program, but this would be unusably slow I think.

flowerdealer
01-04-2005, 09:31 PM
The second approach would probably be too slow as you mentioned, but your first suggestion is in fact very interesting, I will look into it as soon as I have the time, thanks!

Blur1
01-05-2005, 01:50 AM
I was reading the documentation for Twixtor 4 (revisionfx.com then downloaded the demo) and it explains a lot about calculating motion vectors based on an xy offset. Seems totally doable. One thing I was thinking about was how to calculate the vector map for a frame sequence. An approach could be this: make the unwrapped sphere normal map the first frame of your existing bent normal pass (resize to match xy frame dimensions) and calculate the motion vectors for this frame sequence. The result is a vector map animation. Apply this to your blurred environment map using xy pixel displacement. Now get rid of the first frame of this new sequence because this was only used to warp the environment map from it's unwrapped state to the first frame of the bent normals pass. In theory you should be left with an IBL animation that registers nicely with your 3D scene.

GatorNic
05-02-2005, 03:01 PM
digging this thread up again....

with the Sexy Light or normal2light macro wouldn't you have to recalculate a new normal map for each frame when you have a deforming mesh, bacause the normals change? I know you can do a tangential normal map instead of a world space normal map, but it doesn't seem like that kind of map would work.

maybe I am just missing something. I am trying to read up but my normal map knowledge is not that great.

Blur1
05-02-2005, 04:11 PM
Jacob at Illuminate Labs has shown me how to use the bent normals pass with the displaceX node in Shake. It's very basic , and obvious when you think about it. In essence you type in the equation to spherically warp a longitudinal environment map (that is scaled to the frame aspect ratio) and you do this for displace x and y with the red and green of your bent normals pass. I can't remember exactly why but the blue channel is somehow implied by the red and green. I have it written down.
There are other issues such as need to oversample/smooth the warped image etc but that is the gist of it.
Perhaps my long-winded vector method would work, but it is much more complicated.
Note that if you are doing an environment reflection then you use a conventional normals pass for a typical spherical mapping and for simulated bounce light and IBL-style lookup you use bent normals.

Cheers

Michael

GatorNic
05-05-2005, 05:34 PM
Hey thanks for the info Michael, I will have to give a try.

Although what is the equation for a warping a spherical lontgitudinal map? I tried looking it up, but couldn't find much on it.

Oh one other thing, the bent normals pass you are using for this, is it in 'camera space' or 'world space'? They are pretty different, wasn't sure if it would work with both.

thanks again! :thumbsup:
Nic

Blur1
05-05-2005, 10:22 PM
Hey thanks for the info Michael, I will have to give a try.

Although what is the equation for a warping a spherical lontgitudinal map? I tried looking it up, but couldn't find much on it.

Oh one other thing, the bent normals pass you are using for this, is it in 'camera space' or 'world space'? They are pretty different, wasn't sure if it would work with both.

thanks again! :thumbsup:
Nic

Hiya Nic,

I re-read my post and it's not really accurate to what Jacob showed me, I think it was about 2 am when I wrote it! Here's what he said (note that this is for spherically warping an env map with a normal pass but the theory applies):

1. I started with three renders: A diffuse pass, a specular pass and a reflection occlusion pass.

2. I added a environment map on the "Mirrored Ball" format.

3. To the reflection occlusion pass, I added a ColorX node, where I remap the first two components of the
color values to lookup (u,v) index in the env-map. According to a standard spherical mapping:

Color N = {r, g, b} from refl occ pass

u = arcsin((r-0.5)*2)/pi + 0.5
v = arcsin((g-0.5)*2)/pi + 0.5

u,v will now be in the interval [0,1]

4. A DisplaceX node is used for the env-lookup.
I scale the u,v values with the height and width of the env map to get the pixel value, and also added a bias value, to remove border artefacts.
[nb. in the xExpr enter 0.5*width*(1.0-bias)+r*width*bias and in yExpr enter 0.5*height*(1.0-bias)+g*height*bias]

Further to this, the bent normals example, was in camera space (I think), but I would like to try it in world space, because that's meant to be more appropriate for doing it in 2d, right? PM me if you can send me a world space bent normal image and I'll try it out. A sphere on a plane is fine (maybe throw a cube in there for good measure).

I wouldn't have figured this out on my own but it seems pretty straightforward once I've been shown. Also, the results will need massaging but that's the gist of it.

Cheers

Michael

DiGiman
05-06-2005, 04:37 AM
Michael,
this is an interesting post. I have been experimenting with this setup that you mention, but I just can't get my head around it.

I scale the u,v values with the height and width of the env map to get the pixel value, and also added a bias value, to remove border artefacts.
[nb. in the xExpr enter 0.5*width*(1.0-bias)+r*width*bias and in yExpr enter 0.5*height*(1.0-bias)+g*height*bias]

Im not sure how to scale the u,v values of something in shake. Would you mind explaining this alittle further? Maybe put up some screenshots, a tutorial or even a scenefile.

Thanks
-Ryan

GatorNic
05-07-2005, 05:24 PM
Wow, thanks Michael! I definitly have to say that explaination was a little clearer, heh.

Haven't had a chance to try it out yet though. But while thinking about it, it should only be necessary to do the longitudinal warp if you are working will a mirrored ball image, correct? If you already have converted it in hdr shop to a logintudinal map, didn't think it would be necessary.

check your PM

Blur1
05-07-2005, 07:03 PM
Nic, thanks for sending me the comparison images. I checked it out and it's world space. It makes sense of course, because otherwise the environment is going to always orient to the camera when warped.

Haven't had a chance to try it out yet though. But while thinking about it, it should only be necessary to do the longitudinal warp if you are working will a mirrored ball image, correct? If you already have converted it in hdr shop to a logintudinal map, didn't think it would be necessary.


I just finished a tough job and was a bit groggy so was not communicating effectively regarding the format for the source env image. I said a longitudinal env map but the example supplied to me was using a mirrored ball image going into the WarpX so I would stick with that until you work out a few more tricks.

Also, as mentioned by Jacob, a bias value with a range of 0-1 is added to the WarpX as a local variable. Create this before plugging in your expressions.


Michael

flowerdealer
05-13-2005, 05:27 PM
Hi Michael, would it be possible to send or post some example pics of the process to better understand it?

Aneks
05-14-2005, 01:57 AM
Hi All,

I had a look at this function and it is really quite nice. With some help from the man (Blur1) himself I can see that is a great proof of concept for how to apply a image based lighting setup in shake. This setup will not give variable lighting direction like sexy light or that 'norman' tool that mpc use. I'll check with Michael first but if its ok I will add this feature to my ambient occlusion compositing tutorial. (and up my version of script !)

Blur1
05-17-2005, 02:17 PM
flowerdealer, apologies for the delay. I will post screengrabs asap. I hope you can use it as some impetus to create a cool little tool for shake.

BTW anyone interested in relighting may want to check this thread as well:

http://www.cgtalk.com/showthread.php?t=229137&page=4&pp=15

There is a bit of talk about relighting the ambient diffuse AND distant light sources, with soft shadows, either in 3D with hardware, or in 2D and a "deep image buffer" of relevant data. There is a link to a good movie:

http://www.cs.virginia.edu/%7Egfx/pubs/vdrt/

fr3drik
09-14-2005, 12:37 PM
No further updates on this? ;)

Blur1
09-14-2005, 12:56 PM
No further updates on this? ;)

Yeah I'm thinking it may be less hassle to just use VRay

PeterSanitra
06-16-2006, 03:55 PM
Hi guys. can somebody give me a tip how to do relighting in Combustion? the passes are no problem, but the how to do the graph...

Aneks
06-17-2006, 12:43 AM
Its been a bit of a while since I have played with C* but I seem to remember that you could just use the gbuffer builder and nominate your normal pass and your depth pass there, then combustion would let you relight things in the scene with the built in 3d lights.

there was a small tut on this on the web somewhere. this is it I think :[/url]

[url="http://www.tutorialselect.com/frametut.php?id=10560&tutorial=http://www.digitalvideoediting.com/articles/viewarticle.jsp?id=32332"]http://www.digitalvideoediting.com/articles/viewarticle.jsp?id=32332 (http://www.tutorialselect.com/frametut.php?id=10560&tutorial=http://www.digitalvideoediting.com/articles/viewarticle.jsp?id=32332)

CGTalk Moderation
06-17-2006, 12:43 AM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.