TBN for skinned mesh

[Per-Anders]

The TBN matrix is just used to transform the light vector (normally) into texture space (or tangent space if you prefer). You just do that on a vertex level basis as it's much faster that way rather than calculating it for every single point (i.e. it's like phong). All you then do is treat the normal map as the actual surface normal to calculate the illumination.

[yakul]

Yes I know that. This is how it works for static object.
However, the TBN is precalculated. You dont calculate the TBN in the shader, because the TBN of a specific vertex depends on all the vertices that belong to the triangles this specific vertex is part of.
However, when you deform the body, the TBN for every vertex of the deformed body should change. And since you precalculate once the TBN, you have a problem with a deformed body.
Or I might be wrong.

[Per-Anders]

I think you're slightly confused by the idea that you need to store these matrices, you don't. You calculate them as a part of your vertex shader, but all they are at the tangent, binormal and normal, what you store is the light vector for each vertex, which is the incoming light-vector multiplied by the inverse texture space matrix for the vertex, that's stored simply by dint of generating the vertex-shader in the first place. Your pixel-shader then takes the value that's interpolated for you, normalizes it and now you have a light vector in tangent/texture space ready to input into your illumination model (together with the value from the normal map multiplied by two and with one subtracted to give you a proper normalized surface normal vector). For each pass you will be recalculating this stuff on the fly, it's not that much more of a step from calculating phong shading, and for any static objects of course you can store the matrices if you wish for speed (and if the lights are static you can even store the light vector).

edit - here is an example showing exactly how it works after a seconds search on google, code needs a little tweaking, but it shows the fundamental approach : http://dotnet.org.za/pieterg/archive/2005/07/29/40407.aspx

[yakul]

The code you have linked me to shows how to have normal mapped static object.
This line: float3x3 TBNMatrix = float3x3(IN.tangent, IN.binormal, IN.normal);
Where does IN.tangent, IN.binormal and IN.normal come from? They have been precalculated and stored in the vertex buffer of the static mesh.
This is the problem.
When you have a skinned mesh, those three vectors change according to the deformation of the mesh. So you either need to recalculate them, lock the vertex buffer and write them, then send it to render. Or use something else.

[Per-Anders]

Well yes, you must calculate these in exactly the same way that you calculate the phong. How you store and calculate this is up to you, but yes it needs to be updated constantly on a moving character for any points that change, whether you use a flag, let the deformer do it (only good with a single pass defomer), cache it (morphs and prebuilt loops only) or brute force it. Ideally your spacial conversion code shouldn't be too slow in the first place.

[yakul]

Well yes, you must calculate these in exactly the same way that you calculate the phong. How you store and calculate this is up to you, but yes it needs to be updated constantly on a moving character for any points that change, whether you use a flag, let the deformer do it (only good with a single pass defomer), cache it (morphs and prebuilt loops only) or brute force it. Ideally your spacial conversion code shouldn't be too slow in the first place.
That is exactly what I am trying to figure out.
I can think of some ways to do it myself, but I thought it would be better if I can learn from someone else who already tried all these stuff and can tell what are the pros and cons of every method to achieve this.
I couldnt find material on the web on this matter, perhaps I lack the correct term to describe what I am looking for.
How can I look for this on the web? normal mapped skined mesh?
I can't find any material on this "thing".

[Robert Bateman]

you've got two choices,

a) pre-compute the tangents & normals for the bind pose, and then rotate them via the bone TM's, then do a cross product in the vertex shader to calculate the bi-normal. (assuming you are currently doing all skinning in a vertex shader - in which case just treat the tangents the same way as normals).
b) do the skinning in and TBN calculation in software. (can just do the tangents and normals and let the vertex shader do the tangents if you want). If your skinning is already in software, then just do the same process for your skinned meshes as you currently use for your static meshes.
c) use DX10's geometry shader to calculate the TBN matrix on the fly in hardware.

http://www.robthebloke.org/opengl_programming.html#5 << see the obj loader example for how to calculate the TBN...

[Robert Bateman]

This is the problem.
When you have a skinned mesh, those three vectors change according to the deformation of the mesh. So you either need to recalculate them, lock the vertex buffer and write them, then send it to render. Or use something else.

technically speaking, not really. In the vertex shader you can rotate the normals via the bone transforms (bear in mind that if scaling is used, you should use the inverse transpose of the matrix). Having rotated them, use slerp to blend their weights (rather than linearly interpolate as you do for vertices). I believe nvidia have an example for this somewhere.

I normally just do all that stuff in software though, and save the rest of the GPU for lighting calculations. But hey, that's just me....

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.