Yeah, this does come up a lot. You are on the right track. If you just do this:
Corrected Vertex in Object Space - Skinned Vertex in Object Space = Corrective Delta in Object Space
you get a delta that isn’t that useful. You can stick it on a blendshape that comes before the skinCluster, and turn the skinCluster back on, and get this:
Corrective Delta in Obj Space + Skin Vertex in Obj Space = Corrected Shape
So, yay, I guess. But the problem is that corrective deltas are often very small, and object space is a big place. So even small changes to your deformations can make that delta useless. This is what you are saying is horrible, and I agree. I have seen scripts that turn the skinCluster off, turn each blendshape on one by one and clone them out, then on the new skinned mesh grab all the clones and re-save all the blendshapes. You can try something like that. It’s saved my bacon before. But it is also pretty crude.
It’s much more flexible to store the corrected shape in “pose space”. That means you look at all the joints influencing the vertex, figure out what that ‘skin’ transform is, and store the corrective delta after that transform has been removed.
First, remember that a skincluster works like this:
For every vertex:
skinnedPosition = (0,0,0)
For every joint:
weightedToJointPosition = initialPosition * jointTransform
skinnedPosition += weightedToJointPosition * jointWeight
That is why it’s so important your joint weights all add up to 1.0.
To reverse a skinCluster you would need to do something like:
For every skinned vertex:
unskinnedPosition = currentPosition
For every joint:
weightedToInverseJoint = currentPosition * inverseJointTransform
unskinnedPosition -= weightedToInverseJoint * jointWeight
If you do this to your delta you end up with a much more useful vector. This one will still be useful if the bones change, because it doesn’t care about the bone transforms. In fact, unlike blendshapes, you can apply this transform AFTER the skinCluster, instead of before. It can ride on top of the skin.
Obviously there’s a lot more stuff that wraps around this, and there are issues of properly blending all the poses together when they overlap, so I’ll stop there. Fortunately, if you want to dive into this, Michael Comet wrote a plug-in that handles it all called “Comet PSD.” There’s an entire workflow involved, but you can start saving your corrective shapes as PSDs (pose space deformations) and it will help you a lot in the future.
Website is here:
http://www.comet-cartoons.com/maya.html
Someone compiled for more recent Maya here (or Google for the version you need):
http://www.djx.com.au/blog/2013/04/14/posedeformer-for-maya-2014/
It is also open source under the GPL license, so if you are interested you can dig around and see how it works. There are other pose space solutions available too. It is a pretty popular solution at many studios.