OK, I think I can understand what you are going through. There is a lot of information you have covered there. So lets break it down to simple to understand compartments.
First Control Rigs and Custom Rigs.
The entire HIK system is based on a control rig. The only way to modify this rig is to add Auxiliary Effectors and so on. It is basically a set thing. You don't customize it internally.
What you are doing when you characterize a skelleton, is you are just telling the HIK system which bones will be driven by the control rig. Same as when you define the skeleton.
Based on my theoretical understanding. I have not used this yet -A custom rig is one that you create with other controls already created for animation above the basic deforming bones. In other words, you already have a control rig you have built and use for other purposes in your pipeline. And you don't want to tear this apart. You just want to have the HIK system take over, lets say to retarget animation to this rig.
Custom rigs are typically driven by some combination of ikHandles, constraints and other miscellaneous nodes. The Custom Rig tab provides a visual interface for mapping your custom rig, and retargeting tools to transfer data from other HumanIK defined characters.
So rather than assigning the controls to bones, you are assigning the HIK controls you the controls you already have set up. That is the main difference.
That is why the definition panel looks like this:
And the Cutom rig definition panel looks like this:
Notice the similarity to the Controls tab:
Maybe this makes more sense now.
Now mocap source data:
I think you mean the CMU database correct?
Make sure you are using the MotionBuilder files:
https://sites.google.com/a/cgspeed.com/cgspeed/motion-capture/cmu-bvh-conversion
And second use the later examples from the trials. The early experiments are of lesser quality.
And finally,I seem to remember there is a filter you can apply to fix those bad rotations. But now it escapes me. Otherwise, the best thing to do is first go in and simply clean up the channels by hand in the graph editor. I don’t think you want to try and do that with the animation tools. Give yourself a good clean data file to start with. Then on top of that clean data make your tweaks and changes with layers.
Foot sliding can be just bad mocap data. And you can use an override on that if you want, but I think it is simply better to do it by hand in the graph editor.
In general that mocap database can be very strange to work with. It is not the best quality. So you may find yourself having to do a lot of tweaking and clean up.
I find some of the joint rotations to very strange, usually in the neck and head area. And for this it seems layers can help to compensate. But for limb flipping and rotation it is best to just clean it up.
Then Story and Trax, mixing animation in general is a way to simply join different clips together, and I have found it is best used just for that. Then use the Layers to do non destructive clipping on top of Story edits that I have plotted back to the control rig.
Now the idea is, assuming you have clean mocap data to start, you simply use that control rig and layers to refine and make changes. It is just another way to work other than hand keyframing.
Not saying it is better. But it is another workflow. And it is best to understand how to use it the way it is designed to be used. It does have many advantages.
, “Body Parts”
and "Full Body
just like you can in MB. But I am not sure how to go further and just key T or R like you can in MB. I am sure you can.