Smart plot is for key reduction… so it tries to use interpolation to plot down the motion rather than a key on every frame. This is handy whilst working in MB if you’re doing manual key editing, as its more familiar and similar to a hand animation workflow. I wouldnt necessarily recommend it however for the final export, as only the key values will be exported, not the interpolation type between them. (this may be different if ur using the 2012 versions, send to max/maya, but I cant speak to that).
Now as for rotation filtering, that is basically to filter out the gimbal flips that you often get with mocap, owing ofc to euler rotation order. This, you may or may not care about… obv, if you’re going to a game engine, the euler flips are a big no-no, and the same can also be said for some rendering pipes where motionblur is required (as you’ll get a “pop” blur). As for how the differnt filters work (gimbal killer and unroll) they are each doing the same thing, but in different ways… so far as Ive ever known, there’s no real way to be sure which is right for a given motion… Unroll can work well if you’re using the “roll bones” in limbs (arm, armroll, etc), as it spreads the roll, allowing for less gimbal locking on the main joint… but gimbal killer can also be the best a lot of the time.
The best way to get a feel for the 2 rotation filters, is to try them out directly, and observe their effects on your Fcurves… to do this, plot with NO filter applied, so you jsut get your base keys. Then bring them all up (for a given joint, or even all joints) in the fcurve editor, select the keys, and use the filter controls directly (filters tab next to asset browser, pose, etc)… you can then do, undo, try the other, to see which works the best for a given motion on a given joint. Often, if its absolute perfection in curves you’re after (ala game skele) You’ll find that you want to filter some joints using one filter and other joints using the other… you may even have to go and make some manual changes to the overall curve data too here n there.