its a nice idea to have controllers in the scene so you know what kind of motion you are looking for, but the basic setup for a mechanical rig is placing the joints and testing on how they behave and if they behave in the way you want
parent the geometry to the joints to see if the motion is correct, if not, move the bones around.
when that basic setup is done (and clean) then you can think about how to let the controllers influence the joints.
there are more then enough tutorials on these subjects, main things you need to know is joints, ik handles and constraints