check out the local rotation axes of the joints.the knee joint might have a different orientation than the others,in which case you need to orient from the hip joint downwards using the Skeleton->Orient Joints[].
You must be careful with joint orientation.It’s a very serious matter that can mess everything up.
also try not to rotate joints directly.It changes the orientation.
to see if the orientation is indeed the problem shift select all three joints that the ikHandle controls and go to Display->Transform Display->Local Rotation Axes.This is a great tool.It shows you the rotation axes of anything.So if they are not consistent this is your problem.
afterwards you need to choose an orientation that works for you and which prevents as much as possible the gimbal lock(i realize that you might be a newbie but these are important things to know which can save you a whole lot of redoing work in the future,trust me) 
EDIT:also the pole vectors Panupat mentioned are handy too but you should fix everything else before doing that in order to have a correct and solid model
EDIT2:i advise you not to use perspective view when creating joints.It’s much easier because in perspective the position you see isn’t always the one you want.there are three dimensions to fix.in side view you have only two since everything else goes on the origin of the third.that’s why in side view it’s easier to create joints for the legs for example.