you are on the right track - unfortunately when doing a railsweep on a helix it often gets mixed up because the function uses different reference points on the profile curve(s) - e.g. if u have a square it might take the the lower left corner of and on the next loop it might take the center of the square messing up the thread.
To ensure to get the desired result you sometimes have to do a somewhat cumbersome routine:
put your profile curve at the end and the beginning of the helix and orient/align them identically. Now make copies of the profile curve and rotate them 90, 180, 270 and 360 degrees and align them the same way you did before at the ends of the helix curve (use the quadrants option of the objectsnap-tools). This way you have a set of one full 360° loop with correctly oriented profile curves - now copy this set along the helix axis as often as needed. When you now do a one-railsweep you have a lot of profile curves to create the shape well-defined - but u need to make sure u always use the same reference points within your profile curves along the rail-curve after you picked the last profile curve and before you press enter (or RMB) to finish the command - then it works like a charm.
This also might work with only 180° dividing instead of my suggested 90°, but don’t bet on it - anyway you still need to pick the right reference points.
If this explanation alone doens’t help I’ll provide some screenshot/illustration later…