What I have found so far with my workflow in nCloth is the main quality points - other than a good set up of the mesh objects - are the collision flags, which also affect the trapped options, the trapped check and self trapped check, push out value and most important the nucleus solver settings.
The iterations settings are perhaps a deceiving title. From my understanding those are clamp values not a progressive increase of quality.
Max Iterations Specifies the maximum number of iterations per simulation step for the current nCloth objectÂ’s dynamic properties (for example, Stretch Resistance and Bend Resistance). Max Iterations clamps the number of iterations to prevent high level property values or a large number of substeps from locking up the nCloth.
An iteration is a single computation of a single nCloth property by the Maya Nucleus solver. There are many iterations for each dynamic property value per step. The number of iterations for each dynamic property is automatically set by their current values. Higher dynamic property values generate larger numbers of iterations.
I was not around for classic hair so I am not sure about what it was called if it was substeps or iterations. But I have seen iterations in other apps named so to indicate the same things as sub steps.
Since iterations in this case is a “negative” property, not having the option would mean no clamp on the quality setting rather than the other way around.
For nHair I have found the nucleus solver to have a very small amount of effect on quality issues. I have solved most issues with the settings of the nHair. For instance changing the collision flag to edge among other things.
And in my case had quite a bit of difficulty getting the hair not to stretch. In the docs it said a higher substep value would perhaps help this. And this may be so but I have found that that in conjunction with the stretch value was the only thing what worked. Seems logical. lol
But there was also an issue I had with the hair just dancing all over the place and I was not having any luck at all. My only solution was to apply a small amount of damp value to it. In my case .2 was a nice amount to keep the hair flowing and yet not squirming all over the place which just looked like a quality issue.
There are some setup issues that I also found were the source of some problems. Mainly making the scalp a collision object. This can cause the hair to pop up and keep dancing around. I suspected this was because it was colliding directly with itself as I have seen this in cloth. And the solution seems to be to use another object for collision.
Not sure if any of this will help in your situation but this is my experience so far.
If you describe your problem in detail maybe someone can give you some pointers on a solution.