I have coded up a routine to check how many polygons will intersect a ray shot from the mouse cursor (so I can make polygon selection code)
The result from the routine produces the correct answers most of the time, so if the ray intercepts 2 polygons, then 2 is returned.
The problem is that my code has been made to work with triangles and not quads, because I didn't think about quads until after I started to test it.
I only have to work with tri's and quads so ngons is not a problem.
The quads that cause problems are invalid quads that lie on 2 planes
Polygon = Vector(-1,0,-1), Vector(-1,0,1), Vector(1,0,1), Vector(1,50,-1)
The last vertice is too high and no longer lies on the same plane as the first 3.
Casting a ray onto this polygon, will only intercept the first half of the polygon, because im checking only the first 3 points for the intercepting.
The way im doing this is, checking if the ray intercepts the polygons plane. The polygon only has one plane when all 4 vertices are lined up correctly, so I don't have a problem there.
What im asking is, what shall I do to overcome this.
Ive thought about casting two rays, if it dont find the polygon first time, try again on the second plane of the polygon.
The problem is, that essentially the polygon has 4 planes, and i dont know how to figure out the correct 2 planes.
Maybe I havn't thought this through enough yet, but I thought I'll just ask see if anyone has an idea