View Full Version : Scripting an Edit poly Psuedo Booleen

10 October 2005, 02:20 AM
Let me begin by saying I am reading the docs on this one and am just taking a break to ask if there is anyone who could point me in the right direction. Possibly save me some time if they know a faster easier way.
Now on to the issue:

My team is creating alot of nav meshes for AI and I've began making tools to help them in production. Each are added to a tool bar I have and now I'm looking into a script I can place under a button....and select something like a building or house and the navmesh, or terrain, beneath it and have it cut the shape from the terrain. The script should leave the building intact, but really just use the edge.

Slow method on this is to select the edges on the building make it a shape shape merge it into the scene and delete the center poly.......toooooo long.

So does any one have any idea exactly what Im needing form this to make it work on two object?

I'm just trying to get my head around what I can do in max script on an edge or vertice level.

I know I havent given much detail on the scripting I have done on this so far, but overall I'm looking into simple methods like creating arrays from selected edges. But I would feel better if I knew I was heading in the right direction and that edit poly and maxscript can handle this?

10 October 2005, 03:51 AM
Hi, I'm not exactly sure what you need to do...but using max script you have pretty much complete control over the mesh, or mesh creation.

10 October 2005, 09:11 AM
Perhaps getting all the verts from the edges at the bottom of your building as an array, then looping through each one, raycasting in the -z direction to find where they interupt the terrain and cutting a vert into the terrain mesh at that point might work :thumbsup:

10 October 2005, 09:13 AM
If you start from an outline, that seems feasible.
You transform your outline into a series of vertices. In the right order of course.
You calculate the position of each point on the ground. There are commands like intersectRay for that.
You select the faces under the building with his bounding box.
For each couple of points, you create an new edge with a command like polyOp.cutFace
You select the face inside of the initial outline. That could be more difficult.
You could calculate 2 distances:
- the distance between the center of the outline and the center of the face
- the distance between the center of the outline and the point of intersection with the outline
If the first distance is lower than the second then the face will be inside the outline. (It does work only with a convex outline)
Next you delete faces.

EDIT: sorry Moosley I wrote my text at the same time as you

10 October 2005, 03:53 PM
i'm glad to hear the gathering of verts and using ray intersection "I deas.....this is the direction I was trying to convey Im heading in.
But I probably wasnt too clear in my description.

I'll try and wrap up this portion of the code, then I foresee my next issue will be to get the new section Ive made in the mesh to delete.

Pretty Pixel:
I see you covered this in your description, but Im not too familiar with how I will accomplish getting the distance to determine what polys should be deleted. I'll need to look at that more in depth....

thanks to all of you.

10 October 2005, 07:03 PM
After reflexion, it seems that It is not necessary to calculate the intersection with surface.
The tool polyOp.cutFace seems already to do it.

polyOp.cutFace <Poly poly> <int face> <point3 start> <point3 destination> \ <point3 projdir>

I did not test it but <point3 projdir> seems to be a projection vector.
I suppose if you use [0,0,-1] that create automatically the vertical intersection with the surface (?)

10 October 2005, 12:36 AM
I have just created a kind of 'shape merge' function

- create a plane named Plane01
- convert it to poly

- create a circle named Circle01
- adjust interpolation to 2 or 3
- move the circle completely inside the plane
- convert it to poly
- select all edges of the circle

- evaluate the script

struct vectorizedOutline

fn getFirstItem theBitArray = (for i in theBitArray do return i)

fn vectorizeEdgeList obj theEdges =
if theEdges.numberset!=0 do (
theEdgesMem=copy theEdges
currentEdge=getFirstItem theEdges
currentVerts=polyOp.getEdgeVerts obj currentEdge
append vertsString currentVert
append vertsPosString (polyOp.getVert obj currentVert)
while theEdges.numberset!=0 do (
append edgesString currentEdge
append vertsString nextVert
append vertsPosString (polyOp.getVert obj nextVert)
edgeList=(polyOp.getEdgesUsingVert obj #{nextVert})*theEdgesMem
if edgeList.numberset==0 do exit
currentEdge=getFirstItem edgeList
currentVerts=polyOp.getEdgeVerts obj currentEdge
if currentVerts[1]==nextVert
then ( currentVert=currentVerts[1] ; nextVert=currentVerts[2] )
else ( currentVert=currentVerts[2] ; nextVert=currentVerts[1] )
vectorizedOutline edges:edgesString verts:vertsString vertsPos:vertsPosString

fn cutOutline obj posArray projVector =
if posArray.count!=0 do (
polyOp.setEdgeSelection obj #{}
for pos in posArray do theCenter+=pos
obj2=snapshot obj
retArray=intersectRayEx obj2 (ray theCenter projVector)
if retArray==undefined do ( retArray=intersectRayEx obj2 (ray theCenter -projVector) )
if retArray!=undefined do (
theFaceVerts=(meshop.getVertsUsingFace obj2 #{face})as array
delete obj2
for pos in posArray do ( newVert=polyOp.cutVert obj newVert pos projVector )
outlineEdges=polyOp.getEdgeSelection obj
theFirstEdge=getFirstItem outlineEdges
polyOp.setEdgeSelection obj outlineEdges

theOutline=vectorizeEdgeList objShape (polyOp.getEdgeSelection objShape)
outlineEdges=cutOutline objGround theOutline.vertsPos [0,0,-1]
max views redraw

The edges selection is returned by the function.
Now The only thing which remains to be made is to select the faces inside the outline...

10 October 2005, 03:31 PM
Here a version which removes the inelegant edges.

fn cutOutline obj posArray projVector =
if posArray.count>1 do (
polyOp.setEdgeSelection obj #{}
for pos in posArray do theCenter+=pos
obj2=snapshot obj
retArray=intersectRayEx obj2 (ray theCenter projVector)
if retArray==undefined do ( retArray=intersectRayEx obj2 (ray theCenter -projVector) )
if retArray!=undefined
then (
theFaceVerts=(meshop.getVertsUsingFace obj2 #{face})as array
delete obj2
oldEdgesBitarray=#{1..polyOp.getNumEdges obj}
newVert=polyOp.cutVert obj newVert posArray[1] projVector
badNewEdges+=polyOp.getEdgeSelection obj
polyOp.setEdgeSelection obj #{}
for i=2 to posArray.count do (
newVert=polyOp.cutVert obj newVert pos projVector
thisNewEdges=polyOp.getEdgesUsingVert obj #{newVert}
if (thisNewEdges*oldEdgesBitarray).numberset==0 do badNewEdges+=thisNewEdges
outlineEdges+=polyOp.getEdgeSelection obj
polyOp.setEdgeSelection obj #{}
bit30=bit.set 0 30 true
polyOp.setEdgeFlags obj #{1..polyOp.getNumEdges obj} 0 mask:bit30
polyOp.setEdgeFlags obj outlineEdges bit30
polyOp.setEdgeSelection obj badNewEdges
obj.Remove selLevel:#Edge
outlineEdges=polyOp.getEdgesByFlag obj bit30
polyOp.setEdgeSelection obj outlineEdges
else delete obj2

10 October 2005, 05:58 PM
I might be thinking too simplisticly, but here is what I would do:

*Create a large plane with, say, 20x20 segments.
*Add a Noise Modifier on top and adjust to get some funny terrain-like mesh
*Create a Box (or any other closed volume representing the building) and place on the mesh, intersecting enough at the base.

Now open the MAXscript Listener and type in:

$Plane01 - $Box01

Instant boolean subtraction. The plane should now have a hole where the box penetrated.

This (as all booleans) might not work in all 100% of cases, but it is much faster than scripting own cutting operations. Plus, the higher the mesh density, the better the result.

Note that the original plane was converted to an Editable Mesh. You might want to create a clone of the original terrain node, perform all booleans on it, then replace the mesh of the original node with the TriMesh of the boolean result.

The only thing left is to create the projection volume of your buildings. If the bases of your objects are easily traceable, you can create a shape, extrude to intersect the terrain, boolean.

Hope this gives you an alternative approach to your problem...


10 October 2005, 07:20 PM
Nice trick, Bobo:).

10 October 2005, 09:09 PM
Now open the MAXscript Listener and type in:

$Plane01 - $Box01

Instant boolean subtraction.

I am surprised.
I am happy :o)
yes nice trick :)

I am also interested by the cutting of a form on the surface of a poly. (for other reasons)

The boolean meshes work really well. It is a pity that sometimes the faces are not deleted.
That is a little frustrating because the method is so easy.
Do you know why the faces are not always deleted ?
Are there a way to prevent that ?

I have another question: Are there methods to select the faces inside an outline ?

11 November 2005, 03:01 PM
Thanks so much for the ideas everyone.
Looks like the simplest code has worked for my temporary solution here. Thanks Bobo your idea it took me in the right direction. In my tool I changed it just a little
$NavMesh - $
So all the artists need to do is select the the initial building or Barrier object and it will cut it into the Nav mesh.
This is still very manual in my opinion when you look at a terrain with 300 plus buildings.

By chance is anyone out there in the same boat developing tools to speed up AI navmeshes?

11 November 2005, 03:41 PM
Just a minor tip, (sori if its not completely related to the topic, but i guess i could just add):)
Say you were to subtract a sphere to a plane, tesselating the plane to TRIs (or just adding an edit mesh above it and making all the edges visible) will produce cleaner cuts for booleans.

$sphere01 - $plane01

so i guess its much safer to work with TRIs rather than quads for situations such as these...

11 November 2005, 05:25 PM
This is still very manual in my opinion when you look at a terrain with 300 plus buildings.

ME3D: If the number of buldings is the problem, It's easy to apply the idea of bobo to an selection of objects:

fn isGeometry obj = ( if ( isKindOf obj geometryClass and classof obj != TargetObject ) then true else false )
if isValidNode navMesh do (
sel=for obj in selection where (isGeometry obj and obj!=navMesh) collect obj
for obj in sel do (navMesh - obj)

I added some filters to avoid the most usual errors.
This script suppose that your object name is "NavMesh"

Galagast: very interesting observation.
In fact the boolean objects seem always converted to mesh internally.
It is not a good news for me because i use always polys :-(

CGTalk Moderation
11 November 2005, 05:25 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.