Triangulating arbitrary polygons with ear clipping

2026-08-23 · Go renderer

Building the 3D Go Renderer meant writing a .obj parser with no scene-graph library to lean on — and .obj files don't promise you triangles. A face line can list any number of vertices, and there's no requirement that they're convex, planar, or even non-degenerate. OpenGL, meanwhile, only draws triangles. Something has to turn an arbitrary N-gon into a triangle fan that renders identically to the original face, for every N-gon a modeling tool might export.

The naive approach breaks immediately

The obvious first move — a triangle fan from vertex 0 to every subsequent pair, (v0, v1, v2), (v0, v2, v3), (v0, v3, v4)… — only works for convex polygons. On a concave face, a fan drawn from a "dented in" vertex draws triangles that cut outside the polygon's boundary. It looks correct on a cube and quietly wrong on anything hand-sculpted, which is most real assets.

Ear clipping, briefly

Ear clipping fixes this by working locally instead of fanning from a fixed point. An "ear" is three consecutive vertices (prev, curr, next) where the triangle they form is convex and contains no other vertex of the polygon. Every simple polygon with more than three vertices has at least one ear (this is the two-ears theorem) — so the algorithm just finds one, clips it off as an output triangle, removes the middle vertex from the polygon, and repeats on the shrinking N−1-gon until only a triangle is left:

while len(polygon) > 3:
    for i in range(len(polygon)):
        prev, curr, next = polygon[i-1], polygon[i], polygon[i+1]
        if is_convex(prev, curr, next) and no_other_vertex_inside(prev, curr, next, polygon):
            emit_triangle(prev, curr, next)
            remove(polygon, curr)
            break

That inner loop is O(n) per ear and there are n−2 ears to find, so naive ear clipping is O(n²) — fine for the low vertex counts an .obj face actually has, and simple enough to get right, which matters more than asymptotic elegance for a parser that has to handle whatever a modeling tool produces.

The part that actually took iteration: which plane?

"Convex" and "contains no other vertex" are both 2D tests — cross products and point-in-triangle checks that assume everything lives in a flat plane. Vertices in an .obj file are 3D, and a face is not guaranteed to be planar: normals can be smoothed, or the source mesh can just be slightly warped. Projecting onto the wrong plane — say, always dropping the Z coordinate — silently corrupts any face that happens to be oriented edge-on to that axis, collapsing it into a degenerate sliver with zero area in 2D.

The fix is to compute a face normal with Newell's method before triangulating, rather than assuming an axis-aligned projection:

nx += (y[i] - y[j]) * (z[i] + z[j])
ny += (z[i] - z[j]) * (x[i] + x[j])
nz += (x[i] - x[j]) * (y[i] + y[j])
// for each edge (i, j) around the face, accumulated

Newell's method gives a stable normal even for near-planar or slightly noisy polygons, because it averages the contribution of every edge instead of trusting any single cross product. Ear clipping then runs in the 2D coordinate system defined by that normal — pick the two axes with the largest projection, or build an explicit tangent basis — so the convexity and containment tests stay correct regardless of how the source face is oriented in world space.

Why this mattered beyond "it renders"

This is the kind of bug that doesn't show up on a cube or the Utah teapot's smoother patches — it shows up on one specific concave, slightly non-planar face in one specific model, and looks like a rendering glitch rather than a geometry bug. Testing it properly meant writing degenerate and concave fixtures deliberately rather than trusting whatever test models happened to be lying around, which is the same lesson that runs through the parser, UV generation, and frustum culling code in the rest of the renderer: the failure cases you don't construct on purpose are the ones that ship.

← Back to the 3D Go Renderer project