I'm trying to create a visual effect where an arbitrary part of the main scene is either clipped out by or cropped to an arbitrary volume. That is to say, I might want to draw just the area of the scene inside the volume, enhanced by other elements, or only the area outside the volume, leaving empty an area I can fill in later.
The main scene is mostly a 10k-500k face surface (not solid). The clipping volume is an extruded polygon, usually 200-300 faces for the entire volume, not generally convex but easy to convexify if needed.
I've tried an approach basically identical to stencil shadow volumes, but where instead of drawing a lighter/darker version of the same scene, I just don't draw the pixels either in or out of the stencil area. This is about 85% of the way to the solution, except that (for the clip mode), pixels from the scene inside the clip volume that would normally occlude other parts of the model, but should be ignored, are left in the stencil.
The problem also seems eerily similar to constructive solid geometry. I would rather not solve it by manipulating the geometry itself, as manipulating 500k+ faces in-browser has not shown itself to be terribly performant (yes, I tried). There's a little bit of work on doing arbitrary CSG via stencils (like this) but it doesn't seem to have gotten much attention in the last decade.
In my ideal case, I'd be able to use some set of techniques to individually identify regions of the viewport that contain:
- Pixels outside the clip volume, even if they would normally be occluded by pixels inside the clip volume
- Pixels that the mesh inside the clip volume would occupy, even if normally occluded by other pixels outside the clip volume
My target is WebGL, but happy to look at plain OpenGL techniques that may be applicable, or those enabled by WebGL2.

ctx.stencilOpSeparate(ctx.BACK, ctx.KEEP, ctx.DECR_WRAP, ctx.KEEP);*ctx.stencilOpSeparate(ctx.FRONT, ctx.KEEP, ctx.INCR_WRAP, ctx.KEEP);* Render the clip volume * Clear depth, disable stencil write, enable color write, depth test/write * Stencil test for zero, should be the areas outside the clip volume * Render full scene again \$\endgroup\$