Computer vision / 2026 / Live in the browser
Vortex Builder
Build voxel structures in the air over your webcam feed: pinch to place, pinch and drag to paint, make a fist to turn the build. A second synced page drives presets and shows live status.
01
The problem
Hand-driven building fails in small ways that add up. Aiming with the index tip drags the cursor toward the thumb on every pinch, so blocks land a cell off. MediaPipe's per-landmark z is relative to the wrist, not the camera, so it cannot tell you how far away the hand is. And a pinch with the other fingers curled looks a lot like a fist.
02
How I approached it
The aim point is the midpoint of thumb and index, which barely moves as the pinch closes, run through a One Euro filter and mapped through the object-fit crop and the selfie mirror. Depth comes from a ray cast into the instanced voxel mesh instead of from z: aim at a block and the new one lands on the face you are pointing at, aim at empty space and it lands on the grid plane facing the camera. A held pinch paints along the plane it started on, filling skipped cells and refusing jumps a hand cannot make in one frame. Pinch detection uses metric world landmarks with separate enter and exit thresholds, and the rotate fist comes from MediaPipe's trained gesture classifier rather than a curl threshold. Hands get roles by gesture, not by the left or right label.
03
The outcome
Pinch, paint, erase and rotate run on one GPU gesture model per video frame, and the same picker drives a mouse fallback (Shift+drag to place, right-drag to erase) for visitors without a camera. The gesture math and the placement state machine are unit tested under Node.
Related work
Hand-tracked cursed-technique visualiser: MediaPipe landmarks drive a Three.js particle engine, with an ML classifier picking the technique in real time.
A 3D globe you steer with your hands: pinch to zoom, move your finger in mid-air to spin it. No camera? Drag, scroll, pinch with two fingers or use the arrow keys.
ASL fingerspelling to text from a webcam. It reads hand geometry instead of pixels, so all 26 letters work with no training, and two seconds of calibration per letter fits it to your own hand.