Domain Expansion
Make a Jujutsu Kaisen hand sign at your webcam and the matching effect appears on screen. MediaPipe tracks your hands and Three.js handles the particles. I made it to learn gesture tracking and shaders, and because I wanted to throw a Hollow Purple.
The techniques you can throw
Each hand sign maps to its own particle effect. These are the ones working right now.
Blue
Four arms of particles spiral inward and collapse onto one point.
Red
The opposite of Blue. Five jagged arms push outward and scatter across the screen.
Hollow Purple
Blue and Red meet in the middle, with two vortex arms spiralling into each other. This is usually the first one I show people.
Infinite Void
Three rings, a pillar of light, and 30k particles fading into a starfield.
Malevolent Shrine
A torii gate, ground aura, embers, and sound. It needs both hands, so other gestures are ignored while you form the mudra.
From webcam to particles
The short version, in four steps.
Your webcam
The browser reads your camera feed. Nothing gets uploaded.
Tracking your hands
MediaPipe follows 21 points on each hand, including the fingertips, knuckles, and wrist.
Working out the gesture
The distances and angles between those points map to a gesture. It has to hold for 10 frames before anything fires.
Drawing the effect
The matching GLSL shader takes over, with 30k particles, additive blending, and bloom.
Jujutsu Kaisen is just really good.
That was the motivation. And, it gave me the reason to learn gesture tracking and spend time writing shaders. SAT0RU by reinesana was the starting point, then I kept adding my own effects. The gesture detection is still sensitive to camera angles, but when it locks in, it feels good.
The stuff it's made of
- Three.js
- MediaPipe Hands
- WebGL GLSL Shaders
- UnrealBloom (three/addons)
- Web Audio API
A couple of notes
A gesture has to hold for 10 frames straight before it fires, and there's a 30-frame grace period before it drops back to neutral. That's what stops it triggering every time your hand twitches.
Malevolent Shrine has two-hand suppression built in, so single-hand gestures get blocked while you're forming the Sukuna mudra. It only fires once you've locked it in on purpose.
However, please forgive me for not working as much on this project as I've hoped, for a while. Time hasn't allowed for it and it isn't as polished as I'd like it to be.