Freak'n Genius
Animation made toy-like. Your body puppets a cartoon through a depth camera; your voice puppets any photo on a phone.
What it was
Real-time character animation with no rig, no keyframes and no training. Stand in front of a depth camera and a cartoon moves as you move. Kids as young as three used it. This was novel in 2010.
The architecture
One portable C++ engine across three platforms: Kinect with DirectX 9 on PC, a Kinect build for Xbox, and OpenGL ES on iOS for YAKiT.
Capture and rendering were per-platform shells. The puppet code was shared, which is the only reason a two-person engine team could hold three platforms at once.
The decision that made it work
The Kinect SDK delivered 20-joint skeletons at 30 fps. The core decision was to flatten 3D to 2D on purpose.
Characters were cutout puppets, flat textured segments hinged at joints. Driving one meant mapping joint pairs to segment rotations: shoulder→elbow becomes the upper-arm angle, one atan2 per bone. No 3D rig, no skinning.
That is why any drawn character, a kid’s doodle, or a bag of Doritos could be performing by the afternoon.
Making it not look broken
Raw Kinect data bends characters in ways that look wrong, so every bone was gated by joint confidence and clamped by constraints, torso ±30°, head ±30° relative to torso.
The deepest code was two-bone foot-planting inverse kinematics: measure the puppet’s leg bones, watch the performer’s real ankles, decide which foot is anchored, and solve to keep feet glued to the floor. That is the difference between a character that looks planted and one that looks haunted.
Most of the animation engine was really a trust engine. Confidence gates, angle clamps, drop-the-frame rules, deciding which parts of the skeleton to believe, frame by frame.
YAKiT, same engine, sensor optional
Cut a mouth out of any photo and drive the jaw with your finger, or with your voice, chunked into 60 RMS energy samples per second through a hysteresis gate that flapped the mouth in real time.
Recording captured an event timeline synced to audio, so playback re-performed the show. Export re-rendered it at 30 fps to H.264 via AVAssetWriter. Everything ran on-device at frame rate; the network only ever carried finished videos.
What broke
Foot-planting looked magical until the sensor reported feet farther apart than the puppet’s legs could reach. Then the law of cosines goes out of domain, returns NaN, and the character’s legs vanish.
// out of domain — acos() would return NaN and the legs vanish // Neither leg angle is possible. Let them both hang!
It sits beside an entire abandoned 3D leg solver, preserved in a #if 0 block. For cutout puppets, solving in 2D with guardrails beat solving in 3D correctly.
Flattening a 3D human onto a 2D puppet failed exactly where a human is 3D: point your arm at the camera and it foreshortens to nothing; turn sideways and half your joints become guesses.
YAKiT’s voice-driven mouth was amplitude, not phonemes, a hysteresis gate on mic energy. Beautiful in a quiet room, flapping like a screen door in a loud one. We shipped a sensitivity slider, which is another way of saying we made the user solve it.
But the real breakage was platform risk. We built for Kinect on Windows, ported toward an Xbox indies could not actually ship on, and shared through a Flash player as Flash was dying.
The engine was portable. The ground under it wasn’t. When your product rides someone else’s sensor and someone else’s platform, their roadmap is your risk register.
They found the magic mirror first, and the stage second.
First they just wanted to see themselves in it, pure play, no goal. The record countdown is what turned play into authorship: short skits, messages for one specific person, a character doing a bit. Same tool, completely different activity.
Users
Animated video created
Apps across 4 platforms
App Store, Photo & Video, US and international
Adjacent gesture and projection prototyping from the same years, the sensor work that fed the engine.
What I’d do differently
Think beyond the Kinect. We built the company on one sensor and one platform holder’s roadmap, and when that roadmap moved, our floor moved with it.
The engine was already sensor-optional, YAKiT proved it. I would have made that the strategy from the start instead of the escape hatch.
