Campfire
Voice recognition with IBM Watson, LIFX smart bulbs and speakers layered over a physical children’s book. Screen-free, start to finish.
What it was
A bedtime story that lit the room. A parent read a physical picture book aloud; the app listened, followed along in the text, and cued the room, light colour, sound effects, ambience and score, on the spoken word.
It came out of watching kids put down books to pick up screens. So there is no screen in the experience: the device sat face-down on the nightstand, doing the listening.
How it was built
The iOS app captured the reader’s voice as raw PCM, Audio Queue Services, 22 kHz mono, and streamed it over a WebSocket to IBM Watson Speech-to-Text, running in continuous mode for the whole reading session.
Watson only returned transcripts, the real trick was following along in the story. Every incoming hypothesis was fuzzy-matched against the script using two-level Levenshtein distance: word-sequence edit distance, where each word substitution costs its own character-level edit distance. Add an adaptive confidence threshold and a search biased around where the reader should be, and it survives paraphrasing, skipped words and STT errors.
LIFX smart bulbs driven over the raw LAN protocol, no cloud, no SDK. The UDP binary protocol was hand-rolled in Swift on BSD sockets: broadcast discovery on port 56700, bulbs tracked by MAC, colour cues sent as 16-bit HSBK packets with fade durations rendered by the bulb firmware. Local network meant the lights hit on the spoken word, not a cloud round-trip later.
Sound effects and looping musical score beds played as layered channels via AVAudioPlayer, all synced to the same story timeline.
The authoring tool
Stories were authored in a web tool, an AngularJS app on Node, Express and MongoDB, deployed on Heroku with assets on S3.
Authors laid out trigger lines, light tracks and sound channels on a timeline, then published to a marketplace the iOS app pulled from. Scoring a book was the slow, skilled work; the tool existed to make that work possible for someone who was not an engineer.
Seven-year-olds scored and published their own story experiences.
A few dozen experiences, authored by children, light, sound effects, ambience and music laid underneath their own words, triggered on their own sentences. The tool was simple enough that the kids who were meant to be the audience became the makers instead.
What broke
Recognition was a main driver of the product, and recognition is fragile. Thick accents, a speech impediment, or a noisy room and we could not clear a confident enough threshold to fire a trigger. On a weak connection you dropped to manual mode, tapping cues instead of speaking them.
The rule I work by now: when the magic depends on a recognition threshold, the manual path ships as a first-class mode on day one, not as a fallback to apologise for.
Three things happened that we did not design for.
After two or three reads, the effects went into the background, and the kids started inventing.
First pass they were in awe: the lights changing, the realism of the sounds. Then the experience receded and the story came forward. That is when they got imaginative, vocalising what else might be happening on the page, off the edges of the book.
Kids with ADHD and kids on the spectrum went still and stayed with the story.
Not just quiet, but enraptured. The room did the work of holding attention that a parent otherwise has to do alone.
It made some parents cry the first time they saw it.
That is the outcome I would put above every number on this page.
Families
Stories scored by kids
Hardware partners
Children’s Research Hospital
What I’d do differently
Fewer retail conversations, more grassroots activation, and hire college students to pump out experiences instead of gating the library behind our own authoring time.
And if done today, I would rebuild trigger design with AI doing the heavy lifting. The slow part was scoring a book cue by cue; that work is now largely machine-assistable, which changes the economics of the whole library.
