An Echo-class assistant that keeps the microphone at home
Set out to give a self-hosted home AI a physical voice, from the wake word to the spoken reply, with the household's audio never leaving the house. Firmware, wire protocol, gateway, and narration model are original; the parts that should be off-the-shelf, are.
- Role
- Stack
- Years
- Status
Wrote the smart-speaker firmware from scratch
C firmware for an ESP32-S3 on ESP-IDF: a custom-trained wake-word model on a microWakeWord-compatible engine, 20-millisecond audio frames streamed over a custom WebSocket protocol, and signed over-the-air updates with bootloader rollback.
A real-time voice gateway that knows who's talking
A 27-module Python gateway transcribes speech on a local GPU, matches the voice against enrolled voiceprints, streams the reply back, and stops mid-sentence when you say stop.
A fine-tuned 7B model narrates what the assistant is doing
Fine-tuned Qwen2.5-7B on a single RTX 4090 to narrate the assistant's work while it happens, then served it quantized on a consumer GPU: 78 milliseconds median time to first token, 47 to 59 tokens a second.
Three devices, one voice, nearest speaker wins
Three speakers elect one device per wake inside a 300-millisecond arbitration window, with a deterministic tie-break and an exclusive turn lease so only one of them answers.
Built by a governed fleet of AI agents
Directed AI coding agents through 53 numbered work packages, 26 architecture decision records, and 848 automated tests, with human approval gates at every decision that mattered.
The result: a voice assistant whose microphone stays home
Wake word, transcription, and speaker identification all run in the house. Whisper, Silero, ElevenLabs, and llama.cpp do the jobs they are good at; everything between them is original.