An Echo-class assistant that keeps the microphone secure and intelligent
Set out to give a self-hosted and secure AI a physical voice, from the wake word to the spoken reply, with the recorded audio never leaving the premises. Firmware, wire protocol, gateway, and narration model are original; the result is a reliable voice assistant that knows me.
- Role
- Stack
- Years
- Status
Aelira, the fully capable assistant
Built in Windows Linux Subsystem running as a server, Aelira is a Hermes Agent that can run commands with a full PC at her fingertips. Web search, calendaring, managing spreadsheets, everything you would expect from a modern agentic assistant. The Memory OS bolt-on for Hermes agent means she manages a 7-layer memory system about me and my projects that never sees the outside of my home server.
Wrote the smart-speaker firmware from scratch
C firmware for an ESP32-S3 on ESP-IDF: a custom-trained wake-word model on a microWakeWord-compatible engine, 20-millisecond audio frames streamed over a custom WebSocket protocol, and signed over-the-air updates with bootloader rollback.
A real-time voice gateway that knows who's talking
A 27-module Python gateway transcribes speech on a local GPU, matches the voice against enrolled voiceprints, streams the reply back, and stops mid-sentence when you say stop.
A fine-tuned 7B model narrates what the assistant is doing
Fine-tuned Qwen2.5-7B on a single RTX 4090 to narrate the assistant's work while it happens, then served it quantized on a consumer GPU: 78 milliseconds median time to first token, 47 to 59 tokens a second.
Three devices, one voice, nearest speaker wins
Three speakers elect one device per wake inside a 300-millisecond arbitration window, with a deterministic tie-break and an exclusive turn lease so only one of them answers.
Built by a governed fleet of AI agents
Directed AI coding agents through 53 numbered work packages, 26 architecture decision records, and 848 automated tests, with human approval gates at every decision that mattered.
The result: a voice assistant whose microphone stays home
Wake word, transcription, and speaker identification all run in the house. Whisper, Silero, ElevenLabs, and llama.cpp do the jobs they are good at; everything between them is original.