Aria Voice AI
A human-like AI voice assistant you talk to in the browser — real-time speech in and out, answered from a business's knowledge base via RAG. Paste a URL or PDF to talk to your own.
The problem
Businesses miss calls and can't staff a 24/7 receptionist. A useful voice agent has to listen, understand, answer from the business's own knowledge, and speak back — all fast enough to feel like a real conversation. And to prove it, a visitor should be able to just talk to it, with no phone number and no setup.
The approach
The browser streams raw microphone audio (PCM16, 16 kHz) over a WebSocket to a FastAPI backend. Deepgram Nova-3 transcribes in real time, Claude Haiku 4.5 generates a response grounded in documents retrieved from a Supabase pgvector knowledge base (RAG), and ElevenLabs Flash v2.5 streams natural speech back — all inside one live loop, with the mic gated to real playback so Aria never transcribes her own voice. A 'bring your own knowledge' mode ingests any URL or PDF into an ephemeral, auto-deleted knowledge base in seconds, so anyone can talk to their own Aria. The same pipeline also answers real phone calls through Twilio media streams.
Architecture
Browser mic ─(PCM16 16kHz / WebSocket)─▶ FastAPI /demo-stream
│
├───────────────────────┬───────────────────────┐
▼ ▼ ▼
Deepgram Nova-3 (STT) Claude Haiku 4.5 + RAG ElevenLabs Flash v2.5 (TTS)
speech → text retrieve + reason text → streamed voice
│
▼
Supabase (Postgres + pgvector) · OpenAI embeddings
default persona · ephemeral "bring your own" KB (auto-wiped on disconnect)
Same loop over a phone call: Caller ─▶ Twilio Media Streams ─▶ pipelineKey decisions & trade-offs
- ▸Stream raw PCM over a WebSocket, not request/response — for voice, latency is the product; the mic is gated to real playback so Aria never hears herself.
- ▸RAG grounding so answers come from the business's own documents, not model guesswork.
- ▸Ephemeral bring-your-own-knowledge — ingest a URL/PDF into a throwaway KB in seconds, auto-deleted on disconnect, so the live demo works on anyone's own content with nothing persisted.
- ▸Cost-tiered, swappable services (Deepgram · Claude Haiku · ElevenLabs · OpenAI embeddings) behind one loop — any vendor can be replaced without touching the core.
Results / impact
- ✓Holds a natural, real-time spoken conversation in the browser — mic in, voice out — grounded in a knowledge base via RAG.
- ✓Paste any website or PDF and talk to a custom voice agent built from it in seconds, then it's wiped.
- ✓The same streaming pipeline also answers real phone calls through Twilio.
Roadmap
- ▸Harden listening reliability so Aria never misses a phrase (mic-gating vs endpointing).
- ▸Add a branded custom domain and a small cost/usage dashboard for the live demo.