← Back to work
Applied LLM & RAG

Aria Voice AI

An AI phone receptionist that answers real calls, holds a natural real-time voice conversation, and responds from a knowledge base via RAG.

View code
4
live voice APIs orchestrated
3-stage
STT → LLM → TTS, streamed
RAG
grounded in a knowledge base
FastAPITwilioDeepgramElevenLabsClaudeSupabaseWebSockets

The problem

Small businesses miss calls and can't staff a 24/7 receptionist. A useful phone agent has to listen, understand, answer from the business's own knowledge, and speak back — all fast enough to feel like a real conversation.

The approach

Twilio opens a bidirectional media stream to a FastAPI backend. Deepgram transcribes the caller in real time, Claude generates a response grounded in documents retrieved from a Supabase knowledge base (RAG), and ElevenLabs streams natural speech back — all inside a live streaming loop. A companion Next.js dashboard manages the knowledge base and reviews call transcripts.

Architecture

Caller ─(phone)─▶ Twilio ─▶ FastAPI webhook ─▶ Media Stream (WebSocket)
                                    │
            ┌───────────────────────┼───────────────────────┐
            ▼                       ▼                       ▼
     Deepgram (STT)        Claude + RAG              ElevenLabs (TTS)
   speech → text        retrieve + reason           text → voice
                                    │
                                    ▼
                    Supabase (Postgres) — knowledge, transcripts, call state

Key decisions & trade-offs

  • ▸Streaming media over WebSockets, not request/response — latency is the whole product for voice.
  • ▸RAG grounding so answers come from the business's documents, not model guesswork.
  • ▸Modular services (STT / LLM / TTS / RAG) so any vendor can be swapped — e.g. to Amazon Transcribe + Polly + Bedrock.

Results / impact

  • ✓Holds a natural, real-time voice conversation over a live phone call.
  • ✓Answers are grounded in the business's own documents via RAG — no invented policies.
  • ✓Vendor-agnostic services mean the STT/TTS/LLM stack can be swapped without touching the core loop.

Roadmap

  • ▸Ship a rate-limited, spend-capped browser demo so anyone can talk to Aria live (no phone number needed).
  • ▸Swap the speech stack to AWS-native Transcribe + Polly for a fully cloud-native version.
← All projects