Aria: Real-Time Voice AI Assistant
Real-time conversations that survive interruptions and provider outages.
The problem
Natural phone-style conversation with an AI needs sub-second latency. Most stacks feel like walkie-talkies.
What I built
Low-latency voice loop on Pipecat + WebRTC with Groq Whisper for speech recognition and LLaMA 3.1 for reasoning. Intelligent interruption handling, dynamic knowledge-base lookup, domain-tuned recognition and failover when providers degrade.
The hard part
A voice conversation is only as good as its slowest moment. Transcription, reasoning and speech each add delay, and callers notice any pause that sounds like lag rather than thinking. On top of that, people interrupt, speak over the assistant and use words general speech recognition gets wrong, while every external provider will eventually be slow or unavailable in the middle of a call.
Decisions that made it work
- Treat latency as a budget: every stage streams into the next, so the caller hears the first words of a reply while the rest is still being generated.
- Keep listening while speaking, stop instantly on a real interruption, and trim the conversation history to what the caller actually heard.
- Answer from a knowledge base instead of the model's memory, with short, specific facts that suit spoken answers.
- Give speech recognition the domain vocabulary that matters, such as product names and reference codes.
- Treat every provider as replaceable, with timeouts set by the latency budget and a fallback for each stage.
How a run goes
- 1Caller speaks
- 2Whisper transcribes
- 3LLaMA 3.1 decides
- 4ElevenLabs replies
Where this fits
Voice agents fit conversations people already expect to have by phone and that follow a predictable pattern: booking and rescheduling appointments, answering routine questions, qualifying inbound calls and routing callers to the right person. They are a poor fit where a mistake is expensive and hard to correct by voice, or where the caller needs to see something. The right first project is a frequent, well-understood call that is currently handled by someone reading from a script.
Built with Python, Pipecat, WebRTC, Groq Whisper, LLaMA 3.1, ElevenLabs.
Aria, inbound call
Aria
Voice assistant over WebRTC
Response time
-
Connecting call
Hearing
Groq Whisper
Reasoning
LLaMA 3.1
Speaking
ElevenLabs