Building a voice AI assistant that doesn't feel like a walkie-talkie

Most voice assistants fail in the first three seconds. You say something, there is a pause long enough to wonder whether it heard you, and then it answers a question you have already moved past. People call these systems "walkie-talkies": you talk, you wait, it talks, you wait. Nobody would put up with that from a human on the phone, and they do not put up with it from software either.

Aria is the real-time voice assistant I built to get past that. It runs on Pipecat and WebRTC, uses Groq Whisper for speech recognition, LLaMA 3.1 for reasoning and ElevenLabs for the voice. This post walks through how a voice loop like this fits together, where the time actually goes, and the three things that made the biggest difference: treating latency as a budget, handling interruptions properly, and planning for providers to fail.

What "real time" means in a voice product

In a text chat, a two second response feels fine. In a voice conversation the same two seconds feels broken. Human turn-taking is fast: in ordinary conversation the gap between one person finishing and the other starting is usually a fraction of a second. Anything much longer reads as hesitation, confusion or a dropped call.

So the target for a voice assistant is not "fast" in the abstract. It is "fast enough that the pause sounds like thinking, not like lag". That target is reached by adding up a chain of small delays, which is why the most useful mental model is a latency budget.

The shape of the loop

Every turn in a voice conversation passes through the same stages:

1. Capture. Audio comes in from the caller's microphone over WebRTC in small frames.
2. Detect the end of speech. Voice activity detection decides when the caller has actually finished, rather than just paused for breath.
3. Transcribe. Whisper turns the finished utterance into text.
4. Reason. The language model reads the conversation so far, plus any relevant knowledge, and decides what to say.
5. Speak. Text-to-speech turns the reply into audio.
6. Play back. Audio streams back to the caller.

Pipecat is a framework built around exactly this pipeline. Each stage is a processor, frames flow between them, and the framework handles the plumbing: moving audio frames, passing transcripts along, and cancelling work when something changes. That matters because the hard part of a voice agent is not any single model call. It is coordinating six stages that each want to take their time.

Latency is a budget, not a number

The single most useful habit is to stop asking "is it fast?" and start asking "where does each millisecond go?". When you write the stages down with a rough cost next to each, a few things become obvious.

End-of-speech detection is a trade-off, not a setting. If you wait too long after the caller stops talking, every turn gets slower. If you cut in too early, you interrupt people mid-thought. There is no universally right value. It depends on who is calling and what they are doing. A caller reading out a reference number pauses between digits; a caller answering "yes" does not.

Transcription speed depends on where it runs. Groq's hosted Whisper is fast enough that transcription stops being the bottleneck for short utterances. That frees budget for the part that actually needs it.

The model's first token matters more than its last. A reply does not need to be finished before you start speaking it. If the language model streams tokens, the text-to-speech stage can start on the first sentence while the rest is still being generated. The caller hears the beginning of the answer while the end is still being written. This is the single biggest perceived-latency win in the whole loop.

Speech synthesis should stream too. The same logic applies to the voice. Streaming audio back in chunks means the caller hears the first word before the last word exists.

When every stage streams into the next, the loop stops being a sequence of waits and becomes a pipeline where the stages overlap. The response time a caller notices is the time to the first sound, not the time to the complete answer.

Interruptions are the real test

Streaming solves most of the latency problem. It also creates a new one. If Aria is already speaking a long answer and the caller says "actually, no", what happens?

A naive system keeps talking. It finishes its sentence, maybe its whole paragraph, and only then processes the interruption. This is the most obvious sign that you are talking to a machine, and it is the thing people find most irritating.

Handling interruptions properly means several things happening at once:

• Keep listening while speaking. The input side never switches off. Voice activity detection keeps running while the assistant talks.
• Tell a real interruption from noise. A cough, background chatter or the assistant's own voice echoing back should not stop it. A clear "wait" or "no, Friday" should.
• Stop immediately. When the caller barges in, the playback stops, queued audio is thrown away and any in-flight generation is cancelled. Pipecat's frame model helps here, because an interruption can propagate through the pipeline and cancel downstream work.
• Remember what was actually said. This is the subtle one. If the assistant was halfway through a sentence when it was cut off, the conversation history should reflect what the caller actually heard, not the full reply the model generated. Otherwise the model believes it said something the caller never heard, and the next turn goes wrong.

The replay on the home page of this site shows this moment: the assistant offers two appointment times, the caller interrupts with a different day, and the assistant stops mid-sentence and books the new day instead. Getting that single exchange to feel natural took more tuning than any other part of the system.

Knowledge: answering from facts, not from memory

A language model on its own will happily answer questions it knows nothing about. For a voice assistant representing a business, that is dangerous. The caller hears a confident voice and assumes it is right.

Aria answers from a knowledge base. When a question comes in, the relevant facts are looked up and given to the model alongside the conversation, and the model is instructed to answer from them. This is the same idea as retrieval-augmented generation (RAG) in text systems, with one extra constraint: the lookup has to be fast, because it sits inside the latency budget. A retrieval step that adds a second to every turn undoes all the streaming work.

Two practical points made the knowledge integration work better:

• Keep facts short and specific. Voice answers are short. Feeding the model long documents encourages long answers, and long answers are worse on the phone.
• Say "I don't know" gracefully. When the knowledge base does not cover a question, the right behaviour is to say so and offer a next step, not to guess.

Domain-specific speech recognition

General speech recognition is very good at general speech. It is weaker at the words that matter most in a specific business: product names, place names, reference codes and industry jargon. A misheard product name sends the whole turn in the wrong direction, because the model reasons about the wrong thing.

The fix is to tell the recognition step what vocabulary to expect. Giving Whisper context about the domain improves how it transcribes the terms that matter. It is a small change with a large effect on how often the assistant understands the caller the first time.

Planning for providers to fail

Aria depends on several external services: one for transcription, one for the language model and one for the voice. Each of them will, at some point, be slow or unavailable. A voice call cannot show an error message and ask the caller to refresh the page. It has to keep going.

Resilience came from treating each provider as replaceable:

• Detect degradation, not just failure. A provider that answers in five seconds instead of half a second is, for a phone call, effectively down. Timeouts need to be set with the latency budget in mind.
• Have a fallback for each stage. If the primary provider for a stage fails or slows beyond its budget, the call switches to an alternative for that stage and carries on.
• Fail softly. If everything goes wrong, the assistant should say something sensible, such as offering to call back, rather than going silent.

The outcome I cared about was simple: a conversation that survives both interruptions and provider outages. Callers should never be able to tell that a service behind the scenes had a bad minute.

What I would tell someone building their first voice agent

If you are starting a voice project, these are the lessons I would pass on:

• Measure time to first audio, not total response time. It is the number your callers actually feel.
• Stream everything. Transcription results, model tokens and synthesized audio should all flow into the next stage as soon as they exist.
• Design interruption handling on day one. Retrofitting it is painful, because it touches every stage of the pipeline.
• Keep the conversation history honest. Record what the caller actually heard, not what the model intended to say.
• Treat every external service as something that will fail. Budgets, timeouts and fallbacks are features, not extras.
• Test with real calls. Background noise, accents, speakerphones and people who talk over the assistant will teach you more than any benchmark.

Where voice AI is useful today

Voice agents are not the right answer for everything. They shine where people already expect to talk: booking and rescheduling appointments, answering routine questions, qualifying inbound calls, taking simple orders and routing callers to the right person. In those cases a well-built assistant answers instantly at any hour, never puts anyone on hold and hands over to a human when the conversation needs one.

They are a poor fit where a mistake is expensive and hard to correct over the phone, or where the caller needs to see something. The honest version of a voice project starts by picking a conversation that is frequent, well understood and currently handled by people reading from a script.

One turn, step by step

To make this concrete, here is what happens inside a single exchange, from the caller's first word to the assistant's reply.

The caller says, "Hi, I need to move my appointment to Thursday." As they speak, audio frames arrive over WebRTC and flow into voice activity detection. Nothing else happens yet, because the caller has not finished.

The caller stops talking. Voice activity detection waits a short moment to make sure this is the end of the sentence and not a pause, then marks the turn as complete. The buffered audio goes to Whisper, which returns the text almost immediately.

The transcript is added to the conversation history, relevant facts are looked up (in this case, available appointment slots), and the model starts generating. Its first sentence, "Sure.", is ready before the rest of the reply exists. That first sentence goes straight to text-to-speech, and the caller hears "Sure" while the model is still working out which slots to offer.

Halfway through the list of times, the caller says, "Actually, can we do Friday instead?" Voice activity detection, still running, flags real speech. Playback stops, queued audio is discarded, the in-flight generation is cancelled, and the history is trimmed to the words the caller actually heard. The new utterance goes through the same path: detection, transcription, lookup, generation and speech. The assistant confirms Friday.

From the caller's side, none of this machinery is visible. They had a short, normal phone conversation. That is the whole goal.

Summary

Aria is built from well-known parts: Pipecat, WebRTC, Whisper, LLaMA and ElevenLabs. What makes it feel like a conversation rather than a walkie-talkie is the engineering around those parts: a latency budget where every stage streams into the next, interruption handling that stops instantly and keeps the history honest, a knowledge base that keeps answers factual, vocabulary hints for the words that matter, and fallbacks for every provider.

If you are thinking about a voice agent for your own business, or want to talk through how any of this works, ask Alfred on the home page or send me a message.