The agent took the caller’s “Tuesday”, said “of course” when the caller interrupted mid-request with “actually Wednesday”, fired off a second booking — and both requests came back fine. Two bookings. One cheerful confirmation — and a textbook failure of voice AI interruption handling. A practitioner describing the case on r/AI_Agents put it like this: “If your tests are a happy caller reading a script, the demo passes and the receptionist double-books.”
That is the case practitioners in one practitioner thread keep describing as the real production gap: the demos are conversations, the failures are transactions.
The demo is a script; the phone call isn’t
Voice AI demos are built to show how natural the conversation sounds, and the state of the art there is genuinely impressive. The problem, as the thread’s author u/djgoel123 frames it, starts the moment a caller behaves like an actual caller: changing their mind halfway through a sentence, speaking over background noise, giving an unclear date, asking to speak to a real person while a slow backend takes seconds to respond. The realistic voice is the easy part. The workflow is where the engineering lives.
Even the vendors say so. ElevenLabs published a technical assessment warning that its own native barge-in handling — the mechanism that lets a caller interrupt an agent mid-sentence — is insufficient for production call-centre deployments without additional engineering work, as reported by TechDefused. The assessment reportedly puts the cost of broken turn-taking at an estimated $62 billion a year for US businesses.
Voice AI interruption handling: a third of a second to save the call
Natural human conversation moves fast. Turn gaps typically run under 300 milliseconds, which means a voice AI system has roughly a third of a second to detect an interruption, stop generating audio, and start processing the new input before the exchange starts to feel broken. Under telephony conditions — network latency, compressed audio, background noise, accents — that budget gets even tighter.
Interruption handling is not one component. It is a chain, and every link has to land: voice activity detection to notice someone is speaking, streaming speech recognition that produces stable results fast enough to act on, and text-to-speech cancellation that halts playback without an audible glitch. Telnyx’s evaluation guide lays out production targets: ASR endpointing at 200–350ms, LLM time-to-first-token at 300–400ms, TTS warm-up at 50–100ms, barge-in response under 500ms, total time-to-first-audio under 600ms. Miss one link and the illusion cracks.
Kore.ai’s breakdown of the failure modes shows what cracking looks like. Get it wrong in one direction and the agent talks over the caller — “which reads as rude and erodes trust within two or three interruptions.” Get it wrong in the other direction and the agent barrels ahead on outdated information, “forcing the caller to repeat a correction they already made.” The two failures share a documented thread — the system doesn’t know what state the conversation is actually in — though the sources describe the failures rather than proving a single cause for every interruption failure.
Barge-in is a state problem wearing an audio costume
That phrase comes from engineer Dmitrijs Rutko’s build log of a voice agent, and it names the hard half of interruption handling. The audio half — stop the playback fast — is obvious. The hard half is the conversation history. The language model writes roughly 15 to 45 times faster than the voice speaks, so by the time a caller interrupts, the entire reply usually exists already and sits in the history. Leave it there and the agent reasons about sentences nobody ever heard, refers back to explanations the user never received, and answers follow-ups as if it had.
Tool calls make it worse. If a booking request is in flight when the interruption lands, its result arrives late into a changed world. Rutko’s prescription: cancel the call, or tag the result to the interrupted turn and discard it, “so stale data never enters the context.” The r/AI_Agents commenters independently converged on the same discipline: when testing, interrupt before the request is dispatched, mid-request, and after commit — separately — and if the backend outcome is unknown, check the booking record before retrying. Or as appointment-AI platform BotWorks puts it: check a timed-out update’s status before retrying “so one request does not create duplicate changes.”
Latency contributes to interruption failures rather than standing apart from them. One practitioner in the thread says latency was the hardest problem to solve, because every extra second makes callers talk over the agent, which then breaks interruption handling downstream. The reported fix is architectural: keep the telephony and the model close together — one commenter’s company ran models on the same infrastructure as their LiveKit servers — so round trips shrink. Filler lines like “one sec, checking that” cover the wait for the caller, but they don’t resolve the booking state. Only the state machinery does that.
What production-readiness actually looks like
For the booking scenarios in this thread, the working checklist practitioners describe is almost embarrassingly traditional. Confirm before committing: read the date back, wait for a yes, then call the API — the same confirmation step a human receptionist takes. Allow barge-in, stream the audio so playback can be cancelled mid-sentence, and tune the endpointing threshold to the sweet spot: CallSphere’s guide puts it at 80–150ms with semantic voice activity detection, or 150–250ms energy-based — too short and breaths become interruptions, too long and the agent feels sluggish.
The practitioner’s numbers set expectations. One veteran telephone-systems commenter, posting from decades of experience, says an in-house voice system starts around $10,000, a large cloud deployment runs about $300 a month with roughly one-second response gaps, and he has never got cloud recognition latency under 400ms. These are self-reported, pseudonymous figures — take them as orientation, not a benchmark. But the direction is right: voice AI has a cost structure and a tuning craft that belongs to telecom engineering as much as to machine learning.
One caveat before the close
All of the above runs on a narrow evidence base: the booking case comes from a single discussion thread of mostly pseudonymous practitioners, the vendor isn’t named, and the anecdotes haven’t been independently verified. What makes it worth covering is not the sample size but the engineering pattern — test interruptions at every stage, confirm before committing, check the request state before retrying — and the fact that ElevenLabs’ own assessment independently names interruption handling as the production problem.
Judge the receptionist, not the voice
“If your system can’t handle those behaviors, it dies in production,” Telnyx’s guide warns. The lesson the thread’s practitioners keep circling is this: in these reported booking cases, what failed wasn’t the voice — it was the transaction underneath. The evaluation that matters is not whether the agent sounds human but what it does when the human does human things: interrupts, changes its mind, goes quiet, asks for a person mid-transaction.
The double-booking anecdote is funny until it is your Tuesday. The testing advice practitioners recommend is to script interruptions into test calls at every stage — before dispatch, mid-request, after commit. The point of such a test, as one commenter put it in substance: track which appointment the caller ends up with, not how quickly the audio stops. For booking agents, at least, that’s close to a production standard: which appointment the caller ends up with matters more than how quickly the audio stops. In a booking call, the voice was never the product. The booking was.
