Why Voice Agents Demo Well and Run Badly

A convincing voice demo now takes a weekend. A voice system that survives Monday morning call volume, in three languages, on mobile lines, takes about a year. Almost every voice company being started right now lives in the gap between those two things, and most of them have not found out yet.

Why Everyone Is Building One

The reason is not hype, it is that the three hard parts got bought out from under the problem. Speech recognition, speech synthesis and the language model are all API calls with a free tier. Telephony is an API too. What used to be a speech science team is now four vendor accounts and a weekend, and the result genuinely is impressive: it answers, it sounds human, it books the appointment.

The prize is obvious as well. Voice is where the expensive, repetitive, out-of-hours work still sits: appointment desks, order status, driver check-ins, first-line support, collections, outbound qualification. Any of those at scale is a large payroll line, and unlike a chat widget, voice does not require the customer to change behaviour. So the market is real, the demo is easy, and money is available. That combination produces a lot of companies.

The demo is not lying. It is just being run in conditions that never occur in production: one language, a quiet room, a cooperative speaker with a headset, no interruption, and no consequences for a wrong answer.

Latency Is the Product

In a text interface, a two second wait is fine. In a phone call it is a broken conversation. People start speaking again at around a second of silence, and when the system then talks over them the call degrades in a way that is hard to recover from. This is why voice is unforgiving in a way that chat is not: the quality bar is set by human turn-taking, not by answer accuracy.

The number that matters is time from the caller finishing to the first audio coming back. Here is the budget we work to, with the ranges we actually see rather than vendor best cases.

The tricks for buying time all cost something. Aggressive endpointing shaves 200 milliseconds and starts cutting people off mid-sentence. Filler audio, the small "let me check that", hides a lookup and works until a caller hears it four times. Smaller models respond faster and get more wrong. Streaming synthesis starts talking before the sentence is decided, which is fine until the sentence needs to change. There is no free option, only a set of trades that need to be made deliberately and written down.

Language Is Not a Setting

The demo is in one language, usually English, and vendors list forty. The list is technically true and practically misleading, because the hard part is not supporting a language, it is handling the way people actually speak on a phone.

• Code-switching mid-sentence. A caller in Mumbai or Dubai will start in English, put the important clause in Hindi or Arabic, and return to English for the numbers. Most stacks pick one language per turn, so the important clause is the part they lose.
• Dialect, not language. Gulf and Levantine Arabic are different problems. So are Flemish and Dutch, Mexican and Castilian Spanish, and the English spoken in Chennai versus Manchester. Accuracy quoted at the language level tells you nothing about the dialect your callers actually use.
• Numbers, names and codes. Order references, postcodes and account numbers are where recognition fails most expensively, and they are also the part of the call that has to be exactly right. Register plates, house numbers and lakh or crore quantities all need explicit handling.
• Register. Formal written Arabic or Hindi is not what people speak. A model trained largely on the written form sounds oddly stiff on the phone, which callers read as a machine and treat accordingly.
• Switching cost. Detecting the language, then switching model and voice, adds a turn of latency exactly when the caller is already repeating themselves.

Our own experience of this was in text rather than voice. On a post-merger support desk we automated first-line handling across four languages and cleared about 65 percent of queries without a person, with resolution time falling from 4.2 hours to 38 minutes. The lesson that transfers is the shape of the thing: it worked because the tasks were narrow and verifiable, not because the model was multilingual. Voice adds a real-time constraint on top of that, and does not remove any of the underlying ones.

Two things follow from that table. The first is that the models are not what makes voice expensive. Recognition and synthesis are fractions of a cent a minute, and a small language model is cheaper than either. What costs money is orchestration margin: a platform fee of five to eight cents a minute sits above the whole stack, which is trivial in a pilot and the largest line in the budget at volume. At around 180,000 minutes a month, a five-cent platform fee on its own is roughly $9,000 a month before a single model call. That is the number that decides whether you buy or build, and it is not on the demo slide.

The second is that a slow agent is a more expensive agent. Recognition, model, synthesis and telephony are all billed by the minute or the token, so every second of dead air on a call is paid for twice, once in cost and once in the caller talking over you. Latency and unit economics are the same problem viewed from two directions.

The Indic Stack Is Its Own Market

The code-switching problem has a specific answer now, and it is worth understanding even if you never operate in India, because it shows what purpose-built looks like. Sarvam AI's speech models cover 22 Indian languages plus English and are trained for exactly the failure we described: Hinglish and Tanglish, mid-sentence switching, regional accents, and the audio quality of an actual phone line rather than a studio. They run both as streaming for live calls and as batch for recordings.

The pricing tells you something too. Speech to text is ₹30 an hour of audio, ₹45 with speaker separation, and synthesis is ₹15 per 10,000 characters on Bulbul v2. In dollars that is around 35 cents an hour to transcribe. Their own language models are priced in the same register, at ₹2.50 in and ₹10 out per million tokens for the 30B, with tiered access at ₹10,000 and ₹50,000 a month for higher rate limits. A global English-first stack shoehorned into Hindi or Marathi costs more and performs worse on exactly the calls that matter.

The general lesson is not "use Sarvam". It is that language coverage is a procurement decision with a right answer per market, and the right answer is rarely the vendor with the longest language list. Test on your own audio, in your own languages, and compare the accuracy that matters, which is order numbers and names rather than general transcription.

If your callers speak Hindi, Tamil or Marathi on a mobile line, the model trained on that is not a nice-to-have. It is the difference between an agent that works and one that asks people to repeat themselves.

What Breaks in Month Two

• Barge-in. Callers interrupt. If the agent cannot stop talking, listen, and pick up where the caller took it, everything else is irrelevant.
• Confirmed wrong answers. A voice agent that reads back an incorrect order number confidently is worse than one that fails, because the caller acts on it.
• The handoff. Transfer to a person is where most deployments lose the thread. If the human receives no transcript, no context and no reason for escalation, callers repeat themselves and the saving evaporates.
• No evaluation set. Teams test the happy path by hand. Without a kept set of recorded failures, replayed on every prompt and model change, quality drifts silently.
• Cost per minute at volume. Pilot economics are irrelevant, and the platform fee usually outgrows the model cost long before you notice.
• Consent and recording law. Different in the Netherlands, the Gulf, India and most US states. Worth settling before the pilot, not after legal reads the transcript store.

What We Would Test Before Signing Anything

• Fifty recorded calls from your own queue, not the vendor's samples, with your accents and your background noise.
• Time to first audio measured on a mobile line, at the ninety-fifth percentile, not the average.
• A deliberate interruption, twice, in the middle of the agent's sentence.
• Ten order or account numbers read aloud quickly by a real customer.
• One call that switches language mid-sentence.
• One call the agent should refuse, and evidence that it does.
• A transfer to a human, checked from the human's side.
• The bill for a thousand minutes of that behaviour, not a thousand minutes of the demo.
• The same test on a model trained for your callers’ languages, priced side by side with the global default.

Where It Genuinely Works Today

Narrow, high-volume, verifiable tasks with a clean fallback. Order and delivery status. Appointment booking and rescheduling. Driver and contractor check-ins. Out-of-hours triage that routes rather than resolves. Outbound reminders. In each case the task is short, the correct answer is checkable against a system of record, and the cost of handing to a person is low.

What does not work yet, whatever the demo suggests, is open-ended conversation with commercial consequences, in mixed languages, without a human anywhere near it. That is not a model problem and it will not be fixed by the next release. It is a design and operations problem, which is the reason so many voice companies started this year will be sold for their engineering team rather than their product.

If you are evaluating one: ask for the ninety-fifth percentile latency on your own audio, and ask what happens when the caller interrupts. The answers to those two questions sort the field quickly.