Phone audio is a genuinely harder problem
Speech recognition demos run on clean audio from a good microphone in a quiet room. Phone calls are none of those things. Traditional telephony carries a narrow slice of the frequency range and compresses it aggressively, which strips out exactly the high-frequency detail that distinguishes similar consonants — the difference between "f" and "s", or "b" and "p", often lives in the part of the signal the network threw away. Add packet loss, mobile handoffs, speakerphone echo, and a caller in a car, and the recogniser is working from a degraded copy. This is not a flaw in a particular vendor; it is a property of the medium. It means you should expect a meaningfully higher error rate on calls than in your testing, and design for it rather than treating each mistake as a bug to be fixed individually.
The errors cluster exactly where the value is
Recognition errors are not evenly distributed across a conversation. Ordinary connected speech is handled well, because the model can use context to resolve ambiguity — surrounding words make most guesses recoverable. The errors concentrate on precisely the items that carry no context: proper names, street addresses, email addresses, order numbers, policy numbers, postcodes, and any string of letters and digits read aloud. These are also, inconveniently, the pieces of information most business calls exist to capture. A caller saying "my reference is B as in bravo, four, seven, D" is the hardest input a phone agent will face, and it is a completely routine one. Any evaluation of a voice agent that does not specifically test alphanumeric capture is measuring the easy part.
What actually helps, in order of return
The instinct when recognition is poor is to swap models. That is rarely where the biggest gain is. The highest-return fix is telling the recogniser what to expect: supplying the vocabulary specific to your business — product names, clinic names, local place names, the format your reference numbers take — so that the system weights plausible interpretations correctly. A recogniser that knows your company sells a product called "Aeris" will stop transcribing it as "Eris" or "arrows". After that comes constraining the format at the point of capture, so a postcode is validated as a postcode rather than accepted as free text. Then comes handling the audio path properly — echo cancellation and sensible endpointing. Model choice matters, but it is usually the fourth lever, not the first.
Design for being wrong, because you will be
No configuration makes recognition perfect on a phone line, so the durable fix is conversational rather than technical: confirm anything expensive to get wrong. Read back the address before dispatching an engineer. Confirm the last four digits rather than the whole number. Ask the caller to spell an unusual surname rather than guessing. The craft is in confirming selectively — an agent that repeats every single value back sounds like an interrogation and adds a turn to every exchange, which has its own cost in call length and patience. Confirm what is expensive to get wrong, accept what is cheap to correct later, and let the caller move on. Getting that judgement right is a large part of what separates a voice agent people tolerate from one they do not notice.
Key takeaways
- Phone audio is narrowband and compressed. Recognition on a call is a harder problem than recognition on a laptop microphone, and always will be.
- The errors that hurt are not random — they cluster on names, addresses, alphanumerics, and numbers, which is exactly the data most calls need.
- Prompting the recogniser with the vocabulary you expect fixes more real-world errors than switching models does.
- Confirm high-stakes values back to the caller. Recognition will never be perfect, so design for being wrong occasionally.
