Why voice agents fail on Hindi-English code-switching

A caller asking to reverse a failed payment will produce something structurally like: Hindi pronoun, English noun phrase, Hindi verb, English number, Hindi auxiliary. There is no pause at the switch. The switch happens mid-phrase, sometimes mid-compound, and the English words carry Indian phonology rather than the pronunciation the acoustic model was trained on.

This matters because most production stacks expose a single language hint per session. Set it to Hindi and English terms are transcribed phonetically into Devanagari, so a product name becomes a nonsense word your intent classifier has never seen. Set it to English and the Hindi is rendered as approximate English syllables, which is worse, because the resulting string looks like valid text and passes silently into the next stage.

The HiACC authors report baselines on their own test set that give a sense of the spread between systems: Whisper medium at 16 per cent word error rate for adults and 18 per cent for children, MMS-1b-all at 31 and 36 per cent, and wav2vec2-XLS-R-300M at 38 and 40 per cent. Two things follow. Model choice does matter here, more than it does in retrieval. And the same corpus is 5.24 hours in total, which tells you how thin publicly available code-switched Indian audio still is.

A caller reading a reference number will say double and triple for repeated digits. An amount may arrive as an English numeral, as Hindi words, as a mixture, or in lakh and crore units that a normaliser trained on Western conventions will convert wrongly or not at all. Dates alternate between formats within the same sentence. None of this is exotic; it is the ordinary content of a support call.

Build a numeric normaliser as an explicit component with its own test set, not as a regular expression at the end of a prompt. It needs Indian units, repetition shorthand, mixed-language digit sequences and currency handling. Then verify the result rather than trusting it: read the number back to the caller and require confirmation before any action that depends on it. Read-back is unfashionable because it costs a turn. It is also the single highest-value reliability feature in an Indian voice deployment, and human agents do it for the same reason.

Named entities have the same shape of problem. Indian personal names, place names and branch names are mangled by recognisers trained predominantly on other data. The fix is contextual biasing: pass the caller's own name, city, recent transaction merchants and relevant product names as bias phrases when the session opens, so the recogniser is weighted towards the small set of strings that are actually plausible for this call. Most commercial ASR services support some form of this, and it is routinely left unused.

• A tested numeric normaliser: lakh and crore, repetition shorthand, mixed-language digits
• Read-back and explicit confirmation before any action keyed on a number
• Contextual biasing from the caller's own record at session start
• A pronunciation lexicon for your domain terms, applied to both recognition and speech output

Overall word error rate averages over filler words, greetings and the parts of the utterance no downstream component reads. It is a reasonable research metric and a poor operational one. What we track instead is entity-level accuracy on the fields that drive actions — numbers, names, product identifiers, dates — and task success end to end, which is the only measure the business recognises.

Build the test set from your own recordings, with consent and a lawful basis, transcribed by people who speak the way your callers speak. Public code-switched corpora are useful for sanity checks and too small to tune against; HiACC, one of the better documented, holds 5.24 hours from 44 speakers. Your own call centre generates more than that in an afternoon, in your domain and your accents.

Stratify deliberately: pure Hindi, pure English, heavy switching, background noise, poor line quality, elderly and young speakers, regional accents. Report per stratum. An average conceals that the system works for the calls you demonstrated and fails for the ones you receive at four in the afternoon from a market. Same discipline as how to evaluate a RAG system, applied to audio.

• Entity error rate on numbers, names and identifiers as the primary metric
• Task success and escalation rate as the business-facing pair
• Stratified reporting by language mix, line quality and speaker group
• Your own consented recordings as the test set, because public corpora are small

Why does the agent work in testing and fail on real calls?

Testing is usually done by the team that built it, speaking clearly, on good lines, in one language at a time, with vocabulary the system already knows. Real calls have background noise, poor connections, intra-sentential switching, regional accents and names the recogniser has never seen. Build the test set from recorded production calls and the gap closes, because you are then measuring the thing you deployed.

Should we use a multilingual model or a Hindi model with English support?

Measure both on your own audio; the answer varies by domain and by line quality more than the marketing suggests. The published HiACC baselines show a wide spread between systems on the same test set, from 16 per cent word error rate up to 38 for adult speech, which is a large enough gap that model selection is worth doing empirically rather than by reputation.

How much code-switched training data do we need?

For fine-tuning a recogniser, more than any public Indian corpus currently offers. For evaluating one, a few hundred well-chosen and carefully transcribed calls will already tell you where you stand. Start with evaluation, because it is cheaper and it tells you whether fine-tuning is the constraint at all.

Does a speech-to-speech model solve code-switching?

It removes some pipeline joins and a great deal of latency, which helps with the interruption problems. It does not remove the need for entity accuracy, and it takes away the intermediate transcript that audit, evaluation and dispute handling all depend on. For regulated deployments that trade-off usually decides the architecture.