A dictation app transcribes a podcast perfectly and mangles your name. It is doing exactly what it was trained to do

Dictate a paragraph of news copy and a modern transcription system gets essentially every word. Dictate your own name, your colleague's name and a street address in the same voice, in the same room, and it produces something confidently wrong. The usual explanation offered is that the system "isn't good with accents", which is partly true and mostly beside the point. The failure is structural, and it follows from what the machine is actually doing when it listens.

There are no spaces in speech

Start with the fact that surprises people most. Written language has gaps between words. Spoken language does not.

A recording is a continuous pressure wave. The system converts it into a spectrogram — a picture of how much energy sits at each frequency at each moment — which is what the model actually consumes. Look at that picture for a fluent sentence and you will not find silence at the word boundaries. You will find silence in the middle of words, wherever a consonant closes the vocal tract, and none at all between words that run together.

Worse, individual speech sounds are not fixed shapes. Every sound is smeared by the sounds around it, because the tongue is already moving towards the next position while it finishes the current one. The vowel in your name looks physically different depending on which consonant follows it. There is no clean alphabet in the signal to read off.

So word boundaries are not detected. They are inferred — and inference requires a prior about what words are likely, which is where everything interesting happens.

What the system is really made of

Classical recognisers made the division explicit. An acoustic model mapped chunks of spectrogram to speech sounds. A pronunciation dictionary mapped words to sequences of those sounds. A language model scored how likely any given word sequence was in ordinary text. A decoder searched for the sentence that best satisfied all three at once.

Modern systems mostly collapse this into one neural network trained end to end on pairs of audio and transcript, learning the whole mapping without an explicit dictionary. That change removed a lot of hand-built machinery and improved accuracy substantially, but it did not remove the underlying arrangement: the model still has to resolve an ambiguous acoustic signal using its sense of what sentences look like. It just learned both halves together instead of being told.

The other change that matters is how these models are trained. Self-supervised pretraining lets a model learn the structure of speech from enormous quantities of unlabelled audio — no transcripts needed — by predicting masked portions of its own input. Only afterwards is it fine-tuned on a comparatively small set of transcribed recordings. This is the single most important development for languages that do not have thousands of hours of professionally transcribed audio, which is nearly all of them.

The language model is doing more work than you think

Here is the mechanism behind the failure everyone has experienced.

"Recognise speech" and "wreck a nice beach" are almost identical acoustically. Nothing in the sound reliably separates them. What separates them is that one is a far more likely sequence of English words, and the system chooses accordingly — which it must, because otherwise it could not produce any output at all.

Now apply that to a name it has never encountered. An unfamiliar name has almost no probability under the language model, while some ordinary word that sounds roughly similar has a great deal. The system does exactly what it was built to do: it picks the likely sequence. Your name is not misheard so much as overruled, and the output carries no trace of the doubt, because the mechanism that resolved the ambiguity is the same one that resolves every other sentence correctly.

The system is not transcribing what you said. It is producing the most probable thing a person might have said that would sound like this. Those are the same answer most of the time, and the exceptions are precisely the words that carry the information.

Why it fails on you in particular

Several distinct problems get bundled under "it doesn't understand my accent", and they have different fixes.

Distribution. The model learned from the speech it was trained on. Accents, speaking rates and vocal characteristics that were rare in that data are handled worse, and this is not an opinion the model can be argued out of.

Code-switching. A great deal of everyday Indian speech moves between two languages inside a single sentence. Most recognition systems assume one language per utterance and select a model accordingly, so a sentence that switches halfway violates the architecture rather than merely being difficult. Both the acoustic inventory and the language model are wrong for part of the sentence.

Named entities. Names of people, places, medicines and products are exactly the low-frequency items the language model penalises, and they are also exactly the words whose accuracy determines whether the transcript is useful. A transcript of a medical consultation that gets every word right except the drug name is not 99 per cent useful.

Conditions. Distance from the microphone, reverberation in a hard-walled room and background speech all degrade the signal before the model sees it, and people also change how they speak in noise, which shifts them further from the training distribution.

And there is a measurement problem sitting on top of all this. Systems are compared using word error rate, averaged across a corpus. An average hides structure. A system can post an excellent overall figure while failing on the small percentage of words — names, numbers, technical terms — that the listener actually needed. If you are evaluating a transcription system for real use, the number that matters is accuracy on entities, not the average.

What actually fixes it

The most effective production technique is unglamorous and widely used: contextual biasing. At the moment of transcription, hand the system a list of words that are likely in this specific context — the user's contacts, the medicines on the ward's formulary, the product catalogue, the names of people in the meeting — and let it raise their probability during decoding. This does not retrain anything, takes effect immediately, and is why dictating a message to a saved contact works far better than saying the same name in open dictation.

Beyond that: fine-tuning on audio from the actual setting, multilingual and code-switched training data rather than an assumption of one language per sentence, and, unavoidably, collecting representative speech from the people who will use the system. For Indian languages the binding constraint has rarely been the model architecture. It is transcribed audio across enough languages, dialects and recording conditions, which is slow, expensive and not glamorous.

Why it matters for students and researchers

Speech is the interface that matters most for the people currently least served by computing. Typing is a barrier for anyone not comfortable with a keyboard layout, in a script their phone handles awkwardly, or at all. Voice removes that barrier — but only if recognition works on how people actually speak, which in India means many languages, heavy dialect variation, and constant code-switching as the normal case rather than the edge case.

That makes this an unusually good research area to be in from here. The open problems are genuinely open, they are not solved by simply waiting for a larger model trained elsewhere, and the data advantage is local. The most useful contributions are often not new architectures but carefully collected and honestly evaluated corpora — and the discipline of measuring the thing that matters, rather than the average that flatters.

Frequently asked questions

Why does speech recognition get names wrong?

Because the system resolves acoustic ambiguity using a model of which word sequences are likely. An unfamiliar name has very low probability, so a common word that sounds similar wins — the name is overruled rather than misheard.

How does speech recognition actually work?

Audio is converted into a time-frequency representation, and a neural network predicts the most likely text for it. Because the sound signal alone is ambiguous and contains no gaps between words, the system relies on learned expectations about language to decide where words begin and end.

Why is it worse for Indian languages?

Mainly because of transcribed training data rather than any architectural limit. There is far less labelled audio across Indian languages, dialects and recording conditions, and most systems also assume one language per sentence, which breaks on the routine mixing of two.

What is word error rate and why can it mislead?

It is the proportion of words wrongly substituted, inserted or deleted, averaged over a test set. It can look excellent while the system fails on the small share of words — names, numbers, medical terms — that determine whether the transcript is usable.

Can transcription of names be improved without retraining?

Yes. Contextual biasing supplies a list of likely words at the moment of recognition — contacts, product names, a medicine list — and raises their probability during decoding, which is how production systems handle names reliably.