Oralion
Why does noise reduction sometimes make speech recognition worse? Speech enhancement, seen from the ASR side
In speech recognition, "denoise first, then feed it to the ASR" is close to an article of faith. The reasoning is intuitive enough: less background noise makes speech easier for a person to make out, so a recognition model ought to have an easier time too.
In practice it does not always work that way.
A growing body of research, and a great deal of hands-on testing, has found that for modern large speech-recognition models some denoisers do not reduce the error rate at all — they raise Word Error Rate (WER) or Character Error Rate (CER). The audio sounds cleaner and the transcript gets worse.
The cause is not that noise reduction is immature. It is that Speech Enhancement (SE) and Automatic Speech Recognition (ASR) have been solving different problems from the start.
1. Enhancement and ASR are optimising for different things
Most denoisers are designed to improve what a person hears, so they are trained against perceptual quality metrics — PESQ, STOI, DNSMOS. ASR is not that. What it ultimately cares about is whether the content can be recovered, which makes WER, CER, and the survival of recognisable acoustic features the things that matter.
The two are looking at different properties:
Speech Enhancement cares about:
- Whether it sounds natural and comfortable to a listener.
- Whether the background is quiet enough.
- Whether PESQ, STOI and DNSMOS scores go up.
- It will usually still score well even if some speech detail is sacrificed, as long as the overall impression improves.
Automatic Speech Recognition cares about:
- Whether the words come out right.
- Whether the spectrum still carries enough phonetic information.
- Whether WER and CER go down.
- It generally tolerates background noise reasonably well, but is very sensitive to artifacts.
This is the part that is easy to miss: ASR is not after the best-sounding audio. It is after the most recognisable audio.
Background noise can certainly hurt recognition, but as long as the key features of the speech survive, a modern ASR usually copes. Conversely, if the denoising process alters the speech itself, accuracy can fall even though the background got quieter.
2. The real problem is not noise. It is distribution shift
Modern large ASR models rarely analyse the raw waveform directly. They convert audio into spectral features such as a log-Mel spectrogram first, and a neural network works from there.
Those models were trained on large amounts of real recordings, so what they learned is the distribution formed by real recording environments — not an idealised, noise-free one.
If a front-end denoiser changes the speech spectrum, then even where a listener notices nothing, the input the ASR receives may have drifted away from the distribution it knows (out-of-distribution).
This is distribution shift.
Many neural denoisers reshape the speech spectrum in the course of removing background noise. They may:
- adjust high-frequency energy
- smooth away spectral detail
- shift some formants
- use generative methods to fill in signal that was never there
To the ear these changes usually sound entirely natural. To an ASR they can destroy exactly the features it relies on to tell one phoneme from another.
So what actually moves the error rate is often not how much background noise is left, but whether denoising changed the speech features the model already knew.
3. Over-denoising takes the speech with it
In practice the most common failure is that high-frequency information is attenuated along with the noise.
To get a cleaner background, many denoisers suppress high-frequency energy fairly aggressively. But the high end is also where fricatives and affricates carry most of their information:
- Mandarin "s", "sh", "ch", "zh"
- English /s/, /f/, /θ/
Spectrally, these sounds already resemble wind and airflow noise to some degree. A denoiser that cannot tell them apart will remove real speech along with the noise.
A listener can still infer the word from context and so barely notices. An ASR, deprived of those high-frequency boundaries, confuses similar phonemes more often — and that shows up as a rise in CER or WER.
Which is why testing so often produces the same curious shape:
Audio quality keeps improving, and past a certain point recognition accuracy starts to fall.
4. Modern ASR already treats noise as part of its training data
Recent large speech-recognition models were, for the most part, trained on enormous quantities of real-world recordings: different devices, different room noise, reverberation, overlapping speakers, microphones of every quality. In the course of that training the model learned to extract stable speech features with background noise present.
Put differently, background noise is usually part of the training data rather than an anomaly that has to be eliminated.
Front-end denoising therefore does not necessarily add anything. If the denoiser changes the speech spectrum — by
- redistributing energy across frequencies
- weakening high-frequency information
- introducing new artifacts
- altering the spectral structure of the speech
— the input features can drift away from the distribution the model was trained on. A cleaner background does not guarantee better accuracy, and can cost you some.
So when you evaluate denoising, do not stop at whether the audio got cleaner. Measure what the ASR actually does with it.
5. Choose a denoiser on ASR performance, not audio scores
In a speech-recognition system, no single denoiser suits every situation.
Different models are built on different premises:
- some chase maximum background suppression
- some deliberately preserve more of the original speech detail
- some are meant for voice calls
- some for restoring recordings
- some for real-time streaming
Judging whether a denoiser suits ASR therefore means looking past PESQ and DNSMOS to its effect on recognition, including:
- WER
- CER
- latency
- CPU / GPU usage
- stability over long streaming sessions
And different ASR models can react differently to the same front end. A denoiser that improves accuracy for one is not guaranteed to do so after you switch recognisers.
There is no "best denoiser" — only the denoiser best suited to a particular ASR system.
In closing
The point of speech enhancement should not merely be cleaner audio. It should be more reliable recognition downstream.
For a modern large ASR, background noise is not necessarily the enemy. What is worth avoiding is the change denoising makes to the speech features themselves. Once the input drifts away from the distribution the model knows, accuracy can fall even as the audio becomes more pleasant to listen to.
So when designing a speech-recognition system, treat denoising as part of the ASR pipeline to be evaluated, not as a fixed pre-processing step. It earns its place only when real testing shows it lowering WER or CER — not when it merely sounds better.
For ASR, what matters has never been the quietest audio. It is the audio that keeps the most usable speech information.
