Automatic Speech Recognition

Automatic speech recognition converts an acoustic signal into words, subwords or characters. The problem combines acoustic representation, temporal alignment, language information and search.

Recognition by Task Structure

Speech-recognition systems have historically also been classified by the expected structure of the utterance. Modern models do not always preserve these categories as hard architectural boundaries, but they remain useful for understanding task difficulty.

Isolated-word recognition: the user is expected to speak individual words or short commands. Word boundaries are relatively clear and the vocabulary can be kept small.

Word spotting: the objective is to locate selected words or phrases inside a continuous audio stream. The system does not need to transcribe every surrounding word if detecting the target event is sufficient.

Continuous-speech recognition: natural speech is converted into a sequence of words or subword units. Word boundaries are not explicit, and speaking rate, coarticulation, pauses, and linguistic context all affect recognition.

Modern end-to-end models can address these tasks with related architectures, but segmentation, latency, search space, and error cost still depend on the application.

Template Matching and Dynamic Time Warping

Early small-vocabulary speech-recognition systems often compared a test utterance with stored word templates. When the same word is spoken at different speeds, the time axes of the two feature sequences do not align directly.

Dynamic Time Warping (DTW) searches for a low-cost nonlinear alignment between two sequences. It can therefore compare utterances in which some regions are spoken faster or slower than their counterparts.

DTW is not a substitute for a probabilistic language model or a large-vocabulary continuous-speech decoder. It remains historically important for template matching, small command sets, and time-series similarity. HMM-based systems later provided a broader statistical framework by modeling state transitions and observation probabilities over time.

Classical architectures

Traditional systems combine MFCC-like features, acoustic models, HMM states, pronunciation lexicons and n-gram language models. WFST decoding can integrate these components into a single search graph.

n-gram language models

An n-gram language model approximates sequence probability by conditioning on only the previous n-1 units. A bigram uses one preceding word; a trigram uses the previous two. Smoothing, backoff, or interpolation prevents unseen sequences from receiving an unusable zero probability.

Classical decoding combines acoustic and language-model scores. Linguistic context can disambiguate acoustically similar hypotheses, while an overly strong language model can also favor a plausible but acoustically incorrect sequence. Acoustic/language-model weighting and word-insertion penalties are therefore tuned on validation data.

The probabilistic basis of n-grams and their relation to HMMs is developed in Statistical Learning and Machine Learning.

End-to-end models

In automatic speech recognition, CTC reduces the need for frame-level alignment. RNN-T is well suited to streaming recognition. Transformer and Conformer architectures model long-range context effectively.

End-to-end modelling does not remove the need for VAD, resampling, channel handling and segmentation. These stages determine the structure of the model input.

Conditions That Affect Recognition

The same model does not produce the same error distribution under every recording condition. Speaker variability, speaking rate, accent, microphone and codec characteristics, reverberation, background noise, channel bandwidth, and target vocabulary can all affect recognition.

A system trained on short controlled commands should not be expected to transfer unchanged to spontaneous speech; a model developed on clean close-talk recordings can behave differently on telephony audio; and a model trained in quiet conditions can degrade in a noisy environment.

Error analysis therefore should not end with aggregate WER. Recording source, noise level, speaker group, word type, proper names, and numbers can be evaluated as separate slices.

Pre-emphasis, framing, windowing, and spectral features are discussed in Audio Feature Vectors and Matching, while denoising and filter behavior are covered in Audio Restoration and Enhancement.

Domain adaptation

Technical terms, names and organisation-specific vocabulary may be weakly represented by general language models. Contextual biasing, rescoring and domain adaptation can improve recognition of such terms.

Word Error Rate is the standard aggregate metric. Numbers, names and critical terms can also be evaluated separately.

For real-time systems, WER should be considered together with real-time factor, segment latency, timestamp accuracy and result stability.

Parent topic: Image and Audio Processing

References

• Alex Graves; Santiago Fernández; Faustino Gomez; Jürgen Schmidhuber. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks. Proceedings of ICML 2006, 2006.

• Lawrence R. Rabiner; Ronald W. Schafer. Theory and Applications of Digital Speech Processing. Pearson, 2011.