Audio Feature Vectors and Matching

Audio feature extraction through pre-emphasis, framing, windowing, FFT/STFT, time and spectral features, MFCCs, pitch and formants, and learned audio or speaker embeddings.

Audio features transform raw waveforms into representations that are more stable for matching, classification and recognition. Depending on the task, systems may use time-domain features, spectral representations or learned embeddings.

Preprocessing: Pre-emphasis, Framing, and Windowing

Classical speech-processing pipelines do not usually analyze a long recording as one block. Speech is treated as approximately stationary over short intervals, so the signal is divided into frames that are analyzed separately.

Pre-emphasis was historically used to increase the relative contribution of higher-frequency components. A simple form is:

y[n] = x[n] - α x[n-1]

where α is selected for the application. Pre-emphasis is not mandatory for every modern model; matching the preprocessing used during training is more important, especially for systems that consume raw waveforms or log-Mel features.

Cutting frames with hard boundaries can increase spectral leakage. Hamming and related windows taper frame edges. Window length and frame shift determine the balance between temporal and frequency resolution.

The FFT provides an efficient way to decompose a windowed frame into frequency components. Repeating this calculation over successive frames produces the STFT time-frequency representation. Filter banks, MFCCs, and later cepstral processing can then operate on this short-time spectral structure.

Time and frequency features

Short-time energy and zero-crossing rate are inexpensive time-domain measures. STFT computes frequency components within short windows and produces a time-frequency representation. Window length creates a direct trade-off between temporal and frequency resolution.

MFCC and spectral representations

MFCC extraction pipeline from waveform framing through FFT, Mel filterbank, logarithm, and DCT coefficients — MFCC pipeline

MFCC is derived from log Mel-filterbank energies followed by a cepstral transform. It has long been a standard feature for speech recognition and speaker modelling.

Pitch, formants, spectral slope, jitter and shimmer represent different physical or perceptual properties of the signal.

Learned embeddings

Modern systems can map an audio segment directly to a dense speaker embedding or a more general audio embedding. Matching can use cosine similarity, PLDA or learned scoring models.

Channel, microphone, codec and acoustic environment can change the representation distribution, so normalisation and domain adaptation belong in the feature pipeline.

At scale, vector dimension also determines storage cost, index size and query latency.

Parent topic: Image and Audio Processing

References

Alan V. Oppenheim; Ronald W. Schafer. Discrete-Time Signal Processing, Third Edition. Pearson, 2010.

Lawrence R. Rabiner; Ronald W. Schafer. Theory and Applications of Digital Speech Processing. Pearson, 2011.