Real-Time ASR Latency Budget: VAD, Segmentation, Queueing and Inference

Processing ten seconds of audio in one second does not mean that a user receives the transcript one second after speaking. In real-time speech recognition, latency begins before model inference and ends after the result is delivered. Real-Time Factor is therefore an important capacity metric, but it is not a substitute for end-to-end latency.

A decomposed latency budget

A practical model is:

T_e2e = T_capture + T_vad + T_segment + T_queue + T_inference + T_decode + T_post + T_delivery

The start and end of each interval must be defined explicitly. Otherwise, two services may report "latency" while measuring different portions of the pipeline.

Capture and VAD delay

Frame-based audio capture creates an immediate lower bound. A pipeline operating on 20 or 30 ms frames cannot make a decision before the required frame arrives. VAD smoothing, hangover and minimum-speech constraints may reduce false transitions, but they also delay endpoint decisions.

The interval between actual speech end and the system deciding that the segment is complete is directly visible to the user. VAD classification quality and endpoint latency are therefore related but distinct optimization targets.

Segmentation and context

Very short segments increase invocation frequency, can reduce linguistic context and add orchestration overhead. Very long segments delay the first result and increase memory and queue work per item. In encoder-decoder systems such as the Whisper architecture, segmentation policy needs to be considered together with model behavior.

Percentile segment duration and speech/silence transition distributions are usually more informative than mean segment length alone.

Queueing and the capacity boundary

Even fast inference cannot help a segment that waits for a busy GPU or CPU worker. Queueing delay can become the dominant user-visible component as arrival rate approaches capacity. Little's Law provides a consistency relationship among work in the system, throughput and total time.

Batching has two effects. Larger batches may raise GPU throughput while increasing the time the earliest item waits for a batch to fill. Offline transcription and interactive ASR therefore need not use the same batching policy.

Inference, decoding and RTF

RTF is commonly expressed as:

RTF = processing_time / audio_duration

RTF below 1 indicates that one worker can process faster than real time, but concurrency, warm-up, memory transfer, beam search and decoding still affect capacity. Long-segment tail behavior should be measured in addition to average RTF.

Post-processing and delivery

Timestamp alignment, diarization, punctuation, text normalization, database writes and HTTP/WebSocket delivery all happen outside core model inference. Individually small steps can become significant when serialized. A synchronous database operation on the critical path can make user latency unchanged even after the model itself is accelerated.

Measurement design

For each segment, useful timestamps include capture start, speech start, speech end, endpoint decision, queue enter/leave, inference start/end, decode completion and result publication. Joining them under one correlation identifier exposes not only total latency but the stage responsible for P99.

Capacity engineering for real-time speech recognition answers how much work the system can sustain; a latency budget answers how long that work takes to become useful. Production systems need both measurements and should optimize them together.