Real-Time ASR Latency Budget: VAD, Queueing and RTF

A measurement model for user-visible real-time ASR latency split across capture, VAD, segmentation, queueing, inference, decoding and post-processing.

Processing ten seconds of audio in one second does not mean that a user receives the transcript one second after speaking. In real-time speech recognition, latency begins before model inference and ends after the result is delivered. Real-Time Factor is therefore an important capacity metric, but it is not a substitute for end-to-end latency.

A decomposed latency budget

A practical model is:

T_e2e = T_capture + T_vad + T_segment + T_queue + T_inference + T_decode + T_post + T_delivery

The start and end of each interval must be defined explicitly. Otherwise, two services may report "latency" while measuring different portions of the pipeline.

Capture and VAD delay

Frame-based audio capture creates an immediate lower bound. A pipeline operating on 20 or 30 ms frames cannot make a decision before the required frame arrives. VAD smoothing, hangover and minimum-speech constraints may reduce false transitions, but they also delay endpoint decisions.

The interval between actual speech end and the system deciding that the segment is complete is directly visible to the user. VAD classification quality and endpoint latency are therefore related but distinct optimization targets.

Segmentation and context

Very short segments increase invocation frequency, can reduce linguistic context and add orchestration overhead. Very long segments delay the first result and increase memory and queue work per item. In encoder-decoder systems such as the Whisper architecture, segmentation policy needs to be considered together with model behavior.

Percentile segment duration and speech/silence transition distributions are usually more informative than mean segment length alone.

Queueing and the capacity boundary

Even fast inference cannot help a segment that waits for a busy GPU or CPU worker. Queueing delay can become the dominant user-visible component as arrival rate approaches capacity. Little's Law provides a consistency relationship among work in the system, throughput and total time.

Batching has two effects. Larger batches may raise GPU throughput while increasing the time the earliest item waits for a batch to fill. Offline transcription and interactive ASR therefore need not use the same batching policy.

Inference, decoding and RTF

RTF is commonly expressed as:

RTF = processing_time / audio_duration

RTF below 1 indicates that one worker can process faster than real time, but concurrency, warm-up, memory transfer, beam search and decoding still affect capacity. Long-segment tail behavior should be measured in addition to average RTF.

Post-processing and delivery

Timestamp alignment, diarization, punctuation, text normalization, database writes and HTTP/WebSocket delivery all happen outside core model inference. Individually small steps can become significant when serialized. A synchronous database operation on the critical path can make user latency unchanged even after the model itself is accelerated.

Measurement design

For each segment, useful timestamps include capture start, speech start, speech end, endpoint decision, queue enter/leave, inference start/end, decode completion and result publication. Joining them under one correlation identifier exposes not only total latency but the stage responsible for P99.

Capacity engineering for real-time speech recognition answers how much work the system can sustain; a latency budget answers how long that work takes to become useful. Production systems need both measurements and should optimize them together.

Adding the P99 values of individual stages does not produce a statistically valid end-to-end P99. Stage durations may be dependent within a request: queue congestion, inference time, and delivery behavior can vary together. Record each stage boundary under a common request identity and compute total latency directly from each request's first and last timestamps. Stage-level percentiles are diagnostic signals for locating the tail, not components that can simply be summed to estimate it.

Numerical Example of the Measurement Boundary

For illustration only, consider a sequential path taking 80 ms from speech end to segment closure, 120 ms from closure to inference start and 30 ms from inference start to result publication. Latency within this boundary is 230 ms. Measurement from speech onset also includes speech duration. These values are not measurements from a deployed system.

When stages overlap, total latency comes from the start-to-finish interval of the same request or the critical path of its dependency graph. Audio-sample time and processor-clock time are distinct axes. Resampling and transport buffers require an explicit mapping between them; otherwise transcript timestamps can drift even when inference is fast.

Real-Time ASR Latency Budget: VAD, Segmentation, Queueing and Inference