VAD, Segmentation and Endpointing in Real-Time ASR

How voice activity detection, minimum speech duration, silence gaps, and segment caps jointly shape endpoint latency and inference efficiency in real-time ASR.

Model RTF alone does not explain user-visible latency in real-time speech recognition. The point at which audio is released to the recognizer—segmentation and endpointing policy—is often just as important as inference speed.

A segment roughly follows this lifecycle:

speech begins
→ VAD confirms activity
→ the segment grows
→ sufficient silence appears or a cap is reached
→ inference starts
→ the result is published

Every threshold changes a different part of that path.

Shorter segments are not always faster

Small segments can reduce endpoint delay, but each inference call has fixed work: feature preparation, tensor construction, kernel launch, decoder startup, and result handling. As segment count rises, that fixed cost is paid more often.

Very small segments can also break linguistic context. Minimum segment duration and minimum end-to-end latency are therefore not the same objective.

The cost of long segments

Longer segments amortize fixed inference overhead across more audio, but they keep already-spoken material waiting for closure. Long uninterrupted speech can therefore create unacceptable delay when endpointing relies only on silence. A hard segment cap is useful alongside a target duration.

A hard cap is not merely an error condition; it is a latency contract. Even if speech continues, the system eventually releases work to the recognizer.

VAD creates two kinds of load error

A false negative can miss speech or delay segment opening. A false positive converts noise into speech and creates unnecessary segments and inference calls. The VAD threshold is therefore a capacity parameter as well as an accuracy parameter.

Silence-gap policy has the same dual effect. A gap that is too short can interpret natural pauses as endpoints; one that is too long adds avoidable waiting after every utterance.

Minimum speech and minimum model duration

Rejecting extremely short activity can cheaply remove clicks, impacts, breaths, and false VAD triggers. That minimum-speech rule is different from the minimum duration at which the recognizer is efficient. The first filters whether activity is worth processing; the second concerns inference amortization.

What to measure

A single WER or average RTF cannot tune this system. Useful distributions include:

speech-start to segment-open delay,
speech-end to segment-close delay,
segment-duration distribution,
segments per hour,
model calls per hour,
inference p50/p95/p99,
real-time factor,
VAD false starts and missed speech,
percentage of segments closed by the hard cap.

Together they expose the latency/throughput trade-off.

Tune for the operating regime

There is no universal VAD or silence-gap value. Telephone audio, near-field microphones, distant microphones, noisy field recordings, and multi-speaker material have different SNR and pause distributions. Parameters should be validated on the actual operating corpus.