Capacity Engineering for Real-Time Speech Recognition

In real-time speech recognition, a fast model is not sufficient by itself. User-visible latency is the sum of audio acquisition, speech endpointing, preprocessing, queueing, inference, result assembly and persistence. Capacity engineering asks how much of that pipeline can be sustained under load without losing predictable latency.

This article complements my Whisper architecture note. The subject here is not model internals but the relationship among RTF, segmentation, queues, backpressure, CPU/GPU sharing and end-to-end latency. The examples are generalized and do not expose institutional capacity, topology or operational data.

Decompose Latency Before Optimizing It

A single end-to-end number is useful for an SLA but insufficient for diagnosis. A more useful decomposition is:

L_total =
    L_capture
  + L_endpoint
  + L_preprocess
  + L_queue
  + L_inference
  + L_postprocess
  + L_persist

L_endpoint includes intentional waiting before the system decides that speech has ended. Saving 200 ms in inference will not produce the expected user benefit when endpointing contributes another 800 ms.

L_queue can also grow much faster than component execution time as saturation approaches. Inference measured on an idle system and end-to-end latency under sustained load describe different conditions.

What RTF Actually Measures

Real-Time Factor compares processing duration with media duration:

RTF = processing time / audio duration

If 30 seconds of audio is processed in 3 seconds, the RTF is about 0.1. That shows the individual workload was processed faster than media time.

It does not prove that the machine can safely sustain ten concurrent streams. Under concurrency, CPU preprocessing, GPU memory, batching, queues, I/O and model workers share bounded resources.

Single-job RTF is an important input to a capacity model, not the capacity model itself.

Throughput and Latency Are Different Objectives

The amount of audio processed per unit of wall-clock time is a throughput question. The time one segment waits for a result is a latency question.

A larger batch can improve GPU utilization and total throughput, while increasing the waiting time of the first item in that batch.

That trade-off can be acceptable for offline transcription and unacceptable for interactive speech processing. "Fastest" is therefore undefined until the target is stated:

• minimum single-segment latency,
• maximum sustainable throughput,
• a percentile latency objective,
• maximum concurrent work on fixed hardware.

Segment Length Is a Systems Parameter

Segment size affects quality and capacity at the same time.

Very short segments can cause:

• more model invocations,
• more scheduling/queue overhead,
• less linguistic context,
• boundary errors around words.

Very long segments can cause:

• longer time before a result is available,
• larger temporary memory use,
• less flexible batching,
• larger retries when a job fails.

Segmentation therefore should not be tuned only against an ASR quality score. Model behavior, endpointing, worker capacity and user latency are one optimization problem.

How VAD and Endpointing Affect Capacity

Voice Activity Detection can avoid spending model compute on long non-speech regions, but aggressive cuts can damage transcript quality.

A longer end-of-speech hangover can protect word endings while increasing L_endpoint. Merging short gaps can reduce model-call count but produce longer segments.

Useful VAD evaluation for a production ASR pipeline therefore includes:

• segment-duration distribution,
• segments per hour,
• endpoint latency,
• ASR quality,
• model invocation count.

My VAD library article focuses on the algorithmic layer; this article focuses on its capacity consequences.

Queues, Little's Law and Hidden Waiting Time

For a stable system, Little's Law relates:

L = λW

where L is the average number of items in the system, λ is arrival rate and W is average time in the system.

It is not a complete performance model, but it is a useful consistency check. If arrival rate rises while service capacity is unchanged, queue depth and waiting time should be expected to grow.

The critical condition is sustained overload:

long-term arrival rate > long-term service capacity

No queue size solves that condition. A larger queue only delays visible failure.

Queueing Theory, Backpressure and Tail Latency are therefore fundamental ASR-service concepts even though none is a speech model.

Why Backpressure Matters

An unbounded queue can make a system appear lossless for a while. In reality:

• memory use increases,
• queued jobs age,
• tail latency grows,
• upstream timeouts can trigger retries.

A capacity problem can then become a retry storm.

A safer design uses bounded queues and defines overload behavior explicitly: block the producer, reject new work, or shed selected lower-priority load according to the system contract.

Backpressure does not create capacity. It makes the capacity boundary explicit.

The Real CPU/GPU Work Split

Even when GPU inference is very fast, the entire ASR pipeline does not run on the GPU.

CPU work commonly includes:

• decode,
• resampling,
• channel handling,
• VAD,
• segment creation,
• result merging,
• serialization.

If GPU inference is accelerated while CPU preprocessing is unchanged, the GPU can eventually wait for input. The opposite configuration can leave the GPU queue saturated.

CPU and GPU utilization should therefore be interpreted together with the queue and timing between those stages.

CTranslate2, Threads and Batching

Inference runtimes such as CTranslate2 expose controls for worker/thread count, batching and computation type. Their useful settings depend on the hardware and workload.

On CPU, increasing inter_threads × intra_threads without regard to physical cores can create contention. On GPU, a larger batch can improve throughput while violating a latency target.

Similarly, int8, float16 or bfloat16 should not be selected from model size alone. Hardware support, memory footprint, numerical behavior and total pipeline performance should be measured together.

How Many Workers?

Setting worker count to the largest value that fits in memory is not a reliable capacity strategy.

Each worker can consume:

• model memory,
• workspace,
• input/output buffers,
• threads,
• queue capacity.

More workers initially increase parallelism; later they can add memory pressure, scheduling overhead or GPU contention.

I determine the useful range experimentally while keeping other variables fixed and observing:

• throughput,
• p50/p95/p99 latency,
• queue depth,
• CPU/GPU utilization,
• memory,
• errors/timeouts.

Average Latency Is Not Enough

Real-time experience is often dominated by tail latency rather than the mean.

For example:

p50 = 400 ms
p95 = 900 ms
p99 = 4.5 s

can have a reasonable average while one in every hundred jobs remains visibly slow.

Performance reports should therefore include percentile distributions and sample counts rather than only average and maximum.

Deterministic Media Time

Repeatedly converting media positions into floating-point seconds at every layer can accumulate small rounding differences over long recordings.

Where practical, I keep sample position or another integer timebase as the primary reference and convert to milliseconds/seconds only for presentation.

VAD segments, ASR segments and transcript timestamps can use different time semantics. Those conversion points should be explicit, particularly with stereo input or resampling.

Failures and Retry Belong to Capacity Engineering

Requeueing inference after a temporary failure appears simple. Retrying a timeout caused by saturation, however, can increase the load that caused the timeout.

Retry policy should consider:

• whether the error is transient,
• whether the job is idempotent,
• total time budget,
• queue age,
• remaining user SLA.

This is directly related to Safe Retry Design in Critical Systems.

Measurement Protocol

For a reproducible ASR capacity test, I record at least:

1. hardware and operating system,
2. model and runtime versions,
3. compute_type,
4. worker/thread/batch settings,
5. audio-duration distribution,
6. VAD/segmentation policy,
7. warm-up policy,
8. test duration and sample count,
9. latency percentiles,
10. sustainable throughput and queue behavior.

Without those conditions, an isolated number such as "RTF 0.05" or "20 requests/s" is difficult to transfer to another system.

Conclusion

Real-time ASR capacity is not a model benchmark. Model execution, segmentation, VAD, queues, CPU/GPU sharing, batching, worker count and failure policy belong to the same system.

The optimization sequence I use remains simple:

measure
→ prove the bottleneck
→ change one variable
→ measure again
→ validate under load

A faster model is useful only when the improvement appears in L_total. Otherwise the optimization may have moved the bottleneck to another layer.