Queue Stability in ASR Systems

The performance of an automatic speech-recognition system is often expressed through the real-time factor:

RTF = processing time / audio duration

If ten seconds of audio are decoded in one second:

RTF = 1 / 10 = 0.1

This shows that the model operates ten times faster than the natural flow of the audio. It does not by itself show that the system is real-time, low-latency, or stable under high traffic.

When an ASR server receives data from many audio sources concurrently, the fundamental problem is not only how quickly the model performs inference. Arrival rate, segment-length distribution, batching policy, sharing of CPU and GPU resources, queue order, and reprocessing after failures must be considered together.

A very low model-level RTF can coexist with queue delays lasting seconds or minutes. RTF measures only service time. It does not measure how long a job waits before service begins.

total latency = segment formation time + queue wait + preprocessing + model inference + postprocessing + publication

Real-time ASR capacity arises not from the speed of these components individually, but from keeping the complete pipeline within its stable operating region.

Separating model speed from system capacity

Let the audio duration of an ASR job be D and model service time be S. RTF is:

RTF = S / D

The amount of audio that one worker can theoretically process per unit time can therefore be considered as:

capacity = 1 / RTF

For example, if:

RTF = 0.2

one second of processing time can decode approximately five seconds of audio. If the system receives a continuous stream of audio with the same characteristics and no other cost exists, one worker can carry approximately five concurrent real-time streams.

This result depends on several strong assumptions:

Every job has the same RTF.

There is no preprocessing or postprocessing cost.

There is no model loading or warm-up delay.

The worker is never idle.

There is no context-switch or data-copy cost between jobs.

Segments are ready when they need to be processed.

The GPU or CPU is not shared with other work.

No memory or I/O bottleneck occurs.

In a real system, service time should be modeled more accurately as:

S = Sdecode + Sresample + Sfeature + Sinference + Spost

Where:

Sdecode is media-decoding time.

Sresample is sampling-rate or channel conversion.

Sfeature is preparation of model inputs such as Mel spectrograms.

Sinference is model inference.

Spost is timestamp, text, and output-formatting cost.

If model RTF is measured only as Sinference / D, actual service RTF is higher:

RTFsystem = S / D

Segment duration and service time are not always linear either. The inference cost of a three-second segment may not be exactly one tenth of that of a thirty-second segment. Fixed setup costs dominate short segments:

S(D) = A + B(D)

A is the fixed cost independent of segment length. B(D) is the processing cost that depends on audio duration.

Observed RTF for short segments can therefore be worse than for long segments:

RTF(D) = A / D + B(D) / D

As D decreases, A / D increases. Very short segments may appear to provide low user latency while reducing total throughput.

The fundamental condition of queue stability

Let the arrival rate of ASR jobs be λ and the average service rate of one worker be μ. Utilization for one worker is:

ρ = λ / μ

The fundamental condition of stability is:

ρ < 1

ρ = 1 means full capacity use, but it is not a safe operating point. When service times and interarrival intervals vary, short bursts create a queue. If average capacity equals demand, no spare capacity remains to absorb these fluctuations.

With multiple workers, approximate utilization is:

ρ = λ / (cμ)

Where c is the number of concurrent workers.

In an ASR system, λ should not be treated only as files per second. A three-second and a thirty-second segment each count as one job but do not produce the same load. A more meaningful arrival load is the amount of audio arriving per unit time:

λaudio = total incoming audio duration / time

If eight seconds of audio arrive per second:

λaudio = 8

This represents load equivalent to eight concurrent real-time streams.

If sustainable audio capacity of one worker is approximately:

μaudio = 1 / RTFsystem

then the stability condition for c workers is:

λaudio × RTFsystem < c

Operating at the exact capacity boundary is not appropriate. For example:

λaudio × RTFsystem = 0.95c

means average utilization is 95 percent. A small increase in RTF, a concentration of long segments, storage delay, or retry traffic can make the queue grow rapidly.

In low-latency critical systems, target load should remain below theoretical maximum capacity. The required safety margin depends on the workload and has no universal percentage. The decisive observation is whether the queue grows under measured p95 and p99 service times.

Why average RTF is misleading

Two systems can have the same average RTF and completely different latency behavior.

Assume the first system has stable service times:

0.09 0.10 0.10 0.11 0.10

The second has:

0.02 0.03 0.04 0.06 0.55

The averages can be similar. In the second system, however, an occasional high service time delays every job behind it. In a single-worker FIFO queue, one long-running job creates head-of-line blocking for short segments that arrive later.

Queue latency is sensitive not only to mean service time but also to its variance. In general, as service-time variance increases, waiting time increases under the same average load.

ASR performance should therefore monitor more than average RTF. At minimum:

RTF p50 RTF p95 RTF p99 maximum RTF segment-duration distribution queue-wait p95 and p99

Grouping RTF by segment duration is also important:

0-3 seconds 3-10 seconds 10-20 seconds 20-30 seconds

Otherwise, good RTF on long segments can conceal high fixed cost on short segments.

The relation known as Little's law connects average queue length and average waiting time:

L = λW

Where:

L is the average number of jobs in the system.

λ is the completed or steady-state arrival rate.

W is the average time a job spends in the system.

For example, if a stable system receiving two segments per second contains an average of ten segments:

W = L / λ = 10 / 2 = 5 seconds

This is not only model-inference time. It is total time in the system, including queue wait. Even with a very low RTF, end-to-end latency remains high when the queue grows.

How segmentation policy changes capacity

If an ASR model processes segments produced by VAD or another method rather than a raw continuous stream, capacity is directly linked to segmentation policy.

Negative effects of very short segments include:

Fixed preparation cost is paid again for every segment.

The number of GPU kernel calls increases.

Queue and task-object counts grow.

The number of file or JSON outputs increases.

ASR context is fragmented.

Short and meaningless sounds can be sent to the model.

Very long segments have different effects:

One job's service time grows.

Head-of-line blocking increases.

The first text reaches the user later.

More audio is reprocessed after a failure.

GPU memory use and intermediate tensor sizes can grow.

One job holds a worker's time slice for a long period.

Target segment duration and hard upper limit should therefore be treated as separate concepts. The target duration is the point at which a segment should close if natural silence is available. The hard limit bounds job size even when speech continues without interruption.

For a system with an approximate target of 25 seconds and a hard limit of 29 seconds, the objective is not to make every segment 29 seconds long. Segments can close earlier at natural speech boundaries, but no job is permitted to grow without bound.

From a queuing perspective, the hard limit indirectly bounds maximum service time:

Smax ≈ S(Dmax)

The relation may not be exact because hardware latency and model behavior can vary. Bounding segment length nevertheless narrows the tail of the service-time distribution and makes p99 latency more predictable.

Minimum segment duration is likewise a capacity property. Rejecting VAD events shorter than one second can prevent thousands of small model calls caused by noise. If the model requires at least three seconds of input, valid short speech can be padded with silence, but padding duration should not be treated as real audio load.

Batching can increase throughput while increasing latency

In GPU inference, processing several segments in one batch can increase total throughput. Jobs must accumulate before the batch can be formed.

Let batch waiting time be Bwait and inference time be Binfer:

Tbatch = Bwait + Binfer

Although this can appear slower than individual inference:

Tsingle = Sinfer

throughput can be higher because several jobs are completed per batch.

The central tradeoff is:

large batch → high throughput small batch → low queue latency

In real-time systems, waiting indefinitely for a fixed batch size is inappropriate. A safer policy combines two limits:

run when the batch is full or run the current jobs when maximum waiting time expires

For example, batch capacity can be eight with a maximum wait of 20 milliseconds. If eight jobs arrive early, inference begins immediately. If fewer arrive, a smaller batch runs after 20 milliseconds.

When segment durations differ within a batch, padding cost can occur. If a three-second and a thirty-second segment are placed in the same batch and the shorter one is padded to the longer shape, computational efficiency decreases:

waste = Σ(batchMaxDuration - segmentDuration)

Limited grouping by duration classes can reduce this waste. Overly strict grouping, however, delays formation of a suitable batch. The design must balance padding waste against batch waiting time.

The effect of batching on CPU inference can differ. Because of memory bandwidth, core sharing, and quantized matrix operations, a larger batch does not always provide lower latency or linearly higher throughput. Actual capacity can be determined only by benchmark.

Routing work across heterogeneous servers

In a system containing ASR servers with different RTF values, distributing an equal number of jobs does not balance load.

If audio-processing capacity of server i is approximately:

Cᵢ = 1 / RTFᵢ

then total capacity is:

Ctotal = ΣCᵢ

The ideal share is:

wᵢ = Cᵢ / Ctotal

If a powerful GPU server is twenty times faster than a CPU server, sending an equal number of segments to both causes the CPU queue to grow continuously while the GPU is sometimes idle.

A fixed weight based only on theoretical RTF is still insufficient. A server's instantaneous state can change:

active job count total queued audio duration recent p95 RTF GPU memory state retry count error rate

Measuring queue length only by job count is misleading for heterogeneous segments. Thirty three-second segments and thirty thirty-second segments have the same queue length but not the same load.

A more meaningful queue metric is:

queuedAudioSeconds = ΣsegmentDuration

Estimated drain time of server i can be calculated as:

drainTimeᵢ ≈ queuedAudioSecondsᵢ × RTFᵢ / workerCountᵢ

A new job can be routed to the server with the lowest estimated completion time:

estimatedFinishᵢ = currentDrainTimeᵢ + newSegmentDuration × RTFᵢ

This remains an approximation. It is nevertheless more meaningful than round-robin or distribution based only on job count.

In systems requiring deterministic behavior, capacity-weighted fixed distribution with a controlled overflow policy can be used instead of fully dynamic routing. A powerful server can carry most normal load, while jobs are redirected to other servers after its queue exceeds a threshold.

Without backpressure, low RTF cannot save the system

If the arrival rate permanently exceeds processing capacity, no queue policy can keep the system stable:

λaudio > Ctotal

The queue then grows continuously:

dQ/dt > 0

Adding faster hardware may solve the problem. If capacity cannot be increased, the system must apply backpressure to the input.

Backpressure can take forms such as:

Temporarily stopping admission of new jobs

Reducing the source consumer's read rate

Blocking the producer when the queue reaches its upper bound

Delaying low-priority work

Putting retries into a separate bounded queue

Separating real-time and historical work

An unbounded queue does not solve overload. It only delays failure. Unlimited audio or task objects retained in memory eventually cause memory pressure, garbage-collection latency, or process failure. An unbounded disk queue increases storage consumption and recovery time.

Queue capacity can be defined in time units:

maximum queued audio duration = 30 minutes

This is more meaningful than a segment-count limit. The action taken at the maximum must be explicit according to system requirements. If dropping critical data is unacceptable, the producer must be stopped or redirected to a persistent spool layer.

Allowing real-time jobs and backfill jobs to share the same FIFO queue is also risky. A large historical workload can delay current segments. Separate queues and weighted scheduling can be used:

priority 1 → current real-time segments priority 2 → retries priority 3 → historical backfill

The low-priority queue must not be starved indefinitely. One backfill job can be taken after a configured number of real-time jobs, or a fixed capacity share can be reserved for backfill.

The performance that should actually be measured

The question "how many times faster is the model?" does not describe operational reality. Metrics should be separated across at least four layers.

Input layer

segments per second total audio duration per second segment-duration p50, p95, p99 active source count

Queue layer

job count in queue total audio duration in queue age of the oldest job wait-time p50, p95, p99

Service layer

preprocessing time inference time postprocessing time end-to-end service time RTF p50, p95, p99

Capacity and reliability layer

worker utilization GPU and CPU utilization memory consumption error rate retry rate timeout count number of dropped or deferred jobs

The slope of the queue over time is particularly valuable:

Q(t₂) - Q(t₁)

If the queue rises during the day and later falls, the system may be absorbing temporary bursts. If it rises continuously, capacity is insufficient. If the oldest-job age increases while job count remains constant, some long or failing jobs may be starved while short jobs progress.

The denominator of RTF must be actual audio duration. If silence padding, batch padding, or synthetic samples introduced by resampling are added to media duration, the result can appear artificially better or worse.

Warm and cold startup should also be measured separately. Model loading, CUDA-context creation, or first-kernel compilation should not be mixed into steady-state service RTF. They should nevertheless be monitored as first-job latency after process restart.

Real-time behavior is a system state, not a model property

A model with RTF < 1 does not by itself create a real-time system. It shows only that the model runs in less than the audio duration under specified test conditions.

A real-time and stable ASR system requires the stronger condition:

long-term input load < long-term sustainable service capacity

The latency objective must also hold:

p99 end-to-end latency < system limit

These conditions are not the same. A system can be stable in capacity terms but exceed its latency objective because of large batches or queue policy. Conversely, a system showing excellent latency under low traffic can become unstable during peak periods.

RTF is valuable for model selection, but incomplete when used alone for capacity planning. The actual engineering problem is controlling temporal behavior across the entire path, from segment generation and server routing to queue bounds and backpressure policy.

The real performance of an ASR system is not demonstrated by its fastest sample, but by its ability to drain the queue under load. A low average RTF can be impressive. A reliable production system is one that remains stable under p99 latency, service-time variability, and increasing demand.