Queueing Theory
Queueing Theory — The mathematical study of systems where work arrives, waits for limited service capacity and leaves after processing.
Arrival, Service and Waiting
Queueing models separate how work arrives from how quickly servers process it. Even when average service capacity exceeds average load, variability can create waiting and tail latency.
Utilization near 100% is therefore not a safe low-latency operating point.
Model Assumptions Matter
M/M/1, M/G/1 and multi-server models make different assumptions about arrivals, service distributions and server count. A closed-form result is only as useful as those assumptions are appropriate.
For production work, measured distributions should complement analytical models.
I apply these ideas to speech workloads in Queue Stability in ASR Systems.