Inference Engineering
The engineering discipline of running trained models reliably in production under latency, throughput, memory, accuracy, cost and hardware constraints.
Inference engineering concerns the execution of trained models under operational requirements for accuracy, service capacity, latency and resource usage. The request path includes preprocessing, movement of input tensors, operator execution, postprocessing and delivery of results. Model execution time alone does not represent the latency of an entire request.
Service time and capacity
An end-to-end budget can be expressed as T = T_pre + T_transfer + T_infer + T_post + T_queue. The terms represent preprocessing, data movement, inference, postprocessing and queueing. Latency percentiles such as p50, p95 and p99 are measured alongside successfully completed requests per second. Sustained arrival rate is compared with service capacity, while short bursts are considered separately.
Batch size, CPU parallelism, GPU memory residency and concurrency affect computational utilization and data movement. Quantization changes memory and arithmetic requirements and is evaluated together with the model-quality metric. Meaningful comparisons retain the same input distribution, model version, warm-up policy and hardware constraints.
An Inference Engine executes model operators. Capacity Engineering for Real-Time Speech Recognition develops the relationship between service rate and bounded queues. LLM Inference on FPGA with High-Level Synthesis covers hardware-oriented implementation.