Inference Engineering
The engineering discipline of running trained models reliably in production under latency, throughput, memory, accuracy, cost and hardware constraints.
A strong model metric does not automatically produce a strong production inference system. Inference engineering measures how a trained graph behaves on real hardware together with preprocessing, batching, data movement, runtime choice, quantization, concurrency and post-processing.
The useful distinction is between an Inference Engine and the system built around it. The engine executes operators; the engineering problem owns the end-to-end latency budget, capacity and failure behaviour. The same model can expose very different bottlenecks on CPUs, GPUs and FPGAs.
LLM Inference on FPGA with High-Level Synthesis illustrates the hardware side, while Capacity Engineering for Real-Time Speech Recognition shows the queueing and service-capacity side.