Inference Engine
A runtime that executes trained model graphs on target hardware while managing kernels, graph optimization, memory planning, batching and device placement.
Model Format vs Runtime
A model file can describe graph structure and parameters; an inference engine decides how that graph executes on a CPU, GPU or other accelerator. A model exchange format such as ONNX is therefore not itself an execution engine.
Runtime Optimizations
An engine may perform operator fusion, constant folding, kernel selection, tensor-memory planning, quantized execution, batching and device placement. These choices can make the same model exhibit different latency, throughput and memory consumption across runtimes.
Numerical behavior can also vary slightly with precision and kernel implementations. Loading the same model format does not guarantee bit-identical output.
Production Perspective
An engine benchmark is not end-to-end system latency. Preprocessing, data transfer, queueing and post-processing need separate measurement.