Architectural Comparison of ONNX and CTranslate2 Models Through ArcFace

ArcFace loss and facial embedding inference

ArcFace is not the name of the convolutional backbone that processes facial images, but an additive angular margin loss used during training. Its objective is to bring facial representations of the same person closer in angular space while increasing angular separation among different classes. A fixed margin is added to the angle of the true class over normalized features and class weights.

A simplified ArcFace loss is:

                    exp(s · cos(θyi + m))
Lᵢ = -log ------------------------------------------------
           exp(s · cos(θyi + m)) + Σj≠yi exp(s · cos θj)

Where:

• θyi is the angle between, for example, the true class weight and the feature vector.
• m is the additive angular margin.
• s is the coefficient that scales normalized logits.
• yi is the true identity class, for example.

After training, the classification head is not used in many face-verification systems. The required component during inference is the backbone that transforms an aligned facial image into a fixed-dimensional embedding vector:

x ∈ R^(3×112×112) → f(x) ∈ R^512

The embedding is commonly normalized with the L2 norm:

             f(x)
f̂(x) = -------------
          ||f(x)||₂

Similarity between two faces can be computed as the inner product of normalized embedding vectors:

similarity(a, b) = f̂(a)ᵀ f̂(b)

Therefore, the object intended for conversion is not an “ArcFace layer.” It is an inference graph of an image backbone, such as ResNet, iResNet, MobileFaceNet, or a similar architecture, with weights trained using the ArcFace loss.

ONNX format does not guarantee general conversion

A model being available in ONNX format does not mean that it can be converted to any inference engine. ONNX carries a computation graph and weights in a common representation. The target runtime must additionally be able to parse the graph, support its nodes, and satisfy its execution semantics.

The following distinctions must therefore be maintained:

• Model format is not model architecture.
• Operator support is not full graph support.
• Weight quantization is not model conversion.
• Output dimensionality is not runtime compatibility.

An ArcFace-based InsightFace model may be a valid .onnx file and may produce a 512-dimensional facial representation. Nevertheless, it cannot be directly converted into a CTranslate2 model using the current CTranslate2 conversion tools.

General-purpose ONNX runtimes use a different execution approach:

read the ONNX graph
        ↓
resolve each node's operator and attributes
        ↓
schedule the graph in topological order
        ↓
dispatch to an appropriate kernel or hardware provider
        ↓
produce intermediate tensors

ONNX Runtime can load supported graphs, optimize them, and distribute subgraphs to CPU, CUDA, TensorRT, OpenVINO, or other Execution Provider components. However, the opset version, custom operators, dynamic shapes, and the operator coverage of a hardware provider may still prevent execution.

The architectural boundary of CTranslate2

CTranslate2 is not a general-purpose ONNX graph interpreter. It is a specialized C++ and Python inference library developed for efficient inference of Transformer-based models. Its optimizations, such as weight quantization, layer fusion, and batch reordering, target Transformer execution patterns.

The current converters do not provide a general entry point that accepts arbitrary ONNX graphs. Architecture-aware converters exist for specific model families in the OpenNMT, Fairseq, and Hugging Face Transformers ecosystems. Supported families include selected Transformer models such as BERT, T5, Llama, Whisper, and NLLB; general CNN image models are outside this scope.

CTranslate2 conversion approximately follows this flow:

source framework model
        ↓
identify a known model class
        ↓
match expected parameters by name
        ↓
create a CTranslate2 model specification
        ↓
store weights in the target format

This approach knows the model family and inference flow in advance. No built-in ArcFaceConverter, ResNetConverter, or general ONNXConverter exists for an InsightFace face-recognition graph. It is not sufficient for a converter merely to read weights and map them to similar layers; the target must also provide a model specification and execution class representing the same graph.

Operators, topology, and conversion validation

Evaluating compatibility solely by operator names is misleading. The presence of matrix multiplication, activation, normalization, or certain convolution operations in a runtime does not demonstrate that an arbitrary CNN graph can be executed.

Operator semantics

Operations with the same name may use different data layouts, axes, padding, or broadcasting rules. A convolution node may, for example, include the following parameters:

• kernel_size
• stride
• padding
• dilation
• groups
• input_layout
• weight_layout

The common input layout in image models is [N, C, H, W]. The existence of a Conv operation in a codebase does not mean that all attribute combinations of the ONNX Conv operator are supported.

Graph topology and model specification

InsightFace backbones do not consist only of sequential convolution layers. They include residual connections, normalization layers, activations, and shape transformations:

x ────────────────┐
                  │
Conv → BN → PReLU → Conv → BN
                  │
                  Add → output

Even if a target runtime can execute nodes individually, it must support branching, merging, and tensor lifetime rules. CTranslate2 works with defined structures of supported model types rather than free-form operator graphs.

For a facial embedding backbone, a built-in model class would need to define the following questions:

• What is the input image shape?
• How are residual blocks ordered?
• Where are feature-map dimensions reduced?
• How is the final embedding layer constructed?
• How are batch-normalization parameters applied?
• Will the output be normalized?

Loadability after conversion is also insufficient. Numerical equivalence must be tested:

yONNX = fONNX(x)
ytarget = ftarget(x)

max_abs_error = max |yONNX - ytarget|

Directional drift is additionally important in embedding systems:

cosine_drift =
1 - cosine_similarity(yONNX, ytarget)

If face-verification decisions are made for samples near a cosine-similarity threshold, small numerical differences can alter an accept-or-reject result. A working conversion does not mean that biometric decision behavior has been preserved.

Appropriate inference engine and the embedding contract

For an ArcFace-based ONNX facial embedding model, ONNX Runtime is a natural starting point. Without changing the model format, Execution Provider options such as CPU, CUDA, TensorRT, and OpenVINO can be used under the same API.

TensorRT can also be evaluated on NVIDIA GPUs with fixed input shapes and high batch loads. For Intel CPU, iGPU, or NPU targets, OpenVINO can load the model directly or convert it in advance to its own model format. On ARM-based edge devices, suitable ONNX Runtime CPU providers or mobile-oriented runtimes such as ncnn can be compared. ncnn provides a path for converting ONNX models to its own model format through pnnx.

Engine selection should not be based only on single-image inference time. The following metrics should be measured:

• Single-request latency
• Sustained throughput
• Batch scaling
• p95 and p99 latency
• Model loading time
• Runtime memory consumption
• Host-device copy cost
• Multi-session behavior
• Embedding drift under precision changes

Total latency in a face-recognition pipeline does not consist only of the embedding model:

Ttotal =
Tdecode +
Tdetect +
Talign +
Tcopy +
Tembedding +
Tsearch

An acceleration in embedding inference may have limited effect on the overall system if face detection or vector search is dominant.

Moreover, two models producing 512-element vectors does not show that those vectors occupy the same embedding space:

f₁(x) ∈ R^512
f₂(x) ∈ R^512

These expressions do not imply f₁(x) ≈ f₂(x). The embedding space is formed jointly by training data, backbone, loss function, preprocessing, alignment, color-channel order, and normalization rules.

The following contract must be preserved when changing the runtime:

• Input size
• RGB/BGR order
• Pixel scaling
• Mean and standard deviation
• Face-alignment geometry
• Output tensor name and order
• L2 normalization
• Similarity metric
• Decision threshold

With INT8 quantization, model size or inference time alone should not be assessed. Similarity scores, decision thresholds, and ranking behavior in nearest-neighbor search must also be validated.

Adding CNN facial embedding support to CTranslate2 is theoretically possible at the source-code level; however, it requires a new model specification, a converter, CPU and GPU kernel paths, memory planning, API extensions, and equivalence tests for different backbones. This cost would most likely exceed that of using an engine already optimized for CNNs and ONNX graphs.

While ONNX carries a general computation graph, tensors, and operators, a CTranslate2 model represents a supported model family, a known inference flow, and an optimized weight layout. Unless a general translator exists between these abstraction levels, the .onnx extension does not provide CTranslate2 compatibility.