LLM Inference on FPGA with High-Level Synthesis

FPGA LLM inference requires algorithm/hardware co-design across quantization, memory bandwidth, pipelining, on-chip buffers and KV cache.

Moving a Transformer to an FPGA is not equivalent to translating a CUDA kernel into HDL. Model representation, memory traffic and pipeline structure all affect the synthesized hardware.

The bottleneck is not always MAC count

Transformer layers contain large matrix multiplications, but inference can be memory-bandwidth bound. If weights are fetched repeatedly from external memory, additional multipliers may remain idle.

What HLS contributes

High-Level Synthesis speeds up architectural exploration by translating C/C++-like descriptions into RTL. It does not remove hardware design: loop structure, pipelining, array partitioning and memory interfaces still determine performance and resource use.

Quantization changes the architecture

Lower precision changes DSP use, bandwidth, buffer capacity, accumulator width and achievable parallelism. Quantization is therefore part of hardware architecture, not merely a model-compression switch.

Latency and throughput are different

A deep pipeline can have significant first-result latency while accepting new work at a high rate:

latency ≠ throughput

The correct optimization depends on whether the workload is streaming, batch or interactive.

KV cache is another memory problem

Autoregressive inference keeps previous key/value state. As context grows, KV cache can dominate capacity and traffic. The design must decide what stays on-chip and what moves through external memory.

Deterministic hardware has value

A synthesized pipeline can provide predictable timing and resource ownership, although host scheduling, PCIe and external DRAM still contribute to end-to-end variation.

HLS is a design-space problem

Unroll factor, initiation interval, quantization width and memory partitioning interact. More parallelism can exceed DSP or routing capacity; more buffering can reduce DRAM traffic while consuming BRAM/URAM.

FPGA versus GPU

FPGA is not a universal GPU replacement. It is attractive when customized dataflow, unusual precision, energy constraints or deterministic pipelines justify the engineering cost.

Recent work such as HLSTransform reinforces the same direction: efficient FPGA inference increasingly depends on algorithm/hardware co-design.

References

HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis
HLSTransform source code
ELiTeFormer: An Efficient Transformer for FPGAs