# LLM Inference on FPGA with High-Level Synthesis

> FPGA LLM inference requires algorithm/hardware co-design across quantization, memory bandwidth, pipelining, on-chip buffers and KV cache.

- Author: Muhammet Ali Köker
- Language: en
- Canonical: https://alikoker.com.tr/en/llm-inference-on-fpga-with-hls
- Translation: https://alikoker.com.tr/fpga-uzerinde-llm-cikarimi-ve-hls
- Published: 2024-04-29T12:00:00+03:00
- Modified: 2026-08-30T14:55:00+03:00
- Type: article

Moving a [Transformer](/en/wiki/transformer) to an FPGA is not equivalent to translating a CUDA kernel into HDL. Model representation, memory traffic and pipeline structure all affect the synthesized hardware.

## The bottleneck is not always MAC count

Transformer layers contain large matrix multiplications, but inference can be memory-bandwidth bound. If weights are fetched repeatedly from external memory, additional multipliers may remain idle.

## What HLS contributes

High-Level Synthesis speeds up architectural exploration by translating C/C++-like descriptions into RTL. It does not remove hardware design: loop structure, pipelining, array partitioning and memory interfaces still determine performance and resource use.

## Quantization changes the architecture

Lower precision changes DSP use, bandwidth, buffer capacity, accumulator width and achievable parallelism. Quantization is therefore part of hardware architecture, not merely a model-compression switch.

## Latency and throughput are different

A deep pipeline can have significant first-result latency while accepting new work at a high rate:

```text
latency ≠ throughput
```

The correct optimization depends on whether the workload is streaming, batch or interactive.

## KV cache is another memory problem

Autoregressive inference keeps previous key/value state. As context grows, [KV cache](/en/wiki/kv-cache) can dominate capacity and traffic. The design must decide what stays on-chip and what moves through external memory.

## Deterministic hardware has value

A synthesized pipeline can provide predictable timing and resource ownership, although host scheduling, PCIe and external DRAM still contribute to end-to-end variation.

## HLS is a design-space problem

Unroll factor, initiation interval, quantization width and memory partitioning interact. More parallelism can exceed DSP or routing capacity; more buffering can reduce DRAM traffic while consuming BRAM/URAM.

## FPGA versus GPU

FPGA is not a universal GPU replacement. It is attractive when customized dataflow, unusual precision, energy constraints or deterministic pipelines justify the engineering cost.

Recent work such as HLSTransform reinforces the same direction: efficient FPGA inference increasingly depends on algorithm/hardware co-design.

## References

- *HLSTransform: Energy-Efficient Llama 2 Inference on FPGAs Via High Level Synthesis*
- HLSTransform source code
- *ELiTeFormer: An Efficient Transformer for FPGAs*

## Cite This Work

Köker, M. A. (2024). LLM Inference on FPGA with High-Level Synthesis. alikoker.com.tr. https://alikoker.com.tr/en/llm-inference-on-fpga-with-hls

- BibTeX: https://alikoker.com.tr/en/llm-inference-on-fpga-with-hls.bib
- RIS: https://alikoker.com.tr/en/llm-inference-on-fpga-with-hls.ris
- CSL-JSON: https://alikoker.com.tr/en/llm-inference-on-fpga-with-hls.csl.json