BF16
A 16-bit floating-point format with an 8-bit exponent that preserves the dynamic range of FP32 while reducing memory and compute cost.
Machine-Learning Context
BF16 keeps the eight-bit exponent width of FP32 while reducing mantissa precision and total storage to 16 bits. Its wide exponent range can reduce overflow problems compared with FP16 during training and inference, and modern accelerators may provide dedicated BF16 matrix hardware.
Measurement Boundary
BF16 is not automatically faster or sufficiently accurate for every workload. Hardware support, accumulation datatype, operator mix, numerical sensitivity, and memory behavior must be measured for the target model.
Related Machine-Learning Concepts
- Quantization
- FP16
- INT8
- Mixed Precision