INT vs. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats

Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats to handle the pervasive activation outliers in Large Language Models (LLMs). Despite this industry trend, a unified comparison of FP and integer (INT) quantization across varying granularities has been missing, leaving algorithm and hardware co-design without clear guidance. This paper fills that gap by systematically investigating the trade-offs between FP and INT formats. We reveal a critical performance crossover: while FP excels in coarse-grained quantization, INT consistently surpasses it as the quantization block size shrinks. Our comprehensive comparison demonstrates that for popular fine-grained formats like MX (block size 32), MXINT8 and MXINT4 are superior to their FP counterparts in both algorithmic accuracy and hardware efficiency. We also introduce a symmetric clipping method that resolves gradient bias in fine-grained low-bit INT training, enabling nearly lossless performance for MXINT8 training. These findings challenge the current hardware trajectory and advocate for prioritizing fine-grained INT formats in future AI accelerators to achieve a better balance of accuracy, power, and efficiency. Code is available at \url{https://github.com/ChenMnZ/INTvsFP}.

Modern AI systems require massive computing power, leading the industry to favor floating-point (FP) number formats to compress data and handle extreme numerical values called outliers. However, this study reveals that this industry-wide shift ignores how data behaves when compressed into microscopic blocks. By systematically comparing FP and traditional integer (INT) formats, the researchers found that a one-size-fits-all FP approach is actually suboptimal. For 8-bit data grouped in small blocks, integer formats are significantly more accurate and hardware-efficient than their FP counterparts. While FP formats generally perform better for ultra-compressed 4-bit data, integer formats can still surpass them if specific mathematical techniques are used to smooth out the extreme values. Furthermore, by fixing a mathematical bias during training, 8-bit integer formats can train AI models without any loss in performance. Ultimately, this research challenges current hardware design trends, demonstrating that prioritizing fine-grained integer formats can lead to much more energy-efficient and powerful AI computer chips.