pith. sign in

arxiv: 2409.16626 · v2 · pith:3WA5SV3Hnew · submitted 2024-09-25 · 💻 cs.LG · cs.AI· cs.AR

Ascend HiFloat8 Format for Deep Learning

Yuanyong Luo , Zhongxing Zhang , Richard Wu , Hu Liu , Ying Jin , Kai Zheng , Minmin Wang , Zhanying He
show 11 more authors
Guipeng Hu Luyao Chen Tianchi Hu Junsong Wang Minqi Chen Mikhaylov Dmitry Korviakov Vladimir Bobrin Maxim Yuhao Hu Guanfu Chen Zeyi Huang
This is my paper
classification 💻 cs.LG cs.AIcs.AR
keywords hif8valuesexponentmantissabinadesdeepdynamicencoding
0
0 comments X
read the original abstract

This preliminary white paper proposes a novel 8-bit floating-point data format HiFloat8 (abbreviated as HiF8) for deep learning. HiF8 features tapered precision. For normal value encoding, it provides 7 exponent values with 3-bit mantissa, 8 exponent values with 2-bit mantissa, and 16 exponent values with 1-bit mantissa. For denormal value encoding, it extends the dynamic range by 7 extra powers of 2, from 31 to 38 binades (notice that FP16 covers 40 binades). Meanwhile, HiF8 encodes all the special values except that positive zero and negative zero are represented by only one bit-pattern. Thanks to the better balance between precision and dynamic range, HiF8 can be simultaneously used in both forward and backward passes of AI training. In this paper, we will describe the definition and rounding methods of HiF8, as well as the tentative training and inference solutions. To demonstrate the efficacy of HiF8, massive simulation results on various neural networks, including traditional neural networks and large language models (LLMs), will also be presented.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Boundary-Protection W8A8 HiFloat8 Quantization for Large-Scale Text-to-Video Diffusion Transformers

    cs.CV 2026-05 unverdicted novelty 5.0

    A boundary-protection PTQ strategy for Wan2.1-T2V-14B matches BF16 VBench performance by retaining boundary blocks in higher precision and quantizing the rest to W8A8 HiF8.

  2. Analysis of Floating-Point Matrix Multiplication Computed via Integer Arithmetic

    math.NA 2025-06 unverdicted novelty 5.0

    Error analysis and cost estimator for recasting floating-point matrix multiplication as accumulated integer products on mixed-precision hardware.

  3. GoldenFloat: A Phi-Derived Static-Split Floating-Point Family from GF4 to GF1024 with a Lucas-Exact Integer Identity

    cs.AR 2026-06 unverdicted novelty 4.0

    GoldenFloat introduces a phi-derived rule for setting exponent and fraction widths across floating-point formats from 4 to 1024 bits, backed by open RTL generator, Lucas-exact accumulator, and FPGA implementation.

  4. OSP-Next: Efficient High-Quality Video Generation with Sparse Sequence Parallelism, HiF8 Quantization, and Reinforcement Learning

    cs.CV 2026-05 unverdicted novelty 4.0

    OSP-Next reports 83.73% VBench score and up to 2.27x speedup via hybrid sparse attention, SSP parallelism, HiF8 quantization, and Mix-GRPO on diffusion transformers.