Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Pushing the Limits of BFP on Narrow Precision LLM Inference

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A median-pivot format runs LLM Softmax on integer hardware

desk verdict Solid accuracy results and a neat bit-level trick for integer-only softmax, but the 10x FOM gain is an artifact of comparing different input widths. read the letter →

arxiv 2502.00026 v2 pith:TYDN3OCH submitted 2025-01-21 cs.AR cs.AI

Hui Wang , Yuan Cheng , Xiaomeng Han , Zhengpeng Zhao , Dawei Yang , Zhe Jiang This is my paper · ORCID
classification cs.ARcs.AI
keywords blockfloatingpointLLMinferenceSoftmaxaccelerationhardware-softwareco-designlookuptableattentionquantizationFPGA/ASICaccelerator
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that block floating point (BFP), a format in which one exponent is shared across a block of numbers, can be extended from linear operations like matrix multiplication to nonlinear operations like Softmax without losing model accuracy. It introduces DBFP, which aligns each block to the median exponent rather than the maximum and adaptively groups numbers with similar magnitudes so outliers do not distort the whole block. On top of that it builds DH-LUT, a hierarchical lookup table that evaluates the exponential in DBFP using integer arithmetic, and an RTL engine that pipelines max, exponent, shared-exponent subtraction, and division. If the paper is right, attention, the bottleneck of long-context LLM inference, can run on cheap integer-only hardware with accuracy close to floating point.

What carries the argument

The load-bearing object is DBFP, a block floating-point format whose shared exponent is chosen by a pivot-focus policy, meaning median instead of maximum alignment, and whose grouping is adaptive rather than fixed-size. The paper formalizes the grouping as an optimization over membership weights and shared exponents, solved by alternating updates, with an empty-set weight that keeps outliers from dragging the pivot. The companion mechanism is DH-LUT, a non-uniform hierarchical lookup table indexed by shared exponent and high mantissa bits, which evaluates $e^{x_i - x_{\max}}$ in DBFP so that Softmax becomes a ratio of integer table lookups. This machinery does two jobs: it keeps conversion error small enough to preserve model accuracy, and it turns exponent alignment, addition, and division into single-cycle or shift-based integer operations that the hardware pipeline exploits.

What would settle it

Recompute Eq. 11 with every design normalized to the same input count, for example by measuring each accelerator's throughput per input at $N=1024$; if the proposed engine's FOM advantage over ISCAS'23 falls below 10x, the headline comparison is an artifact of workload size rather than architectural speed.

Watch

Extended reading notes

Core claim

The paper claims that BFP failed on nonlinear operations for three reasons: outlier sensitivity, coarse exponent granularity, and hardware complexity, and that all three can be addressed by letting the shared exponent follow the data distribution. In DBFP, each block uses a representative value such as the median as its alignment pivot, and blocks are adaptively partitioned so that elements with similar magnitudes share an exponent; the grouping choice is formalized as minimizing $\sum_{i} \sum_{S_j} \mu_{ij}^{\beta} d_{ij}^2$ with an empty-set term that absorbs outliers. With DBFP, the Softmax exponential can be evaluated by a two-dimensional hierarchical lookup table, and the whole attention path, including matmul, Softmax, and division, can be expressed as integer operations because shared exponents factor out. The paper reports that replacing only the attention layer in LLaMA, ViT, Swin, and DETR keeps zero-shot accuracy and perplexity essentially unchanged, while a CUDA Softmax emulation cuts LLaMA Softmax latency by 74% and the FPGA engine reports a 10x figure-of-merit gain over prior Softmax accelerators.

Load-bearing premise

The load-bearing premise is that the figure of merit $\mathrm{FOM}=F_{\max}NW/(\mathrm{LUT}+\mathrm{FF})$ is a fair comparison across designs with different input counts $N$, because the paper's 10x advantage comes from evaluating its engine at $N=1024$ while the prior accelerators process $N=1$ to $16$.

Editorial extensions

If this is right

  • If DBFP is correct, nonlinear layers no longer force a separate floating-point datapath; the same integer engine used for BFP matmul can perform Softmax, with shared exponents cascading from linear to nonlinear stages without format conversion.
  • Long-context inference, where attention cost grows quadratically with sequence length, would see the largest relative gain because Softmax's exponential and division become lookup and shift operations instead of multi-cycle transcendental computations.
  • The DH-LUT approach is described as general, so GELU, LayerNorm, and other elementwise nonlinearities in Transformers could be accelerated the same way by fitting their value ranges into a DBFP-indexed lookup table.
  • The paper's own results imply that narrow-precision, integer-only LLM inference is a realistic target on FPGA and ASIC, since the attention path is the part that previously forced floating-point hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the 10x figure-of-merit advantage over prior Softmax accelerators is not yet established as a fair architectural comparison, because Eq. 11 counts input count N directly and the proposed design is evaluated at N=1024 while the cited designs process N=1 to 16; normalizing to equal N would likely shrink the gap substantially.
  • My inference: the median-pivot and adaptive-grouping strategies shift the accuracy burden to a per-block preprocessing step that must select the pivot and partition, so the hardware should also account for data-dependent sorting or grouping cost, which the reported pipeline latency may undercount.
  • My inference: the 74% GPU speedup is measured with a custom CUDA operator that emulates DBFP, not with the RTL engine, so the software and hardware numbers are separate claims; a fair comparison would report the RTL engine's latency on the same LLaMA Softmax workload.
  • A testable extension of the same idea would be to apply DBFP to the denominator pass of attention and to GELU in the feed-forward network, then measure whether the integer-only claim extends to a full transformer layer rather than the attention path alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DB-Attn, a hardware-software co-design framework for attention in LLM inference. It proposes DBFP, a block floating-point variant with a median-pivot alignment and an adaptive grouping strategy; DH-LUT, a hierarchical lookup-table method for computing Softmax in a BFP-like format; and an RTL-level engine evaluated on FPGA and ASIC. The accuracy experiments on LLaMA, LLaMA2, LLaMA3, ViT, Swin, and DETR show that DBFP plus DH-LUT matches FP16/FP32 accuracy much more closely than vanilla BFP or FP8, and the paper reports a 74% GPU Softmax speedup and a 10x FOM improvement over prior softmax accelerators.

Significance. If the central claims hold, the paper makes a useful contribution to narrow-precision LLM inference by showing that attention nonlinearities, especially Softmax, can be approximated with a block floating-point format and integer-friendly lookup tables without meaningful accuracy loss. The accuracy study is broad, covering both language and vision models with external benchmarks, and the hardware work includes an RTL implementation with FPGA and 28nm ASIC synthesis, which is a strength. However, the headline hardware comparison is confounded by an input-width mismatch in the FOM metric, and the integer-only derivation in Eq. (9) is not reconciled with the multi-group DBFP definition, so the hardware and algorithmic significance claims need substantial revision.

major comments (4)
  1. [Hardware Implement Evaluation, Eq. (11), Table 3] The FOM comparison in Table 3 is not normalized by input count, and this invalidates the stated 10x advantage over ISCAS'23. Eq. (11) defines FOM = Fmax x N x W / (LUT + FF), yet the prior accelerators process N = 1 to 16 inputs while Ours processes N = 1024. Since N appears linearly in the numerator, comparing these numbers conflates supported input width with architectural quality. Per input, ISCAS'23 gives 49.056/8 ≈ 6.13, while Ours gives 509.563/1024 ≈ 0.50, so the claimed margin reverses under normalization. The abstract's '10x low overhead performance improvement over SOTA designs' therefore rests on an unnormalized metric and needs to be replaced by a normalized comparison, or the claim must be reformulated as a bandwidth-scaling capability.
  2. [DB-Attn Algorithm Design, Eq. (9)] Eq. (9) claims integer-only Softmax by factoring out a shared exponent s from both numerator and denominator, but this is inconsistent with the adaptive grouping definition of DBFP given earlier. In the Methodology section, DBFP partitions a set X into k subsets, each with its own unique shared exponent s_j. For a Softmax denominator that sums over elements from multiple subgroups, the terms are 2^{s_j} e_int_j with different s_j, so a common factor does not exist and the cancellation shown in Eq. (9) does not hold globally. The paper needs to state explicitly whether each Softmax row is a single DBFP block with one shared exponent, or, if multiple groups are used, how cross-group exponent alignment is performed before the denominator sum. Without this clarification, the central 'integer-only computation' property is not established.
  3. [DBFP GPU Run-time Analysis and Implementations] The claimed '74% GPU speedup on Softmax of LLaMA' and 'at least 30%' speedups are not reproducible because the baseline is undefined. The Implementations section says a custom CUDA operator emulates DBFP formats and replaces the Attention layer, but it does not specify whether the baseline is a PyTorch Softmax, a cuDNN kernel, a FlashAttention variant, or a custom FP16/FP32 CUDA kernel, nor does it give sequence lengths, batch sizes, or timing methodology. Since the speedup is a headline result in the abstract, the authors must report a precise baseline definition and the measurement conditions.
  4. [Theoretical Analysis, Eqs. (2)-(6)] The constrained optimization in Eqs. (2)-(6) is not tied to the implemented algorithm or hardware. The paper states that alternating minimization yields an optimal set S, but no implementation step, pseudocode, or ablation shows that the membership variables, the hyperparameter beta, or the empty-set outlier term are actually used in the DBFP conversion or in the DH-LUT construction. As written, the theoretical analysis appears disconnected from the empirical pivot-focus and grouping choices, so the claim of an 'optimal solution' is unsupported. The authors should either show how the optimization maps to the implemented grouping or explicitly present Eqs. (2)-(6) as motivation rather than as a solved design procedure.
minor comments (5)
  1. [Table 3] The table header 'Methods NUM Format' is ambiguous; please use a separate column for the number of inputs N and a second column for the numeric format, and label units consistently.
  2. [Fig. 5] The caption 'Pipeline's balanced proportion under input sequences length growth' is incomplete, and the label 'Redunction' appears to be a typo for 'Reduction'; please correct both.
  3. [Table 1] Please explain the NaN entries for the FP8 e4m3 rows; if the format cannot represent the required dynamic range, the table should say so explicitly rather than leaving the entries unexplained.
  4. [DB-Attn Algorithm Design] The phrase 'for the first time, completes nonlinear operations in a BFP-like format' is a strong novelty claim and should be softened or supported with a more precise comparison to existing integer-only or BFP-based nonlinear-operation work.
  5. [Optimization of Softmax, Eq. (8)] The notation N_gamma, p_gamma_i, and L_E is introduced quickly; please define each symbol in one place and state the range of the shared exponent explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the DB-Attn derivation is self-contained and its main claims are checked against external benchmarks.

full rationale

The paper's claimed derivation chain is not circular. The central algorithmic step, Eq. (9), shows that the shared exponent 2^s cancels in the Softmax ratio, so Softmax reduces to integer mantissa arithmetic; this is a genuine algebraic identity rather than a renaming of the input. The pivot-focus and adaptive-grouping choices are justified by the empirically reported 9.6x greater loss for maximum alignment and by measured accuracy on LLaMA, ViT, and DETR against FP16/FP32 baselines, so they are not fitted to the reported speedup or FOM numbers. DH-LUT is a standard LUT approximation whose parameters are chosen by an MAE-minimizing partition over FP16 values and then evaluated on external model benchmarks. The hardware results come from RTL synthesis with Chisel/Verilator/Vivado/Design Compiler, and the GPU Softmax speedup is measured on an A800, not derived from the paper's own definitions. No load-bearing self-citation or imported uniqueness theorem is used. The FOM comparison in Eq. (11) and Table 3 is questionable as a benchmark because N differs by 64x-1024x across designs, but that is a correctness/evaluation weakness, not circularity: the paper's accuracy and measured speedup claims do not reduce to the FOM definition.

Assumptions & free parameters 5 free parameters · 3 assumptions · 2 invented entities

The central accuracy claim relies on several hand-chosen parameters (block size, mantissa bits, LUT width) and two domain assumptions about data distributions and convergence of the grouping algorithm. The hardware performance claim additionally relies on the FOM comparison being fair across different input widths, which is questionable. No new physical entities are introduced; DBFP and DH-LUT are algorithmic constructs with empirical support.

free parameters (5)
  • DBFP block size = 128
    Chosen for hardware efficiency and accuracy; no sensitivity analysis across block sizes is provided.
  • mantissa bit length = 8
    Same as prior BFP work; affects quantization error per Eq.8.
  • shared exponent bits = 5
    Chosen empirically to balance dynamic range and table size.
  • DH-LUT bit width = 7
    Selected from Pareto frontier in Fig.5; not a globally optimal value.
  • clustering exponent beta = 2
    Default in fuzzy clustering; the paper does not test sensitivity.
assumptions (3)
  • standard math The quantization error of BFP follows the zero-mean variance model in Eq.8 with rounding-to-nearest.
    Standard BFP error analysis, used as an input to justify median alignment.
  • domain assumption For Softmax inputs, the distribution is such that aligning to the median exponent preserves accuracy better than aligning to the maximum exponent.
    Observation 1 is empirical, not proven; it is the motivation for the pivot-focus policy.
  • domain assumption The alternating minimization of J_DBFP in Eq.2 converges to a useful grouping and the resulting assignment can be implemented in hardware.
    The paper asserts this but does not show convergence, initialization, or how the hardware reads the group boundaries.
invented entities (2)
  • DBFP (Dynamic Block Floating-Point) independent evidence
    purpose: A number format with median pivot and adaptive exponent grouping to enable integer-only nonlinear operations.
    Evaluated on LLaMA and vision models; accuracy results in Tables 1-2 provide external evidence.
  • DH-LUT (Dynamic Hierarchical Lookup Table) independent evidence
    purpose: A memory-efficient LUT for approximating exp in DBFP format.
    Evaluated via GPU speedup and hardware implementation, though no code is released.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Pushing the Limits of BFP on Narrow Precision LLM Inference." pith.science (2026). https://pith.science/paper/TYDN3OCH

@misc{pith2026250200026,
  author       = {Pith},
  title        = {Pith review of: Pushing the Limits of BFP on Narrow Precision LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYDN3OCH}},
  note         = {Machine review of arXiv:2502.00026}
}
read the original abstract

The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear operations, a cornerstone of LLM workloads. However, as sequence lengths grow, nonlinear operations, such as Attention, increasingly become performance bottlenecks due to their quadratic computational complexity. These nonlinear operations are predominantly executed using inefficient floating-point formats, which renders the system challenging to optimize software efficiency and hardware overhead. In this paper, we delve into the limitations and potential of applying BFP to nonlinear operations. Given our findings, we introduce a hardware-software co-design framework (DB-Attn), including: (i) DBFP, an advanced BFP version, overcomes nonlinear operation challenges with a pivot-focus strategy for diverse data and an adaptive grouping strategy for flexible exponent sharing. (ii) DH-LUT, a novel lookup table algorithm dedicated to accelerating nonlinear operations with DBFP format. (iii) An RTL-level DBFP-based engine is implemented to support DB-Attn, applicable to FPGA and ASIC. Results show that DB-Attn provides significant performance improvements with negligible accuracy loss, achieving 74% GPU speedup on Softmax of LLaMA and 10x low overhead performance improvement over SOTA designs.

Figures

Figures reproduced from arXiv: 2502.00026 by the authors.

Figure 1
Figure 1. Memory overhead and latency of prefill stages for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Number system comparison between floating [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. DB-Attn algorithm-driven Softmax hardware architecture and enhancement compared with FP16 design. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Pipeline’s balanced proportion under input sequences length growth. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Pipeline’s balance under sequences scaling up. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 27 canonical work pages

  1. [1]

    Bachrach, J.; Vo, H.; Richards, B.; Lee, Y.; Waterman, A.; Avi z ienis, R.; Wawrzynek, J.; and Asanovi \'c , K. 2012. Chisel: constructing hardware in a scala embedded language. In Proceedings of the 49th Annual Design Automation Conference, 1216--1225

    work page 2012

  2. [2]

    Burgess, N.; Milanovic, J.; Stephens, N.; Monachopoulos, K.; and Mansell, D. 2019. Bfloat16 processing for neural networks. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), 88--91. IEEE

    work page 2019

  3. [3]

    C.; Di Nunzio, L.; Fazzolari, R.; Giardino, D.; Nannarelli, A.; Re, M.; and Span \`o , S

    Cardarilli, G. C.; Di Nunzio, L.; Fazzolari, R.; Giardino, D.; Nannarelli, A.; Re, M.; and Span \`o , S. 2021. A pseudo-softmax function for hardware-based high speed image classification. Scientific reports, 11(1): 15307

    work page 2021

  4. [4]

    Choquette, J.; Lee, E.; Krashinsky, R.; Balan, V.; and Khailany, B. 2021. 3.2 the a100 datacenter gpu and ampere architecture. In 2021 IEEE International Solid-State Circuits Conference (ISSCC), volume 64, 48--50. IEEE

    work page 2021

  5. [5]

    Darvish Rouhani, B.; Lo, D.; Zhao, R.; Liu, M.; Fowers, J.; Ovtcharov, K.; Vinogradsky, A.; Massengill, S.; Yang, L.; Bittner, R.; et al. 2020. Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point. Advances in neural information processing systems, 33: 10271--10281

    work page 2020

  6. [6]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

  7. [7]

    Drumond, M.; Lin, T.; Jaggi, M.; and Falsafi, B. 2018. Training dnns with hybrid block floating point. Advances in Neural Information Processing Systems, 31

    work page 2018

  8. [8]

    Du, G.; Tian, C.; Li, Z.; Zhang, D.; Yin, Y.; and Ouyang, Y. 2019. Efficient softmax hardware architecture for deep neural networks. In Proceedings of the 2019 on Great Lakes Symposium on VLSI, 75--80

    work page 2019

Show all 40 references
  1. [9]

    Frantar, E.; and Alistarh, D. 2023. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, 10323--10337. PMLR

    2023

  2. [10]

    K.; Prasad, K.; Srivastava, V

    Jha, C. K.; Prasad, K.; Srivastava, V. K.; and Mekie, J. 2020. FPAD: a multistage approximation methodology for designing floating point approximate dividers. In 2020 IEEE International Symposium on Circuits and Systems (ISCAS), 1--5. IEEE

    2020

  3. [11]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    2023 arXiv

  4. [12]

    P.; Yoon, D

    Jouppi, N. P.; Yoon, D. H.; Kurian, G.; Li, S.; Patil, N.; Laudon, J.; Young, C.; and Patterson, D. 2020. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7): 67--78

    2020

  5. [13]

    W.; and Keutzer, K

    Kim, S.; Gholami, A.; Yao, Z.; Mahoney, M. W.; and Keutzer, K. 2021. I-bert: Integer-only bert quantization. In International conference on machine learning, 5506--5518. PMLR

    2021

  6. [14]

    W.; Shao, Y

    Kim, S.; Hooper, C.; Wattanawong, T.; Kang, M.; Yan, R.; Genc, H.; Dinh, G.; Huang, Q.; Keutzer, K.; Mahoney, M. W.; Shao, Y. S.; and Gholami, A. 2023. Full Stack Optimization of Transformer Inference: a Survey. arXiv:2302.14017

    2023 arXiv

  7. [15]

    A.; Do, A

    Koca, N. A.; Do, A. T.; and Chang, C.-H. 2023. Hardware-efficient Softmax Approximation for Self-Attention Networks. In 2023 IEEE International Symposium on Circuits and Systems (ISCAS), 1--5. IEEE

    2023

  8. [16]

    Kouretas, I.; and Paliouras, V. 2020. Hardware implementation of a softmax-like function for deep learning. Technologies, 8(3): 46

    2020

  9. [17]

    Lin, Y.; Tang, H.; Yang, S.; Zhang, Z.; Xiao, G.; Gan, C.; and Han, S. 2024. Qserve: W4a8kv4 quantization and system co-design for efficient llm serving. arXiv preprint arXiv:2405.04532

    2024 arXiv

  10. [18]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  11. [19]

    Lo, Y.-C.; Lee, T.-K.; and Liu, R.-S. 2023. Block and subword-scaling floating-point (BSFP): An efficient non-uniform quantization for low precision inference. In The Eleventh International Conference on Learning Representations

    2023

  12. [20]

    Ma, Y.; Li, H.; Zheng, X.; Ling, F.; Xiao, X.; Wang, R.; Wen, S.; Chao, F.; and Ji, R. 2024. Affinequant: Affine transformation quantization for large language models. arXiv preprint arXiv:2403.12544

    2024 arXiv

  13. [21]

    Mellempudi, N.; Kundu, A.; Das, D.; Mudigere, D.; and Kaul, B. 2017. Mixed low-precision deep learning inference using dynamic fixed point. arXiv preprint arXiv:1701.08978

    2017 arXiv

  14. [22]

    Meta. 2024. Meta Llama 3: Advancing Generative AI Responsibly. https://ai.meta.com/blog/meta-llama-3/. Accessed: 2024-08-01

    2024

  15. [23]

    v.; Blankevoort, T.; and Welling, M

    Nagel, M.; Baalen, M. v.; Blankevoort, T.; and Welling, M. 2019. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1325--1334

    2019

  16. [24]

    NVIDIA. 2020. Int4 precision for AI inference. https: //developer.nvidia.com/blog/int4-for-ai-inference/. Accessed: 2024-08-01

    2020

  17. [25]

    NVIDIA. 2022. TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x. https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/. Accessed: 2024-08-01

    2022

  18. [26]

    D.; Zhao, R.; More, A.; Hall, M.; Khodamoradi, A.; Deng, S.; Choudhary, D.; Cornea, M.; Dellinger, E.; Denolf, K.; et al

    Rouhani, B. D.; Zhao, R.; More, A.; Hall, M.; Khodamoradi, A.; Deng, S.; Choudhary, D.; Cornea, M.; Dellinger, E.; Denolf, K.; et al. 2023. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10537

    2023 arXiv

  19. [27]

    S, R. 2021. Reduced Softmax Unit for Deep Neural Network Accelerators. arXiv:2201.04562

    2021 arXiv

  20. [28]

    Song, Z.; Liu, Z.; and Wang, D. 2018. Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32

    2018

  21. [29]

    R.; Venkatesan, R.; Dai, S.; Khailany, B.; and Raghunathan, A

    Stevens, J. R.; Venkatesan, R.; Dai, S.; Khailany, B.; and Raghunathan, A. 2021. Softermax: Hardware/software co-design of an efficient softmax for transformers. In 2021 58th ACM/IEEE Design Automation Conference (DAC), 469--474. IEEE

    2021

  22. [30]

    Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozi \`e re, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  23. [31]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  24. [32]

    Wang, M.; Lu, S.; Zhu, D.; Lin, J.; and Wang, Z. 2018. A high-speed and low-complexity architecture for softmax function in deep learning. In 2018 IEEE asia pacific conference on circuits and systems (APCCAS), 223--226. IEEE

    2018

  25. [33]

    Xia, H.; Zheng, Z.; Li, Y.; Zhuang, D.; Zhou, Z.; Qiu, X.; Li, Y.; Lin, W.; and Song, S. L. 2023. Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity. arXiv:2309.10285

    2023 arXiv

  26. [34]

    Xia, T.; and Zhang, S. Q. 2023. Softmax Acceleration with Adaptive Numeric Format for both Training and Inference. arXiv preprint arXiv:2311.13290

    2023 arXiv

  27. [35]

    Yeh, T.; Sterner, M.; Lai, Z.; Chuang, B.; and Ihler, A. 2022. Be Like Water: Adaptive Floating Point for Machine Learning. In International Conference on Machine Learning, 25490--25500. PMLR

    2022

  28. [36]

    Zhang, Y.; Peng, L.; Quan, L.; Zhang, Y.; Zheng, S.; and Chen, H. 2023. High-precision method and architecture for base-2 softmax function in dnn training. IEEE Transactions on Circuits and Systems I: Regular Papers, 70(8): 3268--3279

    2023

  29. [37]

    Zhang, Y.; Zhang, Y.; Peng, L.; Quan, L.; Zheng, S.; Lu, Z.; and Chen, H. 2022. Base-2 softmax function: Suitability for training and efficient hardware implementation. IEEE Transactions on Circuits and Systems I: Regular Papers, 69(9): 3605--3618

    2022

  30. [38]

    Zou, L.; Zhao, W.; Yin, S.; Bai, C.; Sun, Q.; and Yu, B. 2024. BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization. In Forty-first International Conference on Machine Learning

    2024

  31. [39]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.