Pith. sign in

REVIEW 3 major objections 6 minor 43 references

BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that BBFP, a block floating point format with a 1-bit flag and a lowered shared exponent, reduces quantization error enough to make sub-4-bit linear and nonlinear LLM inference practical, and that its BBAL accelerator…

desk verdict BBFP is a genuine and well-tested tweak on block floating point for LLMs, but the paper's error analysis ignores its own left-shift path and the overlap width is tuned on the reported test perplexity, so it needs a serious referee and careful revision rather than a desk reject. read the letter →

arxiv 2504.15721 v1 pith:ZXHGAX7R submitted 2025-04-22 cs.AR

Xiaomeng Han , Yuan Cheng , Jing Wang , Junyang Lu , Hui Wang , X.x. Zhang , Ning Xu , Dawei Yang
show 1 more author
Zhe Jiang
This is my paper · ORCID
classification cs.AR
keywords blockfloatingpointquantizationlargelanguagemodelshardwareacceleratoroutliersnonlinearactivationpost-trainingsoftmax
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Block floating point (BFP) quantization packs LLM weights and activations into fixed-point blocks that share one exponent, but aligning every value to the block maximum washes out small and moderate values. BBFP adds a 1-bit flag to each mantissa and sets the shared exponent to $E_{\text{shared}} = \max(E) - (m-o)$, so ordinary values are right-shifted less and outliers are left-shifted into a protected high group. The paper claims this lowers quantization error enough that 3–4 bit mantissas beat or match wider BFP formats, and that the same format makes nonlinear layers like Softmax and SiLU nearly lossless. The accompanying BBAL accelerator realizes the format with a BBFP processing-element array and a segmented lookup-table nonlinear unit, reporting a 22% accuracy improvement over an outlier-aware accelerator at similar efficiency and a 40% efficiency improvement over BFP at similar accuracy.

What carries the argument

The load-bearing object is the BBFP data format itself: a word with a sign bit, a 1-bit flag, a fixed 5-bit shared exponent, an $m$-bit mantissa, and an $o$-bit overlap field. Its defining identity is the shared-exponent choice $E_{\text{shared}} = \max(E) - (m-o)$ paired with the flag-controlled reconstruction factor $f=1$ for the low group and $f = 2^{m-o}$ for the high group, which effectively widens the mantissa's dynamic range by $2^{m-o}$ without storing more mantissa bits. The flag also makes the dot product a fixed-point operation, so the same format carries linear computation, and it creates regular zero patterns in partial sums that a sparse carry-chain adder exploits, saving 15% adder resources. For nonlinear layers, the shared exponent indexes a segmented lookup table, letting each BBFP mantissa directly address its sub-table.

What would settle it

Take any real LLM activation block with a few extreme outliers, compute the actual BBFP(4,2) reconstruction mean-squared error with $E_{\text{shared}} = \max(E) - 2$, and compare with vanilla BFP4's max-exponent alignment: if for a non-negligible fraction of blocks the BBFP error exceeds BFP's error (or if the true optimal exponent is closer to the max than to $\max - (m-o)$), the central error-reduction claim fails. An end-to-end check: quantize OPT-6.7B and Llama-7B linear layers with BBFP(3,1) and measure Wikitext2 perplexity; if the improvement over BFP4 is not reproduced, the empirical result is not robust.

Watch

Extended reading notes

Core claim

BBFP attacks the main failure of BFP: aligning every value in a block to the maximum exponent destroys small and moderate mantissas. Instead, each block chooses the shared exponent $E_{\text{shared}} = \max(E) - (m-o)$, where $m$ is the mantissa width and $o$ is the overlap width, and a 1-bit flag records whether an element's original exponent exceeded this shared value. Elements above the shared exponent (the outliers) are left-shifted and stored in a high group; the rest are right-shifted into a low group, and the overlap bits rescue bits that would be lost in the left shift. Relying on the BFP roundoff-variance formula, the paper argues that this non-maximum shared exponent reduces the quantization error variance, and its experiments on OPT and Llama models show BBFP(3,1) improves perplexity by 6% over BFP4, BBFP(4,2) comes within 4% of BFP6, and BBFP(6,3) nearly matches FP16. For nonlinear layers, a BBFP(10,5) segmented lookup-table unit keeps perplexity within 0.44 of FP32, where BFP10 raises perplexity by at least 3×. The full accelerator, BBAL, reports a 22% accuracy improvement over an outlier-aware accelerator at similar efficiency and a 40% throughput improvement over a BFP-based accelerator at similar accuracy.

Load-bearing premise

The load-bearing premise is that choosing the shared exponent as $\max(E) - (m-o)$ and marking outliers with a 1-bit flag lowers quantization error in the way the standard BFP roundoff formula predicts, even though that formula does not model the left-shift truncation of the flagged high group and the exponent offset is justified empirically, not derived.

Editorial extensions

If this is right

  • Lower-bit linear layers become viable without calibration: BBFP(3,1) improves perplexity by about 6% over BFP4, and BBFP(4,2) lands within 4% of BFP6.
  • BFP-style nonlinear computation becomes practical: BBFP(10,5) adds at most 0.44 perplexity on Llama-family models, where BFP10 adds 3× or more.
  • The accelerator's 22% accuracy gain over an outlier-aware design at equal area and its 40% throughput gain over BFP at equal accuracy make a specific efficiency–accuracy tradeoff available to edge LLM inference.
  • The flag-driven zero patterns let the sparse adder cut resource use by 15%, with savings growing as mantissa width grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The flag bit is effectively a one-bit outlier detector; the fraction of high-flag values per block could be used as a free calibration signal to make the exponent offset $(m-o)$ adaptive per layer or tensor, a testable extension the paper does not explore.
  • The error analysis assumes the standard BFP round-to-nearest variance model, but Eq. (4) truncates instead of rounding; estimating and subtracting the resulting per-block bias during accumulation could shrink error further.
  • The exponent-segmented lookup table should generalize to any monotone transcendental function, so the same nonlinear unit with reloaded LUT contents could compute GELU, swish variants, or attention masks without new hardware, which the paper only sketches.
  • The iso-area comparisons use fixed outlier-ratio baselines; an extension is to benchmark BBFP against per-channel scaling methods on 13B-70B models to see whether the fixed exponent offset or the flag remains the better outlier mechanism at scale.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes BBFP, a block floating-point variant that adds a 1-bit flag, two mantissa groups, and overlap bits, and selects the shared exponent as Max(E) - (m - o). The authors claim that this reduces the error caused by BFP's maximum-exponent alignment, enabling lower-bit linear quantization and, for the first time, practical BFP-style quantization of nonlinear layers. They then present BBAL, an accelerator with a BBFP processing-element array and a nonlinear unit based on an exponent-segmented lookup table. The evaluation reports WikiText2 perplexity for Llama and OPT models across many bit configurations and compares the accelerator against BFP- and outlier-aware designs using TSMC 28nm synthesis, CACTI memory models, and a cycle-level simulator. The headline claims are a 22% accuracy improvement over an outlier-aware accelerator at similar efficiency and a 40% throughput improvement over a BFP-based accelerator at similar accuracy.

Significance. If the central claims hold, the paper is a useful contribution to edge-oriented LLM inference: it extends BFP-style fixed-point computation from linear layers to nonlinear layers and provides a full hardware stack with concrete area, energy, and throughput estimates. The strengths are the breadth of the perplexity evaluation (12 model variants), the use of a real hardware implementation flow (Chisel, Design Compiler, CACTI, DnnWeaver-based simulation) rather than abstract estimates, and the explicit comparisons to prior outlier-aware accelerators. The data format idea is simple and plausible. However, the theoretical error analysis that motivates the format is incomplete, and the hyperparameter selection procedure raises a risk of selection on the evaluation metric, so the empirical results, as presented, are not yet conclusive evidence for the mechanism claimed.

major comments (3)
  1. [Section III-B, Eq. (8)] The variance formula cited from [31] applies to round-to-nearest block floating point with a single shared exponent and no left-shift path. BBFP, as defined in Eq. (4), uses truncation (Clip) and a second alignment group that left-shifts before truncating. For values above E_shared, the high group's error is not the roundoff error modeled by Eq. (8), so the statement that 'the only factor influencing the quantisation error is pγi' is not established. This is load-bearing because the claimed error reduction of BBFP is the mechanism for all downstream accuracy results. The paper should either derive an error model for the flagged two-group format or provide a direct numerical comparison of Eq. (8) against measured BBFP error on representative activation blocks, including blocks with strong outliers.
  2. [Section III-C, Eq. (9)] The choice E_shared = Max(E) - (m - o) is justified by intuition and by Fig. 3, but Fig. 3 itself shows non-monotone behavior: Max-3 (offset m-o+1) is reported as causing 'significant error', and the optimum at Max-(m-o) is not derived. The assumption that lowering the shared exponent monotonically reduces quantization error is not generally true when the left-shift/truncation path can saturate or lose most-significant bits for outliers. The paper should provide an analytic derivation of the optimal offset or an empirical validation on a held-out set. As written, the reported PPL tables cannot separate a genuine format advantage from tuning the exponent selection to the data.
  3. [Section III-D, Algorithm 1 and Section V-B] Algorithm 1 selects the overlap width o by minimizing a score that includes WikiText2 PPL, and the main accuracy tables (Tables II and IV) also report WikiText2 PPL. This is a selection-on-the-evaluation-metric procedure: the reported numbers are the result of optimizing a hyperparameter on the same data used for evaluation. Comparisons with prior methods that use fixed, un-tuned hyperparameters are therefore not on equal footing. The paper should use a held-out validation split for the hyperparameter selection and report test PPL, or provide a sensitivity analysis demonstrating that the conclusions are robust to the choice of o across all models.
minor comments (6)
  1. [Section II-B] The sentence beginning 'An format for optimizing...' contains a typo and should read 'A format...'.
  2. [Section IV-A, Eq. (10)] Eq. (10) gives shift amounts of 2 and 4 bits, which are specific to BBFP(4,2); for a general BBFP(m,o) configuration the shifts should be m-o and 2(m-o). Please state the general form or clearly restrict Eq. (10) to the BBFP(4,2) case.
  3. [Section III-B, Eq. (8)] The notation in Eq. (8) is not fully defined: Lm, LE, and the range of the sum over γi should be stated explicitly, and 'probabilities mass function' should read 'probability mass function'.
  4. [Figure 3] The y-axis of Fig. 3 is labeled 'MSE' while the caption and text refer to 'quantisation error'; please reconcile the units or define the metric.
  5. [Algorithm 1] The 'Overhead' computed in Algorithm 1 is not defined; the paper should state which hardware metric (area, energy, latency, or a combination) is used when selecting the overlap width.
  6. [Table V] The entries in Table V under 'Methods' are unclear (e.g., '10 Int8', '8 Int 27'); please format the table so that each baseline's numerical format and bit width are legible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central error-reduction and accelerator claims rest on an external roundoff formula and independent experiments, not on a self-citation or a fitted prediction.

full rationale

The paper's derivation chain is: BBFP uses a non-maximum shared exponent and a flag/overlap scheme to reduce BFP quantization error (Eqs. 4-9), then hardware units are built around this format and evaluated against BFP, Oltron, etc. No step equates an output to an input by construction. Eq. (8) is taken from an independent prior source [31], not from the authors' own work, and the claim that lowering the shared exponent reduces error is a stated application of that formula to the BBFP scheme. Whether that application is valid (the flagged high group's left-shift/truncation path is not modeled by Eq. (8)) is a correctness or rigor concern, not a circularity: the formula is not defined in terms of the paper's target result. The choices E_shared = Max(E) - (m-o) (Eq. 9) and overlap width o (Algo. 1) are selected empirically using MSE/PPL and hardware overhead; the paper reports PPL tables on WikiText2 for many configurations. This is a design-space sweep, and the selected configurations are not presented as predictions derived from the fitted parameters; the headline accuracy/efficiency comparisons are direct experimental measurements. There is no hidden equation in which the reported improvement equals the fitted score by construction. The only self-citation ([26], I-LLM, by co-authors Cheng and Yang) appears in a background statement on INT4 quantization and is not load-bearing. No uniqueness theorem or ansatz is imported from the authors' prior work. Accordingly, the derivation is not circular, though the overlap-width selection on the evaluation metric may raise a separate overfitting/evaluation concern.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The central claim depends on two tuned format parameters (overlap width and exponent offset) and on the applicability of a prior BFP error formula to a two-group format. No new physical entity is introduced.

free parameters (2)
  • Overlap width o = 1 to 5 depending on configuration (e.g., BBFP(3,1), BBFP(6,3), BBFP(10,5))
    Algorithm 1 selects o by minimizing a weighted sum of WikiText2 perplexity and hardware overhead, and the same perplexity dataset is then reported in Table II and Table IV.
  • Shared exponent offset = m - o (e.g., 2 for BBFP(4,2))
    Eq. (9) sets E_shared = Max(E) - (m-o); Section III-C chooses this offset by comparing MSE for Max-1, Max-2, and Max-3 in Fig. 3 rather than deriving it.
assumptions (3)
  • domain assumption The BFP roundoff error variance formula (Eq. 8, from reference [31]) remains valid for BBFP's two-group flag scheme.
    The paper says the only factor changing is the exponent probability p_gamma_i, but BBFP also changes the quantization step and introduces left-shift truncation of the high group, which Eq. (8) does not model.
  • domain assumption Selecting a lower shared exponent monotonically reduces total quantization error for LLM activations.
    Section III-C assumes a smaller Max(E) selection reduces right-shift loss and therefore error; Fig. 3 shows this only empirically for one set of activations.
  • domain assumption WikiText2 perplexity is a sufficient proxy for the accuracy of the quantized models.
    All accuracy claims and the overlap selection use WikiText2 PPL; no other tasks or held-out sets are used.
invented entities (1)
  • BBFP data format (1-bit flag, high/low mantissa groups, Max-(m-o) shared exponent)
    purpose: Reduce BFP quantization error by preserving small and moderate mantissas while covering outliers.
    No falsifiable prediction outside the paper's own experiments; its value is measured only on the models and datasets reported here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models." pith.science (2026). https://pith.science/paper/ZXHGAX7R

@misc{pith2026250415721,
  author       = {Pith},
  title        = {Pith review of: BBAL: A Bidirectional Block Floating Point-Based Quantisation Accelerator for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXHGAX7R}},
  note         = {Machine review of arXiv:2504.15721}
}
read the original abstract

Large language models (LLMs), with their billions of parameters, pose substantial challenges for deployment on edge devices, straining both memory capacity and computational resources. Block Floating Point (BFP) quantisation reduces memory and computational overhead by converting high-overhead floating point operations into low-bit fixed point operations. However, BFP requires aligning all data to the maximum exponent, which causes loss of small and moderate values, resulting in quantisation error and degradation in the accuracy of LLMs. To address this issue, we propose a Bidirectional Block Floating Point (BBFP) data format, which reduces the probability of selecting the maximum as shared exponent, thereby reducing quantisation error. By utilizing the features in BBFP, we present a full-stack Bidirectional Block Floating Point-Based Quantisation Accelerator for LLMs (BBAL), primarily comprising a processing element array based on BBFP, paired with proposed cost-effective nonlinear computation unit. Experimental results show BBAL achieves a 22% improvement in accuracy compared to an outlier-aware accelerator at similar efficiency, and a 40% efficiency improvement over a BFP-based accelerator at similar accuracy.

Figures

Figures reproduced from arXiv: 2504.15721 by the authors.

Figure 1
Figure 1. (a) Distribution of activation and weight values in OPT-6.7B. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Comparison of the impact of different selection of shared [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. The PPL and hardware overhead for BBFP with a width of 6 [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: (a) Multiplication operation between two BBFP blocks; (b) [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Proposed nonlinear unit with Softmax as an example and the [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 8
Figure 8. Figure 8: Various quantisation methods compared by average [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]
Figure 9
Figure 9. Figure 9: Average normalized energy consumption under identical PE [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 14 canonical work pages

  1. [31]

    Roundoff errors in block-floating-point systems,

    K. Kalliojarvi and J. Astola, “Roundoff errors in block-floating-point systems,” IEEE transactions on signal processing , vol. 44, no. 4, pp. 783–790, 1996

    work page 1996

  2. [1]

    Language models are few-shot learners,

    B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal et al. , “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165 , vol. 1, 2020

    arXiv 2005

  3. [2]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” Advances in neural information processing systems , vol. 35, pp. 36 479–36 494, 2022

    2022

  4. [3]

    Prestu: Pre-training for scene-text understanding,

    J. Kil, S. Changpinyo, X. Chen, H. Hu, S. Goodman, W.-L. Chao, and R. Soricut, “Prestu: Pre-training for scene-text understanding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 270–15 280

    work page 2023

  5. [4]

    New trends in machine translation using large language models: Case examples with chatgpt,

    C. Lyu, J. Xu, and L. Wang, “New trends in machine translation using large language models: Case examples with chatgpt,” arXiv preprint arXiv:2305.01181, 2023

    arXiv 2023

  6. [5]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  7. [6]

    A survey on efficient inference for large language models,

    Z. Zhou, X. Ning, K. Hong, T. Fu, J. Xu, S. Li, Y . Lou, L. Wang, Z. Yuan, X. Li et al., “A survey on efficient inference for large language models,” arXiv preprint arXiv:2404.14294 , 2024

    arXiv 2024

  8. [7]

    Hardware acceleration of llms: A comprehen- sive survey and comparison,

    N. Koilia and C. Kachris, “Hardware acceleration of llms: A comprehen- sive survey and comparison,” arXiv preprint arXiv:2409.03384 , 2024

    arXiv 2024

Show all 43 references
  1. [8]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2015

    2015 arXiv

  2. [9]

    Billm: Pushing the limit of post-training quantization for llms,

    W. Huang, Y . Liu, H. Qin, Y . Li, S. Zhang, X. Liu, M. Magno, and X. Qi, “Billm: Pushing the limit of post-training quantization for llms,” arXiv preprint arXiv:2402.04291 , 2024

    2024 arXiv

  3. [10]

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,” Advances in Neural Information Processing Systems , vol. 35, pp. 30 318–30 332, 2022

    2022

  4. [11]

    Rptq: Reorder-based post-training quantization for large language models,

    Z. Yuan, L. Niu, J. Liu, W. Liu, X. Wang, Y . Shang, G. Sun, Q. Wu, J. Wu, and B. Wu, “Rptq: Reorder-based post-training quantization for large language models,” arXiv preprint arXiv:2304.01089 , 2023

    2023 arXiv

  5. [12]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  6. [13]

    Bfloat16 processing for neural networks,

    N. Burgess, J. Milanovic, N. Stephens, K. Monachopoulos, and D. Mansell, “Bfloat16 processing for neural networks,” in 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH) . IEEE, 2019, pp. 88–91

    2019

  7. [14]

    Fp8 formats for deep learning,

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisen- thwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu et al., “Fp8 formats for deep learning,” arXiv preprint arXiv:2209.05433 , 2022

    2022 arXiv

  8. [15]

    Revisiting block-based quantisation: What is important for sub-8-bit llm inference?

    C. Zhang, J. Cheng, I. Shumailov, G. A. Constantinides, and Y . Zhao, “Revisiting block-based quantisation: What is important for sub-8-bit llm inference?” arXiv preprint arXiv:2310.05079 , 2023

    2023 arXiv

  9. [16]

    Be like water: Adaptive floating point for machine learning,

    T. Yeh, M. Sterner, Z. Lai, B. Chuang, and A. Ihler, “Be like water: Adaptive floating point for machine learning,” in International Confer- ence on Machine Learning . PMLR, 2022, pp. 25 490–25 500

    2022

  10. [17]

    A block mini- float representation for training deep neural networks,

    S. Fox, S. Rasoulinezhad, J. Faraone, P. Leong et al. , “A block mini- float representation for training deep neural networks,” in International Conference on Learning Representations , 2020

    2020

  11. [18]

    Bie: Bi-exponent block floating-point for large language models quantization,

    L. Zou, W. Zhao, S. Yin, C. Bai, Q. Sun, and B. Yu, “Bie: Bi-exponent block floating-point for large language models quantization,” in Forty- first International Conference on Machine Learning , 2024

    2024

  12. [19]

    Fpga-based convolutional neural network accel- erator with resource-optimized approximate multiply-accumulate unit,

    M. Cho and Y . Kim, “Fpga-based convolutional neural network accel- erator with resource-optimized approximate multiply-accumulate unit,” Electronics, vol. 10, no. 22, p. 2859, 2021

    2021

  13. [20]

    High-performance acceleration of 2-d and 3-d cnns on fpgas using static block floating point,

    H. Fan, S. Liu, Z. Que, X. Niu, and W. Luk, “High-performance acceleration of 2-d and 3-d cnns on fpgas using static block floating point,” IEEE Transactions on Neural Networks and Learning Systems , vol. 34, no. 8, pp. 4473–4487, 2021

    2021

  14. [21]

    Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design,

    Z. Song, Z. Liu, and D. Wang, “Computation error analysis of block floating point arithmetic oriented convolution neural network accelerator design,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018

    2018

  15. [22]

    Attention is all you need. advances in neural information processing systems,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need. advances in neural information processing systems,” Advances in neural information processing systems, vol. 30, no. 2017, 2017

    2017

  16. [23]

    Softermax: Hardware/software co-design of an efficient softmax for transformers,

    J. R. Stevens, R. Venkatesan, S. Dai, B. Khailany, and A. Raghunathan, “Softermax: Hardware/software co-design of an efficient softmax for transformers,” in 2021 58th ACM/IEEE Design Automation Conference (DAC). IEEE, 2021, pp. 469–474

    2021

  17. [24]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin et al. , “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068 , 2022

    2022 arXiv

  18. [25]

    Llm-fp4: 4-bit floating-point quantized transformers,

    S.-y. Liu, Z. Liu, X. Huang, P. Dong, and K.-T. Cheng, “Llm-fp4: 4-bit floating-point quantized transformers,” arXiv preprint arXiv:2310.16836, 2023

    2023 arXiv

  19. [26]

    I- llm: Efficient integer-only inference for fully-quantized low-bit large language models,

    X. Hu, Y . Cheng, D. Yang, Z. Yuan, J. Yu, C. Xu, and S. Zhou, “I- llm: Efficient integer-only inference for fully-quantized low-bit large language models,” arXiv preprint arXiv:2405.17849 , 2024

    2024 arXiv

  20. [27]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099

    2023

  21. [28]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  22. [29]

    Post-training quantization for vision transformer,

    Z. Liu, Y . Wang, K. Han, W. Zhang, S. Ma, and W. Gao, “Post-training quantization for vision transformer,” Advances in Neural Information Processing Systems, vol. 34, pp. 28 092–28 103, 2021

    2021

  23. [30]

    Q8bert: Quantized 8bit bert,

    O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8bert: Quantized 8bit bert,” in 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS). IEEE, 2019, pp. 36–39

    2019

  24. [32]

    A pseudo-softmax function for hardware-based high speed image classification,

    G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, A. Nannarelli, M. Re, and S. Span `o, “A pseudo-softmax function for hardware-based high speed image classification,” Scientific reports , vol. 11, no. 1, p. 15307, 2021

    2021

  25. [33]

    High- precision method and architecture for base-2 softmax function in dnn training,

    Y . Zhang, L. Peng, L. Quan, Y . Zhang, S. Zheng, and H. Chen, “High- precision method and architecture for base-2 softmax function in dnn training,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 70, no. 8, pp. 3268–3279, 2023

    2023

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    Meta llama 3: Advancing generative ai responsibly,

    Meta, “Meta llama 3: Advancing generative ai responsibly,” https://ai. meta.com/blog/meta-llama-3/, 2024

    2024

  28. [36]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

    2016 arXiv

  29. [37]

    Omniquant: Omnidirectionally calibrated quan- tization for large language models,

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . Qiao, and P. Luo, “Omniquant: Omnidirectionally calibrated quan- tization for large language models,” arXiv preprint arXiv:2308.13137 , 2023

    2023 arXiv

  30. [38]

    Oltron: Algorithm-hardware co-design for outlier-aware quantization of llms with inter-/intra-layer adaptation,

    C. Xue, C. Zhang, X. Jiang, Z. Gao, Y . Lin, and G. Sun, “Oltron: Algorithm-hardware co-design for outlier-aware quantization of llms with inter-/intra-layer adaptation,” in Proceedings of the 61st ACM/IEEE Design Automation Conference , 2024, pp. 1–6

    2024

  31. [39]

    Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization,

    C. Guo, J. Tang, W. Hu, J. Leng, C. Zhang, F. Yang, Y . Liu, M. Guo, and Y . Zhu, “Olive: Accelerating large language models via hardware- friendly outlier-victim pair quantization,” in Proceedings of the 50th Annual International Symposium on Computer Architecture , 2023, pp. 1–15

    2023

  32. [40]

    Chisel: constructing hardware in a scala embedded language,

    J. Bachrach, H. V o, B. Richards, Y . Lee, A. Waterman, R. Avi ˇzienis, J. Wawrzynek, and K. Asanovi ´c, “Chisel: constructing hardware in a scala embedded language,” in Proceedings of the 49th Annual Design Automation Conference, 2012, pp. 1216–1225

    2012

  33. [41]

    Kurup and T

    P. Kurup and T. Abbasi, Logic synthesis using Synopsys® . Springer Science & Business Media, 1997

    1997

  34. [42]

    Cacti 6.0: A tool to model large caches,

    N. Muralimanohar, R. Balasubramonian, and N. P. Jouppi, “Cacti 6.0: A tool to model large caches,” HP laboratories, vol. 27, p. 28, 2009

    2009

  35. [43]

    Dnnweaver: From high-level deep network models to fpga acceleration,

    H. Sharma, J. Park, E. Amaro, B. Thwaites, P. Kotha, A. Gupta, J. K. Kim, A. Mishra, and H. Esmaeilzadeh, “Dnnweaver: From high-level deep network models to fpga acceleration,” inthe Workshop on Cognitive Architectures, 2016

    2016

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.