Pith. sign in

REVIEW 3 major objections 5 minor 36 references

This paper claims that a fully integer attention pipeline, built around IndexSoftmax — a 32-entry lookup-table softmax replacement — eliminates the dequantize-to-softmax-to-requantize bottleneck in quantized attention, delivering up to 3.7x

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 19:58 UTC pith:WJ72VQG4

load-bearing objection A well-engineered, training-free integer attention pipeline with surprisingly large speedups; worth reviewing, but code release and baseline details need to be conditions of acceptance. the 3 major comments →

arxiv 2511.21513 v2 pith:WJ72VQG4 submitted 2025-11-26 cs.LG

IntAttention: A Fully Integer Attention Pipeline for Efficient Edge Inference

Wanli Zhong , Haibo Feng , Zirui Zhou , Hanyang Peng , Shiqi Yu This is my paper
classification cs.LG
keywords integer attentionsoftmax approximationlookup tablequantizationedge inferencetransformerINT8energy efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that once transformer matrix multiplications are quantized to INT8, the softmax path — with its dequantize, floating-point exponential, normalize, and requantize steps — becomes the dominant attention bottleneck, consuming up to 65% of latency on edge CPUs. To remove it, the authors propose IntAttention, a fully integer attention pipeline whose core operator IndexSoftmax replaces floating-point exponentials with integer clipping, a 32-entry lookup table, and integer row-sum normalization. The result is a training-free, drop-in replacement for conventional quantized attention that reportedly runs up to 3.7x faster than FP16 and 2.0x faster than a standard INT8 pipeline on Armv8 CPUs, while using 61% less energy and matching baseline accuracy across several language and vision models. If true, it means the costly datatype-conversion detour can be removed entirely, restoring an end-to-end integer dataflow for edge inference.

Core claim

IntAttention is a fully integer attention pipeline that runs entirely in the integer domain from the QK^T logits to the PV output. Its key operator, IndexSoftmax, performs row-wise max-subtraction in integers, clips small logit distances using a threshold derived from the quantization scale, approximates the exponential with a fixed 32-entry UINT8 lookup table, and normalizes the result with integer arithmetic to produce a UINT8 attention probability matrix. The paper claims this is the first fully integer, plug-and-play attention pipeline that requires no retraining, and that it achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and up to 2.0x over conventional INT8 qu

What carries the argument

IndexSoftmax — a three-stage integer operator: (1) sparsity-aware clipping in the integer domain, where logit distances are saturated at a quantization-aligned threshold c_int; (2) a fixed 32-entry UINT8 lookup table that maps clipped indices to approximate exp(-x) values; (3) integer row-sum normalization with UINT8 scaling to produce attention probabilities. The LUT occupies about 32 bytes and, unlike prior LUT-based softmax methods (e.g., EXAQ), uses a fixed clipping threshold (c≈6.6) and a moderate resolution (b=5), avoiding per-tensor dynamic statistics and global reductions that are costly on edge processors.

Load-bearing premise

The headline 2.0–3.7x speedups rest on the assumption that the FP16 and INT8 'quant-only' baseline pipelines are fairly and well-optimized implementations of conventional attention; the paper gives no kernel-level detail, variance, or comparison against widely deployed production kernels for those baselines.

What would settle it

Benchmark IntAttention against a carefully optimized INT8 attention implementation with a fused dequant-softmax-requant kernel on the same Armv8 CPUs; if the 2.0x speedup over INT8 shrinks substantially, the central claim is inflated. Separately, evaluate the same models on a long-context task (16k+ tokens) against FP16; if perplexity or reasoning accuracy degrades by more than the small margins reported at 1k–16k lengths, the fidelity claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Existing INT8 quantized inference stacks can replace their softmax stage with IndexSoftmax without retraining or calibration, potentially recovering accuracy on most models while removing the dequant/requant detour.
  • The attention bottleneck shifts back to QK and PV GEMMs, so further speedups on edge devices should come from optimizing integer matrix multiplication kernels rather than the softmax path.
  • The measured 61% energy reduction makes fully integer attention practical for battery-powered transformer deployment on edge hardware.
  • The unsigned UINT8 format for attention probabilities is measurably more faithful than signed INT8 (higher cosine similarity, lower L1/RMSE), which could influence quantization scheme design beyond this paper.
  • The robustness of the fixed clipping threshold and LUT resolution across language and vision models suggests an integer softmax surrogate that transfers without per-model tuning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The fixed 32-entry LUT is essentially a universal exponential surrogate for softmax on quantized logits; it would be worth testing whether it holds for much longer contexts (e.g., 32k tokens) and larger models, where attention distributions can become sparser or flatter.
  • Because the clipping step already prunes near-zero exponentials, the method could combine with sparse or variable-length GEMM support on edge hardware to skip memory traffic entirely, not just LUT lookups.
  • The paper suggests combining IntAttention with per-block or per-channel quantization and input smoothing; this is a natural next experiment that could close the residual accuracy gap seen on Qwen3.
  • The reported speedups depend on the fairness of the FP16 and INT8 'quant-only' baseline implementations; an independent, production-grade re-implementation of those baselines would be the most direct way to confirm the 2.0–3.7x claims.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes IntAttention, a fully integer attention pipeline for edge inference that eliminates the dequantize→softmax→requantize path in quantized Transformers. The core operator, IndexSoftmax, uses integer-domain max-subtraction and sparsity-aware clipping followed by a fixed 32-entry UINT8 lookup table to approximate the exponential, with integer row-sum normalization and UINT8 quantization of the attention probability matrix. The method is training-free and intended as a drop-in replacement for conventional quantized attention. Experiments on Armv8 CPUs (RK3588S2 and Apple M2) report up to 3.7× speedup and 61% energy reduction over FP16 attention, up to 2.0× speedup over an INT8 'Quant Only' pipeline, and accuracy comparable to baselines on several language and vision models. The paper also presents hyperparameter sensitivity analysis and ablations against EXAQ-based LUT softmax approximations.

Significance. If the reported efficiency gains are reproducible, the work addresses a real bottleneck in quantized Transformer inference on edge devices: the softmax path dominates once GEMMs are quantized. The proposed design is simple, portable (uses only integer add/multiply/shift/lookup), and does not require retraining or per-input statistics, which are practical advantages. The paper includes useful ablations: comparison with EXAQ variants, sensitivity to the two hyperparameters (c and b), and a comparison of INT8 vs UINT8 probability quantization. The breadth of models and benchmarks is a strength. However, the central efficiency claims currently rest on an incompletely specified baseline, and the accuracy comparison shows a notable degradation on Qwen3-1.7B that is not fully reflected in the paper's summary claims.

major comments (3)
  1. [§4.1/§4.2, Figures 6–8] The headline speedup and energy claims are unverifiable without a precise description of the baselines. The 'Quant Only' pipeline is said to be a conventional INT8 attention pipeline, but the manuscript does not state whether the softmax/QDQ path is a scalar FP32 loop, a fused kernel, or an ACL composition, nor does it report kernel-level implementation details, thread configuration, or measurement variance. Without error bars or standard deviations, the reported 'up to 3.7x' and 'up to 2.0x' figures could be inflated by a suboptimal baseline. Additionally, the abstract states code is available, while Section 1 says 'Code will be released in later version of this work.' This contradiction prevents independent replication. The authors should describe the baseline kernels precisely, provide error bars over multiple runs, and either release code or provide enough detail for exact reproducti
  2. [Table 1, §4.3] The claim that IntAttention maintains accuracy comparable to baselines is weakened by the Qwen3-1.7B results. The average accuracy for IntAttention is 55.02%, versus 59.49% for FP16 and 55.12% for Quant Only. On HellaSwag, PIQA, WinoGrande, and ARC-Challenge, IntAttention is numerically below FP16 by 2–6 points, and on average it is not better than Quant Only. The text in §4.3 says IntAttention 'narrows the gap' and 'yields a clear perplexity gain on WikiText,' but the average accuracy does not improve over Quant Only. This should be acknowledged explicitly, and the authors should either improve the method for this model or temper the claim that accuracy is 'comparable to baseline' across diverse language models.
  3. [Equation (15) and §3.2] The integer normalization in Equation (15) uses rowSum(Ē), where Ē is a UINT8-quantized lookup table. Since the LUT is quantized to 8 bits, small exponential values, especially those near the clipping boundary, are rounded to zero. For c=6.6, exp(-6.6)*255 ≈ 0.35, so a range of entries become exactly zero. This is presumably intentional sparsity, but the error analysis is not provided. The paper should quantify the approximation error of this combined LUT quantization and integer normalization, and show its effect on the final attention output, particularly for long sequences where many logits may be near the clipping threshold.
minor comments (5)
  1. [Abstract vs. Section 1] The abstract says 'Code is available at https://github.com/WanliZhong/IntAttention,' but Section 1 says 'Code will be released in later version of this work.' Please clarify the actual availability and, in the meantime, avoid claiming public availability if the repository is not yet accessible.
  2. [§4.1] The notation is inconsistent: 'Quant Only' appears in figures and tables, while the text also uses 'Quantized-Only' and 'Quantization-only.' Please standardize.
  3. [§4.2] The phrase 'studied in Table 4.4' appears to be a typo; there is no Table 4.4, and the relevant discussion is in Section 4.2/Figure 2.
  4. [§3.2 title] The title 'LUT Rebuild and Integer Scale Normalization' is confusing; the LUT is not rebuilt at runtime. Rename to 'LUT-based Integer Normalization' or similar.
  5. [Figures 2, 6, 7] The figures lack error bars and statistical significance. Even a brief statement of run-to-run variance would help support the quantitative claims.

Circularity Check

0 steps flagged

No circularity found: IndexSoftmax is a direct LUT construction with empirical benchmarks, not a reduction to its own inputs.

full rationale

The paper's central derivation is the construction of IndexSoftmax as a fixed lookup-table approximation to exp over the clipped interval [0,c], with integer normalization. The LUT entries are defined directly from exp (Eq. 10), the index mapping is a deterministic rescaling of clipped integer distances (Eq. 11), and normalization is integer fixed-point scaling (Eq. 15). There is no step where a fitted parameter is renamed as a prediction: hyperparameters (b,c) are chosen by a validation sweep (Section 4.4) and then applied across models, which is standard hyperparameter selection rather than a construction that forces the reported accuracy. Accuracy and speed claims are empirical measurements against FP16/INT8 baselines, not consequences of the LUT definition. No self-citation is load-bearing; references to prior integer-softmax works (I-BERT, I-ViT, I-LLM, EXAQ, TurboAttention) are comparisons, not justifications of this paper's results. The only methodological caveat is that the hyperparameter sweep used the same benchmarks later reported (LLaMA-3.2-1B/WikiText and DeiT-B/ImageNet-1K), which can mildly inflate apparent fidelity on those datasets, but this is a validation concern, not circularity. Thus no circular step is established.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The central technical components (LUT softmax approximation, integer clipping, fixed-point normalization) are all software constructs, no new physical entities. The main free parameters are the clipping threshold and LUT size, chosen by validation; the main load-bearing premises are the empirical transferability of these choices and the fairness of the baseline pipelines.

free parameters (2)
  • clipping threshold c = 6.6
    Chosen from a joint sweep on LLaMA-3.2-1B WikiText and DeiT-B ImageNet (Fig. 9); controls the truncation range of the exponential and affects how many logits are saturated.
  • LUT resolution b = 5 (32 entries)
    Chosen from the same sweep; higher b improves approximation but increases table size; the paper fixes b=5 for all experiments.
axioms (4)
  • domain assumption Attention logits after INT8 quantization have a distribution such that distances greater than c≈6.6 contribute negligibly to softmax; a single fixed clipping threshold is transferable across models and sequence lengths.
    Empirical premise tested on 6 models; no formal characterization of logit distributions; if the logit spread changes, clipping may distort attention.
  • domain assumption The 32-entry piecewise-constant LUT plus UINT8 quantization and integer normalization has error small enough to preserve downstream model accuracy.
    No error bound; empirically validated on selected benchmarks; may degrade on other models or long-context distributions.
  • ad hoc to paper c_int = round(c / alpha) is positive and the division is well-defined for all inputs.
    Equation (8) and index mapping in Eq. (11) divide by c_int; the paper does not handle the case c_int = 0, which could occur if alpha is large.
  • domain assumption The Arm Compute Library implementations of FP32/FP16/INT8 GEMMs used as baselines are representative of production kernels.
    The claimed speedups depend on these baselines; no kernel-level comparison or third-party validation is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 15038 in / 17310 out tokens · 147989 ms · 2026-08-03T19:58:03.845078+00:00 · methodology

0 comments
read the original abstract

Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax-related path as the dominant bottleneck. This stage incurs a costly dequantize -> softmax -> requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer attention pipeline that serves as a training-free drop-in replacement. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup table approximation, and direct integer normalization, thereby eliminating datatype conversion overhead along the attention path. Experiments on Armv8 CPUs show that our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines, and up to 2.0x speedup over conventional INT8 attention pipelines. Across diverse language and vision models, as well as additional reasoning and long-context evaluations, IntAttention maintains strong overall fidelity and demonstrates a more favorable trade-off than existing LUT-based softmax approximations. Code is available at https://github.com/WanliZhong/IntAttention

Figures

Figures reproduced from arXiv: 2511.21513 by Haibo Feng, Hanyang Peng, Shiqi Yu, Wanli Zhong, Zirui Zhou.

Figure 1
Figure 1. Figure 1: Comparison between conventional quantized attention and the proposed IntAttention, where IntAttention maintains an end-to-end integer dataflow from QK⊤ to PV. model to the performance level of 32B models on multi￾ple reasoning benchmarks (Li et al., 2025). This migration shifts inference from cloud servers to mobile and embedded processors, where end-to-end latency and energy efficiency become primary cons… view at source ↗
Figure 2
Figure 2. Figure 2: Breakdown of time share for the dequantize → softmax → requantize path across different precisions. Once GEMMs are accelerated to INT8, this path emerges as the domi￾nant latency and becomes the next optimization target. execution of matrix multiplications, and a sparsity-based Softmax approximation that avoids FP32 dequantization during exponentiation(Kang et al., 2025). These results confirm that the sof… view at source ↗
Figure 3
Figure 3. Figure 3: Overview of the proposed IntAttention pipeline. determines dynamic optimal clipping ranges to quantize attention scores to as low as 3 bits (Shkolnik et al., 2024). TurboAttention uses a small LUT for the integer part of the exponent and a 3rd-order polynomial for the fractional part, plus sparsification of negligible exponentials (Kang et al., 2025). These techniques eliminate heavy floating-point ex￾pone… view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of the exponential activation in Softmax. Most logits lie in the near-zero region, where e x contributes neg￾ligibly to the normalization. Only a small subset of higher logits significantly affects the output distribution. wise max-subtraction for stability: ∆ˆ = rowmax(Aˆ ) − Aˆ , (7) which yields nonnegative distances from the dominant value in each row. We then use a quantization aligned cl… view at source ↗
Figure 5
Figure 5. Figure 5: IndexSoftmax achieves 4x higher LUT resolution under the same memory budget, enabling higher-fidelity exponential approximation without dynamic clipping or global statistics, which are costly on edge devices. our exponential lookup table is also quantized to UINT8, so that each entry is compact yet expressive enough to repre￾sent the clipped exponential curve. Over the clipped interval [0, c], the floating… view at source ↗
Figure 6
Figure 6. Figure 6: Speed comparison among different attention implemen￾tations on RK3588S2 across varying sequence lengths with headdim = 128. 1K 2K 4K 8K 16K Sequence Length 0 200 400 600 800 1000 Speed (GFLOP/s) 191 238 263 275 258 260 320 334 338 342 345 367 383 406 402 618 821 807 884 964 FP32 FP16 Quant Only IntAttention [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Normalized energy consumption per iteration across dif￾ferent precision settings, using FP16 as the baseline for compari￾son. 4.1 Experimental Setup Models. We evaluate IntAttention across both language and vision models to verify its generality. For language, we adopt LLaMA-3.2-1B (Dubey et al., 2024), OPT-1.3B (Zhang et al., 2022), and Qwen3-1.7B (Yang et al., 2025). For vision, we include DeiT-B-224 (To… view at source ↗
Figure 9
Figure 9. Figure 9: Hyperparameter sensitivity of IntAttention over LUT resolution b and clipping threshold c. Red indicates noticeable degradation (> 1 PPL or > 0.3% Top-1), while green denotes high-fidelity regions [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 7 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Gao, J., and Choi, Y

    Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  3. [3]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    Pith/arXiv arXiv 2018

  4. [4]

    Flashattention: Fast and memory-efficient exact attention with io-awareness

    Dao, T., Fu, D., Ermon, S., Rudra, A., and R \'e , C. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35: 0 16344--16359, 2022 a

    2022

  5. [5]

    Y., Ermon, S., Rudra, A., and Re, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. Flashattention: Fast and memory-efficient exact attention with IO -awareness. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022 b

    2022

  6. [6]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  7. [7]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in neural information processing systems, 35: 0 30318--30332, 2022

    2022

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021

    2021

  9. [9]

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Rozière, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra...

    Pith/arXiv arXiv doi:10.48550/arxiv.2407.21783 2024

  10. [10]

    The language model evaluation harness, 07 2024

    Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac'h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. The language model evaluation harness, 07 2024. URL https://zenodo.or...

    arXiv 2024

  11. [11]

    I-llm: Efficient integer-only inference for fully-quantized low-bit large language models

    Hu, X., Cheng, Y., Yang, D., Yuan, Z., Yu, J., Xu, C., and Zhou, S. I-llm: Efficient integer-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405.17849, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 2704--2713, 2018

    2018

  13. [13]

    Turboattention: Efficient attention approximation for high throughputs llm

    Kang, H., Bharadwaj, S., Hensman, J., Krishna, T., R \"u hle, V., and Rajmohan, S. Turboattention: Efficient attention approximation for high throughputs llm. In Eighth Conference on Machine Learning and Systems, 2025

    2025

  14. [14]

    W., and Keutzer, K

    Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. In International conference on machine learning, pp.\ 5506--5518. PMLR, 2021

    2021

  15. [15]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pp.\ 611--626, 2023

    2023

  16. [16]

    Questa: Expanding reasoning capacity in llms via question augmentation

    Li, J., Lin, H., Lu, H., Wen, K., Yang, Z., Gao, J., Wu, Y., and Zhang, J. Questa: Expanding reasoning capacity in llms via question augmentation. arXiv preprint arXiv:2507.13266, 2025

    arXiv 2025

  17. [17]

    and Gu, Q

    Li, Z. and Gu, Q. I-vit: Integer-only quantization for efficient vision transformer inference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17065--17075, 2023

    2023

  18. [18]

    Consmax: Hardware-friendly alternative softmax with learnable parameters

    Liu, S., Tao, G., Zou, Y., Chow, D., Fan, Z., Lei, K., Pan, B., Sylvester, D., Kielian, G., and Saligane, M. Consmax: Hardware-friendly alternative softmax with learnable parameters. In Proceedings of the 43rd IEEE/ACM International Conference on Computer-Aided Design, pp.\ 1--9, 2024

    2024

  19. [19]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017

    2017

  20. [20]

    Fp8 formats for deep learning

    Micikevicius, P., Stosic, D., Burgess, N., Cornea, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.05433, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    The lambada dataset: Word prediction requiring a broad discourse context

    Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fern \'a ndez, R. The lambada dataset: Word prediction requiring a broad discourse context. In 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016-Long Papers, volume 3, pp.\ 1525--1534. Association for Computational Lin...

    2016

  22. [22]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.\ 8748--8763. PmLR, 2021

    2021

  23. [23]

    L., Bhagavatula, C., and Choi, Y

    Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

    2021

  24. [24]

    Flashattention-3: Fast and accurate attention with asynchrony and low-precision

    Shah, J., Bikshandi, G., Zhang, Y., Thakkar, V., Ramani, P., and Dao, T. Flashattention-3: Fast and accurate attention with asynchrony and low-precision. Advances in Neural Information Processing Systems, 37: 0 68658--68685, 2024

    2024

  25. [25]

    Shkolnik, M., Fishman, M., Chmiel, B., Ben-Yaacov, H., Banner, R., and Levy, K. Y. EXAQ : Exponent aware quantization for LLM s acceleration. In Workshop on Machine Learning and Compression, NeurIPS 2024, 2024

    2024

  26. [26]

    R., Venkatesan, R., Dai, S., Khailany, B., and Raghunathan, A

    Stevens, J. R., Venkatesan, R., Dai, S., Khailany, B., and Raghunathan, A. Softermax: Hardware/software co-design of an efficient softmax for transformers. In 2021 58th ACM/IEEE Design Automation Conference (DAC), pp.\ 469--474. IEEE, 2021

    2021

  27. [27]

    Gemma 3 technical report

    Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ram \'e , A., Rivi \`e re, M., et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021 a

    2021

  29. [29]

    Going deeper with image transformers

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and J \'e gou, H. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 32--42, 2021 b

    2021

  30. [30]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  31. [31]

    Qwen3 technical report

    Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  33. [33]

    Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization

    Zhang, J., Huang, H., Zhang, P., Wei, J., Zhu, J., and Chen, J. Sageattention2: Efficient attention with thorough outlier smoothing and per-thread int4 quantization. In International Conference on Machine Learning (ICML), 2025 a

    2025

  34. [34]

    Sageattention3: Microscaling FP 4 attention for inference and an exploration of 8-bit training

    Zhang, J., Wei, J., Zhang, P., Xu, X., Huang, H., Wang, H., Jiang, K., Zhu, J., and Chen, J. Sageattention3: Microscaling FP 4 attention for inference and an exploration of 8-bit training. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025 b . URL https://openreview.net/forum?id=JbJVWljk7r

    2025

  35. [35]

    Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration

    Zhang, J., Wei, J., Zhang, P., Zhu, J., and Chen, J. Sageattention: Accurate 8-bit attention for plug-and-play inference acceleration. In International Conference on Learning Representations (ICLR), 2025 c

    2025

  36. [36]

    V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P

    Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/2205.01068

    Pith/arXiv arXiv 2022