{"id":"1474feff-28f4-4bc4-a3f1-3fdd217bfc73","arxiv_id":"2511.21513","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A lookup-table-based integer softmax replaces the floating-point softmax in quantized transformer attention, delivering up to 3.7x speedup and 61% energy reduction on ARMv8 CPUs with roughly unchanged accuracy.","lead":"IntAttention replaces the floating-point softmax in quantized transformer attention with a lookup table and integer normalization, keeping the whole attention path in 8/32-bit integer arithmetic. On ARMv8 edge chips the authors measure up to 3.7x lower attention latency and 61% lower energy than FP16, with small accuracy changes on language and vision models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline fairness and absence of released code make the 2.0–3.7x speed/energy claims unverifiable; requires third-party replication with an optimized baseline.","rationale":"The reader's weakest assumption is baseline fairness; I agree. This is the most load-bearing point because all central efficiency claims are relative. The accuracy tables are present and largely favorable, but the speed/energy numbers are the headline and they rest on the unstated quality of the baseline implementation. The abstract/full-text code contradiction is an explicit missing-support flag; as of submission there is no way to independently verify the kernels. I also note the Qwen3 result: IntAttention's average is 55.02 vs FP16 59.49 (a 4.5pt drop), which weakens the 'comparable' wording, but the paper explicitly attributes remaining gaps to Q/K/V quantization rather than softmax, and Table 3 shows IndexSoftmax alone matches FP16; so this does not change the verdict. The proposed test—replacing the baseline softmax path with an optimized fused implementation and re-measuring—would settle whether the 2.0x speedup is real. If it survives, the main engineering claim is credible; if not, the paper's usefulness is substantially reduced. Verdict remains CONDITIONAL/UNCHANGED.","tokens_in":15327,"tokens_out":11602,"duration_ms":113290,"concrete_test":"Re-run the Section 4.2 benchmark after obtaining the code, using identical ACL INT8 GEMM kernels for both pipelines and replacing the baseline dequant→softmax→requant path with Arm Compute Library's optimized CpuSoftmax/CpuCast (or XNNPACK's fused softmax). Measure at L=1K, 4K, 16K on RK3588S2, 10 trials each, and report mean ± std. If the IntAttention-to-Quant-Only speedup drops below 1.5x (or overlaps the noise), the 2.0x claim is a baseline artifact; if it stays above 1.8x, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that IntAttention is a training-free drop-in replacement that is up to 2.0x faster than conventional INT8 attention and up to 3.7x faster than FP16, with 61% energy reduction. The decisive condition is that the 'Quant Only' baseline (Section 4.1) implements the dequant→softmax→requant path at production quality, because the speedup is computed relative to that baseline. The manuscript gives no kernel-level description of this baseline, no variance/error bars, and no comparison against an established quantized attention kernel beyond Arm Compute Library. The abstract says code is available at github.com/WanliZhong/IntAttention, but Section 1 says 'Code will be released in later version of this work' — so the implementation cannot currently be inspected. If the baseline softmax is a scalar FP32 loop or a naive composition of ACL kernels, the 58–65% softmax share in Figure 2 and the resulting 2.0x speedup are inflated. A production fused softmax/QDQ baseline could reduce the apparent gain substantially. Thus the headline efficiency numbers are not yet established by the evidence in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IntAttention, a fully integer attention pipeline for edge inference that eliminates the dequantize→softmax→requantize path in quantized Transformers. The core operator, IndexSoftmax, uses integer-domain max-subtraction and sparsity-aware clipping followed by a fixed 32-entry UINT8 lookup table to approximate the exponential, with integer row-sum normalization and UINT8 quantization of the attention probability matrix. The method is training-free and intended as a drop-in replacement for conventional quantized attention. Experiments on Armv8 CPUs (RK3588S2 and Apple M2) report up to 3.7× speedup and 61% energy reduction over FP16 attention, up to 2.0× speedup over an INT8 'Quant Only' pipeline, and accuracy comparable to baselines on several language and vision models. The paper also presents hyperparameter sensitivity analysis and ablations against EXAQ-based LUT softmax approximations.","tokens_in":15629,"tokens_out":3640,"duration_ms":40538,"significance":"If the reported efficiency gains are reproducible, the work addresses a real bottleneck in quantized Transformer inference on edge devices: the softmax path dominates once GEMMs are quantized. The proposed design is simple, portable (uses only integer add/multiply/shift/lookup), and does not require retraining or per-input statistics, which are practical advantages. The paper includes useful ablations: comparison with EXAQ variants, sensitivity to the two hyperparameters (c and b), and a comparison of INT8 vs UINT8 probability quantization. The breadth of models and benchmarks is a strength. However, the central efficiency claims currently rest on an incompletely specified baseline, and the accuracy comparison shows a notable degradation on Qwen3-1.7B that is not fully reflected in the paper's summary claims.","major_comments":[{"comment":"The headline speedup and energy claims are unverifiable without a precise description of the baselines. The 'Quant Only' pipeline is said to be a conventional INT8 attention pipeline, but the manuscript does not state whether the softmax/QDQ path is a scalar FP32 loop, a fused kernel, or an ACL composition, nor does it report kernel-level implementation details, thread configuration, or measurement variance. Without error bars or standard deviations, the reported 'up to 3.7x' and 'up to 2.0x' figures could be inflated by a suboptimal baseline. Additionally, the abstract states code is available, while Section 1 says 'Code will be released in later version of this work.' This contradiction prevents independent replication. The authors should describe the baseline kernels precisely, provide error bars over multiple runs, and either release code or provide enough detail for exact reproducti","section":"§4.1/§4.2, Figures 6–8"},{"comment":"The claim that IntAttention maintains accuracy comparable to baselines is weakened by the Qwen3-1.7B results. The average accuracy for IntAttention is 55.02%, versus 59.49% for FP16 and 55.12% for Quant Only. On HellaSwag, PIQA, WinoGrande, and ARC-Challenge, IntAttention is numerically below FP16 by 2–6 points, and on average it is not better than Quant Only. The text in §4.3 says IntAttention 'narrows the gap' and 'yields a clear perplexity gain on WikiText,' but the average accuracy does not improve over Quant Only. This should be acknowledged explicitly, and the authors should either improve the method for this model or temper the claim that accuracy is 'comparable to baseline' across diverse language models.","section":"Table 1, §4.3"},{"comment":"The integer normalization in Equation (15) uses rowSum(Ē), where Ē is a UINT8-quantized lookup table. Since the LUT is quantized to 8 bits, small exponential values, especially those near the clipping boundary, are rounded to zero. For c=6.6, exp(-6.6)*255 ≈ 0.35, so a range of entries become exactly zero. This is presumably intentional sparsity, but the error analysis is not provided. The paper should quantify the approximation error of this combined LUT quantization and integer normalization, and show its effect on the final attention output, particularly for long sequences where many logits may be near the clipping threshold.","section":"Equation (15) and §3.2"}],"minor_comments":[{"comment":"The abstract says 'Code is available at https://github.com/WanliZhong/IntAttention,' but Section 1 says 'Code will be released in later version of this work.' Please clarify the actual availability and, in the meantime, avoid claiming public availability if the repository is not yet accessible.","section":"Abstract vs. Section 1"},{"comment":"The notation is inconsistent: 'Quant Only' appears in figures and tables, while the text also uses 'Quantized-Only' and 'Quantization-only.' Please standardize.","section":"§4.1"},{"comment":"The phrase 'studied in Table 4.4' appears to be a typo; there is no Table 4.4, and the relevant discussion is in Section 4.2/Figure 2.","section":"§4.2"},{"comment":"The title 'LUT Rebuild and Integer Scale Normalization' is confusing; the LUT is not rebuilt at runtime. Rename to 'LUT-based Integer Normalization' or similar.","section":"§3.2 title"},{"comment":"The figures lack error bars and statistical significance. Even a brief statement of run-to-run variance would help support the quantitative claims.","section":"Figures 2, 6, 7"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the paper is generally well organized, but the efficiency claims are currently too underspecified to be accepted. The most important fix is to make the baselines and code available; without that, the 2.0–3.7× speedup and 61% energy reduction claims cannot be assessed by reviewers or readers. The Qwen3 accuracy degradation should also be handled honestly, either by improving the method or by reducing the scope of the accuracy claim. I would not reject the paper — the method is plausible and the ablation set is useful — but it needs substantial revision and independent verification before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I think this is a solid systems contribution that deserves a serious referee. The core idea is straightforward: replace the dequantize->softmax->requantize path in quantized attention with a fixed 32-entry lookup-table softmax approximation, integer normalization, and UINT8 probability storage, all without retraining or calibration. The measured speedups (2.1-3.7x over FP16, 1.6-2.4x over an INT8 quant-only pipeline, 61% energy reduction on RK3588S2) are consistent across six models and two ARMv8 platforms, and the ablations against EXAQ are fair. The hyperparameter sensitivity analysis shows a stable plateau around (b=5, c=6.6), so the method isn't a knife's edge.\n\nThe main soft spot is verification. The abstract promises code at a GitHub URL, but the body says code will be released later. That's a discrepancy that has to be fixed. The baseline pipelines are not described at kernel level, and there are no error bars on the latency/energy numbers; energy is measured on one board. If the 'Quant Only' baseline softmax is a naive scalar FP32 loop, the 2x speedup over it would be inflated. I don't think the central claim is wrong (the consistency of speedups across two ISAs and the large softmax time share in Figure 2 make the effect plausible), but the magnitude needs third-party confirmation with optimized baselines. On the accuracy side, Qwen3 degrades noticeably relative to FP16, so calling that 'strong fidelity' is slightly generous; still, IntAttention improves on Quant-Only for Qwen3, so it's not a regression caused by the operator.\n\nThe writing is clear, the related work is handled honestly, and the method is genuinely training-free, unlike I-BERT/I-ViT/I-LLM which need QAT or calibration. The 'first fully integer' claim is defensible once you add 'training-free' and 'without calibration.'\n\nI'd send this to peer review. Reviewers should push for a clean code release, kernel-level descriptions of all baselines, and repeated measurements with error bars. If those appear, this becomes a very usable paper for people doing edge Transformer inference.","headline":"A well-engineered, training-free integer attention pipeline with surprisingly large speedups; worth reviewing, but code release and baseline details need to be conditions of acceptance.","tokens_in":16089,"tokens_out":2735,"would_cite":true,"duration_ms":27636,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a fully integer attention pipeline, built around IndexSoftmax — a 32-entry lookup-table softmax replacement — eliminates the dequantize-to-softmax-to-requantize bottleneck in quantized attention, delivering up to 3.7x","keywords":["integer attention","softmax approximation","lookup table","quantization","edge inference","transformer","INT8","energy efficiency"],"falsifier":"Benchmark IntAttention against a carefully optimized INT8 attention implementation with a fused dequant-softmax-requant kernel on the same Armv8 CPUs; if the 2.0x speedup over INT8 shrinks substantially, the central claim is inflated. Separately, evaluate the same models on a long-context task (16k+ tokens) against FP16; if perplexity or reasoning accuracy degrades by more than the small margins reported at 1k–16k lengths, the fidelity claim fails.","tokens_in":15238,"feed_emoji":"⚡","tokens_out":3922,"duration_ms":41347,"temperature":0.7,"pith_summary":"This paper claims that once transformer matrix multiplications are quantized to INT8, the softmax path — with its dequantize, floating-point exponential, normalize, and requantize steps — becomes the dominant attention bottleneck, consuming up to 65% of latency on edge CPUs. To remove it, the authors propose IntAttention, a fully integer attention pipeline whose core operator IndexSoftmax replaces floating-point exponentials with integer clipping, a 32-entry lookup table, and integer row-sum normalization. The result is a training-free, drop-in replacement for conventional quantized attention that reportedly runs up to 3.7x faster than FP16 and 2.0x faster than a standard INT8 pipeline on Armv8 CPUs, while using 61% less energy and matching baseline accuracy across several language and vision models. If true, it means the costly datatype-conversion detour can be removed entirely, restoring an end-to-end integer dataflow for edge inference.","feed_headline":"Integer-only softmax cuts attention latency up to 3.7x on edge CPUs","feed_subtitle":"A 32-entry lookup table replaces floating-point softmax, removing the costliest detour in quantized attention without retraining.","key_machinery":"IndexSoftmax — a three-stage integer operator: (1) sparsity-aware clipping in the integer domain, where logit distances are saturated at a quantization-aligned threshold c_int; (2) a fixed 32-entry UINT8 lookup table that maps clipped indices to approximate exp(-x) values; (3) integer row-sum normalization with UINT8 scaling to produce attention probabilities. The LUT occupies about 32 bytes and, unlike prior LUT-based softmax methods (e.g., EXAQ), uses a fixed clipping threshold (c≈6.6) and a moderate resolution (b=5), avoiding per-tensor dynamic statistics and global reductions that are costly on edge processors.","core_discovery":"IntAttention is a fully integer attention pipeline that runs entirely in the integer domain from the QK^T logits to the PV output. Its key operator, IndexSoftmax, performs row-wise max-subtraction in integers, clips small logit distances using a threshold derived from the quantization scale, approximates the exponential with a fixed 32-entry UINT8 lookup table, and normalizes the result with integer arithmetic to produce a UINT8 attention probability matrix. The paper claims this is the first fully integer, plug-and-play attention pipeline that requires no retraining, and that it achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and up to 2.0x over conventional INT8 qu","pith_inferences":["The fixed 32-entry LUT is essentially a universal exponential surrogate for softmax on quantized logits; it would be worth testing whether it holds for much longer contexts (e.g., 32k tokens) and larger models, where attention distributions can become sparser or flatter.","Because the clipping step already prunes near-zero exponentials, the method could combine with sparse or variable-length GEMM support on edge hardware to skip memory traffic entirely, not just LUT lookups.","The paper suggests combining IntAttention with per-block or per-channel quantization and input smoothing; this is a natural next experiment that could close the residual accuracy gap seen on Qwen3.","The reported speedups depend on the fairness of the FP16 and INT8 'quant-only' baseline implementations; an independent, production-grade re-implementation of those baselines would be the most direct way to confirm the 2.0–3.7x claims."],"forward_implications":["Existing INT8 quantized inference stacks can replace their softmax stage with IndexSoftmax without retraining or calibration, potentially recovering accuracy on most models while removing the dequant/requant detour.","The attention bottleneck shifts back to QK and PV GEMMs, so further speedups on edge devices should come from optimizing integer matrix multiplication kernels rather than the softmax path.","The measured 61% energy reduction makes fully integer attention practical for battery-powered transformer deployment on edge hardware.","The unsigned UINT8 format for attention probabilities is measurably more faithful than signed INT8 (higher cosine similarity, lower L1/RMSE), which could influence quantization scheme design beyond this paper.","The robustness of the fixed clipping threshold and LUT resolution across language and vision models suggests an integer softmax surrogate that transfers without per-model tuning."],"fun_headline_variants":["Integer-only softmax: 3.7x faster attention on edge","No FP32 detour: IntAttention speeds edge inference 3.7x","Drop-in integer attention cuts latency and energy","IntAttention: fully integer pipeline, 3.7x faster than FP16","Replacing FP softmax with integers: 3.7x edge speedup"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline 2.0–3.7x speedups rest on the assumption that the FP16 and INT8 'quant-only' baseline pipelines are fairly and well-optimized implementations of conventional attention; the paper gives no kernel-level detail, variance, or comparison against widely deployed production kernels for those baselines.","fun_headline_variants_meta":{"raw":{"variants":["Integer-only softmax: 3.7x faster attention on edge","No FP32 detour: IntAttention speeds edge inference 3.7x","Drop-in integer attention cuts latency and energy","IntAttention: fully integer pipeline, 3.7x faster than FP16","Replacing FP softmax with integers: 3.7x edge speedup"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1251,"prompt_tokens":797,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":541,"tokens_out":454,"duration_ms":5004,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T19:58:03.845078+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Benchmark IntAttention against a carefully optimized INT8 attention implementation with a fused dequant-softmax-requant kernel on the same Armv8 CPUs; if the 2.0x speedup over INT8 shrinks substantially, the central claim is inflated. Separately, evaluate the same models on a long-context task (16k+ tokens) against FP16; if perplexity or reasoning accuracy degrades by more than the small margins reported at 1k–16k lengths, the fidelity claim fails.","supporting_citations":[],"review_version":1}