Pith. sign in

REVIEW 4 major objections 5 minor 63 references

All LLM activations, including attention, can use block floating point without meaningful accuracy loss—cutting the KV cache to ~31% and enabling faster, more efficient inference accelerators.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:31 UTC pith:BL5OZJSC

load-bearing objection A credible BFP-everywhere accelerator with honest RTL details, but the KV bit-allocation premise and the unverified full-precision baselines keep the headline numbers from being fully convincing. the 4 major comments →

arxiv 2602.04595 v3 pith:BL5OZJSC submitted 2026-02-04 cs.AR

Harmonia: Algorithm-Hardware Co-Design for Memory- and Compute-Efficient BFP-based LLM Inference

Xinyu Wang , Jieyu Li , Yanan Sun , Weifeng He This is my paper
classification cs.AR
keywords block floating pointKV cache compressionLLM inferenceattention layersasymmetric bit allocationoutlier smoothinghardware co-designactivation quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Harmonia is an algorithm-hardware co-design that tries to make every activation in an LLM—including the attention mechanism's keys, values, and scores—use block floating point (BFP) instead of FP16. Prior BFP accelerators stopped at linear layers because attention suffered accuracy loss; Harmonia argues that with a group size of 32 and 8-bit mantissas, all-layer BFP is safe, and that the KV cache can be pushed to 4-bit mantissas for most tokens without meaningful degradation. Two algorithmic tricks carry the argument: an asymmetric bit allocation that keeps 8-bit mantissas for the first 32 and most recent 64 tokens, and a hybrid offline-online outlier smoother that prevents a few large key channels from wrecking the shared exponent. On LongBench, the accuracy loss averages about 0.3% against a weight-only-quantized baseline, while the KV cache shrinks to 31.25% of its FP16 size. The companion hardware—a reconfigurable PE array, a real-time FP16-to-BFP converter, and a tiling-aware dataflow—turns that compression into reported average gains of 3.84x area efficiency, 2.03x energy efficiency, and 3.08x speedup over prior accelerators.

Core claim

The paper's central discovery is that block floating point (BFP)—where a group of values shares one exponent and each value keeps a mantissa—can be extended from linear layers to the entire Transformer, including attention, without the accuracy collapse earlier designs assumed was unavoidable. With a group size of 32, an 8-bit mantissa for all non-KV activations, and an asymmetric KV-cache allocation that gives 8-bit mantissas only to the first 32 and most recent 64 tokens while the rest receive 4 bits, Harmonia reports an average accuracy drop of about 0.3% on LongBench across five instruction-tuned models, and near-lossless perplexity on WikiText2 at 8-bit KV precision. The KV cache is the

What carries the argument

The central object is block floating point (BFP): a number format in which a group of values (here, group size 32) shares a single 5-bit exponent, and each value stores only its mantissa, so the multiply-accumulate work becomes integer arithmetic with no per-element exponent alignment. Three mechanisms carry the argument. First, an asymmetric bit-allocation strategy assigns 8-bit mantissas to the KV-cache tokens that attention scores show matter most—the first 32 and most recent 64 tokens—and 4-bit mantissas to everything else, cutting KV-cache storage to 31.25% of FP16. Second, a hybrid offline-online outlier smoother suppresses channel-wise outliers in K: per-channel scaling factors are le

Load-bearing premise

The whole accuracy story rests on the assumption that the KV-cache tokens that need 8-bit mantissas are always the first 32 and most recent 64 tokens—a pattern observed in short 20-token attention measurements and then applied to 4K-token contexts and all eight evaluated models.

What would settle it

Run Harmonia's KV-cache compression on a long-context retrieval task where a key fact is deliberately placed in the middle of a 4K-token document (e.g., a needle-in-a-haystack test with the needle at position 2000). If the 4-bit region causes accuracy to drop comparably to uniformly quantized 4-bit KV, or by more than the reported ~0.3%, the asymmetric allocation premise is falsified for that workload.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Attention layers no longer need FP16 storage or FP-FP arithmetic: the same integer-style MAC units serve both linear and attention layers, so hardware utilization stays high across short- and long-context workloads.
  • KV-cache memory traffic, the dominant bottleneck in decode, drops to 31.25% of FP16 size, directly relieving the memory-bandwidth pressure that limits generation throughput.
  • The conversion is fully online and requires no retraining: only the per-channel smoothing factors need calibration, so the approach applies to already-deployed weight-only quantized models.
  • BFP-based KV compression is orthogonal to eviction and sliding-window cache techniques, so it can combine with those methods for further memory reduction.
  • If the asymmetric allocation holds generally, aggressive 4-bit KV-cache compression becomes practical on the majority of tokens, not just a research curiosity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The asymmetric allocation's premise—that attention always concentrates on the first 32 and last 64 tokens—is drawn from 20-token attention measurements on one model family; a natural extension would be to measure attention concentration on longer, non-conversational retrieval documents and, if needed, make the 8-bit region dynamic rather than fixed.
  • The paper reports prefill-focused hardware gains; because decode is even more KV-cache-bound, one testable extension is to measure Harmonia's speedup and energy specifically in the decode stage, where the 31.25% cache reduction should produce outsized benefits.
  • The group size of 32 and the 8/4-bit split are chosen empirically rather than derived from an error bound; a tighter analytical link between KV-cache mantissa width and attention-score error could predict when 4-bit is safe and when a 5- or 6-bit fallback is needed.
  • The same BFP-plus-asymmetric-allocation recipe could be tried on other shared-exponent formats, such as microscaling, or on multimodal attention patterns where the initial/local concentration assumption may not hold.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Harmonia, an algorithm-hardware co-design framework that applies block floating point (BFP) to all LLM activations, including those in attention layers, with the KV cache compressed to 4-bit mantissas via an asymmetric bit-allocation scheme and an offline-online outlier-smoothing method. The authors evaluate accuracy on WikiText2 and LongBench across eight LLMs, and hardware efficiency using an RTL implementation of Harmonia synthesized at 28nm plus re-implemented baselines in a cycle-accurate simulator. The central claims are that the KV cache can be reduced to 31.25% of its original size with an average LongBench accuracy loss of only 0.3% relative to a weight-only-quantized baseline, and that the proposed architecture achieves 3.84x higher area efficiency, 2.03x better energy efficiency, and 3.08x speedup on average over prior accelerators.

Significance. If the results hold, this is a significant contribution to architecture-aware LLM inference: it extends BFP beyond linear layers into attention, aggressively compresses the KV cache, and backs the proposal with a full RTL-level implementation and synthesis. The paper's strengths include held-out LongBench evaluation, ablation of the proposed algorithmic components, a reconfigurable PE design that is clearly described, and hardware results from actual synthesis rather than an abstract cost model. However, the accuracy claims rest on the validity of the asymmetric KV-cache bit allocation, and the hardware claims rest on re-implemented baselines and an author-built simulator, with no artifacts or code released. The paper is promising but needs additional evidence and methodological tightening before it can be accepted.

major comments (4)
  1. [§III-B, Fig. 8, Table III] The asymmetric KV-cache allocation (8-bit mantissas for the first 32 and last 64 tokens, 4-bit elsewhere) is justified by attention-score evidence from only 128 length-20 sentences of Llama-3.1-8B (Fig. 8). This is a weak basis for applying the same fixed allocation to all eight models and to 4K-token LongBench contexts. The task-level results in Table III already show a consistent penalty on retrieval-heavy tasks: on multifieldqa, Harmonia is below Omniquant in all five models (e.g., 47.61 vs 48.95 for Llama-3.1-8B; 44.87 vs 46.97 for Llama-3.2-3B), indicating that 4-bit middle-token compression does degrade accuracy where distributed retrieval is required. Such task-level losses are masked by the 0.3% average. Please provide longer-context attention analysis and per-task or middle-token ablations to establish that the asymmetric allocation generalizes beyond the short conversational pa
  2. [§II-C, Table II] The BFP configuration (group size 32, 8-bit mantissas, 5-bit shared exponent) is selected by observing perplexity on WikiText2 in Figs. 5-6 and then validated on the same WikiText2 benchmark in Table II. This makes the reported near-lossless PPL results partially circular: they are not an independent test of the configuration. The LongBench results are independent, but the configuration itself is selected on the same distribution that is later used for validation. Please use a calibration split, or report selection on one benchmark and validation on another, to break the circularity.
  3. [§V-A, §V-D] The hardware gains (e.g., 3.08x average speedup, 3.84x area efficiency) are computed against accelerator baselines that are 're-implemented and synthesized under the same technology node' and evaluated in a cycle-accurate simulator built on ANT and DNNWeaver. No code, netlists, or configuration details are released, so the baseline implementations cannot be independently checked. Please release the artifacts or provide validation of the simulator against published FIGNA, Anda, or M-ANT results, and itemize the design choices made in re-implementing these baselines. Without this, the hardware comparisons are not reproducible.
  4. [Table III] All LongBench numbers are reported as single runs without error bars or statistical significance. Given that some Harmonia scores exceed the Omniquant baseline by small margins (e.g., Llama-3.2-3B 44.11 vs 44.07; Llama-3.1-8B 46.74 vs 46.35), it is unclear whether the observed 0.3% average difference is meaningful or within run-to-run noise. Please report multiple runs with standard deviations, or explain why the evaluation protocol is deterministic enough that error bars are unnecessary.
minor comments (5)
  1. [§II-B] Typo: 'correspongding' should be 'corresponding'.
  2. [§III-C, Eq. (3)] The notation F(W, Convert_BFP(X); S) is not fully defined. Clarify whether S enters through the Q/K linear-layer weight rescaling, what Convert_BFP includes, and how the MSE objective is computed over calibration data.
  3. [§III-C] The online K-offset step sets each selected channel offset to 'half of' the maximum absolute value in the initial 32-token window. The choice of the factor 0.5 is not justified; please provide a derivation, an ablation, or a reference.
  4. [Abstract / Table III caption] The abstract says 'less than 1% average accuracy loss', while the body reports 0.3% relative to the weight-only quantization baseline. State clearly that the comparison baseline is Omniquant, not full precision, and consider reporting the full-precision gap as well.
  5. [Fig. 8] Please specify the exact context length, the number of heads averaged, and how 'initial-token attentive' and 'local-token attentive' are defined; the caption currently gives only 'average attention scores'.

Circularity Check

1 steps flagged

Partial circularity: BFP/KV configurations selected on WikiText2 are then validated on the same WikiText2 PPL; LongBench and hardware claims remain held out.

specific steps
  1. fitted input called prediction [Sec. II-C (Figs. 5-6) and Table II]
    "To assess the impact of different BFP configurations on model accuracy, we adopt perplexity on the WikiText2 dataset [45] as the evaluation metric... Using 8-bit mantissas keeps the accuracy degradation around 1% for most models, which we consider an acceptable loss. Therefore, all subsequent analyses and experiments adopt 8-bit mantissas for activations except those in the KV cache."

    The BFP configuration (group size 32, 8-bit mantissas; later KV 4-bit) is selected by reading WikiText2 perplexity/relative-accuracy curves in Figs. 5-6. Table II then reports WikiText2 perplexity for these exact configurations ('Harmonia BFP8 INT4 BFP8 BFP8 5.78...') as evidence that the method is near-lossless. The reported accuracy is the selection criterion itself, so Table II does not independently confirm the configuration; it restates the input. The LongBench and hardware results are held out and independent, so this is partial, not central.

full rationale

The central derivation chain is not circular. The BFP configuration and KV bit widths are selected from WikiText2 PPL curves in Sec. II-C, and the same WikiText2 PPL values reappear in Table II as evidence; this is a selection-validation loop for that table, but it is not the paper's main accuracy claim. The headline LongBench results (0.3% average drop) are on a separate benchmark not used for configuration selection, the offline scaling S is calibrated on held-out blocks with evaluation on LongBench, and all hardware numbers come from independent RTL synthesis and cycle-accurate simulation against re-implemented baselines. There are no load-bearing self-citations, imported uniqueness theorems, or ansatz-via-citation moves. The 20-token attention-pattern basis for the asymmetric KV allocation is an extrapolation concern (external validity), not circularity. Score reflects the one partial same-benchmark selection/validation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

Central claims rest on several empirical tunables: BFP configuration, KV bit width, asymmetric window sizes, learned per-channel scales, and the top-k offset rule. These are chosen from data rather than derived from first principles. The hardware comparison also assumes faithful baseline re-implementations. No new physical entities are introduced.

free parameters (5)
  • BFP configuration (group size, mantissa bits, shared exponent) = group_size=32, mantissa_bits=8, shared_exponent_bits=5
    Chosen empirically after Fig. 5 to keep WikiText2 relative accuracy near 1%; no analytical derivation.
  • Asymmetric bit-allocation window sizes = initial_tokens=32, local_tokens=64
    Hand-picked from attention-score analysis on 128 length-20 sentences of Llama-3.1-8B (Fig. 8); applied unchanged to all models.
  • Per-channel K scaling factors S = C-dimensional vector learned on calibration set (Eq. 3)
    Learned by MSE minimization between full and quantized block outputs; transferability to evaluation benchmarks is assumed.
  • Online K-offset magnitude = half of max absolute value in first 32-token window; top-k channel count k not disclosed
    Empirical rule from Sec III-C; k not specified in text, so the rule is not fully specified.
  • KV cache mantissa target = 4 bits for non-window tokens
    Aggressive compression target selected from Fig. 6 and Table II/III; the reported accuracy depends on this choice.
axioms (5)
  • domain assumption Attention scores concentrate on initial and most recent tokens (attention-sink/local pattern)
    Inferred from Fig. 8 measured on 128 sentences of length 20 for Llama-3.1-8B; the whole 4-bit KV accuracy story depends on this pattern holding across models and 4K contexts.
  • domain assumption Calibration-learned scales S and online offsets transfer to evaluation benchmarks
    Eq. (3) minimizes MSE on calibration data; no distribution-shift analysis is provided.
  • standard math Softmax shift-invariance used for K-offset (subtracting a per-channel offset from K does not change attention scores)
    True mathematical property; used in Sec III-C.
  • domain assumption Hidden dimension is an integer multiple of the BFP group size (32) for all evaluated layers
    Required for per-token grouping without partial groups; holds for the tested models but is not guaranteed.
  • domain assumption Re-implemented baseline accelerators are representative and fair
    All baselines re-synthesized by the authors; no public artifacts to audit; results could be sensitive to implementation choices.

pith-pipeline@v1.3.0-alltime-deepseek · 17975 in / 17661 out tokens · 172479 ms · 2026-08-03T04:31:36.089019+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) incur substantial memory and computation costs. Prior works reduce FP-INT arithmetic overhead by converting linear-layer activations to block floating point (BFP), but retain FP activations in attention layers due to accuracy concerns. We propose Harmonia, an algorithm-hardware co-design framework that enables BFP representation and computation across both linear and attention layers. Harmonia first explores BFP configurations to balance model accuracy and activation compression. It then combines asymmetric bit allocation with hybrid offline-online outlier smoothing to compress the KV cache from FP16 to 4-bit-mantissa BFP with less than 1% average accuracy loss on LongBench. To efficiently support all-layer BFP computation, Harmonia incorporates a reconfigurable PE unit for mixed data formats and precisions, a real-time FP16-to-BFP converter, and a flexible tiling-aware dataflow that reduces external memory traffic. Evaluations across eight widely used LLMs show that Harmonia achieves 3.84x higher area efficiency, 2.03x better energy efficiency, and 3.08x speedup on average, with maximum improvements of 5.05x, 3.90x, and 4.62x, respectively.

Figures

Figures reproduced from arXiv: 2602.04595 by Jieyu Li, Weifeng He, Xinyu Wang, Yanan Sun.

Figure 1
Figure 1. Figure 1: Comparison of different compute schemes: (a) customized FP-INT [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of the weight-only quantized LLM architecture. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of (a) the FP16-to-BFP numerical conversion procedure, [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Relative accuracy of different models under various preserved mantissa [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relative accuracy of different models under various preserved mantissa [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Average attention scores in Llama-3.1-8B over 128 sentences, each [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Magnitude distribution of values in the KV cache of Llama-3.1-8B. [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Effect of outlier suppression using the offline-online hybrid [PITH_FULL_IMAGE:figures/full_fig_p006_11.png] view at source ↗
Figure 14
Figure 14. Figure 14: Illustration of (a) output dataflow of the PE array, (b) two data [PITH_FULL_IMAGE:figures/full_fig_p007_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Configurable output dataflows: column-first and row-first modes. [PITH_FULL_IMAGE:figures/full_fig_p008_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Area efficiency and energy efficiency comparison across accelerators in (a) linear-layer execution and (b) joint linear- and attention-layer execution. [PITH_FULL_IMAGE:figures/full_fig_p010_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: PE-level comparison of area efficiency and energy efficiency, with [PITH_FULL_IMAGE:figures/full_fig_p010_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Speedup and energy breakdown comparison across accelerators in (a) linear-layer execution and (b) joint linear- and attention-layer execution. [PITH_FULL_IMAGE:figures/full_fig_p011_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Speedup and energy comparison across accelerators under varying [PITH_FULL_IMAGE:figures/full_fig_p011_19.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

63 extracted references · 23 linked inside Pith

  1. [1]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacardet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stoneet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Menschet al., “Mistral 7b,”arXiv preprint arXiv:2310.06825, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyalet al., “Opt: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    Phi-4 technical report,

    M. Abdin, J. Aneja, H. Behlet al., “Phi-4 technical report,”arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwalet al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xueet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighanet al., “Scaling laws for neural language models,”arXiv preprint arXiv:2001.08361, 2020. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 12

    Pith/arXiv arXiv 2001

  9. [9]

    {InfiniGen}: Efficient generative inference of large language models with dynamic{KV}cache management,

    W. Lee, J. Lee, J. Seoet al., “{InfiniGen}: Efficient generative inference of large language models with dynamic{KV}cache management,” in18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 155–172

    2024

  10. [10]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuanget al., “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

    2023

  11. [11]

    Flexgen: High-throughput generative inference of large language models with a single gpu,

    Y . Sheng, L. Zheng, B. Yuanet al., “Flexgen: High-throughput generative inference of large language models with a single gpu,” in International Conference on Machine Learning. PMLR, 2023, pp. 31 094–31 116

    2023

  12. [12]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhuet al., “Atom: Low-bit quantization for efficient and accurate llm serving,”Proceedings of Machine Learning and Systems, vol. 6, pp. 196–209, 2024

    2024

  13. [13]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznecet al., “Smoothquant: Accurate and efficient post-training quantization for large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 38 087–38 099

    2023

  14. [14]

    Quarot: Outlier-free 4-bit inference in rotated llms,

    S. Ashkboos, A. Mohtashami, M. L. Crociet al., “Quarot: Outlier-free 4-bit inference in rotated llms,”Advances in Neural Information Processing Systems, vol. 37, pp. 100 213–100 240, 2024

    2024

  15. [15]

    Duquant: Distributing outliers via dual transformation makes stronger quantized llms,

    H. Lin, H. Xu, Y . Wuet al., “Duquant: Distributing outliers via dual transformation makes stronger quantized llms,”Advances in Neural Information Processing Systems, vol. 37, pp. 87 766–87 800, 2024

    2024

  16. [16]

    Spinquant: Llm quantization with learned rotations,

    Z. Liu, C. Zhao, I. Fedorovet al., “Spinquant: Llm quantization with learned rotations,”arXiv preprint arXiv:2405.16406, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Flatquant: Flatness matters for llm quantization,

    Y . Sun, R. Liu, H. Baiet al., “Flatquant: Flatness matters for llm quantization,”arXiv preprint arXiv:2410.09426, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    M-ant: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,

    W. Hu, H. Zhang, C. Guoet al., “M-ant: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,” in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1112–1126

    2025

  19. [19]

    Tender: Accelerating large language models via tensor decomposition and runtime requantization,

    J. Lee, W. Lee, and J. Sim, “Tender: Accelerating large language models via tensor decomposition and runtime requantization,” in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp. 1048–1062

    2024

  20. [20]

    Mixpe: Quantization and hardware co-design for efficient llm inference,

    Y . Zhang, M. Wang, L. Zouet al., “Mixpe: Quantization and hardware co-design for efficient llm inference,”arXiv preprint arXiv:2411.16158, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Oaken: Fast and efficient llm serving with online-offline hybrid kv cache quantization,

    M. Kim, S. Hong, R. Koet al., “Oaken: Fast and efficient llm serving with online-offline hybrid kv cache quantization,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, 2025, pp. 482–497

    2025

  22. [22]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefleret al., “Gptq: Accurate post-training quantization for generative pre-trained transformers,”arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    The case for 4-bit precision: k-bit inference scaling laws,

    T. Dettmers and L. Zettlemoyer, “The case for 4-bit precision: k-bit inference scaling laws,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 7750–7774

    2023

  24. [24]

    Bitnet: Scaling 1-bit transformers for large language models,

    H. Wang, S. Ma, L. Donget al., “Bitnet: Scaling 1-bit transformers for large language models,”arXiv preprint arXiv:2310.11453, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    Norm tweaking: High-performance low-bit quantization of large language models,

    L. Li, Q. Li, B. Zhanget al., “Norm tweaking: High-performance low-bit quantization of large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 18 536– 18 544

    2024

  26. [26]

    Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,

    C. Lee, J. Jin, T. Kimet al., “Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, 2024, pp. 13 355–13 364

    2024

  27. [27]

    Spqr: A sparse- quantized representation for near-lossless llm weight compression,

    T. Dettmers, R. Svirschevski, V . Egiazarianet al., “Spqr: A sparse- quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Figna: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy,

    J. Jang, Y . Kim, J. Leeet al., “Figna: Integer unit-based accelerator design for fp-int gemm preserving numerical accuracy,” in2024 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2024, pp. 760–773

    2024

  29. [29]

    Anda: Unlocking efficient llm inference with a variable-length grouped activation data format,

    C. Fang, M. Shi, R. Geenset al., “Anda: Unlocking efficient llm inference with a variable-length grouped activation data format,” in 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1467–1481

    2025

  30. [30]

    Language models are few-shot learners,

    T. Brown, B. Mann, N. Ryderet al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    1901

  31. [31]

    From generation to judgment: Opportunities and challenges of llm-as-a-judge,

    D. Li, B. Jiang, L. Huanget al., “From generation to judgment: Opportunities and challenges of llm-as-a-judge,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025, pp. 2757–2791

    2025

  32. [32]

    Longbench: A bilingual, multitask benchmark for long context understanding,

    Y . Bai, X. Lv, J. Zhanget al., “Longbench: A bilingual, multitask benchmark for long context understanding,” inProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024, pp. 3119–3137

    2024

  33. [33]

    Chain of agents: Large language models collaborating on long-context tasks,

    Y . Zhang, R. Sun, Y . Chenet al., “Chain of agents: Large language models collaborating on long-context tasks,”Advances in Neural Information Processing Systems, vol. 37, pp. 132 208–132 237, 2024

    2024

  34. [34]

    Kvquant: Towards 10 million context length llm inference with kv cache quantization,

    C. Hooper, S. Kim, H. Mohammadzadehet al., “Kvquant: Towards 10 million context length llm inference with kv cache quantization,” Advances in Neural Information Processing Systems, vol. 37, pp. 1270–1303, 2024

    2024

  35. [35]

    Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics,

    K. Hong, G. Dai, J. Xuet al., “Flashdecoding++: Faster large language model inference with asynchronization, flat gemm optimization, and heuristics,”Proceedings of Machine Learning and Systems, vol. 6, pp. 148–161, 2024

    2024

  36. [36]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmaret al., “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  37. [37]

    Training dnns with hybrid block floating point,

    M. Drumond, T. Lin, M. Jaggiet al., “Training dnns with hybrid block floating point,”Advances in Neural Information Processing Systems, vol. 31, 2018

    2018

  38. [38]

    Ocp microscaling formats (mx) specification,

    B. D. Rouhani, N. Garegrat, T. Savellet al., “Ocp microscaling formats (mx) specification,”Open Compute Project, 2023

    2023

  39. [39]

    Fast: Dnn training under variable precision block floating point with stochastic rounding,

    S. Q. Zhang, B. McDanel, and H. Kung, “Fast: Dnn training under variable precision block floating point with stochastic rounding,” in 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2022, pp. 846–860

    2022

  40. [40]

    With shared microexponents, a little shifting goes a long way,

    B. Darvish Rouhani, R. Zhao, V . Elangoet al., “With shared microexponents, a little shifting goes a long way,” inProceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–13

    2023

  41. [41]

    Boost: Block minifloat-based on-device cnn training accelerator with transfer learning,

    C. Guo, B. Lou, X. Liuet al., “Boost: Block minifloat-based on-device cnn training accelerator with transfer learning,” in2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD). IEEE, 2023, pp. 1–9

    2023

  42. [42]

    Palm: Scaling language modeling with pathways,

    A. Chowdhery, S. Narang, J. Devlinet al., “Palm: Scaling language modeling with pathways,”Journal of machine learning research, vol. 24, no. 240, pp. 1–113, 2023

    2023

  43. [43]

    Training compute-optimal large language models,

    J. Hoffmann, S. Borgeaud, A. Menschet al., “Training compute-optimal large language models,”arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  44. [44]

    {MegaScale}: Scaling large language model training to more than 10,000{GPUs},

    Z. Jiang, H. Lin, Y . Zhonget al., “{MegaScale}: Scaling large language model training to more than 10,000{GPUs},” in21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), 2024, pp. 745–760

    2024

  45. [45]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradburyet al., “Pointer sentinel mixture models,”arXiv preprint arXiv:1609.07843, 2016

    Pith/arXiv arXiv 2016

  46. [46]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tanget al., “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,”Proceedings of machine learning and systems, vol. 6, pp. 87–100, 2024

    2024

  47. [47]

    Omniquant: Omnidirectionally calibrated quantization for large language models,

    W. Shao, M. Chen, Z. Zhanget al., “Omniquant: Omnidirectionally calibrated quantization for large language models,”arXiv preprint arXiv:2308.13137, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    From clip to dino: Visual encoders shout in multi-modal large language models,

    D. Jiang, Y . Liu, S. Liuet al., “From clip to dino: Visual encoders shout in multi-modal large language models,”arXiv preprint arXiv:2310.08825, 2023

    Pith/arXiv arXiv 2023

  49. [49]

    Kivi: A tuning-free asymmetric 2bit quantization for kv cache,

    Z. Liu, J. Yuan, H. Jinet al., “Kivi: A tuning-free asymmetric 2bit quantization for kv cache,”arXiv preprint arXiv:2402.02750, 2024

    Pith/arXiv arXiv 2024

  50. [50]

    Ten lessons from three generations shaped google’s tpuv4i: Industrial product,

    N. P. Jouppi, D. H. Yoon, M. Ashcraftet al., “Ten lessons from three generations shaped google’s tpuv4i: Industrial product,” in 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA). IEEE, 2021, pp. 1–14

    2021

  51. [51]

    In-datacenter performance analysis of a tensor processing unit,

    N. P. Jouppi, C. Young, N. Patilet al., “In-datacenter performance analysis of a tensor processing unit,” inProceedings of the 44th annual international symposium on computer architecture, 2017, pp. 1–12

    2017

  52. [52]

    Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization,

    C. Guo, C. Zhang, J. Lenget al., “Ant: Exploiting adaptive numerical data type for low-bit deep neural network quantization,” in2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO). IEEE, 2022, pp. 1414–1433

    2022

  53. [53]

    Dnnweaver: From high-level deep network models to fpga acceleration,

    H. Sharma, J. Park, E. Amaroet al., “Dnnweaver: From high-level deep network models to fpga acceleration,” inthe Workshop on Cognitive Architectures, 2016

    2016

  54. [54]

    Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables,

    G. Park, H. Kwon, J. Kimet al., “Figlut: An energy-efficient accelerator design for fp-int gemm using look-up tables,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1098–1111

    2025

  55. [55]

    Lut tensor core: A software-hardware co-design for lut-based low-bit llm inference,

    Z. Mo, L. Wang, J. Weiet al., “Lut tensor core: A software-hardware co-design for lut-based low-bit llm inference,” inProceedings of the JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 52nd Annual International Symposium on Computer Architecture, 2025, pp. 514–528

    2021

  56. [56]

    Bitmod: Bit-serial mixture-of-datatype llm acceleration,

    Y . Chen, A. F. AbouElhamayed, X. Daiet al., “Bitmod: Bit-serial mixture-of-datatype llm acceleration,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA). IEEE, 2025, pp. 1082–1097

    2025

  57. [57]

    Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization,

    C. Guo, J. Tang, W. Huet al., “Olive: Accelerating large language models via hardware-friendly outlier-victim pair quantization,” in Proceedings of the 50th Annual International Symposium on Computer Architecture, 2023, pp. 1–15

    2023

  58. [58]

    Wkvquant: Quantizing weight and key/value cache for large language models gains more,

    Y . Yue, Z. Yuan, H. Duanmuet al., “Wkvquant: Quantizing weight and key/value cache for large language models gains more,”arXiv preprint arXiv:2402.12065, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Accurate kv cache quantization with outlier tokens tracing,

    Y . Su, Y . Zhou, Q. Qiuet al., “Accurate kv cache quantization with outlier tokens tracing,”arXiv preprint arXiv:2505.10938, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models,

    Z. Zhang, Y . Sheng, T. Zhouet al., “H2o: Heavy-hitter oracle for efficient generative inference of large language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 34 661–34 710, 2023

    2023

  61. [61]

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,

    Z. Liu, A. Desai, F. Liaoet al., “Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time,” Advances in Neural Information Processing Systems, vol. 36, pp. 52 342–52 364, 2023

    2023

  62. [62]

    Skvq: Sliding-window key and value cache quantization for large language models,

    H. Duanmu, Z. Yuan, X. Liet al., “Skvq: Sliding-window key and value cache quantization for large language models,”arXiv preprint arXiv:2405.06219, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    Efficient streaming language models with attention sinks,

    G. Xiao, Y . Tian, B. Chenet al., “Efficient streaming language models with attention sinks,”arXiv preprint arXiv:2309.17453, 2023

    Pith/arXiv arXiv 2023