Pith. sign in

REVIEW 2 major objections 4 minor 68 references

AdaMX cuts 4-bit microscaling's accuracy loss by 83% by reusing idle exponent bits as per-block metadata.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

AdaMX puts per-block format and enhancement metadata into spare MX exponent bits, recovering 83% of MXFP4's commonsense accuracy loss at the same 4-bit storage budget.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection AdaMX is a genuine format-plus-accelerator contribution with broad evaluation, but the headline zero-EBW advantage is conditional on an E4 residual-exponent assumption the paper only validates on four models. the 2 major comments →

arxiv 2608.03867 v1 pith:GR4O4KEU submitted 2026-08-04 cs.AR

Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference

Junyi Luo , Xinting Jiang , Tai-Hao Wen , Ruichen Qi , Minxing Chu , Hongyi Wu , Gregory Kielian , Ben Laurie
show 4 more authors
Qirui Zhang Quan Cheng Dennis Sylvester Mehdi Saligane
This is my paper
classification cs.AR
keywords microscalingMXFP4low-bit LLM inferenceper-block quantizationquantization heterogeneity4-bit inferencehardware acceleratorKV-cache quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

4-bit microscaling inference is memory-efficient but loses accuracy because existing formats force one element format and one precision-recovery scheme onto every block and every operand. AdaMX shows that within one weight tensor, roughly half the blocks quantize better as FP4 and half as INT4, and that blocks also differ in whether scale refinement or block-maximum mantissa extension helps more. The paper then observes that the MX format's 8-bit exponent field is overprovisioned for LLM blocks: after a per-channel or per-token bias, a 4-bit residual exponent covers over 99.99% of blocks, leaving four spare bits per block. AdaMX spends those bits on per-block metadata at no increase in equivalent bit width, recovering about 83% of MXFP4's accuracy gap to FP16 on commonsense tasks and 82% on MMLU while adding roughly 1% system energy. If correct, 4-bit LLM inference can run at near-FP16 quality at almost the same cost as the simplest MXFP4 path.

Core claim

AdaMX claims that quantization in 4-bit microscaling is heterogeneous in two ways MX formats ignore: FP4 versus INT4 and scale refinement versus block-maximum mantissa extension each win on different blocks, and weights need a different encoding than activations. The paper shows MX's 8-bit E8M0 scale field is overprovisioned for LLM blocks: after a per-channel or per-token bias, over 99.99% of blocks fit a 4-bit residual exponent, leaving four spare bits per block. AdaMX spends those bits as metadata at no EBW increase—a 2-bit T2 selector routes weight blocks among four format-enhancement modes, and activations get round-to-nearest E4M1 scaling plus a lossless FP6 block-maximum encoding via

What carries the argument

The load-bearing mechanism is the per-block metadata header that replaces most of MX's E8M0 exponent field: a 2-bit T2 selector for weights (FP4/INT4 × scale refinement/block-max mantissa extension), an E4M1 scale with a round-to-nearest, division-free rounding rule for activations, and a 1-bit N1 selector that makes the FP6 block-maximum encoding lossless by covering all six reachable offsets with two overlapping 4-value neighborhoods. The enabling fact is the measured per-block exponent-range distribution: after bias, more than 99.99% of blocks fit E4's 0–15 range, so this adaptation costs zero additional bits per element.

Load-bearing premise

The whole scheme assumes that every LLM block's exponent range, after the per-channel or per-token bias, fits in the 4-bit residual exponent range of 0–15; if new models or input distributions exceed that range, AdaMX loses its metadata capacity or clips activations.

What would settle it

Run AdaMX's encoder on a wider model zoo, including code models and very long contexts, and measure the fraction of blocks whose residual exponent after bias falls outside 0–15 or whose block maximum rounds beyond 7.5; if the tail exceeds the 0.0086% seen on the paper's benchmarks, the zero-EBW and lossless-encoding claims break.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At the same 4.5-bit budget as MXFP4, LLMs from 3B to 70B retain near-FP16 accuracy, so memory-bound serving can use 4-bit weights and activations without the usual accuracy penalty.
  • The block size becomes a runtime operating point: AdaMX-16 prioritizes accuracy, while AdaMX-32 at 4.25 bits/element stays more accurate than MXFP4 while cutting decode energy by 2.9–5.2%.
  • The benefit transfers to multimodal models: on Gemma-4 12B, AdaMX-16 leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.
  • The same weight-side encoding works for KV-cache quantization: under W4A4KV4, AdaMX-16 scores 89.7% on RULER at 8K context versus 47.8% for MXFP4, with similar gains at 32K.
  • Hardware overhead is small: a 22nm FD-SOI implementation adds 3.2% array area, 3.5% array power, and at most 1.1% system energy over an equal-throughput MXFP4 accelerator.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not pursue: the N1 lossless-neighborhood trick is generic, so it could be applied to other coarse-scale formats (MXFP8, INT8 microscaling, training-side formats) to eliminate block-maximum clamping at negligible decode cost.
  • The zero-EBW budget is contingent on the E4-range assumption; if future models, code-heavy workloads, or longer contexts produce wider per-block exponent spreads, the bias granularity may need to become finer and cost bits.
  • The T2 routing is an offline, static per-block choice; an online or input-adaptive variant could capture activation-driven shifts per layer or token, trading a small encode cost for additional accuracy.
  • The per-token activation bias is measured on standard benchmarks; testing the 0.0086% worst-case tail-clipping fraction on out-of-distribution or adversarial inputs would show how much safety margin the format actually has.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper introduces AdaMX, an adaptive microscaling format that repurposes the 8-bit E8M0 scale field of MX into a 4-bit residual exponent (E4) plus 4 bits of per-block metadata. For weights, a 2-bit T2 selector routes each block among four combinations of element format (FP4/INT4) and precision-recovery scheme (scale refinement or block-maximum mantissa extension), chosen by an offline per-block MSE search. For activations, the format is E4M1Mt2N1: a 4-bit residual exponent with a 1-bit scale mantissa, a 2-bit block-maximum extension, and a 1-bit neighborhood selector that enables lossless FP6 encoding of the block maximum. The paper also presents a 22nm FD-SOI accelerator design with two block-size operating points (16 and 32) and an energy/area evaluation. Across four LLMs (3B-70B) and a 12B multimodal model, AdaMX-16 is reported to recover 63-72% of the MXFP4 perplexity gap to FP16 and to remove about 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, at the same equivalent bit width (4.5 bits/element) while adding at most about 1% system energy.

Significance. If the results hold, AdaMX is a meaningful advance for low-bit LLM inference: it shows that per-block heterogeneity in both element format and precision-recovery scheme can be captured without increasing storage beyond the MX scale-field budget. The paper contains several self-contained derivations (the round-to-nearest scale-reduction analysis in Section III-C2 and the N1 lossless FP6 neighborhood construction in Section III-C3) and a clean hardware mapping with a unified datapath for both block sizes. The evaluation spans multiple model families and sizes, includes downstream accuracy as well as perplexity, and extends to multimodal and KV-cache settings. The main caveat is that the zero-EBW advantage relies on a distributional assumption about the residual exponent range, which is supported by measurements on a limited set of models and only WikiText-2 for activations.

major comments (2)
  1. [Section III-A and Figure 3] The central 'no increase in EBW' claim rests on every block's residual exponent fitting in E4's range [0,15] after per-channel/per-token bias. Figure 3 shows this holds on four LLMs for weights, and for activations on WikiText-2 with a clamped tail (up to 0.0086% for Qwen2.5-3B). However, the manuscript does not report how this clamping affects the downstream accuracy numbers, nor does it specify a hardware detection/fallback mechanism when E4 overflows. Because activations are encoded online with no calibration, an input distribution whose per-token exponent range exceeds 15 would either distort the entire block (if clamped) or require a wider exponent field, breaking the zero-EBW advantage. I recommend adding a sensitivity analysis (e.g., vary the E4 width, report overflow rates on the actual evaluation datasets, and describe the encoder behavior on overflow).
  2. [Algorithm 1, lines 4 and 24] The weight-side exponent search tries offsets b in {-1,0,+1} and stores E4 = e0 + b. Since b_bias is defined as the minimum block exponent in the channel, the block(s) at that minimum have e0=0, so b=-1 would produce a negative residual, which cannot be represented in an unsigned 4-bit field. The manuscript does not state whether E4 is signed, or whether b=-1 is skipped/clamped for such blocks. This is a correctness detail in the core encoding algorithm; please clarify the encoding space and, if necessary, adjust the search or the bias definition.
minor comments (4)
  1. [Abstract and Section V-A] The word 'prototype' suggests fabricated silicon, but the evaluation is based on synthesis, place-and-route, and simulators. Please rephrase to 'synthesized implementation' or 'accelerator design' to avoid overstatement.
  2. [Tables III-VI] No error bars or multiple-seed runs are reported. If the evaluation is deterministic (e.g., greedy decoding), state this explicitly; otherwise provide variance or confidence intervals for the headline recovery percentages.
  3. [General] No code or RTL is released. Given the algorithmic detail in Sections III and IV, a public release of the quantization framework would substantially aid reproducibility; at minimum, a note about availability would be helpful.
  4. [Table IV] SMX4 perplexity of 468.5 on Qwen2.5-3B is an outlier. Please clarify whether this is a typo or a real divergence, and if real, provide context.

Circularity Check

0 steps flagged

No significant circularity: AdaMX's derivation is self-contained; the zero-EBW claim is a bit-accounting identity, activation encoding is constructively derived, and weight-side routing is an offline MSE search rather than a relabeled prediction.

full rationale

The paper's central derivation chain is self-contained and does not reduce to its own inputs. The activation-side format E4M1Mt2N1 is constructed from the definitions of the FP4 and FP6 grids: Equation (2) maps the block maximum into the range [5.0, 7.5), the scale-rounding analysis bounds the worst-case ratio at 1.25, and N1's two overlapping 4-value neighborhoods are explicitly chosen to cover the six resulting FP6 offsets. This is a constructive coverage argument, not an imported conclusion. The weight-side T2 routing is selected by an offline per-block MSE minimization (Algorithm 1); although AdaMX is guaranteed to have per-block reconstruction error no worse than MXFP4 because MXFP4-like scale choices are among the candidates, the reported task-accuracy recoveries (perplexity, commonsense, MMLU, RULER, multimodal) are externally measured against FP16, MXFP4, NVFP4, M2XFP, and other baselines and are not logically forced by the MSE bound. The zero-EBW claim is a bit-accounting identity: both AdaMX and MXFP4 use 8 bits per block for scale/metadata, giving EBW = 4.5 at N=16 and 4.25 at N=32; no fitted parameter is renamed as a prediction. The only empirical precondition is that residual exponents fit in E4 (with the acknowledged clamped tail of at most 0.0086% on Qwen2.5-3B), which is a stated distributional assumption and an external-validity caveat, not a circular step. There are no load-bearing self-citations: the paper's cited prior formats (MX+, M2XFP, NVFP4, BlockDialect) and the FP4-vs-INT4 analysis are external works, and the novelty is the specific combination and hardware implementation. The selection of round-to-nearest scaling is justified both by the FP6 max-norm bound and by an empirical perplexity comparison; the latter is ordinary model selection, not a prediction forced by construction.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

No physical or ontological entities are introduced; the new AdaMX metadata fields (T2, Mt2, N1, E4M1) are design constructs with direct experimental support in Section V. The central design depends on the E4-sufficiency axiom and the MSE-to-accuracy proxy, both empirically argued but not formally guaranteed. The three free parameters above are the main hand-set or data-tuned choices.

free parameters (3)
  • Scale refinement ratio set R = {1.0, 1.25, 1.5, 1.75}
    Hand-chosen four-value ratio set for weight-side Mt2 scale refinement; not derived from first principles, though only four codes are available.
  • M1 scale mantissa grid = {1, 1.5}
    The 1-bit activation scale mantissa allows exactly two values; the round-to-nearest co-design argument depends on this specific grid and FP6 max-norm 7.5.
  • Operating block sizes 16 and 32 = 16, 32
    Selected from the knee of the Fig. 8 sweep on the same four LLMs that later appear in the main accuracy tables, so the choice is tuned to the evaluation set.
axioms (3)
  • domain assumption After per-channel/per-token bias, a 4-bit residual exponent covers >99.99% of LLM blocks, making E8M0 overprovisioned.
    Load-bearing for zero-EBW metadata. Empirically supported by Figure 3 on four models, but not proven for all future LLMs or inputs; the paper reports a small clamped tail on Qwen2.5-3B.
  • domain assumption Per-block QSNR or MSE is a valid proxy for downstream task accuracy.
    The offline T2 search and block-size selection minimize reconstruction error; the paper then measures task accuracy. This correlation is plausible and partially validated by Tables III-VI, but it is not a mathematical guarantee.
  • domain assumption Weights are static and admit offline per-block search, while activations must be encoded online in one pass.
    This split is the reason for having two different formats and is standard for post-training quantization, but it rules out any runtime-adaptive weight encoding and is load-bearing for the design.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference." pith.science (2026). https://pith.science/paper/GR4O4KEU

@misc{pith2026260803867,
  author       = {Pith},
  title        = {Pith review of: Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GR4O4KEU}},
  note         = {Machine review of arXiv:2608.03867}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix either the element format or the precision-recovery scheme across blocks, and thus capture only limited quantization heterogeneity. Quantization heterogeneity appears at two levels: 1) across blocks, the preferred element format and precision-recovery scheme vary; 2) across operands, weights and activations require different encoding. We introduce AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator. It selects the precision-recovery scheme per block and the representation per operand, at no increase in equivalent bit width (EBW). One design covers two block sizes, giving a higher-accuracy operating point and a lower-EBW operating point that saves storage. We implement a 22nm FD-SOI AI accelerator prototype with the proposed decoder, computing unit, and quantization logic. Against an otherwise identical MXFP4 accelerator with FP4-only multipliers, AdaMX adds about 1% system energy. At the lower-EBW point, AdaMX stays more accurate than the baseline while lowering both memory footprint and energy. Across LLMs from 3B to 70B, AdaMX removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss. AdaMX also generalizes to multimodal models. On Gemma-4 12B, it leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.

Figures

Figures reproduced from arXiv: 2608.03867 by Ben Laurie, Dennis Sylvester, Gregory Kielian, Hongyi Wu, Junyi Luo, Mehdi Saligane, Minxing Chu, Qirui Zhang, Quan Cheng, Ruichen Qi, Tai-Hao Wen, Xinting Jiang.

Figure 1
Figure 1. Figure 1: Bit layouts of MX-family formats. AdaMX repartitions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: CDF of exponent range across four models (block size [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The two axes of T2 routing. (a) FP4 and INT4 differ [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scale-rounding effects: round-to-nearest fits the block [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: E4M1Mt2N1 encoding flow. insufficient: FP7 produces eleven possible offsets, exceeding the coverage of even a 3-bit neighborhood selector. To cover all six offsets, N1 selects between two overlapping 4-value neighborhoods, {0, +1, +2, +3} for N1 = 0 and {−2, −1, 0, +1} for N1 = 1, whose union spans the complete offset set. Mt2 indexes each neighborhood in order, and the two share offsets 0 and +1 so every … view at source ↗
Figure 8
Figure 8. Figure 8: (a) WikiText-2 perplexity, (b) FP4 underflow rate, and [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: AdaMX decoder, supporting both K=16 and K=32. the FP4 or INT4 base path. The ratio modes (T2[0] ⊕ T2[1]) route Mt2w to the scale-refinement field Mt2eff; the block￾max modes instead direct Mt2w to the offset decoder and force Mt2eff to zero. The activation header carries no T2. Its M1 bit drives the second scale-refinement stage (Section IV-C), while Mt2a with N1 feed the offset decoder and the block expo… view at source ↗
Figure 12
Figure 12. Figure 12: AdaMX activation quantization unit. Blue and red [PITH_FULL_IMAGE:figures/full_fig_p009_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Decode energy across four models and three batch [PITH_FULL_IMAGE:figures/full_fig_p011_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 37 canonical work pages · 1 internal anchor

  1. [1]

    AMD Instinct™ MI350 Series GPUs,

    Advanced Micro Devices, Inc., “AMD Instinct™ MI350 Series GPUs,” https://www.amd.com/en/products/accelerators/instinct/mi350.html, 2025

    work page 2025

  2. [2]

    System Card: Claude Opus 4 & Claude Sonnet 4,

    Anthropic, “System Card: Claude Opus 4 & Claude Sonnet 4,” https://www-cdn.anthropic.com/ 07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf, May 2025

    work page 2025

  3. [3]

    QuaRot: outlier-free 4-bit inference in rotated llms,

    S. Ashkboos, A. Mohtashami, M. L. Croci, B. Li, P. Cameron, M. Jaggi, D. Alistarh, T. Hoefler, and J. Hensman, “QuaRot: outlier-free 4-bit inference in rotated llms,” inProceedings of the 38th International Conference on Neural Information Processing Systems, ser. NIPS ’24. Red Hook, NY , USA: Curran Associates Inc., 2024

    work page 2024

  4. [4]

    Cacti 7: New tools for interconnect exploration in innovative off-chip memories,

    R. Balasubramonian, A. B. Kahng, N. Muralimanohar, A. Shafiee, and V . Srinivas, “Cacti 7: New tools for interconnect exploration in innovative off-chip memories,”ACM Trans. Archit. Code Optim., vol. 14, no. 2, Jun. 2017. [Online]. Available: https://doi.org/10.1145/3085572

    doi:10.1145/3085572 2017

  5. [5]

    Piqa: Reasoning about physical commonsense in natural language,

    Y . Bisk, R. Zellers, R. L. Bras, J. Gao, and Y . Choi, “Piqa: Reasoning about physical commonsense in natural language,” in AAAI Conference on Artificial Intelligence, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:208290939

    work page 2019

  6. [6]

    Int v.s. fp: A comprehensive study of fine-grained low-bit quantization formats,

    M. Chen, M. Wu, H. Jin, Z. Yuan, J. Liu, C. Zhang, Y . Li, J. Huang, J. Ma, Z. Xue, Z. Liu, X. Bin, and P. Luo, “Int v.s. fp: A comprehensive study of fine-grained low-bit quantization formats,” 2025. [Online]. Available: https://arxiv.org/abs/2510.25602

    arXiv 2025

  7. [7]

    Boolq: Exploring the surprising difficulty of natural yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,”ArXiv, vol. abs/1905.10044, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:165163607

    Pith/arXiv arXiv 1905

  8. [8]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,”ArXiv, vol. abs/1803.05457, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:3922816

    Pith/arXiv arXiv 2018

  9. [9]

    Four over six: More accurate nvfp4 quantization with adaptive block scaling,

    J. Cook, J. Guo, G. Xiao, Y . Lin, and S. Han, “Four over six: More accurate nvfp4 quantization with adaptive block scaling,” 2025. [Online]. Available: https://arxiv.org/abs/2512.02010

    Pith/arXiv arXiv 2025

  10. [10]

    Efficient precision-scalable hardware for microscaling (mx) processing in robotics learning,

    S. Cuyckens, X. Yi, N. S. Murthy, C. Fang, and M. Verhelst, “Efficient precision-scalable hardware for microscaling (mx) processing in robotics learning,” inISLPED, 2025. [Online]. Available: https: //arxiv.org/abs/2505.22404

    arXiv 2025

  11. [11]

    With shared microexponents, a little shifting goes a long way,

    B. Darvish Rouhani, R. Zhao, V . Elango, R. Shafipour, M. Hall, M. Mesmakhosroshahi, A. More, L. Melnick, M. Golub, G. Varatkar, L. Shao, G. Kolhe, D. Melts, J. Klar, R. L’Heureux, M. Perry, D. Burger, E. Chung, Z. S. Deng, S. Naghshineh, J. Park, and M. Naumov, “With shared microexponents, a little shifting goes a long way,” inProceedings of the 50th Ann...

    arXiv 2023

  12. [12]

    Deepseek-v4: Towards highly efficient million-token context intelligence,

    DeepSeek-AI, “Deepseek-v4: Towards highly efficient million-token context intelligence,” 2026

    2026

  13. [13]

    LLM.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit matrix multiplication for transformers at scale,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY , USA: Curran Associates Inc., 2022

    work page 2022

  14. [14]

    Spqr: A sparse- quantized representation for near-lossless llm weight compression,

    T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse- quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Extreme compression of large language models via additive quantization,

    V . Egiazarian, A. Panferov, D. Kuznedelev, E. Frantar, A. Babenko, and D. Alistarh, “Extreme compression of large language models via additive quantization,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    work page 2024

  16. [16]

    GPTQ: Accurate post-training compression for generative pretrained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “GPTQ: Accurate post-training compression for generative pretrained transformers,”arXiv preprint arXiv:2210.17323, 2022

    Pith/arXiv arXiv 2022

  17. [17]

    The language model evaluation harness,

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou, “The language model evaluation harness,” 07 2024. [Online]. Available: https://zenodo.org/re...

    arXiv 2024

  18. [18]

    Gemma 4 Model Overview,

    Google DeepMind, “Gemma 4 Model Overview,” https://ai.google.dev/ gemma/docs/core, 2026

    work page 2026

  19. [19]

    ANT: Exploiting adaptive numerical data type for low-bit deep neural network quantization,

    C. Guo, C. Zhang, J. Leng, Z. Liu, F. Yang, Y . Liu, M. Guo, and Y . Zhu, “ANT: Exploiting adaptive numerical data type for low-bit deep neural network quantization,” in2022 55th IEEE/ACM International Symposium on Microarchitecture (MICRO), 2022, pp. 1414–1433

    work page 2022

  20. [20]

    BBAL: A bidirectional block floating point-based quantisation accelerator for large language models,

    X. Han, Y . Cheng, J. Wang, J. Lu, H. Wang, X. X. Zhang, N. Xu, D. Yang, and Z. Jiang, “BBAL: A bidirectional block floating point-based quantisation accelerator for large language models,” in Proceedings of the 62nd Annual ACM/IEEE Design Automation Conference, ser. DAC ’25. IEEE Press, 2025. [Online]. Available: https://doi.org/10.1109/DAC63849.2025.11132978

    arXiv 2025

  21. [21]

    Measuring massive multitask language understanding,

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  22. [22]

    1.1 computing’s energy problem (and what we can do about it),

    M. Horowitz, “1.1 computing’s energy problem (and what we can do about it),” in2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 2014, pp. 10–14

    2014

  23. [23]

    Ruler: What’s the real context size of your long- context language models?

    C.-P. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, Y . Zhang, and B. Ginsburg, “Ruler: What’s the real context size of your long- context language models?”arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    M-ANT: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,

    W. Hu, H. Zhang, C. Guo, Y . Feng, R. Guan, Z. Hua, Z. Liu, Y . Guan, M. Guo, and J. Leng, “M-ANT: Efficient low-bit group quantization for llms via mathematically adaptive numerical type,” in2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2025, pp. 1112–1126

    work page 2025

  25. [25]

    M2XFP: A metadata-augmented microscaling data format for efficient low-bit quantization,

    W. Hu, Z. Zhang, H. Zhang, C. Zhang, C. Guo, Y . Feng, T. Hu, G. Li, G. Hu, J. Wang, and J. Leng, “M2XFP: A metadata-augmented microscaling data format for efficient low-bit quantization,” in Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, ser. ASPLOS ’26. New York, N...

    arXiv 2026

  26. [26]

    Blockdialect: block-wise fine-grained mixed format quantization for energy-efficient llm inference,

    W. Jang and T. Tambe, “Blockdialect: block-wise fine-grained mixed format quantization for energy-efficient llm inference,” inProceedings of the 42nd International Conference on Machine Learning, ser. ICML’25. JMLR.org, 2025

    work page 2025

  27. [27]

    A diagram is worth a dozen images,

    A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, “A diagram is worth a dozen images,” inEuropean con- ference on computer vision. Springer, 2016, pp. 235–251

    work page 2016

  28. [28]

    14.2 a 16nm 216kb, 188.4tops/w and 133.5tflops/w microscaling multi- mode gain-cell cim macro edge-ai devices,

    W.-S. Khwa, P.-C. Wu, J.-W. Su, C.-Y . Cheng, J.-M. Hsu, Y .-C. Chen, L.-J. Hsieh, J.-C. Bai, Y .-S. Kao, T.-H. Lou, A. S. Lele, J.-J. Wu, J.-C. Tien, C.-C. Lo, R.-S. Liu, C.-C. Hsieh, K.-T. Tang, and M.-F. Chang, “14.2 a 16nm 216kb, 188.4tops/w and 133.5tflops/w microscaling multi- mode gain-cell cim macro edge-ai devices,” in2025 IEEE International Soli...

    work page 2025

  29. [29]

    SqueezeLLM: dense-and-sparse quantiza- tion,

    S. Kim, C. Hooper, A. Gholami, Z. Dong, X. Li, S. Shen, M. W. Mahoney, and K. Keutzer, “SqueezeLLM: dense-and-sparse quantiza- tion,” inProceedings of the 41st International Conference on Machine Learning, ser. ICML’24. JMLR.org, 2024

    work page 2024

  30. [30]

    Tender: Accelerating large language models via tensor decomposition and runtime requantization,

    J. Lee, W. Lee, and J. Sim, “Tender: Accelerating large language models via tensor decomposition and runtime requantization,” in2024 ACM/IEEE 51st Annual International Symposium on Computer Archi- tecture (ISCA), 2024, pp. 1048–1062

    work page 2024

  31. [31]

    MX+: Pushing the limits of microscaling formats for efficient large language model serving,

    J. Lee, J. Park, S. Cha, J. Cho, and J. Sim, “MX+: Pushing the limits of microscaling formats for efficient large language model serving,” inProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, ser. MICRO ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 869–883. [Online]. Available: https://doi.org/10.1145/3...

    arXiv 2025

  32. [32]

    AWQ: Activation-aware weight quantization for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, G. Xiao, and S. Han, “AWQ: Activation-aware weight quantization for on-device llm compression and acceleration,”GetMobile: Mobile Comp. and Comm., vol. 28, no. 4, p. 12–17, Jan. 2025. [Online]. Available: https://doi.org/10.1145/3714983.3714987

    arXiv 2025

  33. [33]

    Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,

    Y . Lin, H. Tang, S. Yang, Z. Zhang, G. Xiao, C. Gan, and S. Han, “Qserve: W4a8kv4 quantization and system co-design for efficient llm serving,”Proceedings of Machine Learning and Systems, vol. 7, 2025

    2025

  34. [34]

    Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Models

    Y .-C. Lo, G.-Y . Wei, and D. Brooks, “Nanoscaling floating-point (nxfp): Nanomantissa, adaptive microexponents, and code recycling 12 for direct-cast compression of large language models,”arXiv preprint arXiv:2412.19821, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Learn to explain: Multimodal reasoning via thought chains for science question answering,

    P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, “Learn to explain: Multimodal reasoning via thought chains for science question answering,” inThe 36th Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  36. [36]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” inProceedings of the 5th International Conference on Learning Representations (ICLR), 2017

    work page 2017

  37. [37]

    The Llama 3 herd of models,

    Meta, “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Four MTIA Chips in Two Years: Scaling AI Experi- ences for Billions,

    Meta, “Four MTIA Chips in Two Years: Scaling AI Experi- ences for Billions,” https://ai.meta.com/blog/meta-mtia-scale-ai-chips- for-billions/, Mar. 2026

    work page 2026

  39. [39]

    Recipes for pre- training llms with mxfp8,

    A. Mishra, D. Stosic, S. Layton, and P. Micikevicius, “Recipes for pre- training llms with mxfp8,”arXiv preprint arXiv:2506.08027, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    NVIDIA H100 Tensor Core GPU Datasheet,

    NVIDIA, “NVIDIA H100 Tensor Core GPU Datasheet,” Datasheet,

    work page

  41. [41]

    NVIDIA Blackwell Architecture Technical Brief,

    NVIDIA, “NVIDIA Blackwell Architecture Technical Brief,” Technical Brief, 2024

    work page 2024

  42. [42]

    OCP Microscaling Formats (MX) Specification,

    Open Compute Project, “OCP Microscaling Formats (MX) Specification,” Open Compute Project, Tech. Rep., 2023, version 1.0. [Online]. Available: https://www.opencompute.org/documents/ocp- microscaling-formats-mx-v1-0-spec-final-pdf

    work page 2023

  43. [43]

    OpenAI GPT-5 System Card,

    OpenAI, “OpenAI GPT-5 System Card,” 2025

    work page 2025

  44. [44]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K ¨opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala,PyTorch: an imperative style, high- performance deep learning library. Red Hook, NY , USA: Curran Associates Inc., 2019

    2019

  45. [45]

    Scale-sim v3: A modular cycle-accurate systolic accelerator simulator for end-to-end system analysis,

    R. Raj, S. Banerjee, N. Chandra, Z. Wan, J. Tong, A. Samajdhar, and T. Krishna, “Scale-sim v3: A modular cycle-accurate systolic accelerator simulator for end-to-end system analysis,” in2025 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2025, pp. 186–200

    work page 2025

  46. [46]

    Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization,

    A. Ramachandran, S. Kundu, and T. Krishna, “Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization,” inProceedings of the 52nd Annual International Symposium on Computer Architecture, ser. ISCA ’25. New York, NY , USA: Association for Computing Machinery, 2025, p. 1193–1209. [Online]. Available: https://doi.org/10.11...

    work page arXiv 2025

  47. [47]

    Gemma 2: Improving open language models at a practical size,

    G. T. M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Ram’e, J. Ferret, P. Liu, P. D. Tafti, A. Friesen, M. Casbon, S. Ramos, R. Kumar, C. L. Lan, S. Jerome, A. Tsitsulin, N. Vieillard, P. Sta ´nczyk, S. Girgin, N. Momchev, M. Hoffman, S. Thakoor, J.-B. Grill, B. Neyshabur, A. Walton, A. Severyn, A....

    Pith/arXiv arXiv 2024

  48. [48]

    Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point,

    B. Rouhani, D. Lo, R. Zhao, M. Liu, J. Fowers, K. Ovtcharov, A. Vino- gradsky, S. Massengill, L. Yang, R. Bittner, A. Forin, H. Zhu, T. Na, P. Patel, S. Che, L. C. Koppaka, X. Song, S. Som, K. Das, S. Tiwary, S. Reinhardt, S. Lanka, E. Chung, and D. Burger, “Pushing the limits of narrow precision inferencing at cloud scale with microsoft floating point,” ...

    work page 2020

  49. [49]

    Microscaling data formats for deep learning,

    B. D. Rouhani, R. Zhao, A. More, M. Hall, A. Khodamoradi, S. Deng, D. Choudhary, M. Cornea, E. Dellinger, K. Denolf, D. Stosic, V . Elango, M. Golub, A. Heinecke, P. James-Roxby, D. Jani, G. Kolhe, M. Lang- hammer, A. Li, L. Melnick, M. Mesmakhosroshahi, A. C. Rodr ´ıguez, M. Schulte, R. Shafipour, L. Shao, M. Y . Siu, P. Dubey, P. Mi- cikevicius, M. Naum...

    Pith/arXiv arXiv 2023

  50. [50]

    Winogrande: an adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: an adversarial winograd schema challenge at scale,”Commun. ACM, vol. 64, no. 9, p. 99–106, Aug. 2021. [Online]. Available: https://doi.org/10.1145/3474381

    doi:10.1145/3474381 2021

  51. [51]

    Omniquant: Omnidirectionally calibrated quan- tization for large language models,

    W. Shao, M. Chen, Z. Zhang, P. Xu, L. Zhao, Z. Li, K. Zhang, P. Gao, Y . Qiao, and P. Luo, “Omniquant: Omnidirectionally calibrated quan- tization for large language models,”arXiv preprint arXiv:2308.13137, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Towards vqa models that can read,

    A. Singh, V . Natarajan, M. Shah, Y . Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 8317–8326

    work page 2019

  53. [53]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge,

    A. Talmor, J. Herzig, N. Lourie, and J. Berant, “Commonsenseqa: A question answering challenge targeting commonsense knowledge,”

    work page

  54. [54]

    A microscaling multi-mode gain-cell computing-in- memory macro for advanced ai edge device,

    J.-C. Tien, P.-C. Wu, W.-S. Khwa, A. Sanjay Lele, J.-W. Su, C.- Y . Cheng, J.-M. Hsu, Y .-C. Chen, L.-J. Hsieh, J.-C. Bai, Y .-S. Kao, T.-H. Lou, J.-J. Wu, C.-C. Lo, R.-S. Liu, C.-C. Hsieh, K.-T. Tang, and M.-F. Chang, “A microscaling multi-mode gain-cell computing-in- memory macro for advanced ai edge device,”IEEE Journal of Solid- State Circuits, vol. 6...

    work page 2026

  55. [55]

    Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks,

    A. Tseng, J. Chee, Q. Sun, V . Kuleshov, and C. De Sa, “Quip#: Even better llm quantization with hadamard incoherence and lattice codebooks,”Proceedings of machine learning research, vol. 235, p. 48630, 2024

    work page 2024

  56. [56]

    30.1 a 28nm 127.54tflops/w mxfp6 and 117.42tflops/w mxfp8 compute-in-memory macro with adaptive- preserved-bit-width and serial-dual-bit-sliding schemes,

    X. Wang, Y . Du, T. Jiao, D. Wu, X. Chen, M. Tang, Y . Yang, Z. Liu, A. Guo, G. Fu, P. Li, J. Dong, B. Liu, X. Liu, W. Shan, H. Cai, G. Sun, L. Tong, J. Yang, and X. Si, “30.1 a 28nm 127.54tflops/w mxfp6 and 117.42tflops/w mxfp8 compute-in-memory macro with adaptive- preserved-bit-width and serial-dual-bit-sliding schemes,” in2026 IEEE International Solid...

    work page 2026

  57. [57]

    Accelergy: An architecture- level energy estimation methodology for accelerator designs,

    Y . N. Wu, J. S. Emer, and V . Sze, “Accelergy: An architecture- level energy estimation methodology for accelerator designs,” in2019 IEEE/ACM International Conference on Computer-Aided Design (IC- CAD), 2019, pp. 1–8

    2019

  58. [58]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” inProceedings of the 40th International Con- ference on Machine Learning, 2023

    work page 2023

  59. [59]

    Inside Maia 100: Revolutionizing AI Workloads with Microsoft’s Custom AI Accelerator,

    S. Xu and C. Ramakrishnan, “Inside Maia 100: Revolutionizing AI Workloads with Microsoft’s Custom AI Accelerator,” https://techcommunity.microsoft.com/blog/azureinfrastructureblog/ inside-maia-100-revolutionizing-ai-workloads-with-microsofts- custom-ai-accelerat/4229118, Aug. 2024

    work page arXiv 2024

  60. [60]

    An empirical study of microscaling formats for low-precision llm training,

    H. Yang, S. Deng, A. Nagpal, M. Naumov, M. Janani, T. Liu, and H. Guan, “An empirical study of microscaling formats for low-precision llm training,” in2025 IEEE 32nd Symposium on Computer Arithmetic (ARITH), 2025, pp. 1–8

    work page 2025

  61. [61]

    Qwen2.5 technical report,

    Q. A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, 13 J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y . Fan, Y . Su, Y .-C. Zhang, Y . Wan, Y . Liu, Z. Cui, Z. Zhang, ...

    Pith/arXiv arXiv 2024

  62. [62]

    ZeroQuant: efficient and affordable post-training quantization for large- scale transformers,

    Z. Yao, R. Y . Aminabadi, M. Zhang, X. Wu, C. Li, and Y . He, “ZeroQuant: efficient and affordable post-training quantization for large- scale transformers,” inProceedings of the 36th International Conference on Neural Information Processing Systems, ser. NIPS ’22. Red Hook, NY , USA: Curran Associates Inc., 2022

    work page 2022

  63. [63]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y . Liu, W. Huang, H. Sun, Y . Su, and W. Chen, “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” inProceedings of CVPR, 2024

    work page 2024

  64. [64]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” inAnnual Meeting of the Association for Computational Linguistics, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:159041722

    work page 2019

  65. [65]

    Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training,

    J. Zhang, J. Wei, P. Zhang, X. Xu, H. Huang, H. Wang, K. Jiang, J. Chen, and J. Zhu, “Sageattention3: Microscaling fp4 attention for inference and an exploration of 8-bit training,”arXiv preprint arXiv:2505.11594, 2025

    arXiv 2025

  66. [66]

    Atom: Low-bit quantization for efficient and accurate llm serving,

    Y . Zhao, C.-Y . Lin, K. Zhu, Z. Ye, L. Chen, S. Zheng, L. Ceze, A. Krishnamurthy, T. Chen, and B. Kasikci, “Atom: Low-bit quantization for efficient and accurate llm serving,” inProceedings of Machine Learning and Systems, P. Gibbons, G. Pekhimenko, and C. D. Sa, Eds., vol. 6, 2024, pp. 196–209. [Online]. Available: https://proceedings.mlsys.org/paper fi...

    work page 2024

  67. [2019]

    Available: https://arxiv.org/abs/1811.00937

    [Online]. Available: https://arxiv.org/abs/1811.00937

    Pith/arXiv arXiv

  68. [2023]

    Available: https://resources.nvidia.com/en-us-gpu- resources/h100-datasheet-24306

    [Online]. Available: https://resources.nvidia.com/en-us-gpu- resources/h100-datasheet-24306

    work page

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.