Pith. sign in

REVIEW 3 major objections 6 minor 76 references

A matched high-precision trace can audit when low-precision weight updates vanish and, frozen before training, can select the exact 10% of weights to protect, recovering 60.7% of the loss gap on unseen targets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:49 UTC pith:S5322INO

load-bearing objection A careful, self-limiting study of a real low-precision failure mode: the exact freeze event is clean, the audit proxy is honest but fails transfer, and the allocation result is genuinely prospective. Worth refereeing. the 3 major comments →

arxiv 2607.09800 v3 pith:S5322INO submitted 2026-07-09 cs.LG

Reference Traces for Auditing Invisible Weight Updates and Guiding Exact-Budget Protection

Zekai Shang This is my paper
classification cs.LG
keywords invisible weight updatesround-to-neareststochastic roundinglow-precision trainingreference tracesfp32 master weightsproposal contributiongap recovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that the moment a low-precision weight update becomes invisible is an exact, testable arithmetic event, and that a separate high-precision reference trace can be used in two distinct ways: to audit how many weights are frozen over time, and to rank which weights deserve an fp32 master copy. The audit forecasts aggregate visibility accurately in controlled, same-schedule testbeds, but the paper is explicit that the proxy is not a pathwise certificate: in a predeclared transfer test the measured and projected paths diverge by update 3, and in a modern decoder the trajectories are uninformative. The allocation use is stronger: one 600-step source trace, sealed before three unseen targets, selects an exact 10% fp32-master tile mask that recovers 60.7% of the round-to-nearest-versus-fp32 loss gap on average, nearly matching target-specific masks (60.8%) and far exceeding fixed controls (17.9%). The paper also shows that changing only the write-back rule from round-to-nearest to stochastic rounding recovers 89–92% of that gap at 10,000 steps, and that the tested selective-master implementation saves peak memory against a blanket master but is slower than direct write-back, so it currently has no net systems benefit.

Core claim

Central claim: a low-precision update becomes invisible when the quantizer leaves the stored code unchanged — an exact testable event — and a matched high-precision reference trace can both audit how many weights freeze over time and rank which weights deserve an fp32 master copy. The audit forecasts crossing times across an 11–4222-step range (median ratio 1.000, 94.5% within 15%) but is not a pathwise certificate: in a locked 64-cell test paths diverge at update 3, and in a modern decoder the trajectories are uninformative. The allocation use is stronger: one 600-step source trace, frozen before three unseen targets, picks an exact-10% fp32 mask that recovers 60.7% of the gap on average, v

What carries the argument

The load-bearing object is the exact coordinate event Q_m(q_i + d_i) = q_i: the stored low-precision code is bitwise unchanged after quantization of the optimizer's proposal. The audit uses a projected-reference proxy (Eq. 4), which replays a high-precision trajectory onto the target grid to estimate the frozen fraction without any low-precision observation. The allocation study uses a third object: absolute proposal contribution |g_i u_i|, averaged per 768-coordinate tile over a 600-step fp32 reference, which ranks tiles so the top 10% of the eligible fp32-master payload can be protected in advance. Each carries a different burden: the exact event is a fact; the proxy is an empirical aggreg

Load-bearing premise

The audit's projected-reference proxy assumes the high-precision trajectory stays a useful stand-in for the low-precision path near the rounding boundary; once the two paths diverge, every aggregate uncertainty claim loses support, although the exact whether-a-weight-changed event remains valid.

What would settle it

Run the projected-reference proxy (Eq. 4) on a previously unseen model or seed with a predeclared ±15% band and compare its first-majority-crossing forecasts against measured code-equality crossings; the proxy is falsified if fewer than about 90% of forecasts fall in the band, or if measured and projected code vectors diverge by update 3 as they do in the paper's own locked 64-cell test.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Coordinate-level update invisibility is exactly testable: for any optimizer and quantizer, the code-equality condition decides whether a stored weight actually changed, independent of loss behavior.
  • Reference-derived frozen-fraction proxies forecast aggregate visibility timing well in controlled, same-schedule grids (94.5% of forecasts within 15% over an 11–4222-step span), but the paper's own transfer tests show the proxy is not transportable as a calibrated predictor.
  • Changing only the post-optimizer write-back from round-to-nearest to stochastic rounding removes the deterministic never-moved population and recovers 89.2–92.2% of the RNE-vs-fp32 loss gap at 10,000 steps in a 124M-parameter transformer.
  • A single 600-step source trace, frozen before three unseen target seeds, selects an exact-10% fp32-master tile mask that recovers a mean 60.7% of the gap, within 0.0014 of target-specific masks and 0.428 above the best fixed control; a depth-17 migration recovers 66.4%.
  • The current selective-master implementation reduces peak allocation by 584.65 MB versus a blanket master but adds 65.85 MB and lowers arm-median timed-window throughput by 7.196% versus direct round-to-nearest, so as implemented it has no validated break-even.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the near-tie between the pre-frozen source mask (60.7%) and privileged target-specific masks (60.8%) suggests a single short pilot run may suffice to pick protective masks for a whole family of seeds; a direct test would freeze one mask and evaluate across different batch orders, schedules, or widths within the same architecture.
  • Editorial inference: the audit's failure once measured and projected paths diverge suggests a practical hybrid: use the offline proxy for early planning, then switch to exact online code-equality monitoring once divergence is detected, instead of trusting the aggregate throughout training.
  • Editorial inference: the post hoc ECO-Adam comparator recovering 96.5% with unmeasured cost points to a question the paper leaves open — whether error-feedback compensation can replace selective master weights entirely, provided its memory and throughput costs are counted.
  • Editorial inference: because write-back-policy recovery declines between 3,000 and 10,000 steps, the benefit of stochastic rounding may shrink as runs lengthen; testing beyond 10,000 steps would show whether these short-horizon contrasts persist.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies two uses of high-precision reference traces for low-precision training: (i) auditing when stored-weight updates are invisible under round-to-nearest, and (ii) selecting an exact-budget fp32-master tile mask before unseen target runs. The exact visibility event is Eq. (1), Q_m(q_i+d_i)=q_i, with the interval equivalence in Eq. (2). The aggregate audit uses a projected-reference proxy, Eq. (4), which is explicitly not a pathwise certificate and is bounded only by a posterior diagnostic, Eq. (8). Controlled two-layer and char-GPT experiments show good forecast accuracy for first-majority-crossing times, while a locked 64-cell transfer and a modern-decoder bundle fail all predeclared uncertainty/informativeness gates. The write-back policy experiment reports that switching from RNE to stochastic rounding recovers 89.2–92.2% of the RNE-vs-fp32 loss gap after 10,000 GPT-2 steps. The prospective allocation study freezes a source-trace absolute-proposal-contribution mask before three unseen targets and reports mean recovery 0.607, close to target-specific masks (0.608) and well above fixed composition-matched controls (0.179). A systems check shows reduced PyTorch peak allocation versus a blanket master but net overhead and lower throughput versus direct RNE, with no validated break-even. The paper is unusually careful in stating its limitations, including the failures of the aggregate audit and the absence of a systems benefit.

Significance. If the results hold, the exact coordinate-level event in Eq. (1) is a clean, machine-checkable definition of update visibility, and the allocation protocol is a useful template for prospective, pre-registered evaluation of precision-allocation policies. The paper also reports negative results—the failure of the projected-reference proxy under code divergence, the failure of visibility-based allocation, and the lack of systems benefit—with unusual honesty and detail. However, the significance as a general method is limited: the aggregate audit has positive evidence only in small controlled testbeds and has direct negative evidence in the paper's own locked transfer test; the allocation result covers one architecture at two depths with three fixed target seeds; and the systems comparison shows no net benefit. The exact event and the write-back policy contrast are solid, but the broader 'reference traces for auditing' claim needs either substantial additional transfer evidence or a much narrower framing.

major comments (3)
  1. [§2.2, Eq. (4), Appendix B.1] The projected-reference proxy is the load-bearing component of the aggregate-audit claim, yet the paper's own locked transfer test shows it fails outside the controlled testbeds: measured and projected current-code vectors first diverge at update 3; the same-code premise holds in only 0.797% of 762,880,000 coordinate-events; and every predeclared uncertainty gate fails (AUROC 0.722 vs 0.656 comparator, risk 0.105 vs 0.059 required, frozen-fraction interval containment 0.368, onset interval 33/64). The modern-decoder bundle also finds visibility trajectories uninformative (range <0.10 and all directional-added-value gates fail). I acknowledge that the paper states these limitations explicitly, but the conclusion that reference-derived proxies 'give an empirically useful aggregate audit' is supported only in two-layer and small char-GPT regimes. The exact local event Eq. (1) is unaffected,
  2. [Data availability / Code availability] The paper makes extensive empirical claims backed by an elaborate artifact ledger and many hashes, but the Data availability and Code availability sections state that no public archive or repository exists and the Zenodo DOI remains private. A reader cannot verify any headline number—recoveries, gate outcomes, systems timings, or the locked 64-cell failure attribution. For a cs.LG paper whose contribution is primarily empirical and methodological, public release of code, data, and at least the locked protocols and validation outputs should be a condition of publication. This is a load-bearing reproducibility issue, not a stylistic one.
  3. [§5, Tables 17–18] The prospective allocation result is based on three fixed target seeds and five fixed composition-matched controls, and the paper correctly says no population inference is intended. However, the source-P mean recovery of 0.607 is only 0.00143 below the privileged target-specific P, and the post hoc ECO comparator recovers 0.965 (mean) with unmeasured systems cost. The claim that a source trace can 'guide exact-budget protection' is therefore demonstrated only as a fixed-design proof-of-concept against fixed controls, not as an effective or production-viable allocation rule. This limitation is present in the text, but the title and abstract give it less weight than the evidence supports. I would ask the authors to make this scoping prominent in the abstract and conclusion.
minor comments (6)
  1. [Abstract] The phrase '384x in time' should use the multiplication sign (384×) for consistency with the rest of the text.
  2. [§2.2] The sentence 'No low-precision observation enters Eq. (4)' is misleading because the target grid m enters through Q_m; clarify that no low-precision trajectory observation enters.
  3. [§4.2 / Figure 2] Figure 2(c) labels the E4M3 SR never-moved fraction as exactly '0' in one place and '5.9e-09' for E5M2; the text says 'at most 1.18e-8' and 'exactly zero.' Please harmonize the reporting of these near-zero values.
  4. [§5 / Figure 3] The caption says 'No confidence interval or hypothesis test is reported.' This is good, but the same statement should appear in the main text where the mean 0.607 is first introduced, since some readers will otherwise interpret the three fixed targets as a sample.
  5. [Reproducibility statement] The reproducibility statement claims a retained provenance bundle, but the Data/Code availability sections say no public record exists. These statements should be reconciled, ideally by making the artifacts public.
  6. [Throughout] The term 'audit' is used both for the exact event Eq. (1) and for the aggregate proxy Eq. (4). Given that the proxy is not a certificate, consider using 'forecast' or 'aggregate proxy' consistently to avoid conflating the two.

Circularity Check

0 steps flagged

No significant circularity: the exact event, proxy definition, and posterior bound are not reduced to their own outputs, and the paper explicitly disclaims the post hoc selection concerns that could otherwise look circular.

full rationale

The paper's derivation chain does not exhibit the recognized circularity patterns. Eq. (1), Q_m(q_i + d_i) = q_i, is an exact local event that is definitionally correct and not derived from any target outcome. Eq. (4) defines the projected-reference proxy exclusively from a high-precision trajectory: "No low-precision observation enters Eq. (4). Its nontrivial assumption is that the high-precision path remains a useful counterfactual near the visibility boundary." This is an explicit assumption, not a hidden fit; the paper's own locked 64-cell transfer (Appendix B.1) tests and falsifies it, and the manuscript reports that result rather than silently relying on it. Eq. (8) is presented as "a posterior diagnostic, not an a priori guarantee," so it is not claimed as a predicted certificate. The rho* = 1/sqrt(2) boundary is derived from stated uniform-phase, symmetric-spacing assumptions (Appendix A.1), not fitted to the later measurements. The prospective allocation study freezes the source mask before all target observations: "Before any target reference or low-precision outcome, the protocol freezes the top 21,120 tiles," and the paper explicitly acknowledges the score-selection history: "Absolute proposal contribution was selected after earlier GPT-2 outcomes and is a tested candidate rather than a novel or optimal saliency rule." That admission converts a potential post hoc selection concern into a stated limitation, not a circular derivation. There are no load-bearing self-citations: the paper is single-author and does not invoke a prior uniqueness theorem or own-work result as the basis for its central equations. The main weaknesses are external validity, limited regimes, and the acknowledged failure of the proxy transfer, which are correctness/risk issues rather than circularity. Therefore the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

No central parameter is fitted to data; thresholds (10% budget, 600-step window, 90% gate) are protocol choices, and the rho*=1/sqrt(2) threshold is derived from a uniform-phase assumption rather than regression. Load-bearing assumptions are the counterfactual reliability of the reference path, the uniform-phase population summary, and the first-order proposal-contribution proxy. No new physical entities are introduced.

axioms (3)
  • domain assumption A matched high-precision reference trajectory remains a useful counterfactual for the low-precision path near the rounding boundary (used to define bf_t in Eq. 4).
    Paper itself calls this 'nontrivial' in \S2.2, and its locked 64-cell transfer shows measured and projected paths diverge by update 3 with every uncertainty criterion failing; without this assumption the audit has no transfer.
  • domain assumption Within-binade phases of log2|w| are approximately equidistributed and update-to-weight ratios concentrate, so the scalar rho = eta||g||/(epsilon||w||) summarizes the population (Eq. 3).
    Used to justify the rho*=1/sqrt(2) median half-freeze boundary; paper labels it a heuristic population summary, not a replacement for code equality, and notes heterogeneous tensors or boundary mass can make it uninformative.
  • domain assumption Absolute proposal contribution |g_ti u_ti| is a first-order proxy for which coordinates deserve exact-budget protection.
    Invoked for the tile mask in \S5/Appendix E.2; paper explicitly says it is not novel or optimal and top-K ranking does not guarantee realized loss recovery; the experimental result is therefore conditional on this proxy being a useful selector.

pith-pipeline@v1.3.0-alltime-deepseek · 31579 in / 16733 out tokens · 166369 ms · 2026-08-02T07:49:10.577874+00:00 · methodology

0 comments
read the original abstract

Direct low-precision write-back can erase optimizer proposals, while aggregate update visibility need not identify parameters worth protecting. We study two uses of high-precision reference traces: candidate-matched auditing before a low-precision run and regime-matched allocation before unseen targets. The exact target-code event is distinct from aggregate proxies, which are not pathwise certificates. In a controlled two-layer grid, an archived proxy tracks crossings spanning 384x in time, with median predicted/measured ratio 1.000 and 94.5% within 15%. In analytic-grid GPT-2-124M/OpenWebText runs, changing only post-optimizer write-back from round-to-nearest to stochastic rounding recovers 89.2-92.2% of the loss gap through 10,000 steps. The effect repeats in a separate modern-decoder bundle, but its visibility trajectories are uninformative. In a fully prospective 162.2M-parameter allocation test, a 600-step source trace freezes a 10% fp32-master tile mask before three unseen targets. It recovers 60.4-61.0% (mean 60.7%), versus 60.5-61.5% (mean 60.8%) for target-specific masks and 17.5-18.2% for the best fixed composition-matched controls; freeze-only recovers 0.7-0.9%. A separately frozen 197.6M-parameter depth migration recovers 65.9-67.2% (mean 66.4%). A matched systems check reduces PyTorch peak allocation by 584.65 MB versus a blanket master, but adds 65.85 MB and has 7.196% lower arm-median timed-window throughput versus direct round-to-nearest, with no validated break-even. A post hoc same-grid ECO-Adam comparator recovers 96.4-96.6%, versus 60.4-61.0% for the source mask; its systems cost is unmeasured. Thus reference traces can audit aggregate visibility and separately guide exact-budget protection, but the audit did not transfer diagnostically and the current allocator has no net systems benefit.

Figures

Figures reproduced from arXiv: 2607.09800 by Zekai Shang.

Figure 1
Figure 1. Figure 1: FIG. 1. The [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Reference-trace forecast of stored-weight update visibility in controlled systems. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Matched write-back intervention with common target-grid initialization through 10,000 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prospective transfer of a fixed source-P allocation. Exact 10% eligible-2-D fp32-master payload in the tested 162.2M-parameter untied modern decoder/OpenWebText/analytic scale-1 E5M2/update-1000 regime. The scientific units are n = 3 fixed target training seeds: T1=1249306074, T2=1629484925, and T3=695340460. (a) Unclipped same-target RNE–fp32 gap recovery. Thin gray segments span the fixed-target range fo… view at source ↗
Figure 4
Figure 4. Figure 4: Historical effective-mantissa precision–rate diagrams. [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

76 extracted references · 3 canonical work pages

  1. [1]

    Unit scaling: Out-of-the-box low-precision training

    Charlie Blake, Douglas Orr, and Carlo Luschi. Unit scaling: Out-of-the-box low-precision training. In Proceedings of the 40th International Conference on Machine Learning, pages 2548--2576, 2023

    2023

  2. [3]

    Connolly, Nicholas J

    Michael P. Connolly, Nicholas J. Higham, and Theo Mary. Stochastic rounding and its probabilistic backward error analysis. SIAM Journal on Scientific Computing, 43 0 (1): 0 A566--A585, 2021. doi:10.1137/20M1334796

    doi:10.1137/20m1334796 2021

  3. [7]

    Mahoney, and Kurt Keutzer

    Zhen Dong, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. HAWQ : Hessian AW are quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 293--302, 2019. doi:10.1109/ICCV.2019.00038

    arXiv 2019

  4. [9]

    FracTrain : Fractionally squeezing bit savings both temporally and spatially for efficient DNN training

    Yonggan Fu, Haoran You, Yang Zhao, Yue Wang, Chaojian Li, Kailash Gopalakrishnan, Zhangyang Wang, and Yingyan Lin. FracTrain : Fractionally squeezing bit savings both temporally and spatially for efficient DNN training. In Advances in Neural Information Processing Systems, volume 33, 2020

    2020

  5. [10]

    OpenWebText Corpus

    Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. OpenWebText Corpus . Hugging Face dataset, 2019. URL https://huggingface.co/datasets/Skylion007/openwebtext

    2019

  6. [11]

    FlashOptim : Memory efficient optimizers for large-scale training

    Jose Javier Gonzalez Ortiz, Abhay Gupta, Christopher Rinard, and Davis Blalock. FlashOptim : Memory efficient optimizers for large-scale training. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/63497. Spotlight; arXiv:2602.23349v2

    arXiv 2026

  7. [12]

    Deep learning with limited numerical precision

    Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Proceedings of the 32nd International Conference on Machine Learning, pages 1737--1746, 2015

    2015

  8. [13]

    Speeding up and reducing memory usage for scientific machine learning via mixed precision

    Joel Hayford, Jacob Goldman-Wetzler, Eric Wang, and Lu Lu. Speeding up and reducing memory usage for scientific machine learning via mixed precision. Computer Methods in Applied Mechanics and Engineering, 428: 0 117093, 2024. doi:10.1016/j.cma.2024.117093. URL https://doi.org/10.1016/j.cma.2024.117093. arXiv:2401.16645

    arXiv 2024

  9. [15]

    Nicholas J. Higham. Accuracy and Stability of Numerical Algorithms. Society for Industrial and Applied Mathematics, second edition, 2002. doi:10.1137/1.9780898718027

    doi:10.1137/1.9780898718027 2002

  10. [16]

    IEEE Standard for Floating-Point Arithmetic

    IEEE . IEEE Standard for Floating-Point Arithmetic . IEEE Standard 754-2019, IEEE, 2019

    2019

  11. [17]

    Andrej Karpathy. nanoGPT . GitHub repository, 2025. URL https://github.com/karpathy/nanoGPT. Commit 3adf61e

    2025

  12. [19]

    QuRL : Low-precision reinforcement learning for efficient reasoning

    Yuhang Li, Reena Elangovan, Xin Dong, Priyadarshini Panda, and Brucek Khailany. QuRL : Low-precision reinforcement learning for efficient reasoning. In International Conference on Learning Representations, 2026. URL https://iclr.cc/virtual/2026/poster/10008335. Poster; matching arXiv:2602.13953v1 manuscript titled ``QuRL: Efficient Reinforcement Learning ...

    arXiv 2026

  13. [20]

    M+Adam : Low-precision training via additive--multiplicative optimization

    Xiaoyuan Liang, Sebastian Loeschcke, Mads Toftrup, and Anima Anandkumar. M+Adam : Low-precision training via additive--multiplicative optimization. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=XgVv56Y829. ICML 2026; arXiv:2607.10611v2

    Pith/arXiv arXiv 2026

  14. [23]

    Training with fewer bits: Unlocking edge LLMs training with stochastic rounding

    Taowen Liu, Marta Andronic, Deniz G\"und\"uz, and George Anthony Constantinides. Training with fewer bits: Unlocking edge LLMs training with stochastic rounding. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 14531--14546, 2025. doi:10.18653/v1/2025.findings-emnlp.784

    doi:10.18653/v1/2025.findings-emnlp.784 2025

  15. [24]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. arXiv:1711.05101

    Pith/arXiv arXiv 2019

  16. [26]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018

    2018

  17. [28]

    Pruning convolutional neural networks for resource efficient inference

    Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017

    2017

  18. [29]

    Sensitivity-guided mixed-precision quantization-aware training for large language models

    Gautam Krishna Mooppil, Abhijai Sasikumar, Abeer Mathur, and Subramaniyaswamy Vairavasundaram. Sensitivity-guided mixed-precision quantization-aware training for large language models. IEEE Access, 14: 0 84117--84145, 2026. doi:10.1109/ACCESS.2026.3698449

    arXiv 2026

  19. [30]

    ECO : Quantized training without full-precision master weights

    Mahdi Nikdan, Amir Zandieh, Dan Alistarh, and Vahab Mirrokni. ECO : Quantized training without full-precision master weights. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://openreview.net/forum?id=dfVm5yZZ6G. ICML 2026; arXiv:2601.22101v1

    arXiv 2026

  20. [31]

    Stochastic rounding for LLM training: Theory and practice

    Kaan Ozkara, Tao Yu, and Youngsuk Park. Stochastic rounding for LLM training: Theory and practice. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 4402--4410. PMLR, 2025. URL https://proceedings.mlr.press/v258/ozkara25b.html

    2025

  21. [32]

    Quartet II : Accurate LLM pre-training in NVFP4 by improved unbiased gradient estimation

    Andrei Panferov, Erik Schultheis, Soroush Tabesh, and Dan Alistarh. Quartet II : Accurate LLM pre-training in NVFP4 by improved unbiased gradient estimation. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL https://icml.cc/virtual/2026/poster/65532. Poster; arXiv:2601.22813v2

    Pith/arXiv arXiv 2026

  22. [34]

    Why low-precision transformer training fails: An analysis on flash attention

    Haiquan Qiu and Quanming Yao. Why low-precision transformer training fails: An analysis on flash attention. In International Conference on Learning Representations, 2026. arXiv:2510.04212

    Pith/arXiv arXiv 2026

  23. [35]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  24. [37]

    RoFormer : Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer : Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024. doi:10.1016/j.neucom.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063. arXiv:2104.09864

    arXiv 2024

  25. [39]

    Training LLMs with MXFP4

    Albert Tseng, Tao Yu, and Youngsuk Park. Training LLMs with MXFP4 . In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 1630--1638. PMLR, 2025

    2025

  26. [40]

    Training deep neural networks with 8-bit floating point numbers

    Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. In Advances in Neural Information Processing Systems, volume 31, 2018. URL https://arxiv.org/abs/1812.08011. arXiv:1812.08011

    Pith/arXiv arXiv 2018

  27. [41]

    Hochstenbach, and Barry Koren

    Lu Xia, Stefano Massei, Michiel E. Hochstenbach, and Barry Koren. On stochastic roundoff errors in gradient descent with low-precision computation. Journal of Optimization Theory and Applications, 200 0 (2): 0 634--668, 2024. doi:10.1007/s10957-023-02345-7

    work page doi:10.1007/s10957-023-02345-7 2024

  28. [42]

    Escaping the swamping zone: Fully low-precision training via relative ULP noise injection

    Zhenyu Yu. Escaping the swamping zone: Fully low-precision training via relative ULP noise injection. In Proceedings of the 2026 International Conference on Artificial Intelligence and Control, pages 568--572. ACM, 2026. doi:10.1145/3807246.3807335

    arXiv 2026

  29. [44]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, volume 32, pages 12360--12371, 2019. URL https://arxiv.org/abs/1910.07467. arXiv:1910.07467

    Pith/arXiv arXiv 2019

  30. [45]

    CoopQ : Cooperative game inspired layerwise mixed precision quantization for LLMs

    Junchen Zhao, Ali Derakhshan, Jayden Hyman, Junhao Dong, Sangeetha Abdu Jyothi, and Ian Harris. CoopQ : Cooperative game inspired layerwise mixed precision quantization for LLMs . In Findings of the Association for Computational Linguistics: ACL 2026, pages 7566--7578, 2026. doi:10.18653/v1/2026.findings-acl.373

    work page doi:10.18653/v1/2026.findings-acl.373 2026

  31. [46]

    Balancing fidelity and plasticity: Aligning mixed-precision fine-tuning with linguistic hierarchies

    Changhai Zhou, Shiyang Zhang, Yuhua Zhou, Jun Gao, Qian Qiao, Shichao Weng, Weizhong Zhang, and Cheng Jin. Balancing fidelity and plasticity: Aligning mixed-precision fine-tuning with linguistic hierarchies. In Findings of the Association for Computational Linguistics: ACL 2026, pages 15885--15896, 2026. doi:10.18653/v1/2026.findings-acl.779

    work page doi:10.18653/v1/2026.findings-acl.779 2026

  32. [47]

    International Conference on Learning Representations , year =

    Paulius Micikevicius and Sharan Narang and Jonah Alben and Gregory Diamos and Erich Elsen and David Garcia and Boris Ginsburg and Michael Houston and Oleksii Kuchaiev and Ganesh Venkatesh and Hao Wu , title =. International Conference on Learning Representations , year =

  33. [48]

    Proceedings of the 32nd International Conference on Machine Learning , pages =

    Suyog Gupta and Ankur Agrawal and Kailash Gopalakrishnan and Pritish Narayanan , title =. Proceedings of the 32nd International Conference on Machine Learning , pages =

  34. [49]

    Aberger and Christopher De Sa , title =

    Pedram Zamirai and Jian Zhang and Christopher R. Aberger and Christopher De Sa , title =. arXiv preprint arXiv:2010.06192 , year =

    Pith/arXiv arXiv 2010

  35. [50]

    Advances in Neural Information Processing Systems , volume =

    Naigang Wang and Jungwook Choi and Daniel Brand and Chia-Yu Chen and Kailash Gopalakrishnan , title =. Advances in Neural Information Processing Systems , volume =. 2018 , note =

    2018

  36. [51]

    Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , pages =

    Kaan Ozkara and Tao Yu and Youngsuk Park , title =. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , pages =. 2025 , url =

    2025

  37. [52]

    Hochstenbach and Barry Koren , title =

    Lu Xia and Stefano Massei and Michiel E. Hochstenbach and Barry Koren , title =. Journal of Optimization Theory and Applications , volume =. 2024 , doi =

    2024

  38. [53]

    Connolly and Nicholas J

    Michael P. Connolly and Nicholas J. Higham and Theo Mary , title =. SIAM Journal on Scientific Computing , volume =. 2021 , doi =

    2021

  39. [54]

    Computer Methods in Applied Mechanics and Engineering , volume =

    Joel Hayford and Jacob Goldman-Wetzler and Eric Wang and Lu Lu , title =. Computer Methods in Applied Mechanics and Engineering , volume =. 2024 , doi =

    2024

  40. [55]

    Proceedings of the 40th International Conference on Machine Learning , pages =

    Charlie Blake and Douglas Orr and Carlo Luschi , title =. Proceedings of the 40th International Conference on Machine Learning , pages =

  41. [56]

    arXiv preprint arXiv:2412.19437 , year =

    Pith/arXiv arXiv

  42. [57]

    arXiv preprint arXiv:2209.05433 , year =

    Paulius Micikevicius and Dusan Stosic and Neil Burgess and Marius Cornea and Pradeep Dubey and Richard Grisenthwaite and Sangwon Ha and Alexander Heinecke and Patrick Judd and John Kamalu and Naveen Mellempudi and Stuart Oberman and Mohammad Shoeybi and Michael Siu and Hao Wu , title =. arXiv preprint arXiv:2209.05433 , year =

    Pith/arXiv arXiv

  43. [58]

    2019 , url =

    Alec Radford and Jeffrey Wu and Rewon Child and David Luan and Dario Amodei and Ilya Sutskever , title =. 2019 , url =

    2019

  44. [59]

    2019 , howpublished =

    Aaron Gokaslan and Vanya Cohen and Ellie Pavlick and Stefanie Tellex , title =. 2019 , howpublished =

    2019

  45. [60]

    2025 , howpublished =

    Andrej Karpathy , title =. 2025 , howpublished =

    2025

  46. [61]

    International Conference on Learning Representations , year =

    Ilya Loshchilov and Frank Hutter , title =. International Conference on Learning Representations , year =

  47. [62]

    Higham , title =

    Nicholas J. Higham , title =. 2002 , doi =

    2002

  48. [63]

    Advances in Neural Information Processing Systems , volume =

    Biao Zhang and Rico Sennrich , title =. Advances in Neural Information Processing Systems , volume =. 2019 , note =

    2019

  49. [64]

    Neurocomputing , volume =

    Jianlin Su and Murtadha Ahmed and Yu Lu and Shengfeng Pan and Wen Bo and Yunfeng Liu , title =. Neurocomputing , volume =. 2024 , doi =

    2024

  50. [65]

    arXiv preprint arXiv:2002.05202 , year =

    Noam Shazeer , title =. arXiv preprint arXiv:2002.05202 , year =

    Pith/arXiv arXiv 2002

  51. [66]

    International Conference on Learning Representations , year =

    Haiquan Qiu and Quanming Yao , title =. International Conference on Learning Representations , year =

  52. [67]

    Proceedings of the 2026 International Conference on Artificial Intelligence and Control , pages =

    Zhenyu Yu , title =. Proceedings of the 2026 International Conference on Artificial Intelligence and Control , pages =. 2026 , doi =

    2026

  53. [68]

    arXiv preprint arXiv:2602.03839 , year =

    Erfan Miahi and Eugene Belilovsky , title =. arXiv preprint arXiv:2602.03839 , year =

    Pith/arXiv arXiv

  54. [69]

    arXiv preprint arXiv:2603.16731 , year =

    Kristi Topollai and Anna Choromanska , title =. arXiv preprint arXiv:2603.16731 , year =

    arXiv

  55. [70]

    Proceedings of the 43rd International Conference on Machine Learning , year =

    Xiaoyuan Liang and Sebastian Loeschcke and Mads Toftrup and Anima Anandkumar , title =. Proceedings of the 43rd International Conference on Machine Learning , year =

  56. [71]

    Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , pages =

    Albert Tseng and Tao Yu and Youngsuk Park , title =. Proceedings of the 28th International Conference on Artificial Intelligence and Statistics , pages =

  57. [72]

    Proceedings of the 43rd International Conference on Machine Learning , year =

    Andrei Panferov and Erik Schultheis and Soroush Tabesh and Dan Alistarh , title =. Proceedings of the 43rd International Conference on Machine Learning , year =

  58. [73]

    Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =

    Taowen Liu and Marta Andronic and Deniz G\"und\"uz and George Anthony Constantinides , title =. Findings of the Association for Computational Linguistics: EMNLP 2025 , pages =. 2025 , doi =

    2025

  59. [74]

    arXiv preprint arXiv:2606.22932 , year =

    Dikshant Kukreja and Kritarth Prasad and Avinash Anand and Zhengkui Wang and Erik Cambria and Timothy Liu and Aik Beng Ng and Simon See and Bapi Chatterjee , title =. arXiv preprint arXiv:2606.22932 , year =

    Pith/arXiv arXiv

  60. [75]

    Proceedings of the 43rd International Conference on Machine Learning , year =

    Mahdi Nikdan and Amir Zandieh and Dan Alistarh and Vahab Mirrokni , title =. Proceedings of the 43rd International Conference on Machine Learning , year =

  61. [76]

    arXiv preprint arXiv:2606.04115 , year =

    Giuseppe Franco and Ian Colbert and Pablo Monteagudo-Lago and Felix Marty and Nicholas Fraser , title =. arXiv preprint arXiv:2606.04115 , year =

    Pith/arXiv arXiv

  62. [77]

    arXiv preprint arXiv:2606.13300 , year =

    Mariya Pavlova and Harrison Bo Hua Zhu and Lidia Vitanova and Elizaveta Semenova and Yingzhen Li , title =. arXiv preprint arXiv:2606.13300 , year =

    Pith/arXiv arXiv

  63. [78]

    IEEE Access , volume =

    Gautam Krishna Mooppil and Abhijai Sasikumar and Abeer Mathur and Subramaniyaswamy Vairavasundaram , title =. IEEE Access , volume =. 2026 , doi =

    2026

  64. [79]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages =

    Changhai Zhou and Shiyang Zhang and Yuhua Zhou and Jun Gao and Qian Qiao and Shichao Weng and Weizhong Zhang and Cheng Jin , title =. Findings of the Association for Computational Linguistics: ACL 2026 , pages =. 2026 , doi =

    2026

  65. [80]

    Findings of the Association for Computational Linguistics: ACL 2026 , pages =

    Junchen Zhao and Ali Derakhshan and Jayden Hyman and Junhao Dong and Sangeetha Abdu Jyothi and Ian Harris , title =. Findings of the Association for Computational Linguistics: ACL 2026 , pages =. 2026 , doi =

    2026

  66. [81]

    International Conference on Learning Representations , year =

    Yuhang Li and Reena Elangovan and Xin Dong and Priyadarshini Panda and Brucek Khailany , title =. International Conference on Learning Representations , year =

  67. [82]

    Proceedings of the 43rd International Conference on Machine Learning , year =

    Gonzalez Ortiz, Jose Javier and Abhay Gupta and Christopher Rinard and Davis Blalock , title =. Proceedings of the 43rd International Conference on Machine Learning , year =

  68. [83]

    arXiv preprint arXiv:2607.04422 , year =

    Siyu Ding and Mingchuan Ma and Jiabo Tong and Xingrun Xing and Ziming Wang and Guoqi Li , title =. arXiv preprint arXiv:2607.04422 , year =

    Pith/arXiv arXiv

  69. [84]

    arXiv preprint arXiv:2605.09825 , year =

    Musa Cim and Poovaiah Palangappa and Miro Hodak and Ravi Dwivedula and Meena Arunachalam and Mahmut Taylan Kandemir , title =. arXiv preprint arXiv:2605.09825 , year =

    Pith/arXiv arXiv

  70. [85]

    International Conference on Learning Representations , year =

    Pavlo Molchanov and Stephen Tyree and Tero Karras and Timo Aila and Jan Kautz , title =. International Conference on Learning Representations , year =

  71. [86]

    Mahoney and Kurt Keutzer , title =

    Zhen Dong and Zhewei Yao and Amir Gholami and Michael W. Mahoney and Kurt Keutzer , title =. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages =. 2019 , doi =

    2019

  72. [87]

    Advances in Neural Information Processing Systems , volume =

    Yonggan Fu and Haoran You and Yang Zhao and Yue Wang and Chaojian Li and Kailash Gopalakrishnan and Zhangyang Wang and Yingyan Lin , title =. Advances in Neural Information Processing Systems , volume =

  73. [88]

    arXiv preprint arXiv:2605.04711 , year =

    Kang Liu and Wei Peng and Jianchen Hu , title =. arXiv preprint arXiv:2605.04711 , year =

    Pith/arXiv arXiv

  74. [89]

    arXiv preprint arXiv:2605.00649 , year =

    Michael Helcig and Dan Alistarh , title =. arXiv preprint arXiv:2605.00649 , year =

    Pith/arXiv arXiv

  75. [90]

    arXiv preprint arXiv:2604.06836 , year =

    Minglu Liu and Cunchen Hu and Liangliang Xu and Fengming Tang and Ruijia Wang and Fu Yu , title =. arXiv preprint arXiv:2604.06836 , year =

    Pith/arXiv arXiv

  76. [91]

    arXiv preprint arXiv:2607.10803 , year =

    Shrestha Datta and Hongfu Liu and Anshuman Chhabra , title =. arXiv preprint arXiv:2607.10803 , year =

    Pith/arXiv arXiv