REVIEW 4 major objections 3 minor 11 references
An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits
T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper shows that inserting an extra RMSNorm before every linear projection, alongside a gradual quantization schedule, lets full-precision LLMs be fine-tuned directly to 1.58-bit ternary weights and match or beat…
desk verdict A recipe borrowed from prior work, presented without a single number to back it up; the only new claim is a comparison against KD, and that comparison is not reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is an extra RMSNorm (root-mean-square normalization that divides by the activation vector's norm without subtracting the mean), inserted at the input of every quantized linear projection: in a block $y = W_2\,\sigma(W_1 x)$, one RMSNorm sits on $x$ before $W_1$ and another on $\sigma(W_1 x)$ before $W_2$, with the same pattern on attention projections. It keeps the input scale seen by each ternary weight matrix stable. Two other components carry the argument: the BitLinear layer with straight-through-estimator fake quantization, and the two-phase schedule $\lambda(t)=\min(2t/T,1)$ that keeps the model full-precision at the start and fully quantized by the midpoint of training. The paper reports that removing the extra norms leads to divergence or higher final loss.
What would settle it
Repeat the Qwen-1.5B run twice with the added RMSNorm layers initialized to preserve the original forward pass (identity scale) versus initialized randomly, keeping data, schedule, and all other settings fixed; if final cross-entropy differs materially, the success of the recipe depends on an unstated initialization choice rather than on RMSNorm itself.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that extra RMSNorm at every quantized linear input plus a gradual quantization schedule is enough to fine-tune full-precision LLMs into stable ternary models. The authors replace each dense layer with a BitLinear layer that fake-quantizes weights to {-1, 0, +1} with an average-absolute-value scale and uses a straight-through estimator so gradients ignore the rounding. A lambda schedule interpolates from full precision at step zero to full quantization at the midpoint of training. Direct cross-entropy fine-tuning with the inserted norms reaches lower final cross-entropy than layer-wise knowledge distillation, and adding KD on top does not substantially improve it. On AIME-2024 and MATH-500, the ternary models show only negligible accuracy drops relative to full-precision baselines.
Load-bearing premise
The load-bearing premise is that inserting brand-new normalization layers into an already-trained model shifts the forward pass less than it stabilizes later training, even though the paper never reports how those layers are initialized; if the first forward pass is badly off, no amount of later stabilization can recover the pretrained knowledge.
Editorial extensions
If this is right
- Ternary fine-tuning no longer requires a frozen teacher model: direct cross-entropy with extra RMSNorm outperforms layer-wise knowledge distillation, so low-bit pipelines can be much simpler.
- Existing full-precision checkpoints can be converted to 1.58-bit by replacing linear layers with BitLinear and inserting RMSNorm, then fine-tuning on plain language modeling data, rather than retraining from scratch.
- Memory savings make large models fine-tunable on commodity hardware; the paper reports that 70B-class models fit on a single 24GB GPU at ternary precision.
- Math-heavy benchmarks (AIME-2024, MATH-500) keep near full-precision accuracy, so the recipe is not limited to casual language modeling.
- Because the method works for both Qwen-1.5B and Llama3-8B without architecture-specific changes, it should transfer to other decoder-only transformer checkpoints.
Reading between the lines
- My inference: the same normalization cure likely carries over to activation quantization, because stable input norms make activation ranges more predictable; the paper leaves activation quantization to future work.
- My inference: the unstated initialization of the inserted RMSNorm layers is a hidden critical detail. I would expect identity-preserving initialization to be necessary or strongly beneficial, since a randomly initialized norm would perturb the pretrained forward pass on the first step.
- My inference: the near-parity math results suggest ternary models could act as drop-in reasoners on memory-limited hardware, but two benchmarks are too few to conclude that, and this goes beyond what the paper claims.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a quantization-aware fine-tuning recipe for pretrained large language models: insert an extra RMSNorm layer before every quantized linear projection, remove biases, use straight-through estimation, and anneal a lambda parameter from 0 to 1 to transition gradually from full-precision to ternary (-1,0,+1) weights. The method is tested on Qwen-1.5B and Llama3-8B fine-tuned on OpenThoughts-114k. The authors claim that this direct QAT approach achieves lower final cross-entropy than layer-wise knowledge distillation, that the additional RMSNorm is the key stability ingredient, and that downstream accuracy on AIME-2024 and MATH-500 is negligibly degraded relative to full-precision baselines. The paper also asserts that the resulting memory footprint allows fine-tuning 70B-parameter models on a single 24GB GPU.
Significance. If the central claim is correct, that a simple architectural modification plus a gradual quantization schedule can replace knowledge distillation for ternary LLM fine-tuning, the result would be practically valuable: it would lower the complexity and overhead of ultra-low-bit training and improve accessibility. The paper does not, however, provide any reproducible evidence. It reports no numerical values, no tables, no error bars, no training details, and no code. The figures mentioned in the text are absent. The RMSNorm insertion point is not fully specified, so even the method section is not self-contained. The 70B/24GB claim, as stated, is unsupported and, under standard full-model fine-tuning with master weights and Adam states, physically implausible. The paper is better viewed as a short technical report than as a peer-reviewable research paper in its current form.
major comments (4)
- [Experiments and Results (Figures 1-3)] The central empirical claims — that Direct QAT + RMSNorm achieves the lowest final cross-entropy and that AIME-2024 and MATH-500 accuracy show only a negligible drop — are stated without any numerical results. No cross-entropy values, no perplexity numbers, no accuracy percentages, no standard deviations, no seeds, and no statistical comparisons are given. The three figures are captions only; the figures themselves are not present in the manuscript. This makes the paper's contribution impossible to verify or compare against prior work, and it is load-bearing because the entire value of the paper is an empirical recipe.
- [Method, BitLinear Layer with STE Quantization and RMSNorm Insertion] The manuscript never states how the inserted RMSNorm layers are initialized. Adding RMSNorm(x) before W1 and RMSNorm(sigma(W1 x)) before W2 changes the pretrained forward pass at lambda=0 unless the normalization parameters are initialized to compensate for the original per-layer activation statistics. If the default RMSNorm weight initialization (weight=1) is used, every activation entering the quantized linear layers is rescaled on the first forward pass, and the reported training stability may reflect the model absorbing this forced rescaling rather than the proposed method. Without specifying the initialization, warm-up, or any ablation over initialization choices, the paper is not reproducible and the headline claim is not testable.
- [Fitting Larger Models on Commodity Hardware] The assertion that the method enables loading and fine-tuning a 70B-parameter model on a single 24GB GPU is unsupported and, as written, implausible. Full-model updates with STE require full-precision master weights, gradients, and (for Adam-style optimizers) two momentum buffers; for 70B parameters this alone exceeds 24GB by more than an order of magnitude, even before activations and the inserted RMSNorm layers are accounted for. The manuscript provides no memory profiling, no optimizer specification, no gradient checkpointing details, and no description of any memory-saving technique that would make the claim credible. This claim should either be substantiated with a detailed memory breakdown or removed.
- [Method, Gradual Quantization via Lambda Scheduling] The schedule lambda(t) = min(a*(t/T), 1) with a=2 is reported to be 'more effective than a slow linear ramp or abrupt shifts', but no comparison data, sensitivity analysis, or ablation is shown. Since this schedule is a fixed component of the proposed recipe, its claimed optimality over alternatives is a load-bearing part of the method and requires quantitative support in the form of learning curves or final-loss values for each schedule.
minor comments (3)
- [References] References [4] and [8] point to a Medium blog post and a Reddit thread, respectively; these are not stable or peer-reviewed citations and should be replaced with archival sources where available.
- [Model and Dataset Details] The training setup is underspecified: no learning rate, batch size, number of steps, sequence length, optimizer, or hardware details for the Qwen-1.5B and Llama3-8B runs are given, and the exact Hugging Face model identifiers for 'Qwen-1.5B' and 'Llama3-8B' are not stated.
- [Experiments and Results] The manuscript refers to performance 'on par with full-precision baselines' but never defines the full-precision baseline setup, including whether the baseline includes the extra RMSNorm layers without quantization or the original unmodified checkpoint.
Circularity Check
No circularity: the central claim is an empirical comparison, and no prediction reduces by construction to a fitted parameter or to a self-citation.
full rationale
The paper makes an empirical claim: inserting RMSNorm before quantized linear projections plus a lambda-scheduled straight-through estimator permits stable ternary fine-tuning. This claim is tested by comparing final cross-entropy against a baseline QAT and a KD variant, and by ablating the RMSNorm insertion. No equation in the paper defines the reported success metric in terms of the method's fitted parameters; the only schedules and modifications are fixed ingredients, not parameters fitted to the target benchmarks. The RMSNorm insertion is explicitly attributed to BitNet [5] ('following BitNet [5], we add an additional normalization right before every weight projection that is quantized'), so the paper does not present it as a novel derivation from first principles, and it does not rely on a self-citation to justify the choice. There is a reproducibility concern, since the initialization of the inserted RMSNorm layers is not stated and at lambda=0 the inserted RMSNorm is not an identity operation, but that is a correctness or reporting risk rather than circularity in the sense of an output being equivalent to an input by construction. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (1)
- lambda ramp coefficient a =
2
assumptions (3)
- domain assumption Straight-through estimation provides usable gradients through the ternary quantization step.
- ad hoc to paper Existing pretrained Transformer checkpoints can absorb additional RMSNorm layers and still be fine-tuned to ternary weights.
- domain assumption OpenThoughts-114k next-token prediction is a suitable training signal for preserving AIME-2024 and MATH-500 reasoning accuracy.
Cite this review
Pith. "Pith review of An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits." pith.science (2026). https://pith.science/paper/2ETFNI4M
@misc{pith2026250508823,
author = {Pith},
title = {Pith review of: An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits},
year = {2026},
howpublished = {\url{https://pith.science/paper/2ETFNI4M}},
note = {Machine review of arXiv:2505.08823}
}
read the original abstract
Large language models (LLMs) have transformed natural-language processing, yet their scale makes real-world deployment costly. Post-training quantization reduces memory and computation but often degrades accuracy, while quantization-aware training can recover performance at the cost of extra training. Pushing quantization to the ternary (2-bit) regime yields even larger savings but is notoriously unstable. Building on recent work showing that a bias-free, RMS-normalized Transformer with straight-through estimation can reach 1.58-bit precision, we demonstrate that simply inserting RMS normalization before every linear projection and applying a gradual, layer-wise quantization schedule stably fine-tunes full-precision checkpoints into ternary LLMs. Our approach matches or surpasses more elaborate knowledge-distillation pipelines on standard language-modeling benchmarks without adding model complexity. These results indicate that careful normalization alone can close much of the accuracy gap between ternary and full-precision LLMs, making ultra-low-bit inference practical.
Reference graph
Works this paper leans on
-
[1]
Introduction to Quantization, Hugging Face Blog (https://huggingface.co/blog/merve/quantization)
-
[2]
Quantization-Aware Training for Large Language Models with PyTorch, PyTorch Blog (https://pytorch.org/blog/quantization-aware-training)
-
[3]
Scalable MatMul-free Language Modeling (https://arxiv.org/abs/2406.02528)
-
[4]
Distilling the Knowledge in a Neural Network,
Back to Basics: Understanding the Foundational Paper on “Distilling the Knowledge in a Neural Network,” Medium (https://medium.com/@nayounghoon0223/back-to-the-basic-a-foundational-knowledge-distillation-paper-of-distilling-the-knowledge-in-a-10fba70bab3c)
-
[5]
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (https://arxiv.org/abs/2402.17764)
-
[6]
Zhu, C., et al. Trained Ternary Quantization. OpenReview (https://openreview.net/pdf?id=S1_pAu9xl)
-
[7]
Root Mean Square Layer Normalization
Zhang, B., et al. Root Mean Square Layer Normalization. arXiv:1910.07467 (https://arxiv.org/abs/1910.07467)
arXiv 1910
-
[8]
Straight-through estimator for the argmax function? Reddit (https://www.reddit.com/r/MachineLearning/comments/53uln9/straightthrough_estimator_for_the_argmax_function)
Show all 11 references
-
[9]
Layer-wise Convolutional Neural Network Distillation, OpenReview (https://par.nsf.gov/servlets/purl/10171699)
-
[10]
Model compression through distillation with cross-layer integrated..., ScienceDirect (https://www.sciencedirect.com/science/article/abs/pii/S0925231224019337)
-
[11]
Fine-tuning LLMs to 1.58bit: extreme quantization made easy, Hugging Face Blog (https://huggingface.co/blog/1_58_llm_extreme_quantization)
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.