{"id":"21cd5fe1-c723-4008-94d0-022ca3da756b","arxiv_id":"2505.08823","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning a pretrained LLM with extra RMSNorm before each quantized linear layer and a gradual two-phase quantization schedule can reach ternary 1.58-bit weights, according to the paper, but the supporting evidence is not reported.","lead":"This paper claims that adding an extra RMSNorm layer before every linear projection, plus a gradual quantization schedule, lets you fine-tune large language models down to ternary 1.58-bit weights without hurting accuracy. The practical idea is attractive, but the paper reports almost no numbers, code, or evaluation details to back the claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RMSNorm insertion is not identity at initialization; the method description omits how the new layers are initialized, so the claim that extra RMSNorm alone stabilizes ternary fine-tuning is not reproducible or testable.","rationale":"The reader's verdict is REJECT, and this stress-test pass does not change that. The paper's central empirical claims are unsupported by any reported measurements, tables, or code, but the most technically load-bearing gap in the method itself is the missing initialization and warm-up behavior of the inserted RMSNorm layers. The method section specifies where the norms are inserted, but not what their initial scales are, and the block equation y = W2 * sigma(W1 x) shows that adding RMSNorm before W1 and W2 changes the function implemented by the pretrained model at initialization. Since the lambda schedule only smooths the quantization transition, not the architectural modification, the stability claim depends entirely on how the new norms are initialized. This is exactly the weakest assumption the reader identified, so there is agreement. The recommendation remains REJECT: even if the hidden initialization is benign, the paper as written does not provide enough information to reproduce the experiments, and the absence of numerical results makes the central claims untestable. No change to the reader's verdict is needed.","tokens_in":6351,"tokens_out":4551,"duration_ms":50667,"concrete_test":"Run the same recipe on Qwen-1.5B with three inserted-RMSNorm initializations: (a) scale initialized to the inverse of the pretrained per-layer input RMS so the forward pass is exactly preserved at lambda=0, (b) default RMSNorm weight=1, and (c) a random scale. Record the training loss at step 1 and the final cross-entropy for each. If (b) or (c) diverges or yields substantially higher loss than (a), the reported stability depends on a hidden initialization choice rather than on the mere presence of extra RMSNorm layers, and the method description is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that adding RMSNorm before every linear projection, plus a gradual lambda schedule, is sufficient for stable ternary fine-tuning. But inserting an RMSNorm into a pretrained block is not an identity operation at lambda=0. In the method's own example, y = W2 * sigma(W1 x) becomes y = W2 * RMSNorm(sigma(W1 x)), and the input to W1 is also normalized. Unless the new RMSNorm weights are initialized to compensate for the original per-layer input statistics, every activation entering each quantized linear layer is rescaled on the very first forward pass. The manuscript never states the initialization, nor any warm-up schedule for the inserted norms. Because quantization is enabled only gradually through lambda, the model is already altered from the pretrained checkpoint at lambda=0, so the claim of stable fine-tuning depends on an unstated implementation detail. If the default RMSNorm initialization (weight=1) was used, the step-0 forward pass differs from the full-precision backbone by more than quantization error, and observed training stability may reflect how quickly the model absorbs this forced rescaling rather than the proposed method. Another initialization could produce divergence or much higher final loss, which would directly contradict the headline that the extra RMSNorm alone is all that is needed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a quantization-aware fine-tuning recipe for pretrained large language models: insert an extra RMSNorm layer before every quantized linear projection, remove biases, use straight-through estimation, and anneal a lambda parameter from 0 to 1 to transition gradually from full-precision to ternary (-1,0,+1) weights. The method is tested on Qwen-1.5B and Llama3-8B fine-tuned on OpenThoughts-114k. The authors claim that this direct QAT approach achieves lower final cross-entropy than layer-wise knowledge distillation, that the additional RMSNorm is the key stability ingredient, and that downstream accuracy on AIME-2024 and MATH-500 is negligibly degraded relative to full-precision baselines. The paper also asserts that the resulting memory footprint allows fine-tuning 70B-parameter models on a single 24GB GPU.","tokens_in":6541,"tokens_out":3310,"duration_ms":33237,"significance":"If the central claim is correct, that a simple architectural modification plus a gradual quantization schedule can replace knowledge distillation for ternary LLM fine-tuning, the result would be practically valuable: it would lower the complexity and overhead of ultra-low-bit training and improve accessibility. The paper does not, however, provide any reproducible evidence. It reports no numerical values, no tables, no error bars, no training details, and no code. The figures mentioned in the text are absent. The RMSNorm insertion point is not fully specified, so even the method section is not self-contained. The 70B/24GB claim, as stated, is unsupported and, under standard full-model fine-tuning with master weights and Adam states, physically implausible. The paper is better viewed as a short technical report than as a peer-reviewable research paper in its current form.","major_comments":[{"comment":"The central empirical claims — that Direct QAT + RMSNorm achieves the lowest final cross-entropy and that AIME-2024 and MATH-500 accuracy show only a negligible drop — are stated without any numerical results. No cross-entropy values, no perplexity numbers, no accuracy percentages, no standard deviations, no seeds, and no statistical comparisons are given. The three figures are captions only; the figures themselves are not present in the manuscript. This makes the paper's contribution impossible to verify or compare against prior work, and it is load-bearing because the entire value of the paper is an empirical recipe.","section":"Experiments and Results (Figures 1-3)"},{"comment":"The manuscript never states how the inserted RMSNorm layers are initialized. Adding RMSNorm(x) before W1 and RMSNorm(sigma(W1 x)) before W2 changes the pretrained forward pass at lambda=0 unless the normalization parameters are initialized to compensate for the original per-layer activation statistics. If the default RMSNorm weight initialization (weight=1) is used, every activation entering the quantized linear layers is rescaled on the first forward pass, and the reported training stability may reflect the model absorbing this forced rescaling rather than the proposed method. Without specifying the initialization, warm-up, or any ablation over initialization choices, the paper is not reproducible and the headline claim is not testable.","section":"Method, BitLinear Layer with STE Quantization and RMSNorm Insertion"},{"comment":"The assertion that the method enables loading and fine-tuning a 70B-parameter model on a single 24GB GPU is unsupported and, as written, implausible. Full-model updates with STE require full-precision master weights, gradients, and (for Adam-style optimizers) two momentum buffers; for 70B parameters this alone exceeds 24GB by more than an order of magnitude, even before activations and the inserted RMSNorm layers are accounted for. The manuscript provides no memory profiling, no optimizer specification, no gradient checkpointing details, and no description of any memory-saving technique that would make the claim credible. This claim should either be substantiated with a detailed memory breakdown or removed.","section":"Fitting Larger Models on Commodity Hardware"},{"comment":"The schedule lambda(t) = min(a*(t/T), 1) with a=2 is reported to be 'more effective than a slow linear ramp or abrupt shifts', but no comparison data, sensitivity analysis, or ablation is shown. Since this schedule is a fixed component of the proposed recipe, its claimed optimality over alternatives is a load-bearing part of the method and requires quantitative support in the form of learning curves or final-loss values for each schedule.","section":"Method, Gradual Quantization via Lambda Scheduling"}],"minor_comments":[{"comment":"References [4] and [8] point to a Medium blog post and a Reddit thread, respectively; these are not stable or peer-reviewed citations and should be replaced with archival sources where available.","section":"References"},{"comment":"The training setup is underspecified: no learning rate, batch size, number of steps, sequence length, optimizer, or hardware details for the Qwen-1.5B and Llama3-8B runs are given, and the exact Hugging Face model identifiers for 'Qwen-1.5B' and 'Llama3-8B' are not stated.","section":"Model and Dataset Details"},{"comment":"The manuscript refers to performance 'on par with full-precision baselines' but never defines the full-precision baseline setup, including whether the baseline includes the extra RMSNorm layers without quantization or the original unmodified checkpoint.","section":"Experiments and Results"}],"recommendation":"reject","confidential_remarks":"The paper reads like an extended abstract or student-project report rather than a complete research paper. The missing figures, numerical results, and implementation details are too fundamental to be fixed by minor revision. The 70B/24GB claim in particular would require a substantial redesign of the training setup (e.g., quantized optimizers, offloading) that is not described. I recommend the authors resubmit after adding full experimental tables, ablations on RMSNorm initialization, and a realistic memory analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central claim is that inserting extra RMSNorm before quantized linear layers, plus a gradual lambda schedule, lets you fine-tune full-precision checkpoints to 1.58-bit ternary weights, and that this beats layer-wise knowledge distillation. That's an interesting question, and the paper deserves credit for asking it directly and for honestly attributing the core ingredients to BitNet and the Hugging Face blog rather than pretending they are new. The writing is clear about what the method does, and the ablation logic (remove norms, watch it diverge) is the right kind of ablation to run.\n\nBut the paper as written cannot be evaluated, because it contains no numbers. There are no tables, no final cross-entropy values, no accuracy percentages for AIME-2024 or MATH-500, no hyperparameter settings, no seeds, no error bars. The experiments section is a series of assertions about what the figures show, but the figures themselves are absent from the text. The KD comparison, which is the only thing that might be new, is described qualitatively as \"matches or surpasses\" with no supporting numbers.\n\nThe 70B-on-a-single-24GB-GPU claim is also a red flag. Ternary weights for a 70B model take about 14GB, but full-model fine-tuning requires optimizer states, activations, and gradients. Those do not fit in 24GB for a 70B transformer on commodity hardware without offloading or adapters. No memory profiling is shown, and the claim looks physically implausible.\n\nThe stress-test concern about RMSNorm initialization is a real gap. Adding an RMSNorm before each linear layer does change the forward pass at step 0, since RMSNorm rescales the input vector. The paper never says how the new norms are initialized or whether there is any warm-up. That is a missing detail that matters for reproducibility, but it is a minor issue compared to the complete absence of empirical data. You cannot evaluate the method or the comparison because the evidence is simply not there.\n\nWho gets value from this? A reader who wants a concise summary of a plausible recipe for ternary fine-tuning might skim it, but they would get the same information from the cited BitNet paper and the Hugging Face blog. The paper's only potential contribution, the KD comparison, is unverifiable.\n\nI would not send this to peer review in its current state. It is not that the method is wrong; it is that the paper fails to provide any measurements. I would tell the editor to ask the authors for actual numbers, code, and a memory breakdown, and only then consider a resubmission.","headline":"A recipe borrowed from prior work, presented without a single number to back it up; the only new claim is a comparison against KD, and that comparison is not reported.","tokens_in":7119,"tokens_out":1765,"would_cite":false,"duration_ms":20287,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that inserting an extra RMSNorm before every linear projection, alongside a gradual quantization schedule, lets full-precision LLMs be fine-tuned directly to 1.58-bit ternary weights and match or beat…","keywords":["ternary quantization","1.58-bit LLM","RMSNorm insertion","quantization-aware training","straight-through estimator","knowledge distillation","lambda quantization schedule"],"falsifier":"Repeat the Qwen-1.5B run twice with the added RMSNorm layers initialized to preserve the original forward pass (identity scale) versus initialized randomly, keeping data, schedule, and all other settings fixed; if final cross-entropy differs materially, the success of the recipe depends on an unstated initialization choice rather than on RMSNorm itself.","tokens_in":6096,"feed_emoji":"🧠","tokens_out":9826,"duration_ms":91485,"temperature":0.7,"pith_summary":"Large language models are too big to deploy cheaply, and pushing them to ternary weights (only -1, 0, +1 per weight) usually makes training unstable. This paper argues that most of that instability is an input-scale problem: when a ternary weight matrix multiplies activations whose norm drifts, the output swings wildly. The proposed cure is to insert an extra RMSNorm layer before every linear projection and to ramp the quantization strength from zero to full over the first half of fine-tuning. Fine-tuning Qwen-1.5B and Llama3-8B this way beats both unnormalized quantization-aware training and a layer-wise knowledge-distillation pipeline on final cross-entropy. The practical stake is that 1.58-bit LLMs could be produced by a simple drop-in modification instead of teacher-student training.","feed_headline":"One extra RMSNorm stabilizes 1.58-bit LLM fine-tuning","feed_subtitle":"Direct quantized fine-tuning beats knowledge distillation and holds math accuracy near full precision.","key_machinery":"The central object is an extra RMSNorm (root-mean-square normalization that divides by the activation vector's norm without subtracting the mean), inserted at the input of every quantized linear projection: in a block $y = W_2\\,\\sigma(W_1 x)$, one RMSNorm sits on $x$ before $W_1$ and another on $\\sigma(W_1 x)$ before $W_2$, with the same pattern on attention projections. It keeps the input scale seen by each ternary weight matrix stable. Two other components carry the argument: the BitLinear layer with straight-through-estimator fake quantization, and the two-phase schedule $\\lambda(t)=\\min(2t/T,1)$ that keeps the model full-precision at the start and fully quantized by the midpoint of training. The paper reports that removing the extra norms leads to divergence or higher final loss.","core_discovery":"On the paper's own terms, the discovery is that extra RMSNorm at every quantized linear input plus a gradual quantization schedule is enough to fine-tune full-precision LLMs into stable ternary models. The authors replace each dense layer with a BitLinear layer that fake-quantizes weights to {-1, 0, +1} with an average-absolute-value scale and uses a straight-through estimator so gradients ignore the rounding. A lambda schedule interpolates from full precision at step zero to full quantization at the midpoint of training. Direct cross-entropy fine-tuning with the inserted norms reaches lower final cross-entropy than layer-wise knowledge distillation, and adding KD on top does not substantially improve it. On AIME-2024 and MATH-500, the ternary models show only negligible accuracy drops relative to full-precision baselines.","pith_inferences":["My inference: the same normalization cure likely carries over to activation quantization, because stable input norms make activation ranges more predictable; the paper leaves activation quantization to future work.","My inference: the unstated initialization of the inserted RMSNorm layers is a hidden critical detail. I would expect identity-preserving initialization to be necessary or strongly beneficial, since a randomly initialized norm would perturb the pretrained forward pass on the first step.","My inference: the near-parity math results suggest ternary models could act as drop-in reasoners on memory-limited hardware, but two benchmarks are too few to conclude that, and this goes beyond what the paper claims."],"forward_implications":["Ternary fine-tuning no longer requires a frozen teacher model: direct cross-entropy with extra RMSNorm outperforms layer-wise knowledge distillation, so low-bit pipelines can be much simpler.","Existing full-precision checkpoints can be converted to 1.58-bit by replacing linear layers with BitLinear and inserting RMSNorm, then fine-tuning on plain language modeling data, rather than retraining from scratch.","Memory savings make large models fine-tunable on commodity hardware; the paper reports that 70B-class models fit on a single 24GB GPU at ternary precision.","Math-heavy benchmarks (AIME-2024, MATH-500) keep near full-precision accuracy, so the recipe is not limited to casual language modeling.","Because the method works for both Qwen-1.5B and Llama3-8B without architecture-specific changes, it should transfer to other decoder-only transformer checkpoints."],"supporting_citations":[{"why":"Supplies the BitNet architecture precedent that an extra RMSNorm before each quantized linear is crucial for 1.58-bit LLMs.","marker":"[5]"},{"why":"Provides the open-source fine-tuning implementation whose BitLinear pseudocode and two-phase lambda schedule the method adapts.","marker":"[11]"},{"why":"Documents normalization and stability measures needed in ultra-low-precision language modeling, cited alongside [5] for the importance of normalization.","marker":"[3]"},{"why":"Defines RMSNorm, the normalization layer the paper inserts before every linear projection.","marker":"[7]"},{"why":"Supplies the straight-through estimator used to keep gradients flowing through the ternary quantization step.","marker":"[8]"},{"why":"Establishes trained ternary quantization as the lineage of the -1/0/+1 weight representation.","marker":"[6]"},{"why":"One of the layer-wise distillation baselines the paper compares against and finds unnecessary when RMSNorm is present.","marker":"[9]"}],"fun_headline_variants":["Extra RMSNorm stabilizes fine-tuning LLMs to 1.58 bits","RMSNorm before each linear layer stabilizes 1.58-bit LLMs","Inserting one RMSNorm makes ternary fine-tuning stable","Extra RMSNorm alone matches KD for 1.58-bit LLMs","Gradual quantization plus RMSNorm stabilizes ternary LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that inserting brand-new normalization layers into an already-trained model shifts the forward pass less than it stabilizes later training, even though the paper never reports how those layers are initialized; if the first forward pass is badly off, no amount of later stabilization can recover the pretrained knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Extra RMSNorm stabilizes fine-tuning LLMs to 1.58 bits","RMSNorm before each linear layer stabilizes 1.58-bit LLMs","Inserting one RMSNorm makes ternary fine-tuning stable","Extra RMSNorm alone matches KD for 1.58-bit LLMs","Gradual quantization plus RMSNorm stabilizes ternary LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000683,"raw_usage":{"total_tokens":3060,"prompt_tokens":864,"completion_tokens":2196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":480,"completion_tokens_details":{"reasoning_tokens":2105}},"tokens_in":480,"tokens_out":2196,"duration_ms":14865,"temperature":1.0,"reasoning_tokens":2105,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:03:57.816890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the Qwen-1.5B run twice with the added RMSNorm layers initialized to preserve the original forward pass (identity scale) versus initialized randomly, keeping data, schedule, and all other settings fixed; if final cross-entropy differs materially, the success of the recipe depends on an unstated initialization choice rather than on RMSNorm itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the open-source fine-tuning implementation whose BitLinear pseudocode and two-phase lambda schedule the method adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the straight-through estimator used to keep gradients flowing through the ternary quantization step."},{"cited_title":"Trained Ternary Quantization","cited_arxiv_id":null,"evidence_quote":"Establishes trained ternary quantization as the lineage of the -1/0/+1 weight representation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the layer-wise distillation baselines the paper compares against and finds unnecessary when RMSNorm is present."}],"review_version":1}