{"id":"74db8473-38b3-46d3-bc3e-aabdb4fcca0b","arxiv_id":"2512.18934","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Quantized (INT8/INT4) LLMs can outperform FP16 in later-task forward accuracy and retention during continual learning, though single-seed runs leave the effect unquantified.","lead":"This study asks whether compressing a large language model to 8-bit or 4-bit precision changes how well it remembers old skills while learning new ones. It reports that compressed models sometimes beat full-precision models at retaining and transferring knowledge, suggesting low precision plus small memory buffers could help real-world continual learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-seed runs leave the central precision-ordering claim unverified; the reported 8-15% advantage and 40% vs 20% Code gap could be training seed noise.","rationale":"My concern differs from the reader's weakest_assumption, which focuses on the mechanism (static quantization noise vs. implicit regularization). That is an interpretative issue: even if the proposed mechanism is implausible, the empirical ordering could still hold. The more load-bearing issue is evidential: the paper's entire central claim is a comparison of outcomes from single training runs. The authors themselves flag the absence of multi-seed runs and confidence intervals in Section 5, yet the abstract states strong quantitative advantages. Without repeated runs, we cannot know whether the reported inversion is a reproducible property of quantization or an artifact of one seed. This is especially acute for the Code result, where few evaluation problems make the metric noisy. I therefore agree with the reader's CONDITIONAL verdict, but for a different primary reason. The required fix is straightforward: rerun with multiple seeds and report variance; if the ordering remains, the claim is supported.","tokens_in":6101,"tokens_out":7835,"duration_ms":72247,"concrete_test":"Run the complete three-stage continual learning protocol with at least 5 random seeds per (precision, replay) cell for representative buffers (e.g., 0% and 5%). Report mean and 95% CI for final forward accuracy on Code and retention on NLU. Test whether INT8 > INT4 > FP16 holds across all seeds and whether the 40% vs 20% Code gap persists; also compute bootstrap confidence intervals for HumanEval pass@1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that INT8/INT4 outperform FP16 in continual learning, but each precision–buffer condition is evaluated from one training run. The Limitations section (Section 5) explicitly admits 'our experiments lack multi-seed runs and confidence intervals.' With a single seed, the 8–15% final-forward-accuracy differences and the HumanEval gap (40% vs 20%) cannot be separated from initialization or data-order noise. HumanEval pass@1 has only 164 problems, so a 20-point gap is ~33 correct samples; training seed variance can be several points. Because the result contradicts the conventional expectation that higher precision is better, the onus is to demonstrate reproducibility. Without error bars or repeated runs, the empirical foundation for the inversion is unsupported; the ordering may not persist.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the interaction between weight quantization precision (FP16, INT8, INT4) and replay-buffer size in continual learning of a LLaMA-3.1-8B model. Using LoRA adapters on frozen base weights, the authors fine-tune sequentially on NLU, Math (GSM8K), and Code (CodeAlpaca/HumanEval), varying the replay buffer from 0% to 20%. They report that, contrary to the usual assumption that higher precision is better, quantized models outperform FP16 on final-task forward accuracy by 8–15%, with INT4 achieving roughly double FP16's HumanEval pass@1 (40% vs 20%), and INT8 offering the best plasticity–retention balance. They attribute the effect to quantization-induced noise acting as implicit regularization and give practical buffer-size recommendations. The paper includes a public code repository and proposes a benchmark for quantized continual learning.","tokens_in":6328,"tokens_out":2699,"duration_ms":23677,"significance":"If the empirical inversion is correct, the paper identifies a genuinely useful and counterintuitive design choice: low-precision weights could be not only efficient but also beneficial for continual learning, and small replay buffers might suffice under quantization. The systematic sweep over precision × buffer size is a useful contribution, and the public code link strengthens reproducibility. However, the central quantitative claims are currently unverified: all cells are single-seed runs, the abstract and Section 3 report conflicting numbers, and the proposed mechanistic explanation is not tested against simpler alternatives. The paper's practical recommendations therefore rest on an empirical foundation that the text itself acknowledges is incomplete.","major_comments":[{"comment":"The central 8–15% inversion and the 40% vs 20% Code gap rest on one training run per precision–buffer condition. Section 5 admits 'our experiments lack multi-seed runs and confidence intervals.' Since HumanEval pass@1 has only 164 problems, a 20-point gap is roughly 33 samples, and initialization/order noise can be several points, the reported ordering cannot be distinguished from seed noise. This is the load-bearing claim of the paper; multi-seed runs with intervals (or at least repeated runs for the headline cells) are required.","section":"Section 5, Limitations; Abstract; Section 3"},{"comment":"Reported numbers are internally inconsistent. The abstract states FP16 achieves 74.44% on initial NLU, but Section 3 says FP16 drops 'from 77.26% to 42.50%'. The same paragraph says '4-bit models exhibited sharper degradation,' yet the numbers show FP16 losing 34.76 points (77.26→42.50) versus 4-bit losing 29.81 points (72.31→42.50). These contradictions prevent the reader from determining even the direction of the precision effect, let alone its magnitude.","section":"Section 3, 'Baseline degradation' and 'Amplification under continual learning'"},{"comment":"The introduction states that 'at higher precision, minimal replay is sufficient, while under 4-bit quantization, buffer size becomes a decisive factor' and that '8-bit quantization noise acts as a natural regularizer.' The abstract and Section 3, by contrast, claim FP16 is markedly worse under replay and that INT8 is the best balance. These are substantively different claims: one implies FP16 needs little replay, the other implies FP16 underperforms even with replay. The narrative needs to be reconciled and the precise claim stated consistently.","section":"Section 1 vs Abstract/Section 3"},{"comment":"The proposed mechanism — 'quantization-induced noise acts as implicit regularization' — is not supported by the experimental setup. With frozen quantized base weights and high-precision LoRA adapters, the quantization noise is static and does not enter the adapter gradient updates; it merely changes the forward activations. The observed effects could instead come from altered scales, activation ranges, or LoRA initialization/optimization interactions. The authors correctly label this as a hypothesis ('We speculate'), but the paper's abstract and guidelines lean on it as if established. I recommend either removing the mechanistic claim or testing it directly, e.g., by adding equivalent static noise to FP16 forward passes.","section":"Section 2; Section 3, 'Interpretation'"}],"minor_comments":[{"comment":"Several citations are misassigned: [4] is used for both CodeAlpaca and A-GEM; [11] is cited for the eight NLU datasets but is a GFlowNet paper; [16] is cited for AWQ but is a self-attention paper. Please verify all references.","section":"References"},{"comment":"The figure captions appear duplicated ('Forward Accuracy. Forward Accuracy.') and the axes are not described in the caption. Please clarify what is plotted and label the axes or describe them in the caption.","section":"Figures 1 and 2"},{"comment":"Typo: 'Low-Ranking Adaptation' should be 'Low-Rank Adaptation.' Also, 'with out the mentorship' in the acknowledgments should be 'without the mentorship.'","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful scope and a publicly available codebase, but the core empirical claim is currently unsupported by the presented evidence (single-seed, internally inconsistent numbers). The issues are fixable by additional experiments and a careful rewrite, so I do not recommend rejection, but the revision must supply multi-seed statistics and reconcile the conflicting numbers before the central claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the quick take. The paper reports something genuinely not in the literature: a systematic sweep of quantization precision (FP16/8/4) against replay buffer size in a continual-learning LLM setup, and it claims an inversion where quantized models beat FP16 on later tasks. That is a useful practical claim if it holds. The authors also ship code and give concrete buffer-size recommendations per task type, which is more than many empirical papers do.\n\nWhat it does well: the experimental design is systematic across three tasks and eight buffer sizes, and the practical guidelines (small buffers for NLU, larger for Math/Code) are specific. The paper also honestly labels the implicit-regularization story as a hypothesis rather than a proven mechanism. That is to its credit.\n\nThe soft spots are real and load-bearing. The central 8-15% result and the 40-vs-20 Code gap rest on one training run per condition, with no confidence intervals. The Limitations section says this out loud. On HumanEval pass@1 with 164 problems, a 20-point gap is about 33 samples, and training-seed noise can move that. So the headline inversion is not yet separated from luck. The text also contradicts itself: the abstract says FP16 initial NLU is 74.44%, while the results section says 77.26%; and the same section says 4-bit shows 'sharper degradation' even though the numbers it gives show FP16 dropping more (34.8 points vs 29.8). That kind of mismatch makes a careful reader doubt whether the analysis was run consistently.\n\nAlso, the mechanism is slippery. The base weights are frozen and quantized; the adapters are learned in high precision. Quantization noise in the frozen base is static and does not directly perturb adapter updates. So 'quantization-induced noise acts as implicit regularization' is a plausible hand-wave, but it is not something the experiments actually observe. The authors do flag it as speculation, but the abstract leans on it too hard.\n\nWho is this for? Practitioners building continual-learning pipelines for quantized LLMs will find the trade-off map and buffer recommendations useful. It is a legitimate empirical paper, but only after a multi-seed re-run and number reconciliation. I would not cite it in my own work yet. I would send it to peer review, but with the explicit request that the authors run at least 3-5 seeds, report intervals, and fix the internal inconsistencies. The question deserves referee time.","headline":"Single-seed runs and inconsistent numbers undercut an otherwise novel precision-vs-replay result; the mechanism claim is not backed by the setup.","tokens_in":6727,"tokens_out":1924,"would_cite":false,"duration_ms":19821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Quantized language models forget less than full-precision ones in continual learning, with 8-bit precision striking the best balance between retaining old tasks and learning new ones.","keywords":["continual learning","catastrophic forgetting","quantization","8-bit precision","replay buffer","low-rank adaptation","implicit regularization","large language models"],"falsifier":"Run the same continual-learning protocol in FP16 while injecting noise into the adapter gradients or forward activations with the same magnitude as INT8's quantization error; if noise-injected FP16 reproduces the 8–15% retention advantage, the implicit-regularization story is supported, whereas if it does not, the effect is driven by quantization's changed scales or optimization dynamics, not by noise. Repeating across multiple seeds would also confirm whether the INT8-over-FP16 ordering is stable.","tokens_in":6066,"feed_emoji":"⚡","tokens_out":3420,"duration_ms":37318,"temperature":0.7,"pith_summary":"The paper asks whether lower numerical precision can actually help continual learning, not just save memory. Across a three-stage sequence of NLU, Math, and Code training on an 8-billion-parameter language model, the authors find that FP16 wins on the first task, but quantized models outperform it by 8–15% on later-task accuracy, with INT4 nearly doubling FP16's Code-generation score. INT8 consistently offers the best trade-off between learning new tasks and retaining old ones, and even a 0.1% replay buffer dramatically lifts retention across all precisions. The paper challenges the default assumption that higher precision is always better, and suggests quantization noise may act as a form of implicit regularization.","feed_headline":"8-bit models beat full precision at continual learning","feed_subtitle":"INT8 retains 8–15% more across later tasks than FP16 and needs less replay data to anchor old knowledge.","key_machinery":"The experimental engine is a three-stage continual-learning schedule—NLU, then Math with replayed NLU data, then Code with replayed NLU and Math data—run at three precisions (FP16, INT8, INT4) using frozen base weights and trained low-rank adapters. The paper's proposed mechanism is quantization-induced noise as implicit regularization: discretization error is said to smooth the loss landscape and bias the model toward flatter minima, which amplifies the influence of replayed samples and helps anchor prior knowledge. Replay buffer size is the second lever, interleaving up to 20% of prior-task data into later training stages.","core_discovery":"In a three-stage continual learning setup—train on NLU, then Math with replay of NLU, then Code with replay of both—the authors observe a precision inversion. Full-precision FP16 achieves the highest initial NLU accuracy (74.44%), but after subsequent tasks quantized models beat FP16 by 8–15% on final-task forward accuracy, with INT4 reaching about twice FP16's Code pass@1 (40% vs 20%). Across replay buffer sizes from 0% to 20%, INT8 gives the best balance of plasticity and retention. A replay buffer as small as 0.1% raises NLU retention after Math training from 45% to 65% at every precision level, and quantized models generally require less replay than FP16 to reach comparable retention. Th","pith_inferences":["A direct way to test the proposed mechanism is to inject controlled noise into FP16 adapter gradients or forward activations, matched in magnitude to INT8's quantization error; if that reproduces the retention advantage, the regularization story is supported, whereas a failure would point to changed scales or optimization dynamics as the true cause.","The effect may be specific to frozen quantized base weights with high-precision adapters; adaptive or mixed-precision schemes might produce different orderings, so the practical recommendation should be validated across adapter and quantization configurations.","The same lens could be applied to other continual-learning scenarios, such as vision or multimodal models, where quantization noise might interact differently with replay and task order.","If quantization noise really biases toward flatter minima, then explicit flatness-promoting regularizers (e.g., sharpness-aware updates) might confer similar benefits in full precision, offering an alternative to quantization for stability."],"forward_implications":["INT8 quantization should be considered a default for continual fine-tuning of large language models, since it combines lower memory cost with better retention and comparable learning of new tasks.","Small replay buffers (1–2%) are sufficient for NLU retention across precisions, while Math and Code benefit from moderate buffers (5–10%).","Quantized models need less replay data than FP16 to achieve comparable retention, reducing storage overhead in deployed continual-learning systems.","The precision ordering observed (INT8 over FP16 and INT4 on later tasks) implies that quantization choices interact with learning dynamics in ways static benchmarks miss.","If the implicit-regularization hypothesis is correct, precision becomes a tunable knob for balancing plasticity and stability in sequential training."],"fun_headline_variants":["INT8 beats FP16 as tasks pile up in LLMs","8-bit models retain more across tasks than FP16","Quantization as implicit regularizer: INT8 wins","Lower precision, higher retention in continual learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim relies on quantization noise actively regularizing learning, but since the quantized base weights are frozen and the adapters are trained in high precision, the noise is static and never directly perturbs the adapter updates—so the benefit could instead come from altered forward-pass scales or optimization interactions, and if so, the precision ordering may not transfer to other setups.","fun_headline_variants_meta":{"raw":{"variants":["INT8 beats FP16 as tasks pile up in LLMs","8-bit models retain more across tasks than FP16","Quantization as implicit regularizer: INT8 wins","Lower precision, higher retention in continual learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1058,"prompt_tokens":826,"completion_tokens":232,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":169}},"tokens_in":570,"tokens_out":232,"duration_ms":3538,"temperature":1.0,"reasoning_tokens":169,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T14:49:24.235758+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same continual-learning protocol in FP16 while injecting noise into the adapter gradients or forward activations with the same magnitude as INT8's quantization error; if noise-injected FP16 reproduces the 8–15% retention advantage, the implicit-regularization story is supported, whereas if it does not, the effect is driven by quantization's changed scales or optimization dynamics, not by noise. Repeating across multiple seeds would also confirm whether the INT8-over-FP16 ordering is stable.","supporting_citations":[],"review_version":1}