{"id":"f1f5615c-754d-4424-adda-2bdd1251d74c","arxiv_id":"2502.02007","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Training on all tokens (next token prediction) beats training only on answer tokens (critical token prediction) on small-scale reasoning benchmarks, an effect the authors attribute to noise-induced regularization.","lead":"This paper compares two ways to train a small language model on reasoning problems: predicting every token versus predicting only the final answer. It finds that predicting every token produces better reasoning and robustness, and argues this comes from the regularizing effect of 'noise' tokens.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mechanism claim hinges on calling the facts, rules, and question 'noise'; those tokens are informative reasoning content, so NTP's advantage may come from richer supervision rather than noise regularization.","rationale":"The reader's weakest assumption identifies exactly the load-bearing point: the paper's mechanism claim treats the non-answer tokens in natural-language reasoning sequences as 'noise.' I agree with that assessment, and it is the single most important issue because it separates the empirical finding (NTP outperforms CTP) from the proposed explanation (noise-induced regularization). The anchor-function results are real evidence that random non-key tokens can bias the model toward a reasoning solution in a controlled synthetic setting; I credit that independent support. However, the paper extends the same mechanism to PrOntoQA, LogicInference, CLUTRR, RuleTaker, PARARULE Plus, and other benchmarks where the 'noisy' positions encode the premises and question needed for the answer. The CTP model sees the same input context, so the only difference is which positions contribute to the loss; if NTP's extra loss terms train the model to predict the reasoning chain itself, the advantage could be explained by richer supervision rather than regularization. The proposed ablation would settle this directly, because it holds the amount of auxiliary loss fixed while removing the information content of the non-answer tokens. Until that test is run, the broad mechanistic conclusion should be treated as conditional, which is consistent with the reader's CONDITIONAL verdict; my read does not move the verdict.","tokens_in":30380,"tokens_out":3406,"duration_ms":36748,"concrete_test":"On the original PrOntoQA setup, train three GPT-2-125M models from scratch with matched epochs, batch size, and learning rate: (1) CTP as in Eq. (4); (2) NTP as in Eq. (3); (3) a variant that trains the CTP answer loss plus an auxiliary loss on the same non-answer positions, but with those positions replaced by random tokens (or shuffled within each sequence so they carry no reasoning content). If variant (3) matches NTP's ~100% accuracy and reverse/OOV robustness, the noise-regularization mechanism is supported. If variant (3) falls toward CTP's ~80% accuracy while NTP remains at ~100%, the reasoning content of the non-answer tokens, not generic noise, is the active ingredient. Report at least five seeds with error bars.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central explanatory claim, that NTP's advantage over CTP comes from 'the regularizing influence of noise on the training dynamics' (Abstract), requires the non-critical tokens in Eq. (3) beyond the CTP term in Eq. (4) to be noise. Section 7.1 explicitly labels the residual part of the NTP loss as 'noise loss.' But in PrOntoQA and the other natural-language reasoning benchmarks, those non-answer positions contain the facts, rules, and question that determine the answer (Appendix 7.3, e.g., the underlined inference chain). They are structured, informative supervision, not random perturbations. The anchor-function experiments (Section 4.3) use genuine random noise and provide some support for the mechanism in that synthetic setting, but the broad claim over 'various benchmark reasoning datasets' (Abstract, Fig. 4) transfers the conclusion without isolating which property of the extra tokens matters. Fig. 10(b)'s gradient-norm ratio only shows the model eventually attends less to those positions; it does not establish that their role during optimization was regularization rather than feature learning. If the advantage is driven by predicting informative intermediate tokens, the proposed mechanism is not the explanation, although the empirical ranking could still stand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares next-token prediction (NTP) with critical-token prediction (CTP) when training small GPT-2 models from scratch on reasoning datasets. It reports that NTP outperforms CTP in accuracy, out-of-distribution generalization, robustness to embedding and weight perturbations, and resistance to poisoned labels, despite slower early learning. The authors attribute this advantage to a noise-regularization effect of the non-critical tokens and recommend NTP for pretraining and CTP for finetuning. Evidence includes PrOntoQA variants, nine other reasoning and text-classification datasets, an anchor-function synthetic task, finetuning/transfer experiments, and loss-landscape flatness measurements.","tokens_in":30622,"tokens_out":5826,"duration_ms":58227,"significance":"If the empirical ranking is reproducible, the paper offers a useful counterpoint to token-selection methods such as RHO-1 and suggests a concrete role for full-sequence prediction in reasoning tasks. The anchor-function experiments provide a clean synthetic demonstration that full-sequence training can shift a model from a non-reasoning to a reasoning solution, and the poisoned-label experiments are a valuable robustness probe. However, the central explanatory claim—that the advantage is caused by 'noise' regularization—is not yet established for natural-language reasoning data, because the extra tokens in those datasets are informative rather than random. The paper would be considerably strengthened by ablations that separate informative-token prediction from random-noise regularization, and by quantitative reproducibility information for the main comparisons.","major_comments":[{"comment":"The proposed mechanism identifies the non-critical terms in Eq. (3) with 'noise,' and Section 7.1 calls the residual beyond the CTP term the 'noise loss.' However, in PrOntoQA and the other natural-language reasoning benchmarks, these positions contain the facts, rules, and question that determine the answer (see the underlined inference chain in Appendix 7.3). Predicting those tokens is structured supervision, not random perturbation. The observed NTP advantage could therefore come from access to a richer learning signal rather than from noise-induced regularization. The anchor-function experiments in Section 4.3 use genuinely random padding and support the noise-regularization mechanism in that synthetic setting, but Fig. 10(b) only shows that gradient norms on non-critical positions decrease late in training; it does not establish that their role during optimization was regularization. To sustain the mechanism claim, the authors should isolate the property of the extra tokens, for example by comparing CTP trained with additional loss on the informative chain tokens against CTP trained with added random tokens, and report whether only the former reproduces the NTP advantage.","section":"§3.2, §7.1, Eqs. (3)-(4), Fig. 10(b)"},{"comment":"The abstract and Section 6 claim that NTP reaches flatter minima, but the text of Section 6.3 states that 'the flatness disparity between the two training approaches becomes considerably more nuanced' and that 'the flatness analysis under this specific scenario favors CTP' on PrOntoQA reverse and PARARULE plus, with a post-hoc explanation in terms of search-space size. Because flatness is the main mechanistic evidence offered for the generalization claim, this admission directly weakens the central explanation. Please report the flatness comparison quantitatively for all tasks, specify the range of perturbation strengths used in Eq. (6), and either reconcile the contradictory results with the noise-regularization account or remove flatness from the load-bearing evidence.","section":"§6.3, Fig. 9"},{"comment":"Most accuracy and robustness comparisons are shown for a single training run without error bars or seed information; only Fig. 5 reports standard deviations over three runs. Since several reported gaps (for example, the text-classification results in Fig. 4 and the finetuning curves in Fig. 6) are small or variable, single-run curves are insufficient to support the claim that NTP 'consistently outperforms' CTP. Please report means and variances over at least three seeds for the main comparisons, or clearly mark which differences are reproducible.","section":"Figs. 2, 4, 6, 7, 8"}],"minor_comments":[{"comment":"There are several typos: 'vallina' should be 'vanilla' in Sections 7.4.1 and 7.4.2, 'THe' should be 'The' in the ReCOGS description, and 'LTP' should be 'CTP' in the RobustLR paragraph.","section":"§7.4.1, §7.4.2, §7.3"},{"comment":"Equation (4) defines CTP as the loss on a single final token x_T, but Appendix 7.4.2 says the CTP loss is calculated on all tokens between the [SEP] symbols, which for multi-token answers includes more than one token; please reconcile the formal definition with the experimental implementation.","section":"§3.2, Eq. (4) vs. §7.4.2"},{"comment":"The text says 'we introduce 192,000 training dataset and 1,0000 testing dataset'; the '1,0000' appears to be a typo for '10,000'.","section":"§7.3, SimpleLogic"},{"comment":"The figure caption for panel (c) says '1-hop specific PrOntoQA on OOV data,' while the main text explains that the authors downgraded from 2-hop to 1-hop because both methods struggled on 2-hop OOV; please state this limitation directly in the caption and clarify the scope of the OOV claim.","section":"Figure 2 and §4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's novelty relative to the authors' prior anchor-function work and to the broader noise-regularization literature should be checked by the editor; the main mechanism claim overlaps with standard SGD/flatness arguments without providing a new formal result. No other concerns about research integrity are apparent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is worth taking seriously. The paper runs a clean, systematic comparison: NTP versus loss-on-answer-only (CTP) on a battery of synthetic and natural-language reasoning benchmarks, training GPT-2 125M from scratch, plus two modified PrOntoQA splits (reverse and OOV), transfer experiments, robustness to embedding noise and mislabeled data, and flatness analysis. That is a substantial amount of work, and the main empirical finding—NTP generalizes better than CTP on reasoning tasks even when trained from scratch—is new relative to the cited literature, which was mostly about selective token training or implicit bias in linear models. If that ranking holds up, it matters for token-selection methods like RHO-1 and for when to prefer SFT over continued NTP.\n\nThe soft spot is the mechanism. The paper calls everything beyond the answer token 'noise' and attributes NTP's advantage to noise-induced regularization, drawing an analogy to dropout and SGD noise. But in PrOntoQA and the other benchmarks, those non-answer tokens contain the facts, rules, and question that determine the answer. They are structured, informative supervision. The anchor-function experiments do include genuine random noise and show NTP helps the model find the lower-complexity solution there, which is real evidence for the noise-regularization story in a controlled setting. But that does not carry over automatically to natural-language benchmarks, where the extra tokens could be helping because they provide a richer learning signal. The paper never isolates which property of the extra tokens is responsible. Fig. 10(b) shows the model eventually attends less to those positions, but that is a description, not a mechanism test.\n\nAlso, most figures are single runs without error bars (Figs. 2, 4, 6, 7, 8), and the paper itself concedes that the flatness analysis favors CTP on some reasoning tasks. That makes the breadth of the generalization claims—'NTP is instrumental for pretraining, CTP for finetuning'—more than the evidence supports.\n\nThis is for people working on training objectives and token selection; they should read it for the empirical ranking, not for the mechanism. The paper deserves a serious referee: the empirical comparison is substantial, the question is important, and the mechanism can be sharpened. But the referee should push for either an information-ablation (predict random tokens vs. informative intermediate tokens) or a clear argument for why the informative tokens are functionally equivalent to noise. Without that, the noise-regularization claim is not established.","headline":"Solid empirical comparison of NTP vs CTP on reasoning benchmarks, but the noise-regularization mechanism is not established because the 'noise' tokens are informative reasoning content.","tokens_in":31125,"tokens_out":3378,"would_cite":false,"duration_ms":29975,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Next-token prediction beats answer-only training on reasoning tasks because the extra tokens act as a regularizer.","keywords":["next token prediction","critical token prediction","reasoning bias","implicit regularization","flat minima","generalization","robustness","anchor function"],"falsifier":"Train NTP and CTP on a reasoning dataset where the non-answer tokens are replaced by uniform random tokens of the same length and vocabulary, keeping answer positions fixed. If NTP no longer outperforms CTP (or its accuracy drops to CTP's level), the advantage comes from the information content of those tokens, not from noise-induced regularization; if the advantage persists, the noise-regularization mechanism is supported.","tokens_in":30181,"feed_emoji":"🧠","tokens_out":8818,"duration_ms":72768,"temperature":0.7,"pith_summary":"Next token prediction (NTP) trains a model to predict every token in a sequence, while critical token prediction (CTP) trains it only on the answer tokens. This paper aims to show that, contrary to intuition, NTP produces better reasoning ability than CTP when both are trained from scratch on question-answer style reasoning data. The explanation offered is that the non-answer tokens act as noise, and this noise regularizes the training dynamics, steering the model toward flatter loss minima and away from memorizing surface patterns. The paper further claims this makes NTP the better objective for pretraining, while CTP is the better choice for finetuning, where the noise no longer helps. A sympathetic reader would care because the result challenges the idea that focusing a model's loss on the answer is the most direct way to teach reasoning.","feed_headline":"Next-token training beats answer-only training for reasoning","feed_subtitle":"Training on every token, not just the answer, regularizes the model and improves reasoning generalization.","key_machinery":"The load-bearing object is the decomposition of the next-token-prediction loss into the critical-token loss plus a residual 'noise loss' on all other positions. With the critical token last, the NTP loss averages a cross-entropy term over every position, while the CTP loss keeps only the final term. The paper argues that the noise loss acts like gradient noise during optimization, creating a two-stage learning process (a 'flattening' stage that learns the sequence distribution, then a 'fitting' stage that focuses on the answer) and steering the model toward flatter minima. The anchor function, a synthetic task with one composite operator hidden from training, serves as the probe that shows NTP prefers the lower-complexity reasoning solution that generalizes, while CTP can settle on a memorized non-reasoning solution.","core_discovery":"The paper's central discovery is that training a transformer to predict all tokens in a reasoning sequence—rather than only the answer tokens—imparts a reasoning bias: NTP-trained models generalize to out-of-distribution questions, resist contradictory and poisoned labels, and prefer a compositional 'reasoning' solution over a simpler memorized one on an anchor-function benchmark. The authors attribute this to the non-critical tokens acting as a regularizer that flattens the loss landscape and slows memorization. They find that NTP reaches 100% accuracy on PrOntoQA while CTP plateaus near 80%, that NTP keeps near-perfect accuracy on a reversed and an out-of-vocabulary test set while CTP overfits, and that on the synthetic anchor function NTP shifts the model from the non-reasoning to the reasoning solution. On transfer, NTP-trained models generalize faster during finetuning but also forget the pretrained task more easily. The authors conclude that NTP is instrumental during pretraining and CTP is more effective for finetuning.","pith_inferences":["If the advantage is truly noise-driven, it should scale with the fraction and entropy of non-answer tokens; a quantitative law connecting sequence noise level to reasoning gains could be fitted and tested, which the paper does not provide.","The same mechanism may apply to other sequence-prediction settings where the supervisory signal occupies a small part of the output, such as program synthesis and robot-policy learning, making CTP-like objectives there similarly risky.","An explicit curriculum that anneals from NTP to CTP loss during pretraining could combine NTP's reasoning generalization with CTP's faster convergence; the paper tests pretraining on noise-only loss followed by CTP, but not a continuous schedule.","The catastrophic forgetting asymmetry suggests that NTP-trained models have more malleable representations; this could be an advantage for continual learning if paired with replay, which the paper does not explore."],"forward_implications":["Pretraining on question-answer data should keep next-token prediction on all tokens rather than switching to answer-only loss, if reasoning generalization is the goal.","For finetuning, answer-only (CTP/SFT) loss is faster and matches or exceeds NTP, so the optimal training recipe switches objectives between phases.","NTP-trained models transfer faster to new tasks but forget the old task more readily, so finetuning schedules must budget for catastrophic forgetting.","The noise-regularization view predicts that adding noise to a CTP objective (for example, through noisy embeddings) should recover part of NTP's reasoning advantage.","Comparisons of training objectives should be evaluated on out-of-distribution and robustness tests, not only on held-out in-distribution accuracy, because that is where the reasoning bias appears."],"supporting_citations":[{"why":"Supplies the original PrOntoQA dataset and training protocol used for the main NTP-vs-CTP reasoning comparison.","marker":"(Saparov & He, 2023)"},{"why":"Introduces the anchor function benchmark used to test whether models prefer a generalizing reasoning solution or a memorized one.","marker":"(Zhang et al., 2024b)"},{"why":"Shows that initialization scale controls whether transformers fit composite functions by inference or memorization; the paper builds on this to show NTP shifts the choice.","marker":"(Zhang et al., 2024a)"},{"why":"Defines flat minima, the explanation NTP's generalization advantage is attributed to.","marker":"(Hochreiter & Schmidhuber, 1997)"},{"why":"Links flat minima to better generalization in deep networks, grounding the flatness measurements.","marker":"(Keskar et al., 2017)"},{"why":"Demonstrates that SGD noise regularizes optimization toward flat minima, the analogy for the noise-loss mechanism.","marker":"(Wu et al., 2020; Feng & Tu, 2021)"},{"why":"Provides the embedding-noise perturbation protocol used in robustness experiments.","marker":"(Jain et al., 2023)"},{"why":"Supplies the ProsQA dataset used in the transfer-learning and catastrophic-forgetting experiments.","marker":"(Hao et al., 2024)"}],"fun_headline_variants":["Noise from next tokens sharpens reasoning, beats answer-only","All-token training regularizes, boosting reasoning beyond answer-only","Why next-token prediction wins: noise regularizes reasoning","Answer-only training loses to full-sequence prediction for reasoning","Training on every token, not just answers, aids reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explanation assumes that the non-answer tokens in the training sequences are noise rather than structured information, even though in real reasoning datasets those tokens contain the facts, rules, and questions needed to derive the answer.","fun_headline_variants_meta":{"raw":{"variants":["Noise from next tokens sharpens reasoning, beats answer-only","All-token training regularizes, boosting reasoning beyond answer-only","Why next-token prediction wins: noise regularizes reasoning","Answer-only training loses to full-sequence prediction for reasoning","Training on every token, not just answers, aids reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1442,"prompt_tokens":923,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":539,"tokens_out":519,"duration_ms":4553,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T13:41:08.920161+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train NTP and CTP on a reasoning dataset where the non-answer tokens are replaced by uniform random tokens of the same length and vocabulary, keeping answer positions fixed. If NTP no longer outperforms CTP (or its accuracy drops to CTP's level), the advantage comes from the information content of those tokens, not from noise-induced regularization; if the advantage persists, the noise-regularization mechanism is supported.","supporting_citations":[{"cited_title":"and Schmidhuber, J","cited_arxiv_id":null,"evidence_quote":"Defines flat minima, the explanation NTP's generalization advantage is attributed to."}],"review_version":1}