{"id":"d4d7df42-0057-4f8d-9648-c094f205be29","arxiv_id":"2502.09419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NTP-trained LLMs can predict a second token by marginalizing over first-token probabilities, but adapting them with extra heads only partially closes the gap to that baseline.","lead":"This paper tests whether large language models trained to predict one token at a time can also predict several tokens at once, using probability math plus experiments. It finds they can, but attaching extra prediction heads to a frozen model is hard because the model's internal layers are specialized for one-token prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unquantified top-0.99 truncation in the marginalization baseline (Eq. 6, Appendix A.1) can bias the scaling trend in Fig. 1 and the head-vs-baseline gap in Table 1, because the approximation error shrinks as models become sparser.","rationale":"In good faith, the mathematical core is correct: an NTP model can marginalize over first-token candidates to obtain second-token marginals, and the chain-rule derivation of Eq. 6 is sound. The paper also honestly discloses limitations in Appendix D. However, the quantitative evidence for the headline empirical claims is not yet robust. The truncation threshold is arbitrary, its error is model- and data-dependent, and the evaluation set is extremely small. These are fixable with additional computation, bootstrapping, and sensitivity analysis, so the appropriate verdict remains conditional rather than accept or reject. I considered the KL-probe interpretation of NTP specialization as an alternative concern, but that mainly affects the explanatory narrative; even if the probe is re-interpreted, the empirical gap in Table 1 would remain. The truncation issue directly threatens the numerical claims themselves, making it the most load-bearing concern.","tokens_in":7185,"tokens_out":15133,"duration_ms":134994,"concrete_test":"Recompute the Figure 1 marginalization under two variants: (i) use a top-0.999 probability-mass threshold instead of 0.99, and (ii) for at least 20 randomly chosen contexts, compute the exact sum over the full vocabulary (or an importance-sampling estimate of the tail contribution). If the top-5 accuracies and the ordering across model sizes shift by more than the bootstrap standard error over validation sequences, the truncation is load-bearing. Additionally, report bootstrap confidence intervals for the current 0.99 results; if the scaling trend is not significant after resampling, the claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's quantitative claims rely on Eq. 6, which estimates p(xt+2|X≤t) by summing over y in the top 0.99 probability mass of p(xt+1|X≤t) and renormalizing (Appendix A.1). The error from discarding the 1% tail is never bounded. This is not a harmless numerical detail: Appendix B shows that larger models have sparser first-token distributions, so the truncation set is smaller at larger scales. If tail first-tokens carry non-negligible conditional mass for the true second token, the measured 'improvement with model size' in Figure 1 may partly reflect a decrease in truncation bias rather than a genuine gain in MTP capability. The same truncated marginalization defines the baseline in Table 1, so the reported gap between the MTP heads and the marginalization baseline could be distorted in either direction. The issue is compounded by the evaluation scale: Figure 1 is based on 10 validation sequences (200 token positions per model), and no error bars or bootstrap intervals are reported. The central empirical claims—MTP capability grows with scale, and adaptation heads fall short of the marginalization baseline—therefore rest on an unquantified approximation plus a very small sample.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether LLMs pre-trained for next-token prediction (NTP) already possess multi-token prediction (MTP) capabilities. The authors propose that such capabilities arise through numerical marginalization over intermediate tokens: for K=2, p(xt+2|X≤t) is computed as sum_y p(xt+2|X≤t,y) p(y|X≤t). They evaluate this marginalization on Pythia and Llama-3 models for translation and open-ended generation, reporting that MTP capability improves with model scale and is data-dependent. They then study MTP heads attached to frozen backbones, finding that hidden layers are strongly specialized for NTP, and that joint fine-tuning of backbones and heads improves MTP but does not close the gap relative to the marginalization baseline. The paper also proposes weighted hidden states (WHS) as a partial remedy. The main theoretical identity is the law of total probability; the empirical claims rest on small evaluation sets and a top-0.99 truncation whose error is not quantified.","tokens_in":7488,"tokens_out":6771,"duration_ms":61821,"significance":"If the empirical claims hold, the paper makes a useful contribution by establishing a strong, parameter-free baseline for MTP evaluation: numerical marginalization over the model's own next-token distribution. The finding that NTP specialization makes MTP head adaptation difficult is practically relevant for self-speculative decoding research. The paper also provides an interesting negative result that joint training does not fully overcome this barrier. The proposed WHS mechanism is a reasonable direction. However, the significance is currently limited by the weak empirical support: the central claims of scale dependence and of a head-vs-baseline gap rest on very few validation samples, no error bars, and an unquantified truncation approximation. The marginalization identity itself is straightforward; the value of the paper depends on the reliability of the empirical measurements.","major_comments":[{"comment":"The truncation of the marginalization sum to tokens in the top 0.99 probability mass, followed by renormalization, is never analyzed. The error introduced by discarding the tail can be size-dependent: Appendix B shows that the number of tokens in the top 0.99 decreases with model size, so the approximation error is not constant across the models compared in Figure 1. If excluded first-tokens carry non-negligible conditional mass for the true second token, the reported improvement of MTP capability with scale, and the gap between MTP heads and the marginalization baseline in Table 1, could be artifacts of the truncation rather than genuine MTP capability. The paper should bound this error, for example by reporting the total excluded probability mass and comparing truncated marginalization against full-vocabulary marginalization on a subset of positions, or by providing a theoretical bound on the top-5 accuracy bias.","section":"Appendix A.1, Eq. (6); Appendix B"},{"comment":"The empirical evidence for the two main quantitative claims is based on very small samples: 10 validation sequences for Figure 1 and 50 for Table 1, with no error bars, confidence intervals, or significance tests. For open-ended generation, the references are 10 GPT-4o-generated sequences, not ground-truth human text; this measures agreement with a particular model rather than intrinsic MTP capability. The central claims that MTP capability grows with model size and that MTP heads fall short of the marginalization baseline need stronger statistical support, including more samples, bootstrap intervals, and evaluation on standard text corpora for generation.","section":"Appendix A.2; Figure 1; Table 1"},{"comment":"The KL divergence between intermediate-layer token probabilities and the final NTP distribution shows that intermediate representations become close to the final distribution in large models. However, the conclusion that the network 'has likely discarded information crucial for MTP' does not follow from this measurement. A representation can be specialized for NTP while still retaining linearly accessible information useful for a second-token head. Since the claim that hidden layers are strongly specialized for NTP is load-bearing for the paper's argument that adaptation is non-trivial, the authors should provide more direct evidence, such as probing intermediate representations for second-token predictability, or showing that heads attached to earlier layers, before specialization, outperform heads attached to later layers.","section":"Section 3, Figure 2"},{"comment":"The paper defines MTP as sampling a full subsequence of length K in parallel, yet the evaluation only reports per-token top-5 accuracy for individual positions, not joint sequence accuracy. Equation (3) gives the joint probability only under the stated conditional-independence assumption, which is not validated for natural language and is likely inaccurate for longer spans. The paper should clarify that the empirical claims are about marginal multi-token prediction ability, not joint parallel sampling of a coherent K-token block, or evaluate joint accuracy metrics (e.g., both tokens simultaneously in the top-5).","section":"Section 2, Eq. (3)"}],"minor_comments":[{"comment":"The title promises 'efficient LLM inference,' but the paper does not report any inference speedups, latency, or memory cost measurements. The MTP heads are never evaluated for efficiency, and the marginalization baseline is acknowledged to be computationally expensive. Please either add a brief complexity analysis or adjust the title/scope statement to reflect that the paper studies MTP capabilities, not inference efficiency per se.","section":"General"},{"comment":"The notation in Eq. (3) is confusing: the sum in the k-th factor is over all prefixes s_{1:k-1}, but the product with p(xt+1|X≤t) makes the conditional-independence assumption implicit. Please state explicitly that the expression is the product of per-token marginals under conditional independence.","section":"Section 2, Eq. (3)"},{"comment":"Figure 1 has no error bars and no clear legend mapping markers to the Pythia and Llama-3 model sizes. The reader cannot tell which points correspond to which model family or read off approximate error scales. Please add a legend and describe the marker scheme in the caption.","section":"Figure 1"},{"comment":"The table reports point estimates without standard deviations or confidence intervals, and the row 'Head warmup + Diff. LR' appears to be a duplicate of the row above in structure. Please clarify whether the rows are independent strategies and add variability measures across at least a few random seeds or bootstrap resamples.","section":"Table 1"},{"comment":"The abbreviation WHS is used in Table 1 before it is defined in the text; please define it when the strategy is first mentioned, and also define the temperature T in Eq. (5) in the main text rather than only in Appendix A.4.","section":"Section 3, Eq. (4)-(5)"},{"comment":"The sparsification analysis is based on a single example sequence in Figure 3; please report aggregate statistics (e.g., mean and standard deviation of entropy across validation samples) to support the claim that sparsification is a general trend.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about the top-0.99 truncation is valid and should be a required revision point. The paper's novelty is mostly empirical; the marginalization identity is a direct application of the law of total probability, so the empirical reliability is the main determinant of the paper's contribution. With the current sample sizes and the unquantified approximation, the scale-dependent claims are not yet convincing. The paper is within scope for a CL or ML venue, but the empirical evaluation needs substantial strengthening before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Correct math, useful baseline, but the empirical case is thinner than the prose suggests. The marginalization identity in Eq. 6 is textbook, and the paper is right to make it the explicit comparison point for MTP heads. The genuinely new pieces are the cross-scale evaluation, the KL-divergence diagnostic showing NTP specialization in intermediate layers, and the adaptation strategies (head warmup, differential LR, weighted hidden states). Those are worth having in the literature.\n\nThe soft spots are real but not disqualifying. Evaluation scale is the first: Figure 1 rests on 10 validation sequences per model, Table 1 on 50, and there are no error bars anywhere. Several of the strategy differences in Table 1 fall well within what you'd expect from noise, so the claim that WHS is 'the overall best' is not supported. The open-ended generation eval uses a single GPT-4o reference per prompt, which is a weak gold standard for top-5 accuracy.\n\nThe second, more technical concern is the unquantified truncation in the marginalization baseline. Eq. 6 sums over the top 0.99 probability mass and renormalizes, but the error from discarding the tail is never bounded. Since larger models have sparser distributions (their own Appendix B), the truncation error can shrink with scale, which means the 'MTP improves with scale' trend in Figure 1 could partly be a numerical artifact. That doesn't kill the qualitative conclusion—marginalization works and larger models are more confident—but the quantitative scaling claims are not yet solid. A sensitivity analysis over truncation thresholds or a bound on the tail contribution would settle this.\n\nWho benefits: anyone building speculative decoding or MTP-adaptation methods. The marginalization baseline should become a standard comparison. The central argument holds, but the paper needs statistical rigor and a truncation sensitivity check before the numbers are trustworthy. I'd send it to peer review, with major revisions requested.\n\nBottom line: the idea is sound and the diagnostic is genuinely useful; the evaluation just needs to be more careful.","headline":"Correct marginalization baseline and a useful specialization diagnostic, but the evaluations are too small and the truncation error is unquantified, so the scaling and strategy comparisons are not yet robust.","tokens_in":7955,"tokens_out":3111,"would_cite":true,"duration_ms":30218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Next-token-trained LLMs already encode multi-token prediction through numerical marginalization, and the capability scales with model size, but adaptation heads cannot yet match it.","keywords":["multi-token prediction","next-token prediction","LLM inference","marginalization","self-speculative decoding","model scaling","NTP specialization","weighted hidden states"],"falsifier":"Recompute the two-token marginalization on the same validation sets with a much wider or exact token set, such as the top 0.999 of the probability mass or the full vocabulary for a small model, and compare top-5 second-token accuracies; if the tail contributes enough to change the ranking across model sizes or to erode the reported accuracies, the scaling trend is an artifact of the truncation. Additionally, the open-ended generation result in Figure 1 uses only 10 validation samples, so repeating that evaluation with a larger sample could confirm whether the data-dependent pattern holds.","tokens_in":6965,"feed_emoji":"⚡","tokens_out":5177,"duration_ms":41203,"temperature":0.7,"pith_summary":"This paper asks whether a language model trained only to predict the next token can nevertheless predict several tokens ahead. It answers yes: by summing the model's own probabilities over all possible intermediate tokens, the second-token distribution can be recovered, and this marginalized multi-token accuracy is strong, task-dependent, and grows with model size. The paper then asks why bolting multi-token heads onto a frozen model is hard, and finds that intermediate layers specialize for next-token prediction early, so information needed for later tokens has already been discarded. Jointly fine-tuning the backbone and heads helps, but does not close the gap to the marginalization baseline. The conclusion is that marginalization should be the reference baseline for multi-token prediction adaptation, and that adapting next-token models remains an open problem.","feed_headline":"Trained for one token, LLMs already predict two","feed_subtitle":"Summing over intermediate token choices gives strong two-token accuracy, and bigger models do better.","key_machinery":"The load-bearing object is the marginalization identity for two-token prediction, $p(x_{t+2}|X_{\\leq t}; \\theta) = \\sum_{y \\in \\mathcal{V}} p(x_{t+2}|X_{\\leq t}, y; \\theta)\\, p(y|X_{\\leq t}; \\theta)$, approximated in practice by truncating the sum to tokens in the top 0.99 probability mass of the first-token distribution and renormalizing. This identity turns an NTP model into a reference MTP model without any training. The adaptation experiments use a second mechanism: $N$ independent transformer-layer heads fed by the backbone's final hidden state, or by a learned weighted sum of all intermediate hidden states (weighted hidden states, WHS), sharing a single frozen unembedding. Specialization is measured by the KL divergence between intermediate-layer token probabilities and the final next-token distribution, which shows that large models reach an NTP-specialized representation early.","core_discovery":"The central discovery is that next-token prediction (NTP) models already possess multi-token prediction (MTP) capacity: for K=2, the second-token probability $p(x_{t+2}|X_{\\leq t}; \\theta)$ can be obtained by marginalizing the model's own conditional probabilities, and top-5 accuracy from this estimate is surprisingly strong and improves with model scale. This happens because larger models sharpen the next-token distribution, making the marginalization sum tractable. The second discovery is that this MTP capability is not easily amortized: KL-divergence measurements show that intermediate hidden layers converge to the final next-token distribution early in the network, so representations specialized for NTP lack information needed by additional MTP heads. Joint fine-tuning strategies, including head warmup, differential learning rates, and weighted hidden states, improve second-token accuracy over heads-only training, but remain below the marginalization baseline, implying that fully amortized MTP may require pretraining rather than adaptation.","pith_inferences":["A direct extension the paper does not test: if NTP specialization is the barrier, then heads fed from progressively earlier layers (the multi-layer MTP direction suggested in its Appendix D) should show a monotone accuracy improvement up to a point, measurable with the same IWSLT setup.","The truncation assumption suggests a cheap robustness test for future work: compute the marginalized second-token probability twice, with top-0.99 and top-0.999 masks, and report the difference; when the difference is large, marginalization-based conclusions should be treated as provisional.","The same KL-specialization probe could be used as a diagnostic before training MTP heads: models whose intermediate layers already match final next-token probabilities early are the ones where head-only adaptation is least likely to work.","A testable deployment consequence: for a fixed model, MTP-based speculative decoding speedups should be larger on constrained tasks such as translation than on free-form chat, matching the data-dependent pattern reported in Figure 1."],"forward_implications":["If NTP models already encode MTP through marginalization, then any MTP head that underperforms the marginalization baseline is not adding new capability; it is merely amortizing an existing one.","The scaling result implies that inference-acceleration gains from MTP on larger models will be larger than on smaller models, because the next-token distribution is sparser.","The specialization result implies that freezing the backbone and training only heads has a ceiling; joint fine-tuning with strategies like weighted hidden states is needed to approach it, and even then a gap remains.","For self-speculative decoding, rough drafts from adapted heads can still yield speedups even when exact MTP accuracy is below baseline, since drafts only need to be accepted often enough.","For practitioners, translation-like low-entropy tasks are the favorable regime for MTP; open-ended high-entropy generation is where adaptation is hardest."],"supporting_citations":[{"why":"Supplies the Pythia model suite used for the marginalization scaling experiments and for all MTP head fine-tuning runs.","marker":"Biderman et al., 2023"},{"why":"Supplies the Llama-3 model family used to confirm that MTP capability and its data dependence hold across model families.","marker":"AI@Meta, 2024"},{"why":"MEDUSA is the closest prior MTP adaptation framework; the paper builds on its head-attachment idea, reports its speedups, and motivates the joint fine-tuning strategies tested here.","marker":"Cai et al., 2024"},{"why":"Provides the shared-backbone, independent-head, shared-unembedding architecture that the paper adapts from pretrained NTP models, and shows MTP trained from scratch.","marker":"Gloeckle et al., 2024"},{"why":"LoRA is the low-rank adaptation method applied to the backbone in all joint fine-tuning experiments.","marker":"Hu et al., 2021"},{"why":"Supplies the IWSLT 2017 de-en translation dataset and the supervised task template used for all training and evaluation.","marker":"Cettolo et al., 2017"}],"fun_headline_variants":["Bigger LLMs already predict two tokens by summing probabilities","Two-token prediction emerges from scaling next-token LLMs","Marginalization beats joint training for two-token prediction","Hidden layers resist multi-token heads, keeping next-token focus","Scaling sharpens next-token odds, enabling two-token guesses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The numerical MTP estimates assume that discarding the bottom 1% of the first-token probability mass and renormalizing does not change the marginalized second-token accuracies, yet the paper does not quantify the error from that truncation.","fun_headline_variants_meta":{"raw":{"variants":["Bigger LLMs already predict two tokens by summing probabilities","Two-token prediction emerges from scaling next-token LLMs","Marginalization beats joint training for two-token prediction","Hidden layers resist multi-token heads, keeping next-token focus","Scaling sharpens next-token odds, enabling two-token guesses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000998,"raw_usage":{"total_tokens":4182,"prompt_tokens":860,"completion_tokens":3322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":3235}},"tokens_in":476,"tokens_out":3322,"duration_ms":22534,"temperature":1.0,"reasoning_tokens":3235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:33:16.646148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the two-token marginalization on the same validation sets with a much wider or exact token set, such as the top 0.999 of the probability mass or the full vocabulary for a small model, and compare top-5 second-token accuracies; if the tail contributes enough to change the ranking across model sizes or to erode the reported accuracies, the scaling trend is an artifact of the truncation. Additionally, the open-ended generation result in Figure 1 uses only 10 validation samples, so repeating that evaluation with a larger sample could confirm whether the data-dependent pattern holds.","supporting_citations":[{"cited_title":"Overview of the IWSLT 2017 evaluation campaign","cited_arxiv_id":null,"evidence_quote":"Supplies the IWSLT 2017 de-en translation dataset and the supervised task template used for all training and evaluation."}],"review_version":1}