{"id":"cc2f6028-a5e6-40eb-a725-17df70e9b177","arxiv_id":"2505.22757","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A forward curriculum that gradually increases the number of predicted tokens helps small models retain MTP's decoding speed benefits and often improves generation quality, but does not consistently improve next-token prediction accuracy.","lead":"The paper tests two curriculum schedules for multi-token prediction pre-training on small language models: one that ramps the prediction depth up over time and one that ramps it down. The forward schedule keeps self-speculative decoding speedups and often improves generation quality, but the headline claim of better next-token performance is only partially supported by the tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Forward curriculum's claimed NTP improvement is contradicted by Table 1: byte-level forward models degrade MiniPile/LAMBADA BPB in all 4 configurations, and subword gains appear only at 4 heads.","rationale":"The reader identified the overbroad headline in the rationale and assigned CONDITIONAL, which matches my read. However, the reader's stated weakest_assumption is the arbitrariness of the uniform epoch-based schedule. My load-bearing concern is different: the forward curriculum's advertised NTP improvement is contradicted by the paper's own tables, independent of whether the schedule is optimal. The schedule-arbitrariness issue affects whether the magnitude of the effect is curriculum-optimal, but the claim-evidence mismatch affects whether the headline effect exists at all. I therefore partially agree: the reader and I converge on 'claims need narrowing,' but I locate the core problem in the internal inconsistency between abstract and Table 1 rather than in the schedule design. The paper is otherwise honest: it reports absolute scores in Appendix B, includes limitations, and acknowledges that forward curricula do not beat NTP baselines. The reverse-curriculum results are internally consistent (better NTP, no speedup). So conditional acceptance with revised claims remains appropriate; no verdict change is needed.","tokens_in":20959,"tokens_out":4774,"duration_ms":53840,"concrete_test":"Tally the sign of forward-vs-static differences in Table 1 for MiniPile BPB and LAMBADA BPB across all 12 model configurations. If forward improves both NTP metrics in only 4 of 12 configurations (the subword 4-head cases) and degrades or ties them in the remaining 8, including all byte-level configurations, then the abstract's claim that forward curriculum improves downstream NTP performance should be scoped to subword 4-head models or removed. A complementary check: if raw per-seed evaluation logs are available, run a paired bootstrap across the 12 configurations to estimate whether the median forward-vs-static difference is significantly below zero for byte-level models.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—that the forward curriculum improves downstream NTP performance—is not supported by the paper's own results. In Table 1, comparing forward-curriculum models against static MTP baselines: for byte-level models, forward degrades MiniPile BPB in all 4 configurations (e.g., +3.16%, +2.62% for 4-LL; +0.81%, +0.79% for 8-LL) and degrades or ties LAMBADA BPB in 4/4 cases. For subword models, forward improves MiniPile and LAMBADA BPB only in the 4-head configurations (4-LL and 4-TL, both sizes), while it worsens both metrics in the 2-head configurations. The paper itself states in Section 3.2.2 that the curricula 'do not lead to performance improvements over the NTP baseline when applied to subword-level models,' and the reverse curriculum—not forward—is the one that consistently improves NTP metrics relative to static MTP. Thus the advertised NTP benefit of forward curriculum is at best configuration-specific, and the universal phrasing in the abstract overstates the evidence. This is a claim-evidence mismatch, not merely a question of schedule optimality: even granting the uniform epoch-based schedule as a valid curriculum, the data do not show the forward curriculum improving NTP performance broadly. The output-quality claim is similarly fragile: Table 2 shows forward-vs-static wins on at least two metrics in 8/12 configurations, but with mixed signs (e.g., 1.3B subword 2-LL BLEU -2.70%, ROUGE -0.06%), and no significance testing or repeated runs is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two curriculum schedules for multi-token prediction (MTP) pre-training of small language models: a forward curriculum that starts with next-token prediction (k=1) and adds one prediction head every E/kmax epochs, and a reverse curriculum that starts with k=kmax and removes heads over time. The authors train 1.3B and 3B Llama-style models on MiniPile with subword and byte tokenizers, using linear-layer (LL) and transformer-layer (TL) prediction heads, and compare against static MTP and NTP baselines on NTP benchmarks, self-speculative decoding speed, and output quality metrics. The abstract claims that the forward curriculum improves downstream NTP performance and output quality while retaining self-speculative decoding benefits, and that the reverse curriculum gives stronger NTP and output quality but no speedups. The evidence in the paper is configuration-dependent, and the broad claims are not consistently supported by the reported tables.","tokens_in":21235,"tokens_out":7181,"duration_ms":73041,"significance":"If the reported effects were robust, the paper would make a useful practical contribution: a simple fixed schedule for changing the MTP prediction depth could let small models obtain inference speedups without sacrificing next-token quality. The paper includes a wide configuration matrix, a scale check on FineWeb-Edu, and a Limitations section, which are strengths. The SemScore metric comes from the authors' prior work but is a fixed external embedding-based similarity measure, so it is not fitted to the current results and is not a circularity concern. Nevertheless, the headline claim of forward-curriculum improvements is not supported by the paper's own tables, and the absence of repeated runs prevents a strong conclusion.","major_comments":[{"comment":"The central claim that the forward curriculum improves downstream NTP performance is contradicted by the reported numbers. For byte-level models, forward training degrades MiniPile BPB relative to the static MTP baseline in all four configurations (1.3B 4-LL +3.16%, 1.3B 8-LL +0.81%, 3B 4-LL +2.62%, 3B 8-LL +0.79%) and degrades or ties LAMBADA in all four. For subword-level models, forward improves both MiniPile and LAMBADA only in the 4-head configurations (4-LL and 4-TL), while the 2-head configurations show degradations (e.g., 1.3B 2-LL +0.15%/+0.37%, 3B 2-TL +0.88%/+0.26%). Section 3.2.2 itself states that the curricula 'do not lead to performance improvements over the NTP baseline when applied to subword-level models.' The abstract's unqualified claim should be revised to reflect the configuration-specific evidence; the FineWeb-Edu check in Table 5 also shows the NTP model remains the best performer.","section":"Abstract; Table 1; Section 3.2.2"},{"comment":"The claim that the forward curriculum improves generative output quality is not supported by any uncertainty quantification, and the signs are mixed. While forward wins on at least two metrics in 8/12 configurations, there are substantial degradations in several others: 1.3B subword 2-LL BLEU -2.70% and ROUGE -0.06%; 3B subword 2-TL BLEU -7.44% and ROUGE -3.97%; 3B byte 4-LL BLEU -6.98% and SemScore -5.88%; 1.3B byte 8-LL SemScore -5.20%. Without repeated runs or significance tests, the 'at least two metrics' criterion is not a reliable basis for the qualitative claim that output quality improves.","section":"Table 2; Section 3.4"},{"comment":"All results come from a single training run per configuration. Many of the reported differences are under 3%, and several are under 1%, which is within the range that can arise from optimizer stochasticity and data-ordering effects. The paper should provide at least three seeds for the key comparisons (e.g., forward versus static MTP for the 4-head subword and byte configurations) or otherwise justify why single runs are sufficient. Without this, the central improvement claims cannot be evaluated.","section":"Section 3.1.4; Tables 1 and 2"},{"comment":"The curriculum is defined by a single uniform schedule that changes k every E/kmax epochs. The Limitations section acknowledges that this choice is 'somewhat arbitrary' and may under- or overfit the prediction heads. Because no alternative schedule (different step sizes, loss-triggered scheduling, or a random-schedule control) is tested, the observed differences between forward and reverse curricula could be artifacts of this specific schedule rather than evidence about easy-to-hard versus hard-to-easy learning. A sensitivity analysis with at least one alternative schedule is needed before the paper can claim that curriculum-based MTP training fundamentally benefits SLMs.","section":"Section 2.2; Limitations"},{"comment":"The Transformer Layer (TL) head setup is underspecified for dynamic k. The text says 'we use l - k transformer layers as the model backbone and dedicate k out of l layers to handling token prediction,' but when k changes during a curriculum the partition between backbone and prediction layers must change during training. The paper does not explain which layers are added or removed, how the shared output linear layer is applied, or whether parameters are re-initialized when k changes. This is necessary for reproducibility and for interpreting the TL results in Tables 1 and 2.","section":"Section 3.1.2"}],"minor_comments":[{"comment":"The first bullet states that 'Byte-level dynamic MTP models outperform subword-level ones,' but Table 1 does not directly support this as a general statement about all dynamic models; the reverse curriculum drives most of the byte-level gains, and the forward curriculum often degrades byte-level BPB.","section":"Section 3.2.2"},{"comment":"The abstract says the forward curriculum improves 'downstream NTP performance,' but the comparisons in Table 1 are made against the static MTP baseline, not against the NTP baseline. The comparison target should be stated explicitly.","section":"Abstract"},{"comment":"The paper uses gpt-3.5-turbo as the G-Eval judge but does not report the prompt temperature or the number of evaluation samples per generation; adding these details would improve reproducibility.","section":"Section 3.4.1"},{"comment":"The notation for the epoch-based schedule uses E/kmax, but the Limitations section later refers to 'm/n steps' without connecting this to the earlier definition; the notation should be unified.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is readable and the experimental matrix is extensive, but the abstract overstates the evidence in the paper's own tables. The main issues are a claim-evidence mismatch for the forward curriculum, lack of repeated runs, and an underspecified TL-head dynamic implementation. These are fixable with reframing and additional experiments, so I recommend major revision rather than rejection. No novelty disclosure concern: the paper builds on prior MTP work with an incremental curriculum idea, and the authors acknowledge limitations honestly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2505.22757. The new thing here is simple and testable: instead of training with a fixed number of MTP heads, you change k on a fixed epoch-based schedule. Forward curriculum (NTP→MTP) and reverse (MTP→NTP) are both evaluated, and the forward/reverse comparison is genuinely new relative to Gloeckle et al. That deserves credit. The experimental sweep is broad — 1.3B and 3B, subword and byte tokenization, linear and transformer heads, several NTP and generation metrics — and the paper is honestly written, with a Limitations section that openly calls the schedule 'somewhat arbitrary.' The FineWeb-Edu check is also a good-faith scaling probe.\n\nNow the soft spots. The abstract's central claim — that the forward curriculum improves downstream NTP performance — does not survive contact with Table 1. For byte-level models, forward degrades MiniPile BPB in all four configurations and degrades or ties LAMBADA in four of four. For subword models, it helps only at 4 heads and hurts at 2 heads. The paper's own Section 3.2.2 admits the curricula don't beat the NTP baseline for subword models. The reverse curriculum is the one that consistently improves NTP over static MTP, but it forfeits the self-speculative decoding speedups. So the honest headline would be: reverse improves NTP but no speedup; forward preserves speedup but doesn't reliably help NTP. That is a defensible trade-off story, but it is not what the abstract says.\n\nThe other weaknesses are more standard. No repeated seeds, no error bars, no significance tests, and only a promised code release. The output-quality claim is mixed: forward beats static on at least two metrics in 8/12 configurations, but there are negative signs and no variance. The schedule is admittedly arbitrary, so the forward/reverse difference could be an artifact of that specific schedule rather than evidence about curriculum direction. That said, the paper acknowledges this and frames the result as a proof-of-principle, which is fair.\n\nOn the plus side, the SemScore self-reference is not a problem; it's a fixed sentence-embedding metric, not fit to these results. The citation pattern looks appropriate, including Gloeckle et al., Salhan et al., and the curriculum-learning literature.\n\nVerdict: this deserves a serious referee. The idea is new enough and the experiments are extensive enough to warrant engagement. But a referee should push for a rewritten abstract and conclusion that match the actual tables, repeated runs or at least explicit variance reporting, and ideally a release of the training code. I'd be happy to see it in the reading group as a case study in claim-evidence mismatch.","headline":"Interesting and honest empirical study, but the abstract overclaims the forward curriculum's NTP gains; the reverse curriculum is the one that actually improves NTP, and the paper needs claim-narrowing and variance reporting before it is publishable.","tokens_in":21819,"tokens_out":2542,"would_cite":true,"duration_ms":26212,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Small language models trained with a forward curriculum — starting at next-token prediction and adding heads over time — keep multi-token prediction's speedups while improving on static MTP baselines.","keywords":["multi-token prediction","curriculum learning","pre-training objective","self-speculative decoding","small language models","byte-level tokenization","next-token prediction","inference acceleration"],"falsifier":"Train identical 1.3B and 3B models on MiniPile under four schedules matched for total head-time: forward, reverse, static MTP, and a randomly shuffled order of the same $k$ values, plus an adaptive schedule that advances $k$ only when the current heads' draft-acceptance rate plateaus. If the shuffled schedule reproduces the forward curriculum's main-head gains and near-static-MTP acceptances, ordering is not the causal ingredient and the curriculum claim fails; if the adaptive schedule clearly beats the fixed one, the fixed schedule — the paper's acknowledged weak point — is the binding constraint.","tokens_in":20686,"feed_emoji":"⚡","tokens_out":16783,"duration_ms":158030,"temperature":0.7,"pith_summary":"Multi-token prediction (MTP) — training a model to predict the next $k$ tokens at once through several heads on a shared backbone — promises faster inference through self-speculative decoding, but small language models have been reported to learn the objective poorly. The paper asks whether ordering that objective during pre-training fixes the problem, and proposes two curricula that change the number of active prediction heads over time: a forward curriculum that starts with ordinary next-token prediction ($k = 1$) and adds one head at regular intervals up to $k = k_{\\max}$, and a reverse curriculum that does the opposite. The paper's claim is that the forward curriculum gives 1.3B and 3B models the best trade-off — better downstream next-token performance and output quality than a static MTP baseline, with self-speculative decoding speedups nearly as large — while the reverse curriculum yields stronger next-token models but renders its extra heads useless for decoding. If the claim is right, a small model can get both quality and fast generation from a single fixed training schedule, with no architectural changes and no auxiliary draft model.","feed_headline":"Start easy, then add heads — small models keep multi-token speed","feed_subtitle":"Small models trained on one-token prediction first keep both quality and multi-token decoding speed.","key_machinery":"The object that carries the argument is a schedule over the active prediction-head count $k$, defined per training epoch $e$ as $k_{\\text{current}}(e) = \\min(k_{\\max}, \\lfloor e/(E/k_{\\max}) \\rfloor + 1)$ for the forward curriculum and $k_{\\text{current}}(e) = \\max(1, k_{\\max} - \\lfloor e/(E/k_{\\max}) \\rfloor)$ for the reverse, where $E$ is the total number of training epochs. The schedule decides which of the $k$ independent language-modeling heads contribute to the multi-token loss $L_{\\text{MTP}} = -\\sum_{t=1}^{T}\\sum_{i=1}^{k} \\log P(x_{t+i} \\mid x_1, \\dots, x_t; \\theta)$ at each stage of pre-training. The payoff mechanism is self-speculative decoding: the trained heads draft $k$ tokens in one forward pass and a verification pass accepts or rejects them, so a model's per-head acceptance rate measures how usable the auxiliary heads actually are. That acceptance rate is what separates the two curricula — forward-trained heads are accepted almost as often as static MTP's, reverse-trained heads almost never.","core_discovery":"Training Llama-style 1.3B and 3B models for a single epoch on MiniPile, with subword and byte vocabularies, two head designs (linear layers and transformer layers), and $k = 2, 4,$ or $8$ predicted tokens, the paper reports a consistent split between the two directions of curriculum. The forward curriculum ($k$ grows from 1 to $k_{\\max}$) improves over the static MTP baseline on next-token benchmarks in most configurations — for four-head models LAMBADA bits-per-byte improves by roughly 4–5 percent relative and BLiMP accuracy rises in most cases — and its drafted tokens are accepted during self-speculative decoding almost as often as static MTP's, keeping inference speed as a close second. The reverse curriculum ($k$ shrinks from $k_{\\max}$ to 1) produces the stronger next-token model, and for byte-level tokens it matches or beats plain NTP on BLiMP, ARC-E, and OBQA, but its auxiliary heads are almost never accepted at decoding time, so its speed effectively falls back to the NTP rate. The paper's conclusion is that the forward direction is the useful one: it is the only curriculum that keeps the MTP speed benefit while improving over the static MTP quality baseline, and the pattern persists when the training budget is raised to 10B tokens of FineWeb-Edu.","pith_inferences":["A matched-exposure control — forward versus static MTP with identical total head-epochs — would separate the benefit of ordering from the benefit of spending early training on easier one-token targets; the paper does not run this control, so part of the forward gain could be extra effective training, not curriculum ordering.","The per-head draft-acceptance rate the paper measures at the end of training could serve as an online signal: an adaptive curriculum that adds a head only once the previous head's acceptance plateaus is a direct, testable upgrade to the fixed schedule the authors themselves flag as arbitrary.","Because the clearest wins appear with byte-level tokens, the forward curriculum is most promising for small-vocabulary regimes — multilingual scripts, morphologically rich languages, and code — where subword units are scarce or irregular.","The reverse curriculum's pattern of a good main head with dead auxiliary heads suggests MTP's extra heads can fail silently even when the backbone benefits, so future MTP pre-training evaluations should report draft acceptance alongside perplexity and downstream scores."],"forward_implications":["Small models trained with the forward curriculum can act as their own speculative decoder: draft acceptance nearly matches static MTP while main-head next-token performance improves on it, removing the usual quality-for-speed trade-off of MTP on sub-7B models.","The reverse curriculum effectively yields a next-token model with dead auxiliary heads: its main head is stronger, but its drafted tokens win almost no acceptances, so nothing is gained at inference time — the choice between the two curricula is a choice between speed-plus-quality and quality-only.","Byte-level tokenization is where dynamic MTP earns its keep: reverse-curriculum byte-level models match or beat plain NTP on BLiMP, ARC-E, and OBQA, even though a static MTP objective with the same tokens does not.","The curriculum advantage over static MTP is clearest with four heads and survives a tenfold larger data budget, since the 10B-token FineWeb-Edu run reproduces the same ordering of results.","Linear-layer and transformer-layer heads behave similarly under the curricula, so the observed effect is attributable to the schedule rather than to the extra parameters that linear heads add."],"supporting_citations":[{"why":"Introduced the multi-token prediction objective with independent heads on a shared backbone, supplied the head design choices, and reported that MTP gains grow with model size — the premise the curricula target.","marker":"Gloeckle et al. (2024)"},{"why":"Proposed curriculum learning as a training principle; the forward and reverse schedules are instances of it.","marker":"Bengio et al. (2009)"},{"why":"Defined the blockwise speculative decoding scheme whose forward-pass speedups and per-head acceptances are the paper's main inference-speed measure.","marker":"Stern et al. (2018)"},{"why":"Provided the MiniPile dataset on which all main models are pre-trained and their outputs evaluated.","marker":"Kaddour (2023)"},{"why":"Supplied the Llama-family decoder-only architecture configuration used for all 1.3B and 3B models.","marker":"Touvron et al. (2023b)"},{"why":"The BLiMP benchmark is the primary linguistic-capability evaluation that distinguishes dynamic from static MTP models.","marker":"Warstadt et al. (2020)"},{"why":"Provided the FineWeb-Edu corpus used for the 10B-token scaling check of the curricula.","marker":"Lozhkov et al. (2024)"}],"fun_headline_variants":["Small models gain MTP speed with a forward curriculum","NTP-first training unlocks multi-token speed for small LMs","Forward curriculum beats reverse for small-model MTP speed","Easy-to-hard MTP curriculum preserves decoding speed in SLMs","Start with one token to keep MTP speed in small models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that changing the number of predicted tokens on a fixed schedule — one head added or removed every $E/k_{\\max}$ epochs, with no signal from the model's own loss or acceptance rate — is a faithful realization of curriculum learning; the paper itself concedes this spacing is 'somewhat arbitrary' and can under- or over-train individual heads.","fun_headline_variants_meta":{"raw":{"variants":["Small models gain MTP speed with a forward curriculum","NTP-first training unlocks multi-token speed for small LMs","Forward curriculum beats reverse for small-model MTP speed","Easy-to-hard MTP curriculum preserves decoding speed in SLMs","Start with one token to keep MTP speed in small models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000374,"raw_usage":{"total_tokens":2029,"prompt_tokens":1012,"completion_tokens":1017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":934}},"tokens_in":628,"tokens_out":1017,"duration_ms":12515,"temperature":1.0,"reasoning_tokens":934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:01:00.273653+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train identical 1.3B and 3B models on MiniPile under four schedules matched for total head-time: forward, reverse, static MTP, and a randomly shuffled order of the same $k$ values, plus an adaptive schedule that advances $k$ only when the current heads' draft-acceptance rate plateaus. If the shuffled schedule reproduces the forward curriculum's main-head gains and near-static-MTP acceptances, ordering is not the causal ingredient and the curriculum claim fails; if the adaptive schedule clearly beats the fixed one, the fixed schedule — the paper's acknowledged weak point — is the binding constraint.","supporting_citations":[],"review_version":1}