{"id":"f88cb651-fdc0-46fb-9b27-25ae793a9da4","arxiv_id":"2412.17743","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A 2.42B-parameter base model trained on 1.08T tokens matches or beats several industry baselines trained on 7T to 18T tokens across math, code, and general benchmarks.","lead":"YuLan-Mini is a 2.42-billion-parameter language model trained on 1.08 trillion tokens that reaches benchmark scores similar to larger models trained on far more data. The report details the data cleaning, scheduling, stability methods, and annealing tricks that made this efficiency possible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Data-mixture decisions were tuned on the same benchmarks used for the headline comparison (Sections 4.5 and 5.2), so the reported data-efficiency advantage may reflect benchmark-specific overfitting rather than a general pretraining recipe.","rationale":"I reviewed the paper in good faith. The engineering contributions are substantial and transparent: full data composition per phase (Appendix E), training-stability analysis (Section 3), and released checkpoints are real strengths and should be credited. The central claim, however, is an empirical comparison of data efficiency, and that comparison rests on the assumption that the reported benchmark numbers are unbiased estimates of general capability. The text shows this assumption is not met: Section 4.5 describes adjusting data ratios during training based on model performance, with HumanEval as the example, and Section 5.2 explicitly says annealing data were chosen to improve MATH-500. These are the same benchmarks used for the headline evaluation. This is not n-gram contamination; it is a closed optimization loop on the test set. Combined with Section 6.1.3's reliance on official baseline numbers rather than identical re-evaluation, the data-efficiency claim is underdetermined. The reader identified this concern; my contribution is to point to Section 5.2 as the clearest evidence. A held-out evaluation or a frozen-curriculum retraining run would settle whether the effect is large. No change to the reader's CONDITIONAL verdict is needed.","tokens_in":50951,"tokens_out":6402,"duration_ms":57548,"concrete_test":"Re-train YuLan-Mini with the 27-phase curriculum frozen before any evaluation on GSM8K/MATH-500/HumanEval/MBPP, or at minimum re-run the final checkpoints on a held-out suite never used in any data-mixture decision (e.g., MMLU-Pro, GPQA, AIME 2024, LiveCodeBench), and compare against Qwen2.5-1.5B and SmolLM2-1.7B under the same best-of-two-prompt protocol. If the held-out gap is much smaller than the gap on the tuned benchmarks, the data-efficiency claim should be downgraded to 'competitive under benchmark-driven curriculum tuning.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing premise of the data-efficiency claim is that the reported benchmark scores reflect general capability rather than tuned targets. The paper's own protocol undercuts this premise. Section 4.5 states that at each 40B-token curriculum boundary the data ratios are reassessed and adjusted based on the model's performance on benchmarks, with HumanEval given as an explicit example; Section 5.2 states that annealing data were selected to improve performance on challenging math benchmarks, e.g., MATH-500. HumanEval, GSM8K, MATH-500, and related tasks are exactly the benchmarks used in Tables 6-7 and Figure 1 to argue for superior data efficiency. Thus the reported math/code advantage may be an artifact of iterating the data mixture against the evaluation set, not evidence of a generally more data-efficient pretraining recipe. Section 6.1.3 compounds this by reporting baseline scores from official papers rather than re-evaluating baselines under the identical protocol, so the comparison does not control for this loop. This is a correctable experimental-design confound, not an accusation of misconduct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the pre-training of YuLan-Mini, a 2.42B-parameter decoder-only base model trained on about 1.08T tokens, and claims that it reaches performance comparable to industry models trained on substantially more data. The technical recipe has three pillars: a data pipeline with cleaning, mixing, and curriculum scheduling; a stability-focused optimization setup combining scaled initialization, µP-like rules, and WeSaR re-parameterization; and an annealing stage with targeted data selection, learning-rate annealing, and context extension to 28K. The authors evaluate the model on a suite of math, code, commonsense, and Chinese benchmarks, and release detailed per-phase data compositions and checkpoints. The central claim is that the reported benchmark results demonstrate a data-efficient pre-training recipe reproducible in a university setting.","tokens_in":51218,"tokens_out":3568,"duration_ms":35868,"significance":"If the data-efficiency claim holds, the paper is a valuable resource for the community: it provides unusually detailed disclosure of data composition at each curriculum phase, a systematic discussion of training-stability diagnostics and mitigations, and an open release that lowers the barrier for reproducing competitive small models. The per-phase data tables and the stability analysis are concrete contributions that do not depend on the contested comparison. However, the headline claim depends on the evaluation protocol being a fair and controlled comparison, and that premise is weakened by the paper's own description of how the data mixture and annealing data were chosen.","major_comments":[{"comment":"The central data-efficiency claim is not supported by a controlled evaluation protocol. Section 4.5 states that at each 40B-token curriculum boundary the data ratios are reassessed and adjusted based on the model's overall performance, with HumanEval given as an explicit example, and Section 5.2 states that formal math and o1-like reasoning data were incorporated to improve performance on challenging math benchmarks such as MATH-500. These are the same benchmarks used in Tables 6-7 and Figure 1 to argue for superior data efficiency. The reported math/code advantage may therefore reflect iterative tuning of the data mixture against the evaluation set rather than a generally more data-efficient pre-training recipe. I ask the authors to either hold out a set of benchmarks that were never used for data-mix decisions, report the trajectory of decisions together with the resulting scores, or evaluate frozen checkpoints from the originally scheduled curriculum.","section":"Section 4.5, Section 5.2, Tables 6-7"},{"comment":"The comparison against baselines is not made under identical conditions. The text states that for CoT benchmarks the authors evaluate each model with both a short and a long prompt and select the higher score, while baseline numbers are mostly cited from official reports rather than re-run in the same harness (the table marks several values with an asterisk). This per-model prompt selection, combined with heterogeneous evaluation sources, can systematically inflate the relative standing of YuLan-Mini. The authors should re-evaluate all baselines with the identical evaluation code, generation limits, and a single fixed prompt per task, and report both prompt variants or justify why prompt selection cannot favor their model.","section":"Section 6.1.3, Tables 6-7"},{"comment":"The annealing ratio and annealing function are fitted choices whose selection is part of the final model. The paper estimates the 8% annealing ratio from a scaling law and states that 1-sqrt annealing was chosen because it performed best empirically, and the final model is produced with that exact configuration. Since the headline comparison is made with the final model only, the reader cannot separate the effect of the annealing strategy from the effect of having selected the best-performing configuration on the evaluation benchmarks. At minimum, the paper should report the performance of checkpoints before annealing and, if available, results from alternative annealing ratios or functions.","section":"Section 5.1, Section 2.4"},{"comment":"The paper's stability story relies partly on proxy-model experiments, but the transfer of the stability conclusions from the 0.05B/0.2B proxy models to the 2.42B model is asserted rather than demonstrated. Section 3.2.2 mentions that instability still appeared when migrating to the target size, and the mitigation is then validated mainly on small proxies and on the final successful run. Given that training-stability claims are one of the three advertised contributions, the authors should provide at least a controlled comparison showing that the chosen initialization/re-parameterization combination prevents divergence on the target-scale model under conditions where the baseline diverges.","section":"Section 2.5 and Table 1"},{"comment":"There is a discrepancy in the token count used for the main claim. The abstract and Section 2.3 state 1.08T tokens, but Table 6 reports the 4K checkpoint as trained on 1.04T tokens and the 28K checkpoint as trained on 1.08T tokens. Since the data-efficiency comparison in Figure 1 uses the average scores of the final model, the exact token count attributed to the evaluated checkpoint must be clarified. If the 4K checkpoint is the one used for parts of the comparison, the paper should state whether the comparison uses the 1.04T or 1.08T checkpoint.","section":"Table 6 and Section 2.3"}],"minor_comments":[{"comment":"There are several typographical and grammatical errors: 'diffrent' in Table 1, 'intergration' in Section 2.5, 'have have' in Section 3.3.2, and 'hightlited' in Section 4.2. These should be corrected.","section":"Throughout"},{"comment":"The residual connection scaling factor is listed as 1.4√n_layers in Table 3 for YuLan-Mini, but the text in Section 3.2 does not derive or motivate this specific value; please add a short explanation or reference.","section":"Section 2.4 and Table 3"},{"comment":"The caption says that models larger than 3B are plotted in gray, but the legend and axis labels do not make it easy to identify which points correspond to which models; please add labels to the points or a clearer legend.","section":"Figure 1"},{"comment":"The evaluation section states that gpt-4o-mini is used to verify MATH-500 outputs and manual checks were conducted, but the number of samples checked manually is not given; please specify the verification procedure and sample size.","section":"Section 6.1.3"},{"comment":"The detailed phase tables are useful, but the row for Phase 1 says the first 10B tokens are warmup and the next 30B are stable training; this should be stated directly above the table as well as in the main text for readability.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper is a technical report with unusually high transparency, and the stability and data-schedule sections contain work that is likely useful to practitioners. My main concern is that the headline 'data-efficient' comparison is built on an evaluation protocol that is not fully controlled: the data mixture was adjusted on the evaluation benchmarks during training, baseline numbers are cited from other papers, and per-model CoT prompt selection is used. These are fixable experimental-design issues rather than fundamental flaws, so I recommend major revision rather than rejection. I would also ask the editors to ensure that the authors provide a clear statement of which checkpoint and token count is used in each comparison, since the abstract, Section 2.3, and Table 6 currently give inconsistent numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a technical report, not a new method paper, and it is one of the more honest ones. The authors release phase-level data mixtures, decontamination details, and checkpoints, and they explicitly state that data ratios were adjusted during training based on benchmark performance. That transparency is the paper's main strength.\n\nWhat's actually new: a specific, reproducible configuration—2.4B parameters, 1.08T tokens, WSD schedule with 8% annealing, µP-style initialization plus WeSaR, synthetic math/code reasoning data, and an anneal that extends context to 28K. None of these is new in isolation, but the combination, plus the full data composition in Appendix E, is a valuable resource for university labs trying to train competitive small models on a budget. The training-stability analysis (hidden-state variance as an early warning signal) is also a useful practical contribution.\n\nThe soft spots are in the evaluation, not the engineering. Baselines are cited from official reports rather than re-run under identical settings. For GSM8K and MATH, the authors pick the better of two CoT prompts per model—but since baselines were not re-run, that selection only applies to their own model, which inflates the comparison. More importantly, Section 4.5 says data ratios were adjusted every 40B tokens based on performance on benchmarks like HumanEval, and Section 5.2 says annealing data were chosen to improve MATH-500. Those are exactly the benchmarks used in the headline tables. So the reported math/code advantage is partly a result of tuning against the evaluation set. This is a correctable confound, but it means the 'data efficiency' claim is not yet established in a controlled way.\n\nThe paper's own numbers show the claim is selective: on MMLU and HellaSwag, YuLan-Mini is below Qwen2.5-1.5B. It is very strong on math/code, which is precisely where the curriculum was tuned.\n\nOverall: this paper deserves serious review as a technical report. The recipe is worth having, and the transparency is exemplary. But the efficiency claim should be reframed or tested on a held-out benchmark not used in any curriculum decision. I would accept it for peer review with that caveat.","headline":"A transparent, practical training recipe for a 2.4B model, but the headline data-efficiency claim is weakened by benchmark-tuned data decisions and cited baselines.","tokens_in":51753,"tokens_out":3305,"would_cite":true,"duration_ms":31354,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2.42B-parameter model trained on 1.08T tokens matches small-model rivals trained on up to 18T tokens.","keywords":["data-efficient pre-training","small language models","training stability","data curriculum","synthetic reasoning data","annealing","long context extension","open reproduction"],"falsifier":"Retrain YuLan-Mini from the released checkpoints and per-phase data under a pre-registered protocol — a single fixed chain-of-thought prompt, no curriculum feedback from the test sets, and baselines re-run in the same harness — and check whether the eight-benchmark average still beats Qwen2.5-1.5B and SmolLM2; if it falls behind when prompt choice and data-ratio tuning are taken away, the data-efficiency claim is not robust.","tokens_in":50742,"feed_emoji":"🤖","tokens_out":13063,"duration_ms":105367,"temperature":0.7,"pith_summary":"The paper reports YuLan-Mini, a 2.42B-parameter decoder-only base model pre-trained on 1.08T tokens of open and synthetic data. Its central claim is that a carefully engineered recipe — multi-stage data cleaning and a 27-phase curriculum, stability-focused initialization and optimization, and an 80B-token annealing stage with targeted selection and 28K context extension — lets this model match or beat industry small models trained on 2T to 18T tokens, including Qwen2.5-1.5B (18T), SmolLM2-1.7B (11T), and Llama3.2-3B (9T). If the claim holds, it means competitive base models can be reproduced at university scale, and the released per-phase data composition makes the recipe directly testable.","feed_headline":"Trained on 1.08T tokens, a 2.4B model matches 18T-token rivals","feed_subtitle":"Open weights and per-phase data make the data-efficiency recipe testable at university scale.","key_machinery":"The load-bearing mechanism is the three-part training recipe rather than any single technique. (1) The data pipeline combines MinHash de-duplication, heuristic filters, topic classifiers for math/code/reasoning recall, model-based quality scoring, n-gram decontamination, and large-scale synthetic generation of reasoning documents, chain-of-thought solutions, formal Lean proofs, and reflection data; these streams are arranged into 27 curriculum phases of 40B tokens each under a WSD schedule (10B warmup, 990B stable, 80B annealing) with per-phase mixture shifts kept under 3 percent. (2) The stability package scales initialization to $\\sigma_{\\mathrm{base}} = \\sqrt{2/(5d)}$, scales the embedding output by 10, scales residual branches by $1.4\\sqrt{n_{\\mathrm{layers}}}$, applies µParameterization-style learning-rate scaling to QKV and FFN weights, and adds WeSaR reparameterization $W = \\alpha \\tilde{W}$ to decouple gradient size from direction, allowing a global learning rate of 0.01 with z-loss and a reduced Adam epsilon. (3) The annealing stage spends 80B tokens on a high-value mix selected by an accelerated gradient-based method (a LESS variant with InsTag), decays the learning rate with a 1-sqrt curve, and raises the RoPE base frequency from 10,000 to 490,000 to extend the context window to 28K tokens while using masked cross-document attention to preserve short-text performance.","core_discovery":"YuLan-Mini is a 2.42B-parameter model with 56 layers, a 1,920-dimensional hidden width, grouped-query attention, and a 99K-vocabulary tokenizer with digit splitting. Trained on 1.08T tokens, the 28K-context checkpoint scores 37.80 on MATH-500 (4-shot), 64.00 on HumanEval (0-shot), 68.46 on GSM8K, and 49.10 on MMLU (5-shot). The authors argue that the model's aggregate performance, averaged over eight benchmarks, is competitive with small industry models trained on 2–18T tokens, despite using roughly one-half to one-seventeenth of their training budgets. The paper attributes this efficiency to a data pipeline that combines cleaning, classifier-based recall, decontamination, and a 27-phase WSD curriculum; to a stability package built on µParameterization-style initialization plus WeSaR reparameterization; and to an annealing stage that mixes high-value reasoning data with long-context training. It releases the full per-phase token composition to make the whole recipe reproducible.","pith_inferences":["Whether the same 1.08T-token budget suffices at 7B scale is an open question; the stability package is designed to transfer via µParameterization, but the data curriculum and annealing are tuned for 2.4B.","The evaluation protocol's prompt selection (better of two per model) could inflate YuLan-Mini's margin; a single pre-registered prompt would make the data-efficiency claim sharper.","The annealing mix blends long-thought, formal-math, and gradient-selected data; isolating these components would show which one drives the MATH-500 and HumanEval gains.","The 4K and 28K checkpoints trade off general knowledge (MMLU 51.79 vs 49.10) for reasoning gains (MATH-500 32.60 vs 37.80); deployment choices may favor different checkpoints."],"forward_implications":["A 2.42B base model can reach the top of its size class on math and code benchmarks after 1.08T tokens, with the 28K checkpoint scoring 37.80 on MATH-500, 68.46 on GSM8K, and 64.00 on HumanEval.","The 27-phase WSD curriculum, with per-phase mixture shifts capped at 3%, keeps training stable across 1.08T tokens at a global learning rate of 0.01.","The annealing stage extends the context window from 4K to 28K tokens by raising the RoPE base frequency to 490,000 while using long-context data with masked cross-document attention to preserve short-text skills.","Because the full per-phase data composition is released, the recipe can be reproduced and its components (data selection, annealing mix, stability package) can be ablated by other groups."],"supporting_citations":[{"why":"Supplies the WSD schedule and annealing formulation the paper adapts, as well as the MiniCPM-2.6B baseline and the base tokenizer.","marker":"[Hu et al., 2024]"},{"why":"Defines µParameterization, the zero-shot hyperparameter-transfer scheme used to move proxy-model settings to full scale.","marker":"[Yang et al., 2022]"},{"why":"Introduces WeSaR reparameterization $W = \\alpha \\tilde{W}$, which the paper uses to decouple gradient magnitude from direction and prevent loss spikes.","marker":"[Nishida et al., 2024]"},{"why":"CerebrasGPT demonstrates scaled-initialization and µParameterization-style training at scale, the template for the stability hyperparameters.","marker":"[Dey et al., 2023b]"},{"why":"Shows that longer warmup reduces learning-rate sensitivity and documents small-model logit growth, motivation for the 10B-token warmup.","marker":"Wortsman et al. [2024]"},{"why":"Provides the learning-rate-annealing scaling law used to set the 8% annealing ratio (80B tokens).","marker":"[Tissue et al., 2024]"},{"why":"Gives the scaling-law formula $C = 6ND$ used to estimate FLOPs for the efficiency comparison.","marker":"Kaplan et al. [2020]"},{"why":"Industry baseline Qwen2.5-1.5B, trained on 18T tokens; its reported scores are the paper's primary data-efficiency comparison point.","marker":"[Qwen-Team, 2024]"},{"why":"FineWeb-Edu supplies the largest English web component and its scorer is used for model-based quality scoring of web and math data.","marker":"[Lozhkov et al., 2024a]"},{"why":"LESS gradient-based data selection method, which the paper accelerates and combines with InsTag for the annealing-stage data mix.","marker":"[Xia et al., 2024]"}],"fun_headline_variants":["2.4B model rivals 18T-token LLMs on 1.08T tokens","Open 2.4B LLM matches giants with 17x less data","Data-efficient 2.4B model: 1.08T tokens, open weights","YuLan-Mini: 2.4B params, 1.08T tokens, big-model matching","1.08T tokens, 2.4B params: open model matches 18T rivals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire data-efficiency comparison rests on the evaluation being a fair apples-to-apples contest: baseline scores come from each model's own paper, the better of two chain-of-thought prompts is picked per model, and the training-data mixture is adjusted based on the very benchmarks used in the final ranking, so if those numbers are not directly comparable the claimed token savings could shrink.","fun_headline_variants_meta":{"raw":{"variants":["2.4B model rivals 18T-token LLMs on 1.08T tokens","Open 2.4B LLM matches giants with 17x less data","Data-efficient 2.4B model: 1.08T tokens, open weights","YuLan-Mini: 2.4B params, 1.08T tokens, big-model matching","1.08T tokens, 2.4B params: open model matches 18T rivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2985,"prompt_tokens":944,"completion_tokens":2041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":1919}},"tokens_in":560,"tokens_out":2041,"duration_ms":13568,"temperature":1.0,"reasoning_tokens":1919,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:12:55.342421+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain YuLan-Mini from the released checkpoints and per-phase data under a pre-registered protocol — a single fixed chain-of-thought prompt, no curriculum feedback from the test sets, and baselines re-run in the same harness — and check whether the eight-benchmark average still beats Qwen2.5-1.5B and SmolLM2; if it falls behind when prompt choice and data-ratio tuning are taken away, the data-efficiency claim is not robust.","supporting_citations":[],"review_version":1}