Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Xmodel-2 Technical Report

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read Xmodel-2 reports that a 1.2-billion-parameter model can lead the 1-2B class on complex-reasoning and agent benchmarks by transferring hyperparameters from tiny wind-tunnel models and tuning the SFT data ratio in the WSD decay phase.

desk verdict The paper's central SOTA claim is contradicted by its own benchmark tables, but the released model and the decay-phase data-ratio recipe have real value if the claims are rewritten. read the letter →

arxiv 2412.19638 v1 pith:R7BX4S2P submitted 2024-12-27 cs.AI

classification cs.AI
keywords smalllanguagemodels1.2BparametermodelmaximalupdateparametrizationhyperparametertransferWSDlearningrateschedulerdataratiooptimizationcomplexreasoningagent-basedtasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Xmodel-2 is a 1.2-billion-parameter language model built for reasoning tasks, and the paper's central claim is that such a small model can beat other 1-2B models on complex reasoning and agent benchmarks. The route to that result is a training recipe: use the maximal update parametrization so that optimal hyperparameters found on 6M- and 54M-parameter wind-tunnel models transfer unchanged to the 1.2B model, then anneal the learning rate with the Warmup-Stable-Decay scheduler while mixing 64 percent supervised fine-tuning data into the decay phase. The paper reports that this combination improves complex-reasoning performance by 29.31 percent over its baseline and gives top agent-task scores in the 1-2B size class. If the recipe is right, expensive large-scale hyperparameter sweeps can be replaced by cheap small-model experiments, lowering the cost of building capable small reasoners.

What carries the argument

The load-bearing mechanism is the maximal update parametrization (muP), a scaling scheme from the Tensor Programs line of work that fixes how weights are initialized and how learning rates and output scales change with width and depth, so that the same hyperparameters stay near-optimal across model sizes. It is what makes the 6M and 54M wind-tunnel experiments trustworthy for the 1.2B model. The second mechanism is the Warmup-Stable-Decay (WSD) learning-rate schedule, whose final decay phase is repurposed as a supervised fine-tuning stage: the paper anneals the learning rate exponentially while mixing SFT data into the pretraining stream, with the SFT share searched and set to 64 percent.

What would settle it

Train a mid-size model, for example 300M parameters, using the wind-tunnel-optimal settings and compare the 64 percent SFT decay mix against, say, a 50 percent mix; if the 64 percent mix does not win or the optimal ratio falls outside the 60-69 percent band, the transfer assumption that carries the paper's central claim would be falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a 1.2B decoder-only model pretrained on 1.5 trillion tokens can reach state-of-the-art complex reasoning and agent performance among 1-2B parameter models. The discovery is not a new architecture but a transferable training configuration: maximal update parametrization makes the optimal learning rate, embedding scaling, and depth scaling stable across model sizes, so a 300-configuration Bayesian search on the 6M model, as opposed to a 570,000-configuration grid, yields settings that carry to the 1.2B model. The second half of the recipe is the decay phase of the WSD scheduler, where the paper mixes supervised fine-tuning data with pretraining data and searches the SFT ratio, finding the optimum between 60 and 69 percent and choosing 64 percent. This data-ratio search, combined with rule-based prompt diversification and SimHash deduplication, accounts for the reported 29.31 percent gain over the baseline in complex reasoning. The authors further claim that the same model is state of the art on ReAct-based agent tasks, including HotpotQA, FEVER, AlfWorld, and WebShop, among 1-2B models.

Load-bearing premise

The entire recipe rests on the assumption that hyperparameters and data ratios found optimal on 6M- and 54M-parameter wind-tunnel models remain optimal for the 1.2B model without any validation at an intermediate scale.

Editorial extensions

If this is right

  • Hyperparameter and data-ratio search can be done mostly on 6M and 54M models, cutting the search cost from hundreds of thousands of grid configurations to a few hundred Bayesian trials.
  • The optimal SFT share in the WSD decay phase is a narrow band around 60-69 percent, so practitioners can fix the overall ratio and search only the internal category mix.
  • Chain-of-thought data placed under logic helps complex reasoning, and instruction-formatted math and code data beats pretraining-format data in the decay phase.
  • A 1.2B model with these settings reaches scores competitive with 1.5-1.8B baselines, suggesting strong reasoning ability does not require a large parameter count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 29.31 percent gain bundles several interventions, including the SFT ratio, chain-of-thought inclusion, prompt diversification, and SimHash deduplication, so an ablation isolating the SFT ratio alone would reveal which ingredient actually drives the improvement.
  • If the wind-tunnel transfer holds beyond this model family, the same 6M and 54M search could be used to tune decay-phase data mixes for other small-model releases, making the recipe a cheap template rather than a one-off result.
  • The fitted post-training power law on Wikitext-2 implies that extending context at test time lowers loss with diminishing returns; whether that curve has the same shape on math, code, or agent trajectories is an untested extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This technical report introduces Xmodel-2, a 1.2B-parameter decoder-only language model trained on about 1.5 trillion tokens with a Warmup-Stable-Decay learning-rate scheduler, a deep-and-thin Transformer architecture with GQA, a custom Unigram tokenizer, and embedding sharing. The authors describe wind-tunnel experiments on 6M and 54M models to select hyperparameters and SFT-data ratios for the decay phase, then evaluate the 1.2B model on commonsense reasoning, complex reasoning, and agent-interaction benchmarks. They also fit a test-time loss scaling curve on Wikitext-2. The headline claim is that Xmodel-2 achieves state-of-the-art performance in complex reasoning and agent-based tasks among 1–2B models; checkpoints and code are released publicly.

Significance. If the claims were accurate, the paper would be a useful contribution: it presents an open 1.2B model, a wind-tunnel hyperparameter-search methodology that could reduce HPO cost, a focused data-ratio study for the WSD decay phase, and an agent-task evaluation that is comparatively rare in small-model reports. The agent results in Table 4 are the strongest part of the empirical contribution. However, the central 'state-of-the-art' claim for complex reasoning and commonsense reasoning is directly contradicted by the paper's own Tables 2 and 3, and the key quantitative improvement claim (29.31%) is not defined with a baseline. The methodological transfer claim from 6M/54M to 1.2B is asserted rather than demonstrated. These issues are load-bearing and prevent the paper, as written, from supporting its advertised conclusions.

major comments (4)
  1. [Abstract, §3, Tables 2–3] The claim of state-of-the-art performance in complex reasoning is contradicted by the paper's own results. In Table 3, Xmodel-2's average over GSM8K, MATH, BBH, MMLU, HumanEval, and MBPP is 39.62, while Qwen2.5-1.5B scores 39.98; Qwen2.5 also leads on GSM8K (62.40 vs 55.88), MATH (28.28 vs 25.50), MMLU (59.72 vs 48.87), and MBPP (40.00 vs 29.20). Section 3 additionally states that Xmodel-2 is SOTA 'especially in commonsense reasoning,' yet Table 2 shows Xmodel-2 at 61.79, below Phi-1.5 (65.68), DCLM-1B (63.81), Qwen2.5-1.5B (63.14), and SmolLM-1.7B (61.92). Please either revise the abstract and Section 3 to reflect the actual comparative results or provide a precisely defined alternative notion of 'state-of-the-art' with justification.
  2. [§2.4] The sentence 'These adjustments, combined with optimized data mixing and processing, improved complex reasoning performance by 29.31% compared to our baseline' is not verifiable because the baseline configuration, the evaluation benchmarks, the prompting setup, and the measurement procedure are not specified. Without a defined baseline and without variance or error information, this number cannot be used as evidence for the efficacy of the data-ratio and mixing recipe. Please specify the exact baseline and the evaluation protocol.
  3. [§6 and Abstract] The training configuration of the final 1.2B model is justified solely by 'wind tunnel' experiments on 6M and 54M models. The abstract asserts 'seamless transfer of optimal configurations to larger models,' but Section 6 only states that the small-scale experiments 'confirmed the strategy's suitability'; no intermediate-scale check (e.g., 200M or 300M) or any quantitative comparison of optimal hyperparameters/data ratios across scales is provided. As written, the transfer is an assumption rather than a demonstrated result. Please provide additional transfer evidence or explicitly characterize the transfer as an assumption and soften the abstract's wording.
  4. [§4.2] The 'Post-training Scaling Law' is presented as a general law, but it is a curve fit to a single model on a single dataset (Wikitext-2), with no error bars, no held-out validation, no alternative functional form comparison, and no evidence that the fitted parameters extend to other models or datasets. The name overstates the finding. Please reframe this as an empirical observation about Xmodel-2 or provide evidence of generality across models and datasets.
minor comments (6)
  1. [§2.2] The decay length is given as 'T is set to 5000 steps (20 billion tokens),' but with a batch size of 3.93 million tokens, 5000 steps corresponds to approximately 19.65 billion tokens; please correct the token count or explicitly state that it is approximate.
  2. [§2.4, Figure 2] In Figure 2, the legend order does not match the order in the pie charts, and the listed percentages do not sum to 100 for either panel; please clarify the data-mixture display so the composition is unambiguous.
  3. [§4.2, Eq. (1)] The fitted parameter values are written as 'a ∼ -0.575, b∼ 1.772, c∼ 32.840'; the tilde is nonstandard and ambiguous. Use '=' or report confidence intervals, and make clear that these are fitted values, not exact constants.
  4. [§4.2, Figure 5] The text says 'consistent decrease in perplexity,' but the y-axis of Figure 5 is labeled 'loss'; please make the terminology consistent.
  5. [Table 3] The HumanEval pass@1 score reported for Qwen2.5-1.5B is 5.49, which is substantially lower than Qwen2-1.5B's 20.73 and looks suspicious relative to Qwen2.5's other reasoning scores; please verify this value, as it directly affects the average comparison.
  6. [Title page] The author block is formatted without clear separators between names; add commas or another unambiguous delimiter.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the paper's claims are empirical benchmark results and a training recipe, not a derivation from fitted inputs.

full rationale

The paper makes no load-bearing 'from first principles' derivation. The central claims are (1) a model trained with a specific recipe achieves certain benchmark scores, and (2) data-ratio choices improve performance by 29.31% over an undefined baseline. Both are empirical statements about measured outcomes, not predictions that reduce to their inputs. The only quantitative law presented, L(t)=b+(t/c)^a in Section 4.2, is a power-law fit to the observed Wikitext-2 loss curve; it is descriptive of the same data and is not used to derive any later result, so it does not constitute a circular prediction. The wind-tunnel procedure (Section 6) selects hyperparameters on 6M/54M models and then evaluates the final 1.2B model; the final evaluation is new data not used in the search, so this is a genuine transfer test, though its validity depends on the transfer assumption rather than on circularity. No load-bearing self-citation appears: Tensor Programs [Yang et al.], MiniCPM [Hu et al.], and data-mixing laws [Ye et al.] are external sources used as design priors, not to establish Xmodel-2's own results. The paper's self-reported complex-reasoning average (39.62) is lower than Qwen2.5-1.5B (39.98) in its own Table 3, and the commonsense average (61.79) is below several baselines in Table 2; that is a correctness/support problem for the SOTA claim, not evidence of circularity. Because no claimed result is equivalent by construction to a fitted parameter or to a self-citation chain, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the transferability of hyperparameters and data ratios from tiny wind-tunnel models to the 1.2B target, which is assumed from prior work rather than demonstrated here. The SFT ratio and scaling-law parameters are fitted values that enter the report's claims.

free parameters (4)
  • SFT data ratio in decay phase = 64%
    Chosen from over 400 trials on small models; treated as the optimal ratio and used in final training.
  • WSD decay length T = 5000 steps (20B tokens)
    Set for the annealing phase; no sensitivity analysis shown.
  • Scaling law parameters a, b, c = a=-0.575, b=1.772, c=32.840
    Fitted to the Wikitext-2 loss curve of Xmodel-2; not validated on other models.
  • Hyperparameters from wind tunnel search = learning_rate 0.01-0.02, dim_model_base <256, scale_emb, scale_depth
    Obtained by Bayesian optimization on the 6M nano model; assumed to transfer to 1.2B without intermediate validation.
assumptions (4)
  • domain assumption Tensor Programs (µP) hyperparameter transfer holds from 6M/54M models to 1.2B model.
    Relied on in Sections 2.2 and 6; no validation at an intermediate scale is provided.
  • domain assumption WSD learning rate scheduler improves training stability and reasoning performance.
    Adopted from MiniCPM (Hu et al. 2024), cited but not re-derived.
  • domain assumption Benchmark evaluations are comparable across models.
    All models evaluated in 'the same environment' (Section 3.1), but prompts, decoding settings, and exact harness configurations are not detailed.
  • domain assumption Data ratio search on small models transfers to larger models.
    Relied on in Section 2.3, based on Ye et al. 2024; only the final 1.2B result is shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Xmodel-2 Technical Report." pith.science (2026). https://pith.science/paper/R7BX4S2P

@misc{pith2026241219638,
  author       = {Pith},
  title        = {Pith review of: Xmodel-2 Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7BX4S2P}},
  note         = {Machine review of arXiv:2412.19638}
}
read the original abstract

Xmodel-2 is a 1.2-billion-parameter large language model designed specifically for reasoning tasks. Its architecture enables different model scales to share a unified set of hyperparameters, allowing for extensive experimentation on smaller models and seamless transfer of optimal configurations to larger models. To maximize training efficiency and stability, Xmodel-2 employs the WSD learning rate scheduler from MiniCPM. Pretrained on 1.5 trillion tokens from diverse sources, Xmodel-2 achieves state-of-the-art performance in complex reasoning and agent-based tasks, while maintaining low training costs. These results highlight the potential of efficient model design and training strategies in advancing reasoning capabilities. Model checkpoints and code are publicly available on GitHub at https://github.com/XiaoduoAILab/Xmodel-2

Figures

Figures reproduced from arXiv: 2412.19638 by the authors.

Figure 1
Figure 1. Average Scores on Complex Reasoning Benchmarks (GSM8K, MATH, BBH, MMLU, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Data mixture of different training stages.The left side represents the stable training phase, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Loss curve for Xmodel-2-1.2B. 3 Results We compared Xmodel-2 with recent decoder-only models containing 1–2 billion parameters, as identified in [Lu et al., 2024]. The baselines include TinyLLaMA [Zhang et al., 2024], InternLM2 [Cai et al., 2024], Qwen2 [Yang et al., 2024], MiniCPM [Hu et al., 2024], Llama 3 [Grattafiori et al., 2024], Phi-1.5 [Li et al., 2023b], StableLM-2 [Bellagente et al., 2024], OLMo [Groenevel… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Calibration plot for the pre-trained Xmodel-2-1.2B model on the MMLU dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Post-training Scaling Law for Xmodel-2-1.2B on the Wikitext-2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Grid search over the µP parameterization spaces. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MindFlow: Revolutionizing E-commerce Customer Support with Multimodal LLM Agents

    cs.CL 2025-07 reject novelty 4.0 of 10

    An e-commerce support agent built from known LLM components reports 93.53% relative A/B improvement and 62.5% pass^5 ablation gain, but no code or public benchmark is provided.

Reference graph

Works this paper leans on

22 extracted references · 22 linked inside Pith · cited by 1 Pith paper

  1. [3]

    URL https://arxiv.org/ abs/2107.03374. C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions,

  2. [5]

    URL https://arxiv.org/ abs/2305.14233. L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, Sept

  3. [7]

    URL https://arxiv.org/abs/2402.00838. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring 9 massive multitask language understanding. Proceed- ings of the International Conference on Learning Representations (ICLR), 2021a. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt...

  4. [9]

    org/abs/1804.10959

    URL https://arxiv. org/abs/1804.10959. R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Ko- cetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y . Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Mon- teiro, O. Shliazhko, N. Gontier, N. Meade, A. Ze- baze, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov...

  5. [10]

    URL https://arxiv.org/abs/ 2402.14905. Z. Lu, X. Li, D. Cai, R. Yi, F. Liu, X. Zhang, N. D. Lane, and M. Xu. Small language models: Survey, measurements, and insights,

  6. [11]

    URL https: //arxiv.org/abs/2409.15790. T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering,

  7. [12]

    org/abs/2406.17557

    URL https://arxiv. org/abs/2406.17557. K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi. Winogrande: An adversarial winograd schema chal- lenge at scale. Communications of the ACM, 64(9): 99–106,

  8. [13]

    URL https://arxiv.org/abs/ 2010.03768. S. L. Smith, P.-J. Kindermans, C. Ying, and Q. V . Le. Don’t decay the learning rate, increase the batch size,

Show all 22 references
  1. [14]

    URL https://arxiv.org/abs/2402.00159. M. Suzgun, N. Scales, N. Scharli, S. Gehrmann, Y . Tay, H. W. Chung, A. Chowdhery, Q. V . Le, E. H. hsin Chi, D. Zhou, and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In Annual Meeting of the Associatio...

  2. [15]

    URL https: //arxiv.org/abs/1803.05355. H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma- hairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhar- gava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao,...

  3. [16]

    URL https://arxiv.org/abs/ 2312.02120. J. Welbl, N. F. Liu, and M. Gardner. Crowdsourcing multiple choice science questions,

  4. [17]

    URL https://arxiv.org/abs/ 2304.12244. A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Ya...

  5. [18]

    URL https://arxiv.org/abs/2407.10671. G. Yang, E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer,

  6. [19]

    URL https://arxiv.org/abs/2203.03466. G. Yang, D. Yu, C. Zhu, and S. Hayou. Tensor pro- grams vi: Feature learning in infinite-depth neural networks,

  7. [20]

    URL https://arxiv.org/abs/ 2310.02244. Z. Yang, P. Qi, S. Zhang, Y . Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop ques- tion answering,

  8. [21]

    URL https://arxiv.org/ abs/1809.09600. S. Yao, H. Chen, J. Yang, and K. Narasimhan. Web- shop: Towards scalable real-world web interac- tion with grounded language agents, 2023a. URL https://arxiv.org/abs/2207.01206. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, an...

  9. [22]

    URL https://arxiv.org/abs/2403.16952. R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi. Hellaswag: Can a machine really finish your sentence?,

  10. [2018]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training ver- ifiers to solve math word problems. arXiv preprint arXiv:2110.14168,

  11. [2021]

    URL https://arxiv.org/abs/2108.07732. M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhu- ravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, M. Lee, E. Mostaque, M. Pieler, N. Pinnaparju, P. Rocha, H. Saini, H. Teufel, N. Zanichelli, and C. Riquelme. Stable l...

  12. [2022]

    URL https://arxiv.org/abs/ 2211.15533. T. Kudo. Subword regularization: Improving neu- ral network translation models with multiple sub- word candidates,

  13. [2023]

    URL https://arxiv.org/abs/2311.07361. J. Ainslie, J. Lee-Thorp, M. de Jong, Y . Zemlyanskiy, F. Lebrón, and S. Sanghai. Gqa: Training gener- alized multi-query transformer models from multi- head checkpoints,

  14. [2024]

    URL https://arxiv.org/abs/2407.21783. D. Groeneveld, I. Beltagy, P. Walsh, A. Bhagia, R. Kin- ney, O. Tafjord, A. H. Jha, H. Ivison, I. Magnus- son, Y . Wang, S. Arora, D. Atkinson, R. Authur, K. R. Chandu, A. Cohan, J. Dumas, Y . Elazar, Y . Gu, J. Hessel, T. Khot, W. Merrill...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.