Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Technical Report of TeleChat2, TeleChat2.5 and T1

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that T1-115B, an open-weight 115-billion-parameter model, outperforms proprietary models o1-mini and GPT-4o on math, alignment, instruction-following, and function-calling benchmarks.

desk verdict A genuinely useful model-release report with unusually detailed training details; the benchmark-vs-proprietary claims are plausible but not yet substantiated. read the letter →

arxiv 2507.18013 v3 pith:HY5BIWCU submitted 2025-07-24 cs.CL

Zihan Wang , Xinzhang Liu , Yitong Yao , Chao Wang , Yu Zhao , Zhihao Yang , Wenmin Deng , Kaipeng Jia
show 30 more authors
Jiaxin Peng Yuyao Huang Sishi Xiong Zhuo Jiang Kaidong Yu Xiaohui Hu Fubei Yao Ruiyu Fang Zhuoru Jiang Ruiting Song Qiyi Xie Rui Xue Xuewei He Yanlei Xue Zhu Yuan Zhaoxi Zhang Zilu Huang Shiquan Wang Xin Wang Hanming Wu Mingyuan Wang Xufeng Zhan Yuhan Sun Zhaohu Xing Yuhao Jiang Bingkai Yang Shuangyong Song Yongxiang Li Zhongjiang He Xuelong Li
This is my paper
classification cs.CL
keywords TeleChat2Chat2.5T1reinforcementlearningchain-of-thoughtreasoningdirectpreferenceoptimizationmathematicalopen-weightmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This technical report introduces three closely related open-weight model families and argues that the performance jump over the previous TeleChat comes almost entirely from training strategy, not architecture. TeleChat2 is pre-trained on 10 trillion tokens and then aligned with supervised fine-tuning and direct preference optimization; TeleChat2.5 and T1 add a continual pre-training phase on domain-specific data and a reinforcement learning stage targeting math, code, and tool use. T1 is the long-chain-of-thought 'thinking' variant, while TeleChat2.5 is the fast non-thinking variant, both at 35B and 115B parameter sizes. The report's headline claim is that T1-115B scores above proprietary models on MATH500, AlignBench, IFEval, and BFCL, which matters because the weights and code are public and the claim can therefore be checked and built upon.

What carries the argument

The load-bearing machinery is the post-training pipeline rather than any single layer or objective: SFT data built from verifiable-answer sources, iterative DPO on pairs separated by at least two score points, and RL whose reward functions are rule-based wherever possible — the math equality checker, sandboxed code execution with test cases, and exact tool-call format and content matching — with an LLM-as-judge score linearly mapped to the interval [-1, 1] for open-ended tasks. Dynamic sampling fills each RL batch only with prompts whose pass rate is neither 0 nor 1, keeping training focused on problems the model can learn from. A secondary mechanism is the long-context annealing stage with staged RoPE base increases, which extends the context window while preserving general capability.

What would settle it

Search the sources the report says it trained on — OpenR1-Math-220k, verifiable-math-problems, the MATH and GSM8K training sets, GitHub-derived code, and the internet-scale pre-training corpus — for exact or near-identical strings of the MATH500 problems; if any evaluation problem appears in the training or RL data, the reported 94.0 on MATH500 would not measure generalization to unseen problems.

Watch

Extended reading notes

Core claim

The report claims to establish a transferable recipe: a conventional dense Transformer, kept largely unchanged from TeleChat, reaches strong reasoning performance when pre-trained on 10 trillion tokens, annealed to long contexts (256K for 35B, 128K for 115B), then post-trained through SFT with separate thinking and non-thinking modes, three rounds of iterative DPO, and RL with verifiable rewards for mathematics, code, and tool use. The evidence for the central claim is the post-training evaluation: T1-115B reports 94.0 versus 90.0 on MATH500, 8.22 versus 7.91 on AlignBench, and 80.15 versus 79.07 on IFEval compared with o1-mini, while TeleChat2.5-115B reports 87.0 versus 75.0 on MATH500 and 83.39 versus 78.65 on BFCL compared with GPT-4o-1120. The author's framing is that these gains show reasoning, coding, tool use, and instruction following can be trained into a dense open model without architectural novelty.

Load-bearing premise

The reported superiority over o1-mini and GPT-4o assumes that the evaluation sets, especially MATH500, were never included in the 10-trillion-token pre-training corpus or in the SFT, DPO, or RL data, a fact the report does not state.

Editorial extensions

If this is right

  • If the reported numbers are correct, an open-weight 115B dense model can beat closed proprietary reasoning models on benchmarked math, coding, and instruction-following tasks without a novel architecture.
  • The same base model can serve two products: T1 for long chain-of-thought reasoning and TeleChat2.5 for fast non-thinking responses, which is useful for deployment where latency matters.
  • Long-context ability up to 128K or 256K tokens can be added during an annealing stage rather than designed into the architecture from the start.
  • Rule-based verification in the form of math equality, code execution, and tool-call matching is enough to drive post-training gains, so future work can extend this style of RL to other verifiable domains.
  • Public release at 35B and 115B parameter scales means the recipe can be tested, adapted, and ablated outside the original training infrastructure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The report never states that MATH500 or the other test sets were excluded from the 10-trillion-token pre-training corpus or from the SFT, DPO, or RL data; if those problems leaked in, the margin over o1-mini and GPT-4o would shrink or vanish.
  • The proprietary-model comparisons use the paper's own sampling settings and unknown inference configurations for o1-mini and GPT-4o; an independent replication on uncontaminated held-out splits would be a stronger test of the claim.
  • The separated SFT, DPO, and RL stages invite a component ablation the paper only partially provides: isolating how many MATH500 points RL adds over SFT plus DPO would tell smaller labs where to spend compute.
  • Because the recipe is described with concrete hyperparameters and reward rules, a feasible extension is to apply the same pipeline to a smaller public base model and check whether the gains transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This technical report presents the TeleChat2, TeleChat2.5, and T1 model families, which share a dense Transformer architecture at 35B and 115B scales. It documents a 10T-token pretraining pipeline with long-context annealing, SFT/DPO/RL post-training, targeted data construction for code, math, tool use, and instruction following, and an engineering section on training with Ascend NPUs. The headline claim is that T1-115B outperforms proprietary models such as OpenAI o1-mini and GPT-4o on selected benchmarks (MATH500, AlignBench, IFEval, BFCL), with model weights and code released publicly.

Significance. If the reported evaluation results are reproducible, the work is significant: it would demonstrate that an open-weight 115B dense model can match or exceed proprietary reasoning models on selected math and alignment benchmarks, and it provides unusually detailed engineering and data-curation information (MFU figures, hardware failure statistics, reward designs) that are valuable to practitioners. The release of six model variants and the associated codebase is a concrete asset. The evaluation evidence, however, is the main load-bearing part of the central claim, and it is currently underspecified; the paper would be substantially strengthened by a decontamination statement, a versioned and released evaluation harness, and uncertainty estimates. I found no equation-level circularity; the reported numbers are internally consistent, and the self-citation to the prior TeleChat report for the base architecture is appropriate.

major comments (3)
  1. [Section 7.2 and Section 5.2] The paper does not state that MATH500, or any other evaluation benchmark in Tables 5-8, was excluded from all training stages. Section 5.2 says the second-stage math SFT uses "MATH, GSM8K training sets" and Section 4.3 says mathematical RL uses OpenR1-Math-220k and verifiable-math-problems; MATH500 is derived from MATH (Section 7.2). If any MATH500 problems appear in these sources, the reported margins over o1-mini and GPT-4o (94.0 vs 90.0 on MATH500, Table 8) would be inflated. A precise decontamination statement covering pretraining, continual pretraining, SFT, DPO, and RL is required before the flagship comparison can be accepted.
  2. [Section 7] Evaluation is performed with a "customized evaluation framework" (Section 7.1) and an "internal evaluation framework" (Section 7.2), but no code, prompts, decoding settings for competitor models, or version identifiers are provided. The descriptions of BFCL ("single-turn performance on python-ast track, reporting averages for both non-live and live subtasks") and AlignBench do not specify the exact harness parameters, judge prompts, answer extraction methods, or which reference scores are used for o1-mini and GPT-4o. Without these details, the abstract's claim that T1-115B outperforms o1-mini and GPT-4o cannot be independently verified as reported.
  3. [Section 7.2, Tables 7 and 8] No error bars, standard deviations, or repeated-run statistics are reported. The key MATH500 margin is 4 points on a 500-problem set (20 problems), and AlignBench uses LLM-based judging, which is stochastic; with the T1 decoding temperature of 0.6, single point estimates are insufficient to establish that the observed advantages are outside run-to-run variation. Confidence intervals or per-seed/per-run results should be provided for the comparisons that support the headline claims.
minor comments (5)
  1. [Abstract and Table 8] The abstract states that "T1-115B outperform proprietary models such as OpenAI's o1-mini and GPT-4o," but Table 8 contains no direct T1-115B vs GPT-4o comparison; GPT-4o-1120 is compared only with TeleChat2.5-115B in non-thinking mode. The wording should be adjusted to match the evidence.
  2. [Section 2] There is a typo in the bullet list: "RoPE base frenquency" should be "RoPE base frequency." Similar typos elsewhere include "famliy" in Section 3.3.2, "there predecessor" in the Introduction, and "Direction preference optimization" in the Introduction.
  3. [Section 7.2] MATH500 is described as derived from the original MATH dataset, but no citation or URL is given for the 500-problem split; please add a reference so readers can identify the exact benchmark version used.
  4. [Section 4.1.3] Equation (1) uses notation such as s(t)i and p(t)i without fully defining the domains of s and p or how they are computed from perplexity curves; a brief formal definition would improve reproducibility.
  5. [Section 6.2.2] The abbreviation "IF1B" appears where the standard name is "1F1B" (one forward, one backward); please correct or clarify. In addition, capitalization of "TeleChat" is inconsistent in the engineering section (e.g., "Telechat-115B").

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark claims rest on external measurements, not on the paper's own definitions or fitted parameters.

full rationale

The paper's central claims (T1-115B outperforming o1-mini and GPT-4o on MATH500, AlignBench, IFEval, and BFCL) are empirical measurements against external benchmarks reported in Section 7.2. No equation in the paper defines a predicted quantity in terms of the benchmark scores being reported, and no fitted parameter is renamed as a prediction. The data-composition algorithms in Sections 3.2.3 and 4.1.3 fit internal validation subsets and perplexity curves, but the final evaluation benchmarks are not the objective being optimized; at most this raises a decontamination question. Section 5.2 states math SFT uses 'MATH, GSM8K training sets' and Section 7.2 evaluates on MATH500, which is 'derived from the original MATH dataset'; the paper never states that MATH500 items were excluded from training or RL data. Similarly, IFEval-style constraints (Section 5.4) and BFCL-style function-call categories (Section 5.3) are used to construct training data before the models are evaluated on the corresponding benchmarks. These are potential train/eval overlap or benchmark-contamination risks, which are soundness and reproducibility concerns, not circularity: the reported numbers are not equal by construction to any fitted parameter or training objective. The self-citations to the prior TeleChat report (Section 2, Section 3.2.2, Section 3.2.4) describe architecture, de-duplication, and data-concatenation choices, but the paper's performance claims do not reduce to those citations. The evaluation harness is described as internal (Section 7.1), which hinders independent reproduction, but that is a reporting limitation rather than a circular derivation. Therefore no circular step can be exhibited from the paper's own equations or self-citation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The report does not rest on a mathematical derivation, so there are no axioms in the formal sense. The domain assumptions above are the load-bearing premises for the empirical claims. The free parameters are training and data-mixing choices that are tuned by hand or by validation-based heuristics; they go into the model-building process but are not themselves derived from theory.

free parameters (6)
  • Data-mixing hyperparameters κ and μ = κ=10, μ=15,000
    Eq. (1) in Section 4.1.3; set to adjust SFT data category proportions based on validation perplexity curves.
  • RoPE base frequencies = 1e6, 8e6, 4e7
    Section 3.3.2; hand-set per annealing stage to extend context length to 32K, 128K, or 256K.
  • DPO β and NLL loss coefficient = β=0.1; NLL coefficient=0.2
    Section 4.2.2; tuned for stable preference training.
  • RL training hyperparameters = lr=5e-7; 16 responses/prompt; batch size 128
    Section 4.3; chosen for Reinforce++ training on math/code/tool RL.
  • SFT data mixing proportions = Not fully specified
    Section 4.1.3; proportions adjusted iteratively based on validation perplexity and then used for training.
  • RL dataset difficulty mix = Medium:Hard = 2:1
    Section 4.3; ratio selected by hand to compose RL training data from pass@5-stratified queries.
assumptions (5)
  • domain assumption Data-mix experiments on 3B/7B models transfer to 35B/115B models
    Section 3.2.3: 'we predict the performance of larger models under different data compositions and select the most promising mix'.
  • domain assumption LLM-as-a-judge scoring reliably ranks responses for preference data
    Section 4.2.1: preference annotation uses an LLM-as-a-judge with a 0-10 scale; no human validation rate is reported.
  • domain assumption Evaluation benchmark problems are not present in training or RL data
    Never stated; training uses MATH/GSM8K training sets and OpenR1-Math, while MATH500 derives from MATH.
  • domain assumption The customized evaluation framework produces scores comparable to official benchmark implementations
    Section 7: 'evaluated using a customized evaluation framework with standardized settings'; details not given.
  • domain assumption Verifiable math checking correctly filters RL data
    Section 4.3: math answers verified with the 'math equal function' from simpleRL-reason; no manual audit described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Technical Report of TeleChat2, TeleChat2.5 and T1." pith.science (2026). https://pith.science/paper/HY5BIWCU

@misc{pith2026250718013,
  author       = {Pith},
  title        = {Pith review of: Technical Report of TeleChat2, TeleChat2.5 and T1},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HY5BIWCU}},
  note         = {Machine review of arXiv:2507.18013}
}
read the original abstract

We introduce the latest series of TeleChat models: \textbf{TeleChat2}, \textbf{TeleChat2.5}, and \textbf{T1}, offering a significant upgrade over their predecessor, TeleChat. Despite minimal changes to the model architecture, the new series achieves substantial performance gains through enhanced training strategies in both pre-training and post-training stages. The series begins with \textbf{TeleChat2}, which undergoes pretraining on 10 trillion high-quality and diverse tokens. This is followed by Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to further enhance its capabilities. \textbf{TeleChat2.5} and \textbf{T1} expand the pipeline by incorporating a continual pretraining phase with domain-specific datasets, combined with reinforcement learning (RL) to improve performance in code generation and mathematical reasoning tasks. The \textbf{T1} variant is designed for complex reasoning, supporting long Chain-of-Thought (CoT) reasoning and demonstrating substantial improvements in mathematics and coding. In contrast, \textbf{TeleChat2.5} prioritizes speed, delivering rapid inference. Both flagship models of \textbf{T1} and \textbf{TeleChat2.5} are dense Transformer-based architectures with 115B parameters, showcasing significant advancements in reasoning and general task performance compared to the original TeleChat. Notably, \textbf{T1-115B} outperform proprietary models such as OpenAI's o1-mini and GPT-4o. We publicly release \textbf{TeleChat2}, \textbf{TeleChat2.5} and \textbf{T1}, including post-trained versions with 35B and 115B parameters, to empower developers and researchers with state-of-the-art language models tailored for diverse applications.

Figures

Figures reproduced from arXiv: 2507.18013 by the authors.

Figure 1
Figure 1. The pre-training framework. 3.2 INITIAL PRE-TRAINING STAGE In the initial pre-training phase, our primary goal is to equip the model with broad and comprehensive world knowledge. To achieve this, we train the model on an extensive, high-quality and diverse 5 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Evaluation results of TeleBase2-115B on the “Needle In A Haystack” test. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The development pipelines of TeleChat2, TeleChat2.5 and T1. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Data Distribution of Supervised Finetuning data. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Example of Virtual Pipeline Parallelism (VPP) Scheduling: Demonstrates the interleaved [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-Objective Exploration and Preference Optimization via Mutual Information

    cs.CL 2026-07 unverdicted novelty 6.0 of 10

    MI-EPO maximizes joint conditional mutual information among responses, feedback, and preference vectors, using probabilistic routing to improve alignment and controllability in multi-objective LLM optimization.

Reference graph

Works this paper leans on

75 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ai flow: Perspectives, scenarios, and approaches, 2025

    Hongjun An, Wenhan Hu, Sida Huang, Siqi Huang, Ruanjun Li, Yuanzhi Liang, Jiawei Shao, Yiliang Song, Zihan Wang, Cheng Yuan, Chi Zhang, Hongyuan Zhang, Wenhao Zhuang, and Xuelong Li. Ai flow: Perspectives, scenarios, and approaches, 2025. URL https://arxiv.org/abs/2506.12479

    arXiv 2025

  2. [2]

    Introducing the next generation of claude, 2024

    Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropic.com/news/claude-3-family

    work page 2024

  3. [3]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  4. [4]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  6. [6]

    Theoremqa: A theorem-driven question answering dataset, 2023

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: A theorem-driven question answering dataset, 2023. URL https://arxiv.org/abs/2305.12524

    arXiv 2023

  7. [7]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradb...

    arXiv 2022

  8. [8]

    Dongjie Ji

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, and et al. Dongjie Ji. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model, 2024 a . URL https://arxiv.org/abs/2405.04434

    arXiv 2024

Show all 75 references
  1. [9]

    Dongjie Ji

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, and et al. Dongjie Ji. Deepseek-v3 technical report, 2024 b . URL https://arxiv.org/abs/2412.19437

    2024 arXiv

  2. [10]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  3. [11]

    Data engineering for scaling language models to 128k context

    Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2025

    2025

  4. [12]

    Introducing gemini 2.0: our new ai model for the agentic era, 2024

    Google. Introducing gemini 2.0: our new ai model for the agentic era, 2024. URL https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/?utm_source=deepmind.google&utm_medium=referral&utm_campaign=gdm&utm_content=

    2024

  5. [13]

    Angela Fan

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, and et al. Angela Fan. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

    2024 arXiv

  6. [14]

    Skywork open reasoner 1 technical report, 2025

    Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner 1 technical report, 2025. URL https://arxiv.org...

    2025 arXiv

  7. [15]

    Measuring massive multitask language understanding, 2021 a

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021 a . URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  8. [16]

    Measuring mathematical problem solving with the math dataset, 2021 b

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021 b . URL https://arxiv.org/abs/2103.03874

    2021 arXiv

  9. [17]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021 c

    2021

  10. [18]

    Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025

    Jian Hu, Jason Klein Liu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025. URL https://arxiv.org/abs/2501.03262

    2025 arXiv

  11. [19]

    Le, Yonghui Wu, and Zhifeng Chen

    Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism, 2019. URL https://arxiv.org/abs/1811.06965

    2019 arXiv

  12. [20]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2...

    2023 arXiv

  13. [21]

    Teknium"

    "Teknium" "interstellarninja". Hermes-function-calling-dataset-v1. URL https://huggingface.co/NousResearch/hermes-function-calling-v1

  14. [22]

    Smith, Yejin Choi, and Hannaneh Hajishirzi

    Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback, 2024. URL https://arxiv.org/abs/2406.09279

    2024 arXiv

  15. [23]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  17. [25]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL https://arxiv.org/abs/2309.06180

    2023 arXiv

  18. [26]

    Modelscope-agent: Building your customizable agent system with open-source large language models

    Chenliang Li, Hehong Chen, Mingshi Yan, Weizhou Shen, Haiyang Xu, Zhikai Wu, Zhicheng Zhang, Wenmeng Zhou, Yingda Chen, Chen Cheng, Hongzhu Shi, Ji Zhang, Fei Huang, and Jingren Zhou. Modelscope-agent: Building your customizable agent system with open-source large language mod...

    2023 arXiv

  19. [27]

    Cmmlu: Measuring massive multitask language understanding in chinese, 2023 b

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese, 2023 b

    2023

  20. [28]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv.org/abs/2109.07958

    2022 arXiv

  21. [29]

    Ring attention with blockwise transformers for near-infinite context, 2023 a

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023 a . URL https://arxiv.org/abs/2310.01889

    2023 arXiv

  22. [30]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023 b

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023 b . URL https://arxiv.org/abs/2305.01210

    2023 arXiv

  23. [31]

    Alignbench: Benchmarking chinese alignment of large language models, 2024 a

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Xiaotao Gu, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. Alignbench: Benchmarking chinese alignment of large l...

    2024 arXiv

  24. [32]

    Scaling laws of ro PE -based extrapolation

    Xiaoran Liu, Hang Yan, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of ro PE -based extrapolation. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openreview.net/forum?id=JO7k0SJ5V6

    2024

  25. [33]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  26. [34]

    Mindspore: Advanced ai framework

    MindSpore. Mindspore: Advanced ai framework. https://www.mindspore.cn/, 2025. Accessed: 5 Feb. 2025

    2025

  27. [35]

    Efficient large-scale language model training on gpu clusters using megatron-lm, 2021

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clus...

    2021 arXiv

  28. [36]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024 arXiv

  29. [37]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  30. [38]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  31. [39]

    Iterative reasoning preference optimization, 2024

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024. URL https://arxiv.org/abs/2404.19733

    2024 arXiv

  32. [40]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machi...

    2025

  33. [41]

    Tool learning with foundation models

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models. arXiv preprint arXiv:2304.08354, 2023

    2023 arXiv

  34. [42]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  35. [43]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  36. [44]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. arXiv preprint arXiv:1910.02054, 2020

    1910 arXiv

  37. [45]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022

    2023 arXiv

  38. [46]

    Ai flow at the network edge, 2024

    Jiawei Shao and Xuelong Li. Ai flow at the network edge, 2024. URL https://arxiv.org/abs/2411.12469

    2024 arXiv

  39. [47]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  40. [48]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

    2002 arXiv

  41. [49]

    Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020. URL https://arxiv.org/abs/1909.08053

    2020 arXiv

  42. [50]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864, 2022

    2022 arXiv

  43. [51]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/abs/2210.09261

    2022 arXiv

  44. [52]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019. URL https://arxiv.org/abs/1811.00937

    2019 arXiv

  45. [53]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025 arXiv

  46. [54]

    Openmathinstruct-1: A 1.8 million math instruction tuning dataset

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. ArXiv, abs/2402.10176, 2024. URL https://api.semanticscholar.org/CorpusID:267681752

    2024 arXiv

  47. [55]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023 arXiv

  48. [56]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin F...

    2023 arXiv

  49. [57]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024 a

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language unders...

    2024 arXiv

  50. [58]

    Telechat technical report, 2024 b

    Zihan Wang, Xinzhang Liu, Shixuan Liu, Yitong Yao, Yuyao Huang, Xuelong Li, Yongxiang Li, Zhonghao Che, Zhaoxi Zhang, Yan Wang, Xin Wang, Luwen Pu, Huinan Xu, Ruiyu Fang, Yu Zhao, Jie Zhang, Xiaomeng Huang, Zhilong Lu, Jiaxin Peng, Wenjun Zheng, Shiquan Wang, Bingkai Yang, Xue...

    2024 arXiv

  51. [59]

    Base of ro PE bounds context length

    Mingyu Xu, Xin Men, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, and weipeng chen. Base of ro PE bounds context length. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=EiIelh2t7S

    2024

  52. [60]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  53. [61]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  54. [62]

    Dapo: An open-source llm reinforcement learning system at scale, 2025

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen...

    2025 arXiv

  55. [63]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  56. [64]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. arXiv preprint arXiv:1910.07467, 2019

    1910 arXiv

  57. [65]

    xlam: A family of large action models to empower ai agent systems

    Jianguo Zhang, Tian Lan, Ming Zhu, Zuxin Liu, Thai Hoang, Shirley Kokane, Weiran Yao, Juntao Tan, Akshara Prabhakar, Haolin Chen, Zhiwei Liu, Yihao Feng, Tulika Manoj Awalgaonkar, Rithesh Murthy, Eric Hu, Zeyuan Chen, Ran Xu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Si...

    2024 arXiv

  58. [66]

    Evaluating the performance of large language models on gaokao benchmark, 2024 b

    Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on gaokao benchmark, 2024 b . URL https://arxiv.org/abs/2305.12474

    2024 arXiv

  59. [67]

    Ape210k: A large-scale and template-rich dataset of math word problems, 2020

    Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. Ape210k: A large-scale and template-rich dataset of math word problems, 2020. URL https://arxiv.org/abs/2009.11506

    2020 arXiv

  60. [68]

    Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on sc...

    2023 arXiv

  61. [69]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  62. [70]

    Agieval: A human-centric benchmark for evaluating foundation models, 2023

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023

    2023

  63. [71]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/2311.07911

    2023 arXiv

  64. [72]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  65. [73]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  66. [74]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  67. [75]

    <ȨW^ ӧO۷kٲeSZUVԯ_?6m M4/ڶm+E7o^<))).DZaÆѴihڴio>Zh 5[ov(iii'nXfMlٲxwwމ?? |۟ҥKcҥ1 裏2+V ;v N8!N<ĨQF VY C ߵkڵkCrhѢXhQ|7fUV- :ٳg QR

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.