Pith. sign in

REVIEW 3 major objections 4 minor 8 cited by

Seed-Coder: Let the Code Model Curate Data for Itself

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An 8B code model family whose pretraining data are filtered by LLMs rather than hand-written rules reaches state-of-the-art results among similar-size open models and beats several larger ones.

desk verdict A serious model-centric data-curation report with strong benchmark results, but its central causal claim rests on one under-specified pretraining ablation that peer review should demand be tightened. read the letter →

arxiv 2506.03524 v2 pith:WRYC567V submitted 2025-06-04 cs.CL cs.SE

ByteDance Seed , Yuyu Zhang , Jing Su , Yifan Sun , Chenguang Xi , Xia Xiao , Shen Zheng , Anxiang Zhang
show 19 more authors
Kaibo Liu Daoguang Zan Tao Sun Jinhua Zhu Shulin Xin Dong Huang Yetao Bai Lixin Dong Chao Li Jianchong Chen Hanzhi Zhou Yifan Huang Guanghan Ning Xierui Song Jiaze Chen Siyao Liu Kai Shen Liang Xiang Yonghui Wu
This is my paper · ORCID
classification cs.CLcs.SE
keywords LLMdatacurationcodepretrainingqualityfilteringSeed-Codermodel-centricpipelinegenerationreinforcementlearningLongCoT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main obstacle to better code models is no longer raw data volume but data selection, and that LLMs can do the selecting. It introduces Seed-Coder, a trio of 8B open models — base, instruct, and reasoning — trained on a 6-trillion-token corpus built by a model-centric pipeline: a 1.3B regression scorer, distilled from an LLM oracle that grades code on readability, modularity, clarity, and reusability, scores every GitHub file and discards the bottom ~10%. The same LLM-scoring logic filters code-related web data, while commit data are formatted as code-change prediction tasks. The authors report that the instruct model tops all ~8B open competitors on MBPP, MHPP, BigCodeBench, LiveCodeBench, FullStack Bench, and SWE-bench Verified, and that the reasoning model matches or exceeds much larger models on LiveCodeBench, IOI 2024, and Codeforces. If correct, this makes 'the model curates its own data' a scalable replacement for hand-crafted rules that are costly to maintain across 89 languages.

What carries the argument

The load-bearing object is the quality scorer: a 1.3B regression model (Llama-2-style backbone plus regression head) trained in one epoch to reproduce the oracle's 0-10 file-quality scores, rescaled to [0,1]. It carries the argument by replacing per-language hand-crafted filter rules with a single learned filter that can be applied to billions of GitHub files; the bottom ~10% of files are discarded, balancing quality and diversity. The same LLM-scoring idea is reused for code-related web data with category-specific quality thresholds, and quality scores double as features for selecting high-quality continued-pretraining data and repository-level long-context data.

What would settle it

Train two models with identical token budgets and recipes, one on the curated corpus and one on a corpus that keeps the discarded bottom ~10% and removes the top ~10% (or shuffles the threshold); if the curated model does not consistently beat the shuffled-threshold control on held-out benchmarks like BigCodeBench and LiveCodeBench, the reported gain is an effect of the oracle's score distribution rather than of quality selection. A cheaper check: evaluate the trained 1.3B scorer's predicted quality against human labels on low-resource languages like Solidity or Verilog, where a systematically wrong signal would show up as near-random ranking.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a compact regression scorer trained on LLM judgments can serve as a general, scalable quality filter for code pretraining. The authors sampled 222,066 code files from 21 languages, asked DeepSeek-V2-Chat to rate each file 0-10 on readability, modularity, clarity, and reusability, and fine-tuned a 1.3B Llama-2-architecture model with a regression head to predict those scores. Applying this scorer to all GitHub data and dropping the bottom ~10% of files (~1T tokens) improved benchmark performance during pretraining relative to minimal rules alone, as shown in Figure 6. Extending the same model-centric principle to web data (with category-specific thresholds), commits, and long-context data yielded a base model that outperforms comparable open base models on HumanEval, MBPP, MultiPL-E, CrossCodeEval, and RepoEval; the instruct and reasoning variants built on it reach state-of-the-art ~8B results and beat several larger models on code generation, editing, reasoning, and software-engineering benchmarks.

Load-bearing premise

The quality filter's usefulness rests on the assumption that the LLM oracle's 0-10 scores — distilled into a 1.3B scorer — correctly identify low-quality code across all 89 languages and every GitHub file type, so that discarding the bottom ~10% reliably improves pretraining rather than injecting the oracle's preferences.

Editorial extensions

If this is right

  • Seed-Coder-8B-Base outperforms comparable open base models on HumanEval/MBPP, MultiPL-E, CrossCodeEval, and RepoEval, and beats several 13B+ models on multilingual code generation.
  • Seed-Coder-8B-Instruct sets the ~8B state of the art on MHPP (36.2%), BigCodeBench full (53.3%) and hard (26.4%), LiveCodeBench 2410–2502 (24.7%), FullStack Bench (55.8%), and SWE-bench Verified under Agentless (19.2%).
  • The reasoning model lifts LiveCodeBench overall pass@1 from 39.0% after warmup to 53.6% after GRPO-based LongCoT reinforcement learning, surpassing DeepSeek-R1-Distill-Qwen-14B and slightly edging Claude-3.7-Sonnet-Thinking.
  • The instruct model resolves 11.2% of SWE-bench Verified instances under the fully autonomous OpenHands workflow, far ahead of other ~8B models, which the authors attribute to commit-data supervision and LLM-filtered formatting consistency.
  • The model-centric pipeline scales to 6T tokens with minimal human involvement, processing GitHub files, 74M commits (~100B tokens), and ~1.2T tokens of code-related web data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same 'distill an oracle, regress, filter' recipe could transfer to other data-hungry domains such as mathematical proofs, scientific code, or documentation, where hand-crafted filtering rules are equally costly to maintain; a direct test would swap DeepSeek-V2-Chat for a stronger or fully open oracle and check whether quality-signal fidelity and downstream gains scale accordingly.
  • Inference: the Figure 6 comparison is the crux, and a sharper control would train on the kept bottom ~10% or on a shuffled threshold; if gains vanish, the benefit is the oracle's score distribution, not quality selection.
  • Inference: the paper's category-specific web thresholds implicitly concede that LLM scorers carry domain biases (forums under-scored, documentation over-scored); making those thresholds adaptive or learned could remove the last human knob in the pipeline.
  • Inference: with more than 10% of reasoning samples truncated at 64K tokens, the reported LiveCodeBench and Codeforces numbers likely understate the 8B reasoning model's ceiling; extending the context window or adding longer rollout might close more of the gap to o3-mini and IOI bronze.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Seed-Coder, an 8B-parameter family of code models (base, instruct, reasoning) trained on a 6T-token corpus whose GitHub, commit, and web data are filtered by an LLM-based quality scorer instead of extensive hand-crafted per-language rules. The instruct model is built with synthetic SFT data, sandbox verification, and DPO; the reasoning model is obtained through a LongCoT warmup followed by GRPO reinforcement learning. The manuscript reports large benchmark tables across code generation, completion, reasoning, editing, and software engineering tasks, and claims that the pipeline demonstrates that code LLMs can curate training data for themselves with minimal human effort.

Significance. If the central claim is sustained, the contribution is practically significant: it provides an open-source 8B code model family with competitive or state-of-the-art scores on many benchmarks, and a data pipeline that reduces the burden of hand-written filtering rules. The paper's strengths include a broad evaluation across diverse tasks, a decontamination procedure, and the release of models. However, the core scientific claim about LLM-driven pretraining data curation rests almost entirely on one pretraining ablation (Figure 6) whose setup is under-specified and whose generalization beyond Python is not measured. That gap is load-bearing for the paper's headline, so I cannot recommend acceptance without revision.

major comments (3)
  1. [§2.2.1 / Figure 6 / Appendices A.1, A.2.2] The only direct evidence that LLM-based quality filtering improves pretraining is Figure 6, which reports MBPP, CRUXEval, and HumanEval curves for Rule vs Rule+LLM. The quality scorer is trained on 21 languages (Table 17) yet applied to 89 languages (Appendix A.1), and every benchmark shown in Figure 6 is Python. Since the paper's central claim is that the filter generalizes across 89 languages and diverse file types, this is a load-bearing gap: I would need per-language results on non-Python benchmarks (e.g., MultiPL-E) or an analysis of scorer error by language and file type to separate a general data-curation effect from an artifact of the oracle's training distribution.
  2. [§2.2.1 / Figure 6] The two pretraining arms are not matched in a way that supports the causal attribution. The Rule+LLM arm removes the bottom ~10% of files, aggregating to ~1T unique tokens, so the arms differ in token count, data composition, and presumably compute budget. Without specifying the token budget per arm, reporting multiple seeds, or giving any variance estimate, the gap in Figure 6 could be due to data quantity or run variance rather than to the quality signal itself. The authors should state whether the two arms were trained on matched token budgets and should report repeated runs or confidence intervals.
  3. [Appendix A.2.3 / §2.2.1] The scorer's validation is only against oracle scores (overall MAE 0.91, category-wise MAE 1.37), not against downstream pretraining utility. Agreement among GPT-4-Turbo, DeepSeek-Coder-33B, and DeepSeek-V2-Chat may reflect shared model biases rather than correct filtering; the manuscript itself notes only minor discrepancies and does not quantify the accuracy of the chosen oracle. Since the zero-score policy is central to the filtering decision and the scorer is acknowledged to struggle with zero-condition cases, the reported MAE does not establish that discarding the bottom 10% improves pretraining across all 89 languages. A downstream utility check on a held-out language set is needed.
minor comments (4)
  1. [§5.1.3] In the base model evaluation section, the text says that 'Seed-Coder-8B-Instruct achieves top-tier performance'; this should refer to Seed-Coder-8B-Base, since the section evaluates base models.
  2. [§3.1.5] There is a typo in 'learning rate of2e−5'; a space is missing between 'of' and '2e−5'.
  3. [Figure 6] The horizontal axis label 'Trained T okens' contains a typo, and the legend appears only in the first subplot; a shared legend or explicit line labels in each panel would improve readability.
  4. [§5.3 / Table 16] The naming is inconsistent between 'DeepSeek-R1-Distill-Seed-Coder-8B' and 'Seed-Coder-8B-Reasoning'; please clarify that the former is the warmup checkpoint and the latter is the final RL-trained model.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the curation filter is an external LLM scorer and the central pretraining claim rests on a direct ablation, not on fitted parameters.

full rationale

The paper's causal chain is: collect GitHub and web data; train a 1.3B quality scorer on 222,066 files labeled by DeepSeek-V2-Chat, an external model; filter out the bottom ~10% of files; pretrain Seed-Coder-8B on the filtered corpus; and then measure external benchmarks. Every step is externally grounded: the filter is not a parameter of the final Seed-Coder model, no benchmark score is used as a training label or filter threshold, and the post-training warmup uses external DeepSeek-R1 generations. Figure 6 directly compares two pretraining arms and is therefore an empirical ablation, not a prediction forced by construction. The scorer's MAE (0.91 overall, 1.37 category-wise) validates approximation of the oracle, and the paper does not present that MAE as evidence of downstream improvement. The FullStack Bench and Multi-SWE-bench evaluations involve overlapping author teams, and the title's 'curate data for itself' overstates the role of Seed-Coder relative to the external scorer; these are independence and accuracy caveats, not circular derivations.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the validity of LLM-generated quality labels, the generalization of a small regression scorer, and the sufficiency of 10-gram decontamination. The pipeline also contains several hand-chosen thresholds and dataset sizes that are not fitted but affect the outcome.

free parameters (5)
  • GitHub quality filter cutoff = bottom ~10% of files
    Section 2.2.1: 'we filtered out from the entire GitHub dataset the bottom ~10% files'. Hand-chosen threshold; central to the quality/quantity tradeoff.
  • Web category-specific thresholds and sampling rates = not specified
    Section 2.2.3: 'we implemented category-specific filtering protocols that adjusted quality thresholds and sampling rates for each category'. Values are not reported, making the web pipeline hard to reproduce.
  • SFT difficulty cutoff = 3 out of 10
    Section 3.1.2: 'Instances receiving a difficulty score lower than 3 out of 10 were discarded from the dataset.'
  • RL curriculum correctness-rate cutoff = 87.5%
    Section 3.2.3: 'we further filtered simple problems, those with correctness rates above 87.5%'.
  • DPO preference pair count = approximately 20,000
    Section 3.1.5: 'we constructed approximately 20,000 high-quality preference pairs'. This is a scale choice, not fitted, but affects the result.
assumptions (4)
  • domain assumption The DeepSeek-V2-Chat oracle's 0-10 quality scores reflect true pretraining usefulness of code files.
    Section 2.2.1 and Appendix A.2.1: the scoring prompt defines readability, modularity, clarity, reusability; the paper assumes these criteria capture data quality for pretraining.
  • domain assumption The 1.3B regression scorer trained on 222,066 samples generalizes to the full GitHub corpus across 89 languages.
    Section 2.2.1 and Figure 13: MAE is reported on 1,178 test samples, but no distribution-shift analysis over the full corpus.
  • domain assumption 10-gram overlap decontamination is sufficient to prevent test data leakage.
    Section 4: the method is adopted from Guo et al. and Hui et al., but no leakage verification is reported.
  • domain assumption Using standard pretraining architecture (Llama 3 style) and 6T tokens is a valid baseline for comparing data pipelines.
    Section 2.3: architecture and training policy are stated; the paper does not ablate architecture or token budget against prior models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Seed-Coder: Let the Code Model Curate Data for Itself." pith.science (2026). https://pith.science/paper/WRYC567V

@misc{pith2026250603524,
  author       = {Pith},
  title        = {Pith review of: Seed-Coder: Let the Code Model Curate Data for Itself},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WRYC567V}},
  note         = {Machine review of arXiv:2506.03524}
}
read the original abstract

Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks.

Figures

Figures reproduced from arXiv: 2506.03524 by the authors.

Figure 1
Figure 1. Benchmark performance of instruct and reasoning variants of Seed-Coder-8B. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Processing pipeline for pretraining data. We collected data from GitHub and web archives. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Sample Python script with decent structure but logical errors. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Sample code snippet from a Python script for LED display controlling. The original [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Pipeline of our data quality scorer. 0 100 200 300 400 Trained Tokens (Billions) 40 45 50 55 60 65 Score (%) MBPP Rule+LLM Rule 0 100 200 300 400 Trained Tokens (Billions) 30 33 36 39 42 45 CRUXEval 0 100 200 300 400 Trained Tokens (Billions) 20 25 30 35 40 45 HumanEva…
Figure 6
Figure 6. Figure 6: On-the-fly performance over benchmarks during pretraining, from data with minimal [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Pipeline of extracting high-quality code-related data from web archives. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Pipeline of synthetic data curation for our instruct model training. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: RL training trajectories: smoothed average response token length (left), smoothed average [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Evaluation results of the “Needle in the Code” pressure test. [PITH_FULL_IMAGE:figures/full_fig_p023_10.png]
Figure 11
Figure 11. Figure 11: Performance of reasoning models on IOI’2024 (left) and Codeforces (right). [PITH_FULL_IMAGE:figures/full_fig_p033_11.png]
Figure 12
Figure 12. Figure 12: Comparison of quality scores from three oracles. Score distributions from individual [PITH_FULL_IMAGE:figures/full_fig_p045_12.png]
Figure 13
Figure 13. Figure 13: Boxplot of predicted scores from the quality filter corresponding to each individual [PITH_FULL_IMAGE:figures/full_fig_p045_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Uncovering Business Logic Bugs via Semantics-Driven Unit Test Generation

    cs.SE 2026-04 conditional novelty 7.0 of 10

    SeGa extracts business semantics from requirements to generate unit tests that detect 22-25 more real-world business logic bugs than prior LLM-based methods in industrial Go projects.

  2. DICE: Diffusion Large Language Models Excel at Generating CUDA Kernels

    cs.LG 2026-02 conditional novelty 7.0 of 10

    DICE, a family of diffusion LLMs trained with a bi-phase infilling-to-generation RL curriculum, achieves state-of-the-art CUDA kernel generation at data and token budgets far below prior specialized autoregressive models.

  3. BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

    cs.CL 2026-03 accept novelty 6.0 of 10

    A 500-task benchmark shows AI code agents plateau around 56% accuracy on broader software tasks, with search access providing limited and inconsistent gains.

  4. Rethinking Scientific Modeling: Toward Physically Consistent and Simulation-Executable Programmatic Generation

    cs.SE 2026-02 conditional novelty 6.0 of 10

    A physics-aware two-stage SFT+GRPO training method with period/AST/sandbox rewards raises small open LLMs from ~0-2% to ~68-77% on a strict OpenSeesPy building-modeling benchmark (BMEval).

  5. AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators

    cs.CL 2025-08 conditional novelty 6.0 of 10

    AutoCodeBench is an LLM-generated, sandbox-verified code benchmark with 3,920 problems across 20 languages, where top models reach only 52.4% pass@1.

  6. AutoTriton: Automatic Triton Programming with Reinforcement Learning in LLMs

    cs.LG 2025-07 conditional novelty 6.0 of 10

    An 8B model trained with SFT and GRPO reinforcement learning generates Triton kernels that match or exceed frontier model performance on TritonBench and KernelBench correctness metrics.

  7. GIFT: Games as Informal Training for Generalizable LLMs

    cs.CL 2026-01 conditional novelty 5.0 of 10

    Game-based RL with formal math improves average general-benchmark scores in several settings, but the proposed nested training objective is mathematically the same average-reward objective as mixed training and in-dom...

  8. TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance

    cs.IR 2025-10 conditional novelty 4.0 of 10

    TaoSR-AGRL improves e-commerce search relevance by combining dense rule-aware reward shaping with adaptive ground-truth-guided replay in GRPO, reporting gains on Taobao's private offline and online evaluations.

Reference graph

Works this paper leans on

66 extracted references · 27 canonical work pages · cited by 8 Pith papers

  1. [1]

    Meet Yi-Coder : A small but mighty LLM for code, September 2024

    01.AI. Meet Yi-Coder : A small but mighty LLM for code, September 2024. URL https://github.com/01-ai/Yi-Coder

    work page 2024

  2. [2]

    L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. SantaCoder : don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. URL https://arxiv.org/abs/2301.03988

    arXiv 2023

  3. [3]

    Claude 3.7 sonnet, February 2025

    Anthropic . Claude 3.7 sonnet, February 2025. URL https://www.anthropic.com/claude

    work page 2025

  4. [4]

    Athiwaratkun, S

    B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang. Multi-lingual evaluation of code generation models. In The Eleventh Intern...

    work page 2023

  5. [5]

    Austin, A

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.07732, 2021. URL https://arxiv.org/abs/2108.07732

    arXiv 2021

  6. [7]

    Bavarian, H

    M. Bavarian, H. Jun, N. Tezak, J. Schulman, C. McLeavey, J. Tworek, and M. Chen. Efficient training of language models to fill in the middle. arXiv preprint arXiv:2207.14255, 2022 b

    arXiv 2022

  7. [8]

    Brunsfeld

    M. Brunsfeld. Tree-sitter: An incremental parsing system for programming tools, 2018. URL https://tree-sitter.github.io/tree-sitter/

    work page 2018

  8. [9]

    Seed1.5-Thinking : Advancing superb reasoning models with reinforcement learning, 2025

    ByteDance Seed . Seed1.5-Thinking : Advancing superb reasoning models with reinforcement learning, 2025. URL https://arxiv.org/abs/2504.13914

    arXiv 2025

Show all 66 references
  1. [10]

    Cassano, J

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, A. Guha, M. Greenberg, and A. Jangda. MultiPL-E : A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software...

    2023

  2. [11]

    Cassano, L

    F. Cassano, L. Li, A. Sethi, N. Shinn, A. Brennan-Jones, J. Ginesin, E. Berman, G. Chakhnashvili, A. Lozhkov, C. J. Anderson, and A. Guha. Can it edit? evaluating the ability of large language models to follow code editing instructions. In First Conference on Language Modeling...

    2024

  3. [12]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, ...

    2021 arXiv

  4. [13]

    J. Dai, J. Lu, Y. Feng, D. Huang, G. Zeng, R. Ruan, M. Cheng, H. Tan, and Z. Guo. MHPP : Exploring the capabilities and limitations of language models beyond basic code generation, 2024. URL https://arxiv.org/abs/2405.11430

    2024 arXiv

  5. [14]

    DeepSeek-AI, A. Liu, B. Feng, B. Wang, B. Wang, B. Liu, and C. Z. et al. DeepSeek-V2 : A strong, economical, and efficient mixture-of-experts language model, 2024 a . URL https://arxiv.org/abs/2405.04434

    2024 arXiv

  6. [15]

    DeepSeek-AI, Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, W. Zeng, X. Bi, Z. Gu, H. Xu, D. Dai, K. Dong, L. Zhang, Y. Piao, Z. Gou, Z. Xie, Z. Hao, B. Wang, J. Song, D. Chen, X. Xie, K. Guan, Y. You, A. Liu, Q. Du, W. Gao, X. Lu, Q. Chen, Y. W...

    2024 arXiv

  7. [16]

    DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, and X. Z. et al. DeepSeek-R1 : Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  8. [17]

    Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. CrossCodeEval : A diverse and multilingual benchmark for cross-file code completion. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levin...

    2023

  9. [18]

    Fried, A

    D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. InCoder : A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/f...

    2023

  10. [19]

    A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang. CRUXEval : A benchmark for code reasoning, understanding and execution, 2024. URL https://arxiv.org/abs/2401.03065

    2024 arXiv

  11. [20]

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. DeepSeek-Coder : When the large language model meets programming -- the rise of code intelligence, 2024. URL https://arxiv.org/abs/2401.14196

    2024 arXiv

  12. [21]

    J. Guo, Z. Li, X. Liu, K. Ma, T. Zheng, Z. Yu, D. Pan, Y. LI, R. Liu, Y. Wang, S. Guo, X. Qu, X. Yue, G. Zhang, W. Chen, and J. Fu. CodeEditorBench : Evaluating code editing capability of LLM s. In ICLR 2025 Third Workshop on Deep Learning for Code, 2025. URL https://openrevie...

    2025

  13. [22]

    Hsieh, Y.-S

    C.-Y. Hsieh, Y.-S. Chuang, C.-L. Li, Z. Wang, L. T. Le, A. Kumar, J. Glass, A. Ratner, C.-Y. Lee, R. Krishna, and T. Pfister. Found in the middle: Calibrating positional attention bias improves long context utilization, 2024. URL https://arxiv.org/abs/2406.16008

    2024 arXiv

  14. [23]

    Huang, T

    S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. Liu, C. Zhang, L. Chai, R. Yuan, Z. Zhang, J. Fu, Q. Liu, G. Zhang, Z. Wang, Y. Qi, Y. Xu, and W. Chu. OpenCoder : The open cookbook for top-tier code large language models, 2025. URL https://arxiv.org/abs/2411.04905

    2025 arXiv

  15. [24]

    Open R1 : A fully open reproduction of deepseek-r1, January 2025

    Hugging Face . Open R1 : A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/huggingface/open-r1

    2025

  16. [25]

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, K. Dang, Y. Fan, Y. Zhang, A. Yang, R. Men, F. Huang, B. Zheng, Y. Miao, S. Quan, Y. Feng, X. Ren, X. Ren, J. Zhou, and J. Lin. Qwen2.5-Coder technical report, 2024. URL https://arxiv.org/abs/2...

    2024 arXiv

  17. [26]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025. URL h...

    2025

  18. [27]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE -bench: Can language models resolve real-world GitHub issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=VTF8yNQM66

    2024

  19. [28]

    Joulin, E

    A. Joulin, E. Grave, P. Bojanowski, M. Douze, H. Jégou, and T. Mikolov. FastText .zip: Compressing text classification models, 2016. URL https://arxiv.org/abs/1612.03651

    2016 arXiv

  20. [29]

    Joulin, E

    A. Joulin, E. Grave, P. Bojanowski, and T. Mikolov. Bag of tricks for efficient text classification. In M. Lapata, P. Blunsom, and A. Koller, editors, Proceedings of the 15th Conference of the E uropean Chapter of the Association for Computational Linguistics: Volume 2, Short ...

    2017

  21. [30]

    Kiveris, S

    R. Kiveris, S. Lattanzi, V. Mirrokni, V. Rastogi, and S. Vassilvitskii. Connected components in mapreduce and beyond. In Proceedings of the ACM Symposium on Cloud Computing, SOCC '14, page 1–13, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450332521. ...

    2014

  22. [31]

    K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational L...

    2022 doi

  23. [32]

    R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. StarCoder : may the source be with you! arXiv preprint arXiv:2305.06161, 2023. URL https://arxiv.org/abs/2305.06161

    2023 arXiv

  24. [33]

    Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with AlphaCode . Science, 378 0 (6624): 0 1092--1097, 2022. URL https://www.science.org/doi/10.1126/science.abq1158

    2022 doi

  25. [34]

    J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Process...

    2023

  26. [35]

    R. Liu, J. Wei, F. Liu, C. Si, Y. Zhang, J. Rao, S. Zheng, D. Peng, D. Yang, D. Zhou, and A. M. Dai. Best practices and lessons learned on synthetic data, 2024 a . URL https://arxiv.org/abs/2404.07503

    2024 arXiv

  27. [36]

    S. Liu, H. Zhu, J. Liu, S. Xin, A. Li, R. Long, L. Chen, J. Yang, J. Xia, Z. Y. Peng, S. Liu, Z. Zhang, G. Zhang, W. Huang, K. Shen, and L. Xiang. Fullstack bench: Evaluating llms as full stack coders, 2024 b . URL https://arxiv.org/abs/2412.00535

    2024 arXiv

  28. [37]

    Introducing meta llama 3: The most capable openly available llm to date

    Llama Team . Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, April 2024 a

    2024

  29. [38]

    Introducing llama 3.1: Our most capable models to date

    Llama Team . Introducing llama 3.1: Our most capable models to date. https://ai.meta.com/blog/meta-llama-3-1/, July 2024 b

    2024

  30. [39]

    Lozhkov, R

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y. Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y. Zhuo, E. Zheltonozhskii, N....

    2024 arXiv

  31. [40]

    K. Lu, H. Yuan, Z. Yuan, R. Lin, J. Lin, C. Tan, C. Zhou, and J. Zhou. InsTag : Instruction tagging for analyzing supervised fine-tuning of large language models, 2023. URL https://arxiv.org/abs/2308.07074

    2023 arXiv

  32. [41]

    M. Luo, S. Tan, R. Huang, X. Shi, R. Xin, C. Cai, A. Patel, A. Ariyak, Q. Wu, C. Zhang, L. E. Li, R. A. Popa, and I. Stoica. DeepCoder : A fully open-source 14B coder at o3-mini level. https://pretty-radio-b75.notion.site/DeepCoder-A-Fully-Open-Source-14B-Coder-at-O3-mini-Leve...

    2025

  33. [42]

    Codestral

    Mistral AI . Codestral. https://mistral.ai/news/codestral/, May 2024

    2024

  34. [43]

    OpenAI o3, April 2025

    OpenAI . OpenAI o3, April 2025. URL https://openai.com/index/introducing-o3-and-o4-mini/

    2025

  35. [44]

    Penedo, A

    G. Penedo, A. Lozhkov, H. Kydlíček, L. Ben Allal, E. Beeching, A. Piqueres Lajarín, Q. Gallouédec, N. Habib, L. Tunstall, and L. von Werra. CodeForces-CoTs . https://huggingface.co/datasets/open-r1/codeforces-cots, 2025. Accessed: 2025-04-26

    2025

  36. [45]

    S. Quan, J. Yang, B. Yu, B. Zheng, D. Liu, A. Yang, X. Ren, B. Gao, Y. Miao, Y. Feng, Z. Wang, J. Yang, Z. Cui, Y. Fan, Y. Zhang, B. Hui, and J. Lin. CodeElo : Benchmarking competition-level code generation of llms with human-comparable elo ratings, 2025. URL https://arxiv.org...

    2025 arXiv

  37. [46]

    Code with CodeQwen1.5 , April 2024 a

    Qwen Team . Code with CodeQwen1.5 , April 2024 a . URL https://qwenlm.github.io/blog/codeqwen1.5/

    2024

  38. [47]

    Introducing Qwen1.5 , February 2024 b

    Qwen Team . Introducing Qwen1.5 , February 2024 b . URL https://qwenlm.github.io/blog/qwen1.5/

    2024

  39. [48]

    Qwen2.5: A party of foundation models, September 2024 c

    Qwen Team . Qwen2.5: A party of foundation models, September 2024 c . URL https://qwenlm.github.io/blog/qwen2.5/

    2024

  40. [49]

    Qwen3, April 2025

    Qwen Team . Qwen3, April 2025. URL https://qwenlm.github.io/blog/qwen3/

    2025

  41. [50]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=HPuSIXJaa9

    2023

  42. [51]

    Robertson and H

    S. Robertson and H. Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3 0 (4): 0 333–389, Apr. 2009. ISSN 1554-0669. doi:10.1561/1500000019. URL https://doi.org/10.1561/1500000019

    2009 doi

  43. [52]

    Rozière, J

    B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. ...

    2024 arXiv

  44. [53]

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300

    2024 arXiv

  45. [54]

    Sheng, C

    G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu. HybridFlow : A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297. ACM, Mar. 2025. doi:10.1145/3689031.369...

    2025

  46. [55]

    R. Sutton. The bitter lesson, 2019. URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html

    2019

  47. [56]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hoss...

    2023 arXiv

  48. [57]

    X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, H. H. Tran, F. Li, R. Ma, M. Zheng, B. Qian, Y. Shao, N. Muennighoff, Y. Zhang, B. Hui, J. Lin, R. Brennan, H. Peng, H. Ji, and G. Neubig. OpenHands : An open platform for AI software devel...

    2025

  49. [58]

    Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang. Magicoder: Empowering code generation with oss-instruct, 2024. URL https://arxiv.org/abs/2312.02120

    2024 arXiv

  50. [59]

    C. S. Xia, Y. Deng, S. Dunn, and L. Zhang. Agentless: Demystifying LLM -based software engineering agents, 2024. URL https://arxiv.org/abs/2407.01489

    2024 arXiv

  51. [60]

    Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W.-Y. Ma, Y.-Q. Zhang, L. Yan, ...

    2025 arXiv

  52. [61]

    Z. Yu, Z. Wang, Y. Fu, H. Shi, K. Shaikh, and Y. C. Lin. Unveiling and harnessing hidden attention sinks: enhancing large language models without training through attention calibration. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org,...

    2024

  53. [62]

    Y. Yue, Z. Chen, R. Lu, A. Zhao, Z. Wang, Y. Yue, S. Song, and G. Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837

    2025 arXiv

  54. [63]

    D. Zan, Z. Huang, W. Liu, H. Chen, L. Zhang, S. Xin, L. Chen, Q. Liu, X. Zhong, A. Li, S. Liu, Y. Xiao, L. Chen, Y. Zhang, J. Su, T. Liu, R. Long, K. Shen, and L. Xiang. Multi-SWE-bench : A multilingual benchmark for issue resolving, 2025. URL https://arxiv.org/abs/2504.02605

    2025 arXiv

  55. [64]

    Zhang, B

    F. Zhang, B. Chen, Y. Zhang, J. Liu, D. Zan, Y. Mao, J.-G. Lou, and W. Chen. RepoCoder : Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303.12570, 2023. URL https://arxiv.org/abs/2303.12570

    2023 arXiv

  56. [65]

    Zhang, H

    S. Zhang, H. Zhao, X. Liu, Q. Zheng, Z. Qi, X. Gu, X. Zhang, Y. Dong, and J. Tang. NaturalCodeBench : Examining coding performance mismatch on humaneval and natural user prompts, 2024. URL https://arxiv.org/abs/2405.04520

    2024 arXiv

  57. [66]

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng. WildChat : 1m chatgpt interaction logs in the wild, 2024. URL https://arxiv.org/abs/2405.01470

    2024 arXiv

  58. [67]

    T. Y. Zhuo, V. M. Chien, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. GONG, J. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, D. Lo, B. Hui, N. Mu...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.