{"id":"fe33d699-cfaf-4005-bb19-64c549cb66b8","arxiv_id":"2506.03524","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Seed-Coder shows that an LLM-trained quality scorer can filter 6T tokens of code data and yield 8B models that outperform similar-size open code models.","lead":"ByteDance researchers trained an 8-billion-parameter code model family using a data pipeline where AI models, not hand-written rules, score and filter the training data. The resulting models beat other open models of similar size on coding benchmarks, suggesting LLM-based data curation can reduce human effort in building code datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 6 is the only direct evidence for the data-curation claim, but the comparison is under-specified (no seeds, token-budget match, or per-language results) and the scorer is trained on 21 of 89 languages; the reported gains may reflect oracle bias or run setup rather than general LLM curation.","rationale":"The reader's weakest_assumption is close to my concern: the quality scorer's generalization across languages and GitHub is the load-bearing premise. I agree that oracle bias is a real risk. However, I would sharpen the concern in two ways. The more immediate problem is experimental identification: Figure 6 does not show that the two conditions are matched in token budget and data volume, and it has no error bars or seeds, so the reported advantage may be a confound. Second, validation of the scorer against oracle scores in Appendix A.2.3 does not establish that the oracle's notion of quality improves pretraining, especially for the 68 languages not in the scorer's training set. The central claim therefore rests on a single, under-specified ablation. I still find the paper's overall contribution plausible: the models are released, the benchmark evaluation is extensive, and the decontamination protocol is standard. The concern does not move me from the reader's CONDITIONAL verdict; it reinforces the need for additional evidence before the data-curation claim can be accepted as general.","tokens_in":42074,"tokens_out":5856,"duration_ms":68737,"concrete_test":"Run a matched-token pretraining comparison at 8B scale (or a 1B proxy using the same pipeline) with three GitHub arms: (a) minimal-rule data only, (b) top-90% by the released Seed-Coder quality scorer, and (c) top-90% by a scorer trained on the original 21 languages plus a held-out long-tail language set. Evaluate on MultiPL-E languages outside the original 21 and report per-language deltas with 2-3 seeds. If arm (b) does not robustly beat arm (a), or if arm (c) outperforms arm (b) on long-tail languages, then Figure 6's improvement is not evidence for a general LLM-curation effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LLMs can effectively curate pretraining data with minimal human effort. In the pretraining section, the direct causal evidence for this claim is Figure 6 (Section 2.2.1), which compares minimal-rule filtering against rule-plus-LLM filtering during pretraining. The load-bearing condition is that the LLM quality scorer's signal improves pretraining broadly across languages and data types. That condition is least secure for three concrete reasons. First, the scorer is trained on only 21 languages (Appendix A.2.2, Table 17) but applied to 89 languages (Appendix A.1); the Figure 6 benchmarks are Python-centric (MBPP, HumanEval, CRUXEval), so generalization to long-tail languages is asserted rather than measured. Second, the ablation is under-specified: no exact scores, no seeds or error bars, and no statement that the Rule and Rule+LLM arms were trained on matched token budgets and comparable total data volumes. Because the LLM-filtered arm removes the bottom ~10% of files, the two arms differ in composition and likely in token count, so the purple curve's advantage could reflect data quantity or run variance rather than quality selection. Third, Appendix A.2.3 validates the scorer only against oracle scores (MAE = 0.91 overall, 1.37 category-wise), not against downstream pretraining utility; agreement among GPT-4, DeepSeek-Coder, and DeepSeek-V2-Chat may indicate shared model bias, not correct filtering. Together, these gaps mean the central data-curation effect shown in Figure 6 could be an artifact of the oracle's biases or the comparison setup rather than a general result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Seed-Coder, an 8B-parameter family of code models (base, instruct, reasoning) trained on a 6T-token corpus whose GitHub, commit, and web data are filtered by an LLM-based quality scorer instead of extensive hand-crafted per-language rules. The instruct model is built with synthetic SFT data, sandbox verification, and DPO; the reasoning model is obtained through a LongCoT warmup followed by GRPO reinforcement learning. The manuscript reports large benchmark tables across code generation, completion, reasoning, editing, and software engineering tasks, and claims that the pipeline demonstrates that code LLMs can curate training data for themselves with minimal human effort.","tokens_in":42407,"tokens_out":4895,"duration_ms":50117,"significance":"If the central claim is sustained, the contribution is practically significant: it provides an open-source 8B code model family with competitive or state-of-the-art scores on many benchmarks, and a data pipeline that reduces the burden of hand-written filtering rules. The paper's strengths include a broad evaluation across diverse tasks, a decontamination procedure, and the release of models. However, the core scientific claim about LLM-driven pretraining data curation rests almost entirely on one pretraining ablation (Figure 6) whose setup is under-specified and whose generalization beyond Python is not measured. That gap is load-bearing for the paper's headline, so I cannot recommend acceptance without revision.","major_comments":[{"comment":"The only direct evidence that LLM-based quality filtering improves pretraining is Figure 6, which reports MBPP, CRUXEval, and HumanEval curves for Rule vs Rule+LLM. The quality scorer is trained on 21 languages (Table 17) yet applied to 89 languages (Appendix A.1), and every benchmark shown in Figure 6 is Python. Since the paper's central claim is that the filter generalizes across 89 languages and diverse file types, this is a load-bearing gap: I would need per-language results on non-Python benchmarks (e.g., MultiPL-E) or an analysis of scorer error by language and file type to separate a general data-curation effect from an artifact of the oracle's training distribution.","section":"§2.2.1 / Figure 6 / Appendices A.1, A.2.2"},{"comment":"The two pretraining arms are not matched in a way that supports the causal attribution. The Rule+LLM arm removes the bottom ~10% of files, aggregating to ~1T unique tokens, so the arms differ in token count, data composition, and presumably compute budget. Without specifying the token budget per arm, reporting multiple seeds, or giving any variance estimate, the gap in Figure 6 could be due to data quantity or run variance rather than to the quality signal itself. The authors should state whether the two arms were trained on matched token budgets and should report repeated runs or confidence intervals.","section":"§2.2.1 / Figure 6"},{"comment":"The scorer's validation is only against oracle scores (overall MAE 0.91, category-wise MAE 1.37), not against downstream pretraining utility. Agreement among GPT-4-Turbo, DeepSeek-Coder-33B, and DeepSeek-V2-Chat may reflect shared model biases rather than correct filtering; the manuscript itself notes only minor discrepancies and does not quantify the accuracy of the chosen oracle. Since the zero-score policy is central to the filtering decision and the scorer is acknowledged to struggle with zero-condition cases, the reported MAE does not establish that discarding the bottom 10% improves pretraining across all 89 languages. A downstream utility check on a held-out language set is needed.","section":"Appendix A.2.3 / §2.2.1"}],"minor_comments":[{"comment":"In the base model evaluation section, the text says that 'Seed-Coder-8B-Instruct achieves top-tier performance'; this should refer to Seed-Coder-8B-Base, since the section evaluates base models.","section":"§5.1.3"},{"comment":"There is a typo in 'learning rate of2e−5'; a space is missing between 'of' and '2e−5'.","section":"§3.1.5"},{"comment":"The horizontal axis label 'Trained T okens' contains a typo, and the legend appears only in the first subplot; a shared legend or explicit line labels in each panel would improve readability.","section":"Figure 6"},{"comment":"The naming is inconsistent between 'DeepSeek-R1-Distill-Seed-Coder-8B' and 'Seed-Coder-8B-Reasoning'; please clarify that the former is the warmup checkpoint and the latter is the final RL-trained model.","section":"§5.3 / Table 16"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test concern: the paper's headline claim about LLM-curated pretraining data is supported by a single under-specified ablation, and the missing controls are essential rather than cosmetic. The benchmark evidence is extensive and likely valuable to the community, but the manuscript should not be accepted until the Figure 6 comparison is properly matched and the cross-language generalization of the scorer is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about Seed-Coder before deciding how much weight to give it. First, it is a genuinely integrated, engineering-heavy contribution: a 6T-token code pretraining pipeline where a 1.3B regression scorer, trained on LLM oracle labels, filters GitHub files, commits, and web data, followed by an SFT/DPO instruct stage and a GRPO LongCoT reasoning stage. That full pipeline, with the resulting 8B model family, is not in the prior literature. Second, the benchmark tables are broad and consistently strong across generation, completion, reasoning, editing, and SWE tasks, which makes the model itself credible.\n\nWhat the paper does well: it gives concrete details on the scorer construction, the oracle comparison, the category-specific web filtering, the FIM choice, and the RL curriculum. The decontamination step is standard but real. The acknowledgment of the BigCodeBench evaluation-setting correction is honest. The central circularity worry largely dissolves: the quality filter is trained on DeepSeek-V2-Chat scores before Seed-Coder is trained, and the reasoning warmup uses DeepSeek-R1 data, both external. The 'for itself' framing is a bit generous, but the result is not derived from a fitted parameter of the final model.\n\nThe soft spots are real but concentrated. Figure 6 is the only direct evidence that the LLM-based filter improves pretraining, and it is under-specified: no seeds, no error bars, no statement that the Rule and Rule+LLM arms were trained on matched token budgets. Because the LLM-filtered arm removes the bottom ~10% of files, the two arms differ in composition and likely token count; the purple curve's advantage could partly be a data-quantity or run-variance effect. Likewise, the scorer is trained on 21 languages but applied to 89, and the Figure 6 benchmarks are Python-centric, so generalization to long-tail languages is asserted rather than measured. The scorer validation is only against oracle scores (MAE 0.91), not against downstream pretraining utility. These are fixable with more reporting, not signs of a broken method. The absence of error bars across all benchmark tables is a minor but pervasive annoyance.\n\nWho this is for: anyone building code pretraining data pipelines or comparing open 8B code models. It deserves a serious referee. My recommendation: send it to peer review, and instruct the referee to focus on Figure 6 and the scorer-language generalization gap. With a matched-token-budget ablation and per-language results, the central claim would be solid.","headline":"A serious model-centric data-curation report with strong benchmark results, but its central causal claim rests on one under-specified pretraining ablation that peer review should demand be tightened.","tokens_in":43108,"tokens_out":1087,"would_cite":true,"duration_ms":14687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An 8B code model family whose pretraining data are filtered by LLMs rather than hand-written rules reaches state-of-the-art results among similar-size open models and beats several larger ones.","keywords":["LLM data curation","code pretraining","quality filtering","Seed-Coder","model-centric data pipeline","code generation","reinforcement learning","LongCoT"],"falsifier":"Train two models with identical token budgets and recipes, one on the curated corpus and one on a corpus that keeps the discarded bottom ~10% and removes the top ~10% (or shuffles the threshold); if the curated model does not consistently beat the shuffled-threshold control on held-out benchmarks like BigCodeBench and LiveCodeBench, the reported gain is an effect of the oracle's score distribution rather than of quality selection. A cheaper check: evaluate the trained 1.3B scorer's predicted quality against human labels on low-resource languages like Solidity or Verilog, where a systematically wrong signal would show up as near-random ranking.","tokens_in":41863,"feed_emoji":"💻","tokens_out":8940,"duration_ms":78802,"temperature":0.7,"pith_summary":"This paper claims that the main obstacle to better code models is no longer raw data volume but data selection, and that LLMs can do the selecting. It introduces Seed-Coder, a trio of 8B open models — base, instruct, and reasoning — trained on a 6-trillion-token corpus built by a model-centric pipeline: a 1.3B regression scorer, distilled from an LLM oracle that grades code on readability, modularity, clarity, and reusability, scores every GitHub file and discards the bottom ~10%. The same LLM-scoring logic filters code-related web data, while commit data are formatted as code-change prediction tasks. The authors report that the instruct model tops all ~8B open competitors on MBPP, MHPP, BigCodeBench, LiveCodeBench, FullStack Bench, and SWE-bench Verified, and that the reasoning model matches or exceeds much larger models on LiveCodeBench, IOI 2024, and Codeforces. If correct, this makes 'the model curates its own data' a scalable replacement for hand-crafted rules that are costly to maintain across 89 languages.","feed_headline":"Self-curated data lift 8B code model past larger open rivals","feed_subtitle":"LLM quality scores replace hand-written filter rules to give 8B Seed-Coder top results on seven benchmark families.","key_machinery":"The load-bearing object is the quality scorer: a 1.3B regression model (Llama-2-style backbone plus regression head) trained in one epoch to reproduce the oracle's 0-10 file-quality scores, rescaled to [0,1]. It carries the argument by replacing per-language hand-crafted filter rules with a single learned filter that can be applied to billions of GitHub files; the bottom ~10% of files are discarded, balancing quality and diversity. The same LLM-scoring idea is reused for code-related web data with category-specific quality thresholds, and quality scores double as features for selecting high-quality continued-pretraining data and repository-level long-context data.","core_discovery":"The paper's central discovery is that a compact regression scorer trained on LLM judgments can serve as a general, scalable quality filter for code pretraining. The authors sampled 222,066 code files from 21 languages, asked DeepSeek-V2-Chat to rate each file 0-10 on readability, modularity, clarity, and reusability, and fine-tuned a 1.3B Llama-2-architecture model with a regression head to predict those scores. Applying this scorer to all GitHub data and dropping the bottom ~10% of files (~1T tokens) improved benchmark performance during pretraining relative to minimal rules alone, as shown in Figure 6. Extending the same model-centric principle to web data (with category-specific thresholds), commits, and long-context data yielded a base model that outperforms comparable open base models on HumanEval, MBPP, MultiPL-E, CrossCodeEval, and RepoEval; the instruct and reasoning variants built on it reach state-of-the-art ~8B results and beat several larger models on code generation, editing, reasoning, and software-engineering benchmarks.","pith_inferences":["Inference: the same 'distill an oracle, regress, filter' recipe could transfer to other data-hungry domains such as mathematical proofs, scientific code, or documentation, where hand-crafted filtering rules are equally costly to maintain; a direct test would swap DeepSeek-V2-Chat for a stronger or fully open oracle and check whether quality-signal fidelity and downstream gains scale accordingly.","Inference: the Figure 6 comparison is the crux, and a sharper control would train on the kept bottom ~10% or on a shuffled threshold; if gains vanish, the benefit is the oracle's score distribution, not quality selection.","Inference: the paper's category-specific web thresholds implicitly concede that LLM scorers carry domain biases (forums under-scored, documentation over-scored); making those thresholds adaptive or learned could remove the last human knob in the pipeline.","Inference: with more than 10% of reasoning samples truncated at 64K tokens, the reported LiveCodeBench and Codeforces numbers likely understate the 8B reasoning model's ceiling; extending the context window or adding longer rollout might close more of the gap to o3-mini and IOI bronze."],"forward_implications":["Seed-Coder-8B-Base outperforms comparable open base models on HumanEval/MBPP, MultiPL-E, CrossCodeEval, and RepoEval, and beats several 13B+ models on multilingual code generation.","Seed-Coder-8B-Instruct sets the ~8B state of the art on MHPP (36.2%), BigCodeBench full (53.3%) and hard (26.4%), LiveCodeBench 2410–2502 (24.7%), FullStack Bench (55.8%), and SWE-bench Verified under Agentless (19.2%).","The reasoning model lifts LiveCodeBench overall pass@1 from 39.0% after warmup to 53.6% after GRPO-based LongCoT reinforcement learning, surpassing DeepSeek-R1-Distill-Qwen-14B and slightly edging Claude-3.7-Sonnet-Thinking.","The instruct model resolves 11.2% of SWE-bench Verified instances under the fully autonomous OpenHands workflow, far ahead of other ~8B models, which the authors attribute to commit-data supervision and LLM-filtered formatting consistency.","The model-centric pipeline scales to 6T tokens with minimal human involvement, processing GitHub files, 74M commits (~100B tokens), and ~1.2T tokens of code-related web data."],"supporting_citations":[{"why":"Serves as the oracle whose 0-10 readability/modularity/clarity/reusability scores become ground-truth labels for the quality scorer.","marker":"[DeepSeek-AI et al., 2024a]"},{"why":"Provides the 1.3B Llama-2-style backbone architecture that the regression quality scorer is fine-tuned from.","marker":"[Touvron et al., 2023]"},{"why":"Qwen2.5-Coder is the principal ~8B baseline to beat and the source of the open evaluation suites used for most benchmark comparisons.","marker":"[Hui et al., 2024]"},{"why":"OpenCoder exemplifies the 130+ hand-crafted filtering rules the paper argues against, and is a direct ~8B baseline.","marker":"[Huang et al., 2025]"},{"why":"DeepSeek-Coder supplies the rule-based filtering approach whose limitations motivate LLM-based scoring, plus a baseline.","marker":"[Guo et al., 2024]"},{"why":"StarCoder is the origin of the hand-crafted filtering rules that DeepSeek-Coder and others inherit.","marker":"[Li et al., 2023]"},{"why":"OSS-Instruct generates diverse seed code snippets used in the instruct model's synthetic SFT data.","marker":"[Wei et al., 2024]"},{"why":"Direct Preference Optimization is the post-SFT step that sharpens instruction-following in the instruct model.","marker":"[Rafailov et al., 2023]"},{"why":"GRPO is the reinforcement-learning algorithm used to train the reasoning model after the LongCoT warmup.","marker":"[Shao et al., 2024]"}],"fun_headline_variants":["LLM-judged code data curates 8B model that beats larger rivals","Scorer from LLM ratings lifts 8B code model past bigger models","Self-curated data gives 8B code model edge over larger peers","Compact LLM-scorer filters code data, boosting 8B model","Model-centric data filter powers 8B code model beyond larger ones"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality filter's usefulness rests on the assumption that the LLM oracle's 0-10 scores — distilled into a 1.3B scorer — correctly identify low-quality code across all 89 languages and every GitHub file type, so that discarding the bottom ~10% reliably improves pretraining rather than injecting the oracle's preferences.","fun_headline_variants_meta":{"raw":{"variants":["LLM-judged code data curates 8B model that beats larger rivals","Scorer from LLM ratings lifts 8B code model past bigger models","Self-curated data gives 8B code model edge over larger peers","Compact LLM-scorer filters code data, boosting 8B model","Model-centric data filter powers 8B code model beyond larger ones"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1831,"prompt_tokens":999,"completion_tokens":832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":734}},"tokens_in":615,"tokens_out":832,"duration_ms":7682,"temperature":1.0,"reasoning_tokens":734,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:59:58.944000+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train two models with identical token budgets and recipes, one on the curated corpus and one on a corpus that keeps the discarded bottom ~10% and removes the top ~10% (or shuffles the threshold); if the curated model does not consistently beat the shuffled-threshold control on held-out benchmarks like BigCodeBench and LiveCodeBench, the reported gain is an effect of the oracle's score distribution rather than of quality selection. A cheaper check: evaluate the trained 1.3B scorer's predicted quality against human labels on low-resource languages like Solidity or Verilog, where a systematically wrong signal would show up as near-random ranking.","supporting_citations":[],"review_version":1}