{"id":"80806d92-73d0-42ab-bf84-a838a07e69e7","arxiv_id":"2412.17395","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A code LLM fine-tuned on winner responses from pairwise expert battles, with instructions mined from chat templates, beats same-size baselines without proprietary LLMs.","lead":"WarriorCoder trains a code model by having five open-source expert models battle each other on generated coding problems, then fine-tunes a 6.7B base model on the winning answers. It reports state-of-the-art results on HumanEval, MBPP, CRUXEval, and DS-1000 without using proprietary LLMs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The selection signal is entirely judge-based—both the local votes (Eq. 2) and the global Elo (Eqs. 3–4) are derived from the same LLM judge preferences—and judge reliability is never validated against executable ground truth, so 'learning from the winner' is not actually established.","rationale":"The reader's weakest assumption is that judge votes are unreliable, and I agree. My stress-test sharpens this by showing that the Elo rating, which the paper presents as a global-consistency safeguard, is merely a temporal aggregation of the same judge votes. Therefore, both terms in Eq. 5 are functions of the same noisy preference signal, so the method cannot independently validate response quality. This is load-bearing because the stated contribution is 'learns from the winner of these pairwise competitions'—if the winner is not actually better, the core mechanism fails. The proposed test is decisive because it directly measures the information content of the selection signal against objective, execution-based correctness. A near-chance result would undermine the method's internal logic, even if the final benchmark numbers are high. I also considered contamination of the evaluation benchmarks as a competing concern; while real and worth checking, it would only undermine the reported metrics, whereas judge reliability undermines the proposed training paradigm itself. The reader's verdict is already CONDITIONAL; my concern reinforces the same condition and does not change the verdict.","tokens_in":18636,"tokens_out":5531,"duration_ms":55993,"concrete_test":"Sample 500 battle pairs from the data-construction pipeline, preserving their judge votes and Elo values. For each pair, execute both candidate answers against a held-out or generated unit test suite (for deterministic, executable instructions; use benchmark test generators where the prompt overlaps, or have an independent coder write simple tests). Label the execution-based winner as the response that passes more tests. Then compute: (1) the accuracy of the judge majority vote at predicting the execution-based winner, and (2) the agreement between the Eq. 5 selected winner and the execution-based winner. If either metric is at or near chance (≤55% over 500 pairs, with a binomial confidence interval), the selection signal is not capturing code correctness, and the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's core mechanism is selecting the better response in each battle and training on it (Section 3.3). The local score (Eq. 2) is the judge vote fraction, and the global Elo score (Eqs. 3–4) is updated from the same judge outcomes. Consequently, Eq. 5 combines two views of the same noisy judge signal, not an independent correctness check. The paper itself acknowledges in Section 2.3 that judge models struggle with complex problems and exhibit position, verbosity, and self-enhancement biases; the only mitigation reported (Appendix C) is order shuffling, which does not address correctness. No inter-judge agreement, judge-vs-execution agreement, or judge calibration is reported. If judge votes do not align with code correctness, the selected 'winner' responses are not necessarily the best available, and the claimed benefit over simpler data-mixing (e.g., random selection among experts) is unsupported. The ablation in Table 5 shows gains from more experts but does not isolate the contribution of judge-based selection. This is the most load-bearing concern because it attacks the causal mechanism of the method rather than merely the evaluated metrics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"WarriorCoder proposes a data flywheel for code LLMs in which open-source expert models compete in pairwise battles, LLM judges vote on the responses, and the target model is fine-tuned on the responses that win under a combination of local vote share and global Elo ratings. The pipeline mines instructions by prompting chat models with only the prefix of their chat templates, filters by judged difficulty, compresses via KCenterGreedy, and uses the battle winners as SFT targets. The paper reports pass@1 results on HumanEval, HumanEval+, MBPP, MBPP+, CRUXEval, and DS-1000, claiming state-of-the-art performance for a 6.7B model without proprietary LLM annotation.","tokens_in":19041,"tokens_out":5301,"duration_ms":45741,"significance":"If the central claims hold, the paper makes a useful contribution: it provides a fully open-source data construction pipeline for code instruction tuning, removes dependence on proprietary annotators, and reports large gains over same-scale baselines. The strengths are the concrete pipeline description, the analysis of instruction diversity and difficulty, and the multi-benchmark evaluation. The main limitation is that the selection signal (judge votes and Elo) is never validated against executable correctness, so the causal mechanism behind the gains remains unproven.","major_comments":[{"comment":"The summation \\sum_{B\\in Com\\A} in Eq. (5) is not operationalized by the described arena. In each battle round only one attacker and one defender compete (Section 3.1), and Eq. (2) defines x_i^{A>B} for a single opponent B. If the final score is instead computed against all other competitors, the paper does not specify how responses from every model are obtained for the same instruction or how the Elo ratings used in the sum are updated. Since e_i^A is the selection criterion for the training response, the reader cannot reproduce or interpret Eq. (5) without clarification.","section":"Section 3.3, Eq. (5)"},{"comment":"The Elo term is not an independent source of global consistency: R_A and R_B are updated in Eq. (4) from the same judge votes t_A and t_B that define the local score in Eq. (2). The combined score therefore reweights the same judge signal rather than adding a separate measurement of model strength. The paper should either justify analytically why this reweighting corrects judge noise or validate it empirically, for example by comparing selections made with and without the Elo term against a held-out execution-based correctness signal.","section":"Section 3.3, Eqs. (3)-(5)"},{"comment":"Judge reliability is the core assumption of the method, but it is never validated against executable ground truth. The paper itself notes that judge models struggle with complex problems and exhibit position, verbosity, and self-enhancement biases, and the only reported mitigation is order shuffling. Without reporting inter-judge agreement, judge agreement with test-case execution, or a human-annotated sample, the claim that WarriorCoder learns from the winner (Section 3.4) is not established; the selected responses could merely be the more verbose or stylistically preferred ones.","section":"Section 2.3 and Appendix C"},{"comment":"The ablation varies the number of experts but does not isolate the contribution of the battle-based selection mechanism. A reader cannot tell whether the gains come from selecting the judge-preferred response, from the diversity of multiple expert responses, or simply from fine-tuning on additional open-source generated data. Adding comparisons against random selection among experts, local-vote-only selection, Elo-only selection, or a pooled dataset without any selection would be needed to support the causal claim implicit in the title and in Section 3.4.","section":"Table 5"}],"minor_comments":[{"comment":"The paper states that 'we deduplicate the data and adopt judges to assess their difficulty' but does not specify which models serve as difficulty judges or how the 1-10 score is elicited; please provide the prompt and the judge model.","section":"Section 3.2"},{"comment":"The explanation for setting alpha to 0.7 ('because we need the Elo Rating only when judges' opinions are divided') is unclear, since alpha=0.7 makes the Elo term dominate rather than apply only in tied cases; please clarify the design.","section":"Section 4.1"},{"comment":"The 'Rely on proprietary LLMs?' column uses the symbols '%' and '!' without a legend in the table or caption, so the reader cannot interpret the last column.","section":"Table 1"},{"comment":"The paper should define what counts as a win and a draw when judges vote; currently only the vote counts t_A and t_B are given, and the mapping from votes to the actual scores s_i^{A>B} and s_i^{B>A} in Eq. (4) is not fully specified.","section":"Equations (2)-(4)"},{"comment":"The ROUGE overlap analysis covers CodeAlpaca and CodeUltraFeedback but not the evaluation benchmarks used in Section 4; an overlap check against HumanEval, MBPP, CRUXEval, and DS-1000 would strengthen the contamination discussion.","section":"Section 4.4.1 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and stress-test concern about judge circularity lands on reading the paper: the central selection mechanism is entirely judge-based and unvalidated. The paper's own Limitations section mentions only the time cost of battles, not the reliability of the judges. If the authors can add judge-validation experiments (agreement with execution, inter-judge agreement, or a human sample) and an ablation that isolates the selection mechanism, the contribution would be much stronger. I also note that no code or data are released, which is a significant reproducibility gap for a data-generation paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a read if you care about data flywheels for code LLMs. It proposes WarriorCoder, which generates training data from battles among open-source code experts: one model mines instructions from its own chat template, another responds, and the remaining models judge which response is better. The target model is fine-tuned on the winners. The result is a 6.7B model with 80.5% HumanEval and 75.6% HumanEval+, beating same-size fine-tuned baselines and rivaling larger models, without any proprietary API calls.\n\nWhat's genuinely new here is the packaging: Magpie-style completion mining plus an arena with Elo selection, applied to code and deliberately free of closed models. The closest prior work, Arena Learning, already uses battles and Elo for post-training, so the incremental piece is the code-specific, no-proprietary-LLM adaptation. That's a real contribution: it makes the flywheel cheap and scalable, and the experiments are careful about comparing against same-backbone baselines.\n\nThe soft spots are real but not disqualifying. The main one is that the entire selection signal comes from LLM judges. Both the local votes and the global Elo are derived from the same judge preferences, so the \"global consistency\" is a smoothed reweighting of the same signal, not an independent check. The paper acknowledges judge biases but only mitigates order shuffling; it never validates judge votes against execution, which is possible for code. That leaves the causal story (\"learning from the winner\") under-supported. The ablation with more experts shows gains, but doesn't isolate selection from simple data mixing. Also, Eq. 5's summation over all opponents isn't operationalized, no error bars or contamination check are reported, and no code or data is released.\n\nStill, the empirical result is strong and consistent across HumanEval, MBPP, CRUXEval, and DS-1000. A 6.7B model hitting these numbers with data mined from open models is genuinely interesting, even if the specific mechanism isn't pinned down.\n\nI'd send this to a serious referee. The missing validation of judge reliability and the absent artifacts are fixable in revision, and the core idea deserves scrutiny rather than a desk reject. For a reading group, it would prompt a good discussion about judge-based data selection and what evidence you need to trust a flywheel.","headline":"WarriorCoder is a solid code-data flywheel paper with strong empirical results, undermined by a judge-based selection signal that is never validated against execution.","tokens_in":19432,"tokens_out":2917,"would_cite":true,"duration_ms":28139,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Battles among open-source code LLMs train a 6.7B model to 80.5% on HumanEval without proprietary data.","keywords":["code generation","LLM-as-a-judge","data flywheel","Elo rating","instruction mining","open-source LLMs","expert battles"],"falsifier":"Retrain the same 6.7B base on the same mined instructions with responses chosen randomly instead of by judge-plus-Elo selection; if the random-response model scores within a few points of WarriorCoder on HumanEval, then the battle-selection step is not what produces the gain.","tokens_in":18457,"feed_emoji":"⚔️","tokens_out":7765,"duration_ms":62769,"temperature":0.7,"pith_summary":"The paper tries to establish that high-quality code training data can be generated from scratch by having open-source code LLMs battle each other, with no proprietary LLM annotations and no human-written seed prompts. It builds an arena where one expert attacks another with instructions sampled from its own chat-template distribution, judges vote on the correctness and helpfulness of the two responses, and a blend of local vote share and global Elo rating selects the winner. Fine-tuning a 6.7B base model on these winning responses yields 80.5% pass@1 on HumanEval and competitive results on CRUXEval and DS-1000, surpassing same-size fine-tuned baselines. If true, this offers a low-cost, scalable way to build diverse code instruction data and absorb the strengths of multiple experts.","feed_headline":"LLM battle arena lifts 6.7B code model to 80.5% on HumanEval","feed_subtitle":"No proprietary GPT or Claude data needed: open-source experts judge each other and the winner becomes training data.","key_machinery":"The load-bearing object is the battle arena itself: one attacker expert mines an instruction by completing its own chat-template prefix, a defender expert answers it, and the remaining experts vote as judges. The selection rule combines the local vote share $x^i_{A>B}$ with an Elo rating expectation $X^{Elo}_{A>B}$ through Equation (5), with $\\alpha=0.7$, so that global consistency tempers noisy local votes. Instruction quality is controlled by four-level difficulty filtering and KCenterGreedy embedding-based compression, where KCenterGreedy is a coreset selection algorithm that chooses a diverse subset from embeddings. The mechanism's work is to convert unlabeled expert knowledge into paired responses with a winner label, which then becomes supervised fine-tuning data.","core_discovery":"The central claim is that a code LLM can be improved to state-of-the-art same-size performance by learning from pairwise battles among open-source expert code LLMs, rather than from data expanded by proprietary models. The authors construct an arena in which each expert alternately attacks and defends, generating instructions by completing the prefix of its own chat template, and the other experts act as judges. The winning response for each instruction is selected by combining the local judge-vote share with an Elo-rating-based global expectation, then used as supervised fine-tuning data for a 6.7B DeepSeekCoder base. The paper reports 80.5% pass@1 on HumanEval, 75.6% on HumanEval+, 76.2% on MBPP, and 64.8% on MBPP+, all without relying on proprietary LLMs, and interprets this as direct evidence that competitive data generation can absorb the strengths of multiple experts.","pith_inferences":["An implication the paper leaves implicit: the same battle framework could generate fine-tuning data for other domains, but only in domains where a panel of open-source judges can reliably rank answers; judge quality would need to be validated first.","The Elo-plus-vote blend suggests a general recipe for aggregating pairwise preferences under noisy judges; a natural next test is whether simpler aggregators, such as majority vote alone, lose the gains on harder problems.","Because the instruction pool is mined from the chat templates of the five chosen experts, the diversity ceiling is set by those models; adding more or more diverse open-source experts should push the benchmarks higher, a trend the paper's Table 5 already hints at.","A cautious reader would want a contamination check: the ROUGE comparisons were done against CodeAlpaca and CodeUltraFeedback, not against HumanEval or MBPP; testing overlap with the evaluation sets would clarify how much of the gain is genuine generalization."],"forward_implications":["If the central claim holds, a 6.7B code model trained on battle-selected data reaches 80.5% pass@1 on HumanEval and 76.2% on MBPP, surpassing all same-size fine-tuned baselines in Table 1.","The same model reaches 42.9% and 45.4% pass@1 on CRUXEval input/output and 38.1% overall on DS-1000, indicating gains extend beyond simple generation into code reasoning and library usage.","Learning from more experts monotonically improves all four main benchmarks (Table 5), so the data flywheel should keep benefiting as the competitor pool grows.","Because the pipeline needs no seed dataset, no human prompts, and no proprietary LLM annotations, it lowers the cost and widens access to building code instruction data.","The paper also argues the mined instructions are largely novel, with ROUGE scores below 0.6 against existing datasets, so the approach adds independent training examples."],"supporting_citations":[{"why":"Supplies the completion-based method for mining instructions directly from chat-template prefixes.","marker":"(Xu et al., 2024b)"},{"why":"Source of the Elo rating system used as the global score to temper local judge votes.","marker":"(Bai et al., 2022)"},{"why":"Documents position, verbosity, and self-enhancement biases that motivate the judge rules and the Elo-vote blend.","marker":"(Zheng et al., 2023a)"},{"why":"Provides the KCenterGreedy algorithm used for embedding-based compression of mined instructions.","marker":"(Sener and Savarese, 2018)"},{"why":"The base model DeepSeekCoder-Base-6.7B that WarriorCoder initializes from and fine-tunes.","marker":"(Guo et al., 2024)"},{"why":"A same-backbone fine-tuned baseline (Magicoder) whose results WarriorCoder is compared against and must beat.","marker":"(Wei et al., 2024)"},{"why":"Introduces HumanEval, the primary benchmark where the 80.5% pass@1 result is measured.","marker":"(Chen et al., 2021)"},{"why":"Introduces MBPP, the second code-generation benchmark used in the main results.","marker":"(Austin et al., 2021)"},{"why":"Provides the leaderboard from which the five open-source competitor experts are selected.","marker":"(Zhuo et al., 2024)"}],"fun_headline_variants":["Code LLM trained on expert battles hits 80.5% HumanEval","WarriorCoder: open-source battles forge top code LLM","No proprietary LLMs: WarriorCoder wins via battle arena","Expert battles train code model to 80.5% HumanEval, no GPT-4","Battling open-source models yields 80.5% HumanEval without proprietary data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the open-source judge LLMs being able to tell which of two code answers is more correct and helpful; if their votes are noisy or systematically biased, the winner labels that form the training data may point the model at the wrong answers.","fun_headline_variants_meta":{"raw":{"variants":["Code LLM trained on expert battles hits 80.5% HumanEval","WarriorCoder: open-source battles forge top code LLM","No proprietary LLMs: WarriorCoder wins via battle arena","Expert battles train code model to 80.5% HumanEval, no GPT-4","Battling open-source models yields 80.5% HumanEval without proprietary data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000749,"raw_usage":{"total_tokens":3326,"prompt_tokens":926,"completion_tokens":2400,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2296}},"tokens_in":542,"tokens_out":2400,"duration_ms":16093,"temperature":1.0,"reasoning_tokens":2296,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:27:12.894849+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same 6.7B base on the same mined instructions with responses chosen randomly instead of by judge-plus-Elo selection; if the random-response model scores within a few points of WarriorCoder on HumanEval, then the battle-selection step is not what produces the gain.","supporting_citations":[],"review_version":1}