{"id":"72c1eaab-f6a5-402b-a977-37f7352d3f0a","arxiv_id":"2412.05210","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 397-sample, 44-language code Q&A benchmark with LLM-judged human preference reveals gaps between open and proprietary code models and a 20B-token synthetic instruction corpus.","lead":"This paper introduces CodeArena, a 397-sample benchmark of real-world coding questions judged by GPT-4o for human preference, and SynCode-Instruct, a 20B-token synthetic instruction corpus. It reports that open-source code models trail proprietary models like o1 and Claude on this preference-based measure, and that a synthetic-data-trained Qwen2.5-SynCoder is competitive on execution benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that CodeArena measures human preference rests entirely on GPT-4o judgments; without human validation, all preference-based conclusions may be judge artifacts.","rationale":"The paper makes two main contributions: CodeArena, a benchmark for 'human preference' in code generation, and SynCode-Instruct, a large synthetic instruction corpus. The central claim is that CodeArena effectively measures alignment between model responses and human preference. The only evaluation signal for CodeArena is the GPT-4o judge, and the paper provides no human preference labels, no inter-annotator agreement, and no judge-human correlation. This is the same load-bearing assumption identified by the reader: if GPT-4o's verdicts are biased toward length, formatting, or explanation style rather than actual user satisfaction, then the benchmark scores do not measure human preference, and the observed open-vs-proprietary gap could be a judge artifact. The self-preference concern, where the judge is also an evaluated model, amplifies but does not replace this core issue. I agree with the reader's weakest_assumption. The rest of the paper, including execution-based results and the synthetic data pipeline, is credible and likely salvageable, which supports a conditional verdict rather than rejection. The proposed concrete test, human validation on a stratified subset, would directly settle whether the central claim holds.","tokens_in":19640,"tokens_out":5779,"duration_ms":53566,"concrete_test":"Select a stratified sample of 100 CodeArena items covering all 7 main categories and the three difficulty levels. For each item, have at least 3 independent annotators (experienced programmers) judge the same pairwise comparison that GPT-4o made (model response vs. baseline gpt-4-turbo-2024-04-09), using the same protocol. Then: (1) compute the percentage of items where GPT-4o's verdict (win/tie/loss) matches the human majority verdict; and (2) recompute model win rates using only human verdicts for these 100 items and compute the Spearman rank correlation with the GPT-4o-based win rates for the same models. If the agreement is below about 80% or the rank correlation is below about 0.8, the claim that CodeArena measures human preference is not supported, and the reported gap between open and proprietary models could be a judge artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 ('Evaluation') states that 'we apply GPT-4o-2024-08-06 as the judger to evaluate the model performance,' with no human preference labels, no inter-annotator agreement, and no comparison to human judgments anywhere in the paper. The benchmark is 'human-curated' only in the sense that humans wrote and selected the questions; the preference labels come solely from one LLM judge. The central claim (Section 1, Contribution 3) that 'CodeArena can effectively measure the alignment between the model-generated response and human preference' therefore reduces to the assumption that GPT-4o's pairwise verdicts are a faithful proxy for human preferences. That assumption is load-bearing: the reported open-vs-proprietary gap (e.g., o1-mini 89.3 vs Qwen2.5-Coder-32B 68.9) and the claimed improvement from SynCode-Instruct are all measured through this judge. The paper's own Figure 1 illustrates that the favored response is 'a detailed and beautiful reply,' suggesting a potential length and explanation bias. Additionally, GPT-4o is itself one of the evaluated models (Table 3, 69.1% win rate), raising the possibility of self-preference. Neither possibility is addressed. If the judge's preferences are not human preferences, the benchmark measures alignment with GPT-4o rather than with humans, and the headline gap could be an artifact.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeArena, a human-curated benchmark of 397 code-related questions extracted from real-world Q&A scenarios, spanning 40 categories and 44 programming languages, and evaluates models by pairwise comparison using GPT-4o-2024-08-06 as an LLM judge against a GPT-4-turbo baseline. It additionally introduces SynCode-Instruct, a roughly 20B-token synthetic instruction corpus derived from web data, and uses it to fine-tune Qwen2.5-SynCoder. The authors report CodeArena win rates for more than 40 models, execution-based results on EvalPlus and MultiPL-E, and conclude that open-source code LLMs lag proprietary models in human-preference alignment while large-scale synthetic instruction data improves execution performance.","tokens_in":20021,"tokens_out":5150,"duration_ms":51169,"significance":"If the central claim is validated, CodeArena would fill a genuine gap by providing a preference-oriented, real-world-distributed benchmark for code assistants, complementing correctness-only benchmarks such as HumanEval and MultiPL-E. The paper has concrete strengths: the questions are human-curated with a multi-stage quality-control process; the dataset covers a broad range of languages and task categories; the decontamination step against existing benchmarks is valuable; and the execution-based evaluations are straightforward and reproducible. The SynCode-Instruct corpus and the two-stage SFT recipe are also potentially useful community assets. However, the central claim that CodeArena measures alignment with human preference is currently unsupported, because the only preference signal is an unvalidated GPT-4o judge.","major_comments":[{"comment":"The central claim that CodeArena measures alignment between model responses and human preference rests entirely on GPT-4o-2024-08-06 as the judge, but the paper provides no human preference labels, no inter-annotator agreement measure, and no correlation between GPT-4o verdicts and human verdicts. The paper's own Figure 1 illustrates that the judge favors 'a detailed and beautiful reply,' which suggests a possible length or explanation-style bias, and Table 3 shows that GPT-4o is itself an evaluated model with a 69.1% win rate, raising a self-preference concern that is not addressed. Without a human-validation study on a sample of CodeArena items, the reported open-vs-proprietary gap and all preference-alignment conclusions could be artifacts of the LLM judge. Please add a human evaluation with agreement statistics, or explicitly reframe CodeArena as measuring agreement with GPT-4o rather than with humans.","section":"Section 2, Evaluation; Section 5.1"},{"comment":"The claim that Qwen2.5-SynCoder 'can achieve top-tier performance of open-source code LLMs' and that the results verify the effectiveness of large-scale synthetic instruction data for improving model performance is contradicted by the CodeArena results in Table 3: Qwen2.5-SynCoder achieves a 49.2% average win rate, well below Qwen2.5-Coder-32B-Instruct (68.9%) and Qwen2.5-72B-Instruct (73.8%), and the paper's Section 5.1 says only that SynCoder 'can still get a strong performance on CodeArena.' On the preference benchmark, the synthetic SFT recipe substantially degrades alignment relative to the strong base instruct model, yet this tension is never discussed. The 'top-tier' claim should be explicitly restricted to execution-based benchmarks, and the paper should explain or at least acknowledge the marked drop in CodeArena win rate for Qwen2.5-SynCoder.","section":"Abstract; Section 5.1; Table 3"},{"comment":"All CodeArena results are reported as point estimates from pairwise comparisons on 397 questions, with no confidence intervals, bootstrap estimates, or significance tests. Several headline comparisons in Table 3 are small (e.g., Claude-3.5-Sonnet-20240620 at 77.8% vs Claude-3.5-Sonnet-20241022 at 78.1%, or GPT-4o at 69.1% vs Qwen2.5-Coder-32B-Instruct at 68.9%) and are likely within sampling noise. The open-vs-proprietary gap is large, but the leaderboard ordering and the more fine-grained claims require uncertainty quantification. Please report bootstrap confidence intervals or pairwise significance tests for the main win rates.","section":"Section 4.3; Section 5.1; Table 3"}],"minor_comments":[{"comment":"The category 'Miscellaneous and General Inquiry' appears twice with counts 62 and 7; rename the second occurrence to avoid ambiguity.","section":"Table 1"},{"comment":"The Qwen2.5-Coder-14B-Instruct row reports an average win/tie rate of 60.6/51.5, which is not derivable from the per-category values listed in the same row; please correct this entry and verify all other averages.","section":"Table 3"},{"comment":"The number of evaluated models is inconsistently reported as '40+ LLMs' in the abstract, 'nearly 40' in the introduction, and 'over 20' in the conclusion; please harmonize these counts.","section":"Abstract; Section 1; Section 7"},{"comment":"The final sentence of Related Work states 'In this work, we propose FullStack Bench,' but the paper proposes CodeArena; this appears to be a leftover from another manuscript and should be corrected.","section":"Section 6"},{"comment":"The paper should clarify that the human annotation process validates question quality and difficulty, not the preference labels; the phrase 'HumanAnnotation ✓' in Table 2 could otherwise be misread as human preference annotation.","section":"Section 2, Human Annotation & Quality Control"},{"comment":"The subsection heading 'Recall from Common Crawl' is incomplete as written; it should describe the data-source recall and cleaning pipeline more explicitly.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The dataset and the SynCode-Instruct corpus are potentially useful contributions, and the execution-based evaluation appears sound. The main risk is that the paper currently overclaims human-preference measurement on the basis of a single unvalidated LLM judge. I would encourage the editor to require a human-validation study or a substantial reframing of the claims before acceptance; the SynCoder result also needs to be reconciled with the paper's narrative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CodeArena is a genuinely useful artifact: 397 real-world Q&A-style coding questions across 40 categories and 44 languages, with human curation, quality control, and decontamination. That is the kind of resource the community can build on, and the execution-based evaluation (EvalPlus, MultiPL-E) is standard and credible. The SynCode-Instruct pipeline, with two-stage SFT and a scaling analysis, is also a concrete contribution.\n\nThe load-bearing problem is exactly what the stress test flags: the claim that CodeArena measures alignment with human preference is supported only by GPT-4o pairwise judgments. There are no human preference labels, no judge-human agreement, no inter-annotator reliability. The judge is also one of the evaluated models (GPT-4o scores 69.1% against the baseline), which raises a self-preference risk that is never addressed. Figure 1 even illustrates the judge rewarding \"a detailed and beautiful reply,\" suggesting a length and elaboration bias. Without a small human-validation study, the headline gap between open and proprietary models could be a judge artifact.\n\nThe paper also overstates what its own results show. The abstract claims Qwen2.5-SynCoder achieves \"top-tier performance of open-source code LLMs,\" but on CodeArena it scores 49.2% win rate, well below Qwen2.5-Coder-32B's 68.9%. The synthetic data helps execution benchmarks but not preference alignment; the paper's own data contradict the abstract's framing. There are smaller issues: inconsistent model counts (40+, nearly 40, over 20 in different places), no confidence intervals on the win rates, and a duplicate category name in Table 1.\n\nThis is a salvageable paper, not a flawed idea. The benchmark and the synthetic-data recipe are worth having, but the central validity claim needs real evidence. I would send it to peer review because the artifact deserves referee time and the fix is clear: add a human-judgment correlation study, report confidence intervals, and temper the language. I would not cite it in my own work until the human validation appears.","headline":"A useful new code-preference benchmark and synthetic-data corpus, but the central claim that CodeArena measures human preference rests entirely on an unvalidated GPT-4o judge; the paper needs a human-validation study before the headline results can be trusted.","tokens_in":20451,"tokens_out":1816,"would_cite":false,"duration_ms":19416,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CodeArena measures code-LLM responses by human preference, not just correctness, and shows open models lag proprietary ones.","keywords":["CodeArena","human preference alignment","code LLM benchmark","LLM-as-a-judge","synthetic instruction data","code generation","Qwen2.5-SynCoder","win rate"],"falsifier":"Run a human study in which developers rate the same 397 CodeArena responses that GPT-4o judged, and compute agreement between human and model verdicts; if agreement is near chance, or if GPT-4o's favorite responses are simply longer or more verbose rather than more useful, then CodeArena scores do not measure human preference.","tokens_in":19428,"feed_emoji":"🧑‍💻","tokens_out":8369,"duration_ms":66602,"temperature":0.7,"pith_summary":"The paper argues that execution-based code benchmarks miss a key axis of code assistant quality: whether responses match what real users prefer. To measure that axis, it introduces CodeArena, a set of 397 manually curated real-world coding questions spanning 40 categories and 44 programming languages, scored by pairwise GPT-4o judgments against a fixed baseline. Sweeping more than 40 models through CodeArena, it finds a substantial gap between open-source code LLMs such as Qwen2.5-Coder and proprietary models such as o1-mini and Claude-3.5-Sonnet, even where execution scores are close. The paper also builds SynCode-Instruct, a nearly 20-billion-token synthetic instruction corpus, and shows that training Qwen2.5-SynCoder on it improves both execution benchmarks and CodeArena win rates, though it does not close the preference gap.","feed_headline":"Human-preference benchmark: open code models lag o1-mini by 20 points","feed_subtitle":"CodeArena scores 397 real user queries with an LLM judge; open-source coders trail closed models on response quality.","key_machinery":"The load-bearing object is CodeArena itself: 397 human-curated user queries that deliberately resist unit-test evaluation and are scored by an LLM judge, GPT-4o-2024-08-06, in both presentation orders against a fixed baseline response from gpt-4-turbo-2024-04-09; the averaged win and tie rates are what make human preference alignment measurable. The training side rests on SynCode-Instruct, a nearly 20-billion-token synthetic instruction corpus generated by scaling questions from code Q&A websites, with algorithmic responses verified by generated test cases and non-algorithmic responses selected by an LLM scorer.","core_discovery":"The central claim is that CodeArena measures alignment between model-generated responses and human preference, and that this alignment is a distinct axis from code correctness. On that axis, proprietary models dominate: o1-mini reaches an 89.3% win rate against the gpt-4-turbo-2024-04-09 baseline, Claude-3.5-Sonnet reaches 77.8%, and the best open model, Qwen2.5-Coder-32B-Instruct, reaches 68.9%, with most smaller open models far lower. The paper further claims that large-scale synthetic instruction data is an effective route to improving both execution scores and preference alignment, because Qwen2.5-SynCoder, trained entirely on the synthetic corpus, reaches 78.8% on MultiPL-E and 49.2% win rate on CodeArena, matching some much larger open models on execution while remaining below closed models on preference.","pith_inferences":["If GPT-4o's verdicts track human preferences, CodeArena scores are a proxy for user satisfaction, but the paper offers no human-correlation check, so a crowdsourced validation study would directly test that link.","The preference gap may partly be a training-pipeline artifact: proprietary models are heavily optimized with human or AI preference feedback, and applying the same recipe to open models, with CodeArena as the reward signal, is a natural next step.","Because CodeArena queries are non-algorithmic and diverse, the benchmark could double as a general instruction-following test, not strictly a code benchmark.","The SynCode-Instruct pipeline inherits any bias of its LLM judge and scorer, so the same preferences being measured by CodeArena are also being baked into the training data."],"forward_implications":["Execution benchmarks overstate how ready open-source code LLMs are for real user-facing assistance, because CodeArena exposes a response-quality deficit that Pass@1 does not capture.","The open-vs-proprietary gap on CodeArena points to a concrete failure mode: open models often emit code alone, while preferred responses include explanations, formatting, and comments.","Large-scale synthetic instruction data is a viable path to improve preference alignment in open models, as demonstrated by Qwen2.5-SynCoder's gains over its base.","Two-stage SFT, with synthetic data followed by a small high-quality set, outperforms one-stage training at the same token budget on both CodeArena and execution benchmarks.","CodeArena can serve as a dynamic leaderboard for tracking how well code LLMs align with human preference as new models are released."],"supporting_citations":[{"why":"Supplies the pairwise LLM-as-a-judge methodology and arena-style preference evaluation that CodeArena adapts.","marker":"(Chiang et al., 2024)"},{"why":"HumanEval is the canonical execution-based benchmark whose function-completion framing CodeArena contrasts with real user queries.","marker":"(Chen et al., 2021a)"},{"why":"MBPP provides another execution-based baseline used for comparison and for decontamination.","marker":"(Austin et al., 2021)"},{"why":"MultiPL-E is used for decontamination and as the multilingual execution benchmark in the performance comparison.","marker":"(Cassano et al., 2023)"},{"why":"BigCodeBench represents the execution-based benchmark with complex instructions that CodeArena positions against.","marker":"(Zhuo et al., 2024)"},{"why":"Justifies using an LLM judge by prior evidence that LLM judgments align with human preferences.","marker":"(Zheng et al., 2023a)"},{"why":"Provides the starting point for scaling instructions from the web, which SynCode-Instruct extends by generating new questions.","marker":"(Yue et al., 2024)"},{"why":"Qwen2.5-Coder is the base model family for Qwen2.5-SynCoder and the key open-source comparison point.","marker":"(Hui et al., 2024)"}],"fun_headline_variants":["CodeArena benchmark: human preference gap between open and closed models","Open code LLMs fall 20 points behind proprietary on human preference","Synthetic data helps open code models but still trail o1-mini on CodeArena","New benchmark CodeArena rates code responses by human preference alignment","o1-mini beats open models on CodeArena human-preference benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim leans on GPT-4o-2024-08-06 being a faithful judge of human preference, yet the paper provides no human preference labels, no inter-annotator agreement, and no direct correlation between GPT-4o verdicts and human verdicts.","fun_headline_variants_meta":{"raw":{"variants":["CodeArena benchmark: human preference gap between open and closed models","Open code LLMs fall 20 points behind proprietary on human preference","Synthetic data helps open code models but still trail o1-mini on CodeArena","New benchmark CodeArena rates code responses by human preference alignment","o1-mini beats open models on CodeArena human-preference benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1497,"prompt_tokens":1026,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":374}},"tokens_in":642,"tokens_out":471,"duration_ms":4898,"temperature":1.0,"reasoning_tokens":374,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:48:44.210412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human study in which developers rate the same 397 CodeArena responses that GPT-4o judged, and compute agreement between human and model verdicts; if agreement is near chance, or if GPT-4o's favorite responses are simply longer or more verbose rather than more useful, then CodeArena scores do not measure human preference.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MultiPL-E is used for decontamination and as the multilingual execution benchmark in the performance comparison."}],"review_version":1}