{"id":"07e818a9-68c0-4c0f-bde8-947f0e5b4825","arxiv_id":"2508.13757","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"COMPASS benchmark shows that evaluating code generation with only correctness misses large differences in runtime efficiency and code quality among frontier LLMs.","lead":"The paper introduces COMPASS, a benchmark that scores AI code generation on correctness, speed, and code quality, using 50 real Codility contest problems and nearly 400,000 human submissions. It finds that one frontier model, Claude Opus 4, is much less efficient than its correctness scores suggest, arguing that correctness-only benchmarks hide real differences.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quality-axis orthogonality rests on an undisclosed composite score; if CodeScene's Codility configuration is not validated, the central claim's independent quality dimension is unsupported.","rationale":"The reader's weakest_assumption is exactly the CodeScene composite score, and I agree it is the most load-bearing unresolved point. The strongest_claim has two parts: (1) correctness does not predict efficiency, and (2) correctness does not predict maintainability. Part (1) is well supported by the efficiency test design and the large Claude Opus 4 gap. Part (2) depends on the quality metric, and the paper's own Section III.A.3 concedes that the metric is Codility's internal CodeScene configuration without disclosing thresholds, weighting, or validation. The PCA/correlation results are consistent with an orthogonal quality axis, but they cannot establish that the axis is maintainability as opposed to stylistic surface features. This is not a manufactured concern: the paper's discussion repeatedly generalizes from 'code quality' to 'readable, modular, idiomatic' and 'maintainable,' so the construct validity of the score is central. I do not see a fatal flaw in the work, and the efficiency finding stands on its own. The right verdict remains CONDITIONAL because the authors can resolve the concern by releasing the configuration and the per-submission scores, or by re-labeling the dimension more conservatively.","tokens_in":24368,"tokens_out":1663,"duration_ms":15389,"concrete_test":"Publish the CodeScene configuration (thresholds, sub-metrics, and composite weighting) and the per-submission quality scores, as stated in the benchmark-release plan. Then independently regress the quality score against a structurally meaningful maintainability proxy (e.g., cyclomatic complexity, nesting depth, function length, duplication) on a subset of 50 model-generated solutions per problem. If the composite score is not primarily driven by structural metrics and cannot distinguish deliberately readable code from deliberately obfuscated-but-equivalent code, the quality dimension's orthogonality claim should be weakened or re-scoped as 'style score' rather than maintainability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that correctness alone hides large gaps in efficiency and code quality. The efficiency gap for Claude Opus 4 (72.2% correctness vs 35.4% efficiency) is concrete and largely self-contained: efficiency is measured by test-case pass rates under strict runtime thresholds, an approach that does not require a proprietary oracle. The more fragile part is the quality dimension. Section III.A.3 states that each solution receives a composite code quality score (1-10) using Codility's internal CodeScene configuration, but the paper does not disclose the thresholds, weighting, sub-metric aggregation, or any validation that this composite tracks maintainability. The PCA in Section V.A.2 then uses this opaque composite to conclude that quality is orthogonal (PC2 loading 0.990). That orthogonality is the statistical basis for claiming that quality is an independent axis and that 'models achieving high correctness scores do not necessarily produce maintainable code.' If the CodeScene composite is dominated by surface style (e.g., line length, naming) rather than structural maintainability, the PCA result would still show orthogonality but would not support the paper's qualitative conclusion. The issue is not that the authors are wrong about quality; it is that the manuscript provides no way to check whether the quality score measures what the paper says it measures. This is a load-bearing, addressable gap rather than an internal inconsistency. The human-baseline percentile analysis (Section V.D) also uses a normal CDF on highly skewed human score distributions, but that is secondary to the headline claim because it concerns human comparison, not the correctness-quality separation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces COMPASS, a multi-dimensional benchmark for code-generation evaluation, built from 50 Codility contest problems with 393,150 historical human submissions. Three frontier models—Claude Opus 4, Gemini 2.5 Pro, and O4-Mini-High—are evaluated on correctness, efficiency, and code quality under four prompt conditions and 64 samples per model-problem combination. The headline finding is that correctness alone is misleading: Claude Opus 4 reaches a mean correctness of 72.2% but an efficiency score of only 35.4%, while O4-Mini-High achieves 95.6% and 93.0%, respectively. The paper also reports correlation and PCA analyses to argue that quality is an independent axis, per-task human-percentile comparisons, within-model consistency statistics, and prompt-framing effects.","tokens_in":24594,"tokens_out":6900,"duration_ms":69523,"significance":"If the results hold, COMPASS is a valuable corrective to correctness-only code benchmarks and a useful resource: the human baseline dataset is large, the efficiency finding is concrete and would likely persist under any reasonable runtime-threshold policy, and the per-task tables provide a transparent view of model variability. The principal weakness is that the code-quality dimension is built on an undisclosed and unvalidated composite score, so the claim that maintainability constitutes an independent, meaningfully measured axis is not yet established. In addition, the prompt-analysis section contains a clear factual inconsistency with its own table. These issues are addressable, so I view them as requiring major revision rather than rejection.","major_comments":[{"comment":"The code quality dimension is scored by a composite from 'Codility's internal CodeScene configuration' without disclosure of the thresholds, weighting, sub-metric aggregation, or validation against human maintainability judgments. The PCA in Section V.A.2 then treats this composite as an established quality axis, concluding orthogonality from a PC2 loading of 0.990; without external validation, this conclusion is unsupported, especially since the quality scores are compressed in a narrow range (Table XIX, means 78-100, SDs mostly below 10). Please disclose and validate the configuration, or reframe the quality results as descriptive rather than as evidence of an independent dimension.","section":"III.A.3 / V.A.2"},{"comment":"The first paragraph of Section V.E states that 'correctness prompts produced slightly higher average correctness scores (87.535) with the lowest variability (SD = 27.891),' but Table XII reports a neutral-prompt correctness mean of 87.356 and a correctness-prompt mean of 87.235; the correctness prompt does not outperform the control on the mean. This factual inconsistency undermines the 'directionally consistent' summary and should be corrected.","section":"V.E / Table XII"},{"comment":"The efficiency dimension relies on 'strict runtime thresholds' that are not reported. Because the headline claim that Claude Opus 4 scores only 35.4% on efficiency is threshold-dependent, the paper should list the per-problem time limits (or at least the reference complexity and the chosen multiplier) so that readers can assess whether the threshold policy is reasonable and replicable.","section":"IV.B / III.A.2"}],"minor_comments":[{"comment":"The sentence ending 'Appendix B contains the per-task efficiency score statistics for each model' cites the wrong appendix; Appendix B reports correctness, while efficiency is in Appendix C. The cross-reference should be corrected.","section":"V.C.1"},{"comment":"The caption 'Human percentile baselines' is misleading; the table reports raw means, medians, and standard deviations, not percentiles. Rename to 'Human performance baselines.'","section":"Table XVI"},{"comment":"Equation (2) converts model scores to human percentiles using a normal CDF, but the human score distributions are markedly skewed (average skew +1.36, many zero scores). The text cautions about skewed/multimodal cases, but Table IX still reports the resulting percentiles without non-parametric alternatives; please provide empirical percentile ranks as a robustness check.","section":"V.D / Eq. (2)"},{"comment":"The reported sample size 'N = 3,200 per correlation' is inconsistent with the experimental design (50 problems × 3 models × 4 prompt types × 64 samples), and the paper should specify whether correlations are computed across samples within models, across task-level means, or across some other unit; otherwise the p-values are not interpretable.","section":"V.A.1"},{"comment":"The sampling description says 'k = 64 independent samples per model–problem combination,' while the prompt-variation results imply that each combination was run under four different prompts. The relationship between the 64 samples and the four prompt conditions should be stated explicitly.","section":"IV.B.a / III.B"}],"recommendation":"major_revision","confidential_remarks":"The paper's main asset is the authors' access to Codility's proprietary contest data and CodeScene integration, but this also creates a reproducibility risk because the quality metric is unexposed. I would encourage the editors to require disclosure of the CodeScene configuration or a validation study before publication. The prompt inconsistency in Section V.E is an easy empirical fix and should not be hard to address."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the efficiency result is concrete and worth engaging with. On 50 Codility contest problems, Claude Opus 4 scores 72.2% correctness but only 35.4% efficiency, while O4-Mini-High scores 95.6% and 93.0%. That is a clear demonstration that correctness-only benchmarks hide large gaps, and the efficiency measure is based on test-case pass rates under strict runtime thresholds, so it doesn't depend on a proprietary oracle. The human-baseline dataset (393k submissions) is a genuinely valuable resource. The paper does a service by pushing beyond pass@k.\n\nThe soft spots are real but don't take down the main empirical claim. The quality dimension is the one that worries me. The 1-10 composite from 'Codility's internal CodeScene configuration' is not described: no thresholds, no weighting, no validation that it tracks maintainability. The PCA (quality loading 0.990 on PC2) is only as meaningful as that composite. If the score mostly captures surface style, orthogonality doesn't tell us much about maintainability. This is fixable: report the subcomponents, validate against human judgments, or release the configuration.\n\nOther issues: the benchmark is not released, and for a benchmark paper that is a serious omission. The statistical validation ignores clustering—64 samples per problem are not independent observations, so the correlations and PCA likely overstate precision. The prompt analysis has a small internal inconsistency: the text says correctness prompts raised correctness, but Table XII shows neutral at 87.356 and correctness at 87.235. Reporting error, not a big deal. The human-percentile conversion uses a normal CDF on heavily skewed human distributions (average skew +1.36); the paper flags it in an appendix but the headline numbers should come with a warning. And they don't cite existing efficiency-aware work like LiveCodeBench, so the novelty framing is a bit selective.\n\nIf I were refereeing this, I'd ask for the benchmark to be publicly released, the CodeScene configuration disclosed and ideally validated, and the statistical claims to account for nesting. The core efficiency result is likely to survive. This paper deserves serious review, and it's worth a reading group if you want to argue about what code-quality benchmarks should actually look like.","headline":"A useful efficiency-focused benchmark with a concrete Claude Opus 4 finding, but the quality axis rests on an undisclosed composite and the benchmark isn't released.","tokens_in":25173,"tokens_out":4067,"would_cite":true,"duration_ms":40965,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Correctness scores do not predict efficiency or maintainability, and code quality is its own axis.","keywords":["code generation","LLM evaluation","benchmark","correctness","algorithmic efficiency","code quality","competitive programming","static analysis"],"falsifier":"Re-run the 50 problems on a different family of models and recompute the PCA; if code quality loads strongly on the first principal component alongside correctness and efficiency, the dimensional separation would not generalise. A direct check of the quality axis itself would be to have experienced programmers rate a sample of submissions for maintainability; if their ratings diverge sharply from the tool's composite scores, the quality metric would not be measuring what the paper claims.","tokens_in":24162,"feed_emoji":"📊","tokens_out":6088,"duration_ms":58686,"temperature":0.7,"pith_summary":"COMPASS is a benchmark for code-generation models that scores solutions on three axes instead of one: correctness, algorithmic efficiency, and code quality. The paper claims that correctness alone is a misleading proxy for real programming ability, because efficiency and quality are largely independent of whether the generated code passes its tests. The evidence is statistical: over 3,200 model submissions, correctness and efficiency correlate moderately, while code quality correlates with neither, and principal-component analysis loads quality almost entirely onto its own component. The headline example is Claude Opus 4, which averages 72.2 percent correctness but only 35.4 percent efficiency, against O4-Mini-High's 95.6 and 93.0; a correctness-only leaderboard would rank these two much closer than a multi-dimensional one does. If the paper is right, benchmarks that score only pass/fail correctness overstate how ready models are for real software work.","feed_headline":"Correct code alone hides slow or messy code","feed_subtitle":"A 50-problem benchmark finds code quality is an independent axis, so correctness-only rankings mislead.","key_machinery":"The load-bearing mechanism is COMPASS itself: 50 competitive programming problems from real contests on an online coding-assessment platform, each with a human baseline from 393,150 submissions, scored on correctness (percentage of test cases passed), efficiency (large-input tests with runtime thresholds derived from expert reference solutions), and quality (a composite 1-100 score from a static-analysis tool covering complexity, structure, and maintainability). The argument for treating these as separate axes is carried by correlation and principal-component analyses showing that quality loads on its own component and is nearly uncorrelated with the other two dimensions.","core_discovery":"On its own terms, the paper's central discovery is that the three evaluation dimensions are non-redundant. Correctness and efficiency share a moderate-to-strong relationship across models (aggregate r = .655), but code quality is orthogonal: it correlates at r = .089 with correctness and r = .022 with efficiency, and in a principal-component analysis it loads at 0.990 on the second component while contributing almost nothing to the first. The consequence is that a model can write correct, idiomatic-looking code that is nevertheless algorithmically inefficient or structurally hard to maintain, and no correctness metric will reveal that. The paper therefore argues that code-generation evaluation should treat correctness, efficiency, and quality as separate axes, anchored to human baselines, and that a model's production readiness should be judged on all three.","pith_inferences":["Because all three models score near 90 on quality, the quality axis may currently separate frontier models only on rare failure modes; the orthogonality result is more important for benchmark design than for ranking these three.","The human percentile comparison assumes human scores are normally distributed, but the paper itself reports skew of +1.36; recomputing percentiles with the empirical distribution could shift the ranks, especially in the tails.","A cheap check of the efficiency axis would be to measure how runtime grows with input size across a model's submissions; if the strict thresholds only separate timeouts from non-timeouts, they may be measuring constant factors rather than asymptotic complexity.","The natural extension is to multi-file or project-level tasks: if quality remains orthogonal to correctness there, the claim generalises from contest snippets to real codebases; if not, the orthogonality may be an artifact of short single-file solutions."],"forward_implications":["A model that scores near-perfect on correctness can deliver solutions that time out on large inputs; correctness-only leaderboards therefore systematically overstate production readiness.","Benchmarks should report efficiency and quality as separate scores, because aggregating them into a single number hides the Claude-style case: strong quality, poor efficiency, high variance.","Model rankings change materially: O4-Mini-High's composite mean (92.3) is far above Claude Opus 4's (66.1), even though their median correctness scores are both 100.","Human-relative placement is also different: O4-Mini-High and Gemini 2.5 Pro sit near the 97th percentile of human contestants, while Claude Opus 4 sits near the 76.6th percentile, warning that correctness-only comparisons to humans also mislead.","Prompt conditioning can nudge behavior in the intended direction, especially for the weakest model, so efficiency and quality objectives could be targeted directly in prompt design and training rather than assumed to follow from correctness."],"supporting_citations":[{"why":"Defines the correctness-only evaluation paradigm the paper argues is insufficient.","marker":"[1]"},{"why":"Another correctness-only benchmark showing the same narrow focus.","marker":"[2]"},{"why":"A more complex real-world benchmark that still measures only functional correctness.","marker":"[3]"},{"why":"A coding-competition benchmark that overlooks efficiency and quality.","marker":"[4]"},{"why":"Establishes technical debt and maintainability as long-term costs motivating the quality axis.","marker":"[5]"},{"why":"Provides a metrics suite for code quality that underpins the multidimensional quality notion.","marker":"[6]"},{"why":"Supplies the performance-prediction framework used to justify validating dimensions as non-redundant before aggregating.","marker":"[9]"}],"fun_headline_variants":["Correct code can still be slow and messy","Code quality escapes correctness metrics","Passing tests isn't enough for production code","LLM code quality is an independent axis","Correctness-only rankings hide inefficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The quality dimension and the claim that it is orthogonal rest on the static-analysis tool's composite maintainability score, configured internally at the benchmark's home organization with undisclosed thresholds and weights; if that score does not truly measure maintainability, the paper's quality axis and its orthogonality result collapse.","fun_headline_variants_meta":{"raw":{"variants":["Correct code can still be slow and messy","Code quality escapes correctness metrics","Passing tests isn't enough for production code","LLM code quality is an independent axis","Correctness-only rankings hide inefficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1231,"prompt_tokens":901,"completion_tokens":330,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":267}},"tokens_in":517,"tokens_out":330,"duration_ms":3653,"temperature":1.0,"reasoning_tokens":267,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:10:51.352058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 50 problems on a different family of models and recompute the PCA; if code quality loads strongly on the first principal component alongside correctness and efficiency, the dimensional separation would not generalise. A direct check of the quality axis itself would be to have experienced programmers rate a sample of submissions for maintainability; if their ratings diverge sharply from the tool's composite scores, the quality metric would not be measuring what the paper claims.","supporting_citations":[{"cited_title":"Hackerrank-astra: A benchmark for evaluating llms in coding competitions,","cited_arxiv_id":null,"evidence_quote":"A coding-competition benchmark that overlooks efficiency and quality."},{"cited_title":"Managing technical debt with the sqale method,","cited_arxiv_id":null,"evidence_quote":"Establishes technical debt and maintainability as long-term costs motivating the quality axis."},{"cited_title":"A metrics suite for object oriented design,","cited_arxiv_id":null,"evidence_quote":"Provides a metrics suite for code quality that underpins the multidimensional quality notion."},{"cited_title":"Modeling the performance prediction problem in industrial and organizational psychology,","cited_arxiv_id":null,"evidence_quote":"Supplies the performance-prediction framework used to justify validating dimensions as non-redundant before aggregating."}],"review_version":2}