Pith. sign in

REVIEW 3 major objections 5 minor 9 references

COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Correctness scores do not predict efficiency or maintainability, and code quality is its own axis.

desk verdict A useful efficiency-focused benchmark with a concrete Claude Opus 4 finding, but the quality axis rests on an undisclosed composite and the benchmark isn't released. read the letter →

arxiv 2508.13757 v1 pith:3ATAVV3Z submitted 2025-08-19 cs.SE cs.AI

James Meaden , Micha{l} Jarosz , Piotr Jod{l}owski , Grigori Melnik This is my paper · ORCID
classification cs.SEcs.AI
keywords codegenerationLLMevaluationbenchmarkcorrectnessalgorithmicefficiencyqualitycompetitiveprogrammingstaticanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

COMPASS is a benchmark for code-generation models that scores solutions on three axes instead of one: correctness, algorithmic efficiency, and code quality. The paper claims that correctness alone is a misleading proxy for real programming ability, because efficiency and quality are largely independent of whether the generated code passes its tests. The evidence is statistical: over 3,200 model submissions, correctness and efficiency correlate moderately, while code quality correlates with neither, and principal-component analysis loads quality almost entirely onto its own component. The headline example is Claude Opus 4, which averages 72.2 percent correctness but only 35.4 percent efficiency, against O4-Mini-High's 95.6 and 93.0; a correctness-only leaderboard would rank these two much closer than a multi-dimensional one does. If the paper is right, benchmarks that score only pass/fail correctness overstate how ready models are for real software work.

What carries the argument

The load-bearing mechanism is COMPASS itself: 50 competitive programming problems from real contests on an online coding-assessment platform, each with a human baseline from 393,150 submissions, scored on correctness (percentage of test cases passed), efficiency (large-input tests with runtime thresholds derived from expert reference solutions), and quality (a composite 1-100 score from a static-analysis tool covering complexity, structure, and maintainability). The argument for treating these as separate axes is carried by correlation and principal-component analyses showing that quality loads on its own component and is nearly uncorrelated with the other two dimensions.

What would settle it

Re-run the 50 problems on a different family of models and recompute the PCA; if code quality loads strongly on the first principal component alongside correctness and efficiency, the dimensional separation would not generalise. A direct check of the quality axis itself would be to have experienced programmers rate a sample of submissions for maintainability; if their ratings diverge sharply from the tool's composite scores, the quality metric would not be measuring what the paper claims.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the three evaluation dimensions are non-redundant. Correctness and efficiency share a moderate-to-strong relationship across models (aggregate r = .655), but code quality is orthogonal: it correlates at r = .089 with correctness and r = .022 with efficiency, and in a principal-component analysis it loads at 0.990 on the second component while contributing almost nothing to the first. The consequence is that a model can write correct, idiomatic-looking code that is nevertheless algorithmically inefficient or structurally hard to maintain, and no correctness metric will reveal that. The paper therefore argues that code-generation evaluation should treat correctness, efficiency, and quality as separate axes, anchored to human baselines, and that a model's production readiness should be judged on all three.

Load-bearing premise

The quality dimension and the claim that it is orthogonal rest on the static-analysis tool's composite maintainability score, configured internally at the benchmark's home organization with undisclosed thresholds and weights; if that score does not truly measure maintainability, the paper's quality axis and its orthogonality result collapse.

Editorial extensions

If this is right

  • A model that scores near-perfect on correctness can deliver solutions that time out on large inputs; correctness-only leaderboards therefore systematically overstate production readiness.
  • Benchmarks should report efficiency and quality as separate scores, because aggregating them into a single number hides the Claude-style case: strong quality, poor efficiency, high variance.
  • Model rankings change materially: O4-Mini-High's composite mean (92.3) is far above Claude Opus 4's (66.1), even though their median correctness scores are both 100.
  • Human-relative placement is also different: O4-Mini-High and Gemini 2.5 Pro sit near the 97th percentile of human contestants, while Claude Opus 4 sits near the 76.6th percentile, warning that correctness-only comparisons to humans also mislead.
  • Prompt conditioning can nudge behavior in the intended direction, especially for the weakest model, so efficiency and quality objectives could be targeted directly in prompt design and training rather than assumed to follow from correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because all three models score near 90 on quality, the quality axis may currently separate frontier models only on rare failure modes; the orthogonality result is more important for benchmark design than for ranking these three.
  • The human percentile comparison assumes human scores are normally distributed, but the paper itself reports skew of +1.36; recomputing percentiles with the empirical distribution could shift the ranks, especially in the tails.
  • A cheap check of the efficiency axis would be to measure how runtime grows with input size across a model's submissions; if the strict thresholds only separate timeouts from non-timeouts, they may be measuring constant factors rather than asymptotic complexity.
  • The natural extension is to multi-file or project-level tasks: if quality remains orthogonal to correctness there, the claim generalises from contest snippets to real codebases; if not, the orthogonality may be an artifact of short single-file solutions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces COMPASS, a multi-dimensional benchmark for code-generation evaluation, built from 50 Codility contest problems with 393,150 historical human submissions. Three frontier models—Claude Opus 4, Gemini 2.5 Pro, and O4-Mini-High—are evaluated on correctness, efficiency, and code quality under four prompt conditions and 64 samples per model-problem combination. The headline finding is that correctness alone is misleading: Claude Opus 4 reaches a mean correctness of 72.2% but an efficiency score of only 35.4%, while O4-Mini-High achieves 95.6% and 93.0%, respectively. The paper also reports correlation and PCA analyses to argue that quality is an independent axis, per-task human-percentile comparisons, within-model consistency statistics, and prompt-framing effects.

Significance. If the results hold, COMPASS is a valuable corrective to correctness-only code benchmarks and a useful resource: the human baseline dataset is large, the efficiency finding is concrete and would likely persist under any reasonable runtime-threshold policy, and the per-task tables provide a transparent view of model variability. The principal weakness is that the code-quality dimension is built on an undisclosed and unvalidated composite score, so the claim that maintainability constitutes an independent, meaningfully measured axis is not yet established. In addition, the prompt-analysis section contains a clear factual inconsistency with its own table. These issues are addressable, so I view them as requiring major revision rather than rejection.

major comments (3)
  1. [III.A.3 / V.A.2] The code quality dimension is scored by a composite from 'Codility's internal CodeScene configuration' without disclosure of the thresholds, weighting, sub-metric aggregation, or validation against human maintainability judgments. The PCA in Section V.A.2 then treats this composite as an established quality axis, concluding orthogonality from a PC2 loading of 0.990; without external validation, this conclusion is unsupported, especially since the quality scores are compressed in a narrow range (Table XIX, means 78-100, SDs mostly below 10). Please disclose and validate the configuration, or reframe the quality results as descriptive rather than as evidence of an independent dimension.
  2. [V.E / Table XII] The first paragraph of Section V.E states that 'correctness prompts produced slightly higher average correctness scores (87.535) with the lowest variability (SD = 27.891),' but Table XII reports a neutral-prompt correctness mean of 87.356 and a correctness-prompt mean of 87.235; the correctness prompt does not outperform the control on the mean. This factual inconsistency undermines the 'directionally consistent' summary and should be corrected.
  3. [IV.B / III.A.2] The efficiency dimension relies on 'strict runtime thresholds' that are not reported. Because the headline claim that Claude Opus 4 scores only 35.4% on efficiency is threshold-dependent, the paper should list the per-problem time limits (or at least the reference complexity and the chosen multiplier) so that readers can assess whether the threshold policy is reasonable and replicable.
minor comments (5)
  1. [V.C.1] The sentence ending 'Appendix B contains the per-task efficiency score statistics for each model' cites the wrong appendix; Appendix B reports correctness, while efficiency is in Appendix C. The cross-reference should be corrected.
  2. [Table XVI] The caption 'Human percentile baselines' is misleading; the table reports raw means, medians, and standard deviations, not percentiles. Rename to 'Human performance baselines.'
  3. [V.D / Eq. (2)] Equation (2) converts model scores to human percentiles using a normal CDF, but the human score distributions are markedly skewed (average skew +1.36, many zero scores). The text cautions about skewed/multimodal cases, but Table IX still reports the resulting percentiles without non-parametric alternatives; please provide empirical percentile ranks as a robustness check.
  4. [V.A.1] The reported sample size 'N = 3,200 per correlation' is inconsistent with the experimental design (50 problems × 3 models × 4 prompt types × 64 samples), and the paper should specify whether correlations are computed across samples within models, across task-level means, or across some other unit; otherwise the p-values are not interpretable.
  5. [IV.B.a / III.B] The sampling description says 'k = 64 independent samples per model–problem combination,' while the prompt-variation results imply that each combination was run under four different prompts. The relationship between the 64 samples and the four prompt conditions should be stated explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the multidimensional claims are empirical results from operationally defined metrics; the undisclosed CodeScene composite is a transparency concern, not a circular derivation.

full rationale

The paper's chain is empirical rather than derivational. Correctness is defined as the percentage of test cases passed (Eq. 1), efficiency as the percentage of large-input cases passing strict runtime thresholds derived from expert reference solutions, and quality as a CodeScene composite score using Codility's configuration. The central claim—that high correctness does not imply high efficiency or maintainability—is substantiated by measured scores (e.g., Claude Opus 4 at 72.2% correctness vs. 35.4% efficiency) and by correlation/PCA on those measured scores, not by an equation that makes the conclusion equal to an input. The PCA step is not circular because the quality dimension could have correlated strongly with correctness and efficiency; the observed near-zero correlations and PC2 loading of 0.990 are contingent empirical outcomes. The human-percentile transform (Z = (X - mu)/sigma, Eq. 2) is a standard rescaling of independently measured human baselines. The most serious limitation is that the CodeScene-Codility composite is not disclosed or externally validated, so the construct validity of 'quality' is not fully checkable; however, lack of transparency is a correctness/validity risk, not a circularity reduction. There are no load-bearing self-citations and no fitted parameters renamed as predictions.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical entities or fitted parameters in the physics sense; the main unstated inputs are the proprietary efficiency thresholds and CodeScene configuration, plus the distributional assumption for human percentiles.

free parameters (1)
  • Per-problem efficiency runtime thresholds = not disclosed
    Efficiency scores are binary pass/fail against strict runtime thresholds derived from expert reference solutions (Section III.A.2). These thresholds determine the efficiency dimension but are not published, so they act as hand-chosen parameters that control the main result.
assumptions (3)
  • domain assumption CodeScene quality score with Codility's internal configuration is a valid measure of code maintainability.
    Used in Section III.A.3 to compute the quality dimension; no external validation is provided.
  • domain assumption Human performance percentiles can be approximated with the normal CDF.
    Equation (2) converts z-scores to percentiles despite the authors noting human scores are skewed (Appendix A); this approximation can bias percentile ranks.
  • domain assumption 64 samples per model-problem is sufficient for stable estimates.
    Stated in Section IV.B.a without power analysis or clustering correction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models." pith.science (2026). https://pith.science/paper/3ATAVV3Z

@misc{pith2026250813757,
  author       = {Pith},
  title        = {Pith review of: COMPASS: A Multi-Dimensional Benchmark for Evaluating Code Generation in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3ATAVV3Z}},
  note         = {Machine review of arXiv:2508.13757}
}
read the original abstract

Current code generation benchmarks focus primarily on functional correctness while overlooking two critical aspects of real-world programming: algorithmic efficiency and code quality. We introduce COMPASS (COdility's Multi-dimensional Programming ASSessment), a comprehensive evaluation framework that assesses code generation across three dimensions: correctness, efficiency, and quality. COMPASS consists of 50 competitive programming problems from real Codility competitions, providing authentic human baselines from 393,150 submissions. Unlike existing benchmarks that treat algorithmically inefficient solutions identically to optimal ones provided they pass test cases, COMPASS systematically evaluates runtime efficiency and code quality using industry-standard analysis tools. Our evaluation of three leading reasoning-enhanced models, Anthropic Claude Opus 4, Google Gemini 2.5 Pro, and OpenAI O4-Mini-High, reveals that models achieving high correctness scores do not necessarily produce efficient algorithms or maintainable code. These findings highlight the importance of evaluating more than just correctness to truly understand the real-world capabilities of code generation models. COMPASS serves as a guiding framework, charting a path for future research toward AI systems that are robust, reliable, and ready for production use.

Figures

Figures reproduced from arXiv: 2508.13757 by the authors.

Figure 1
Figure 1. PCA explained variance across components. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of correctness scores by model across 50 tasks. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Distribution of code quality scores by model across 50 tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Distribution of composite scores by model across 50 tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 canonical work pages

  1. [1]

    Evaluating large language models trained on code,

    M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

  2. [2]

    Program synthesis with large language models,

    J. Austin et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  3. [3]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez et al., “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770 , 2023

    arXiv 2023

  4. [4]

    Hackerrank-astra: A benchmark for evaluating llms in coding competitions,

    Z. Xing et al., “Hackerrank-astra: A benchmark for evaluating llms in coding competitions,” arXiv preprint, 2025

    work page 2025

  5. [5]

    Managing technical debt with the sqale method,

    J.-L. Letouzey and M. Ilkiewicz, “Managing technical debt with the sqale method,” IEEE Software, vol. 29, no. 6, pp. 44–51, 2012

    work page 2012

  6. [6]

    A metrics suite for object oriented design,

    S. R. Chidamber and C. F. Kemerer, “A metrics suite for object oriented design,” IEEE Transactions on Software Engineering, vol. 20, no. 6, pp. 476–493, 1994

    work page 1994

  7. [7]

    Measuring coding challenge competence with apps,

    D. Hendrycks et al., “Measuring coding challenge competence with apps,” arXiv preprint arXiv:2105.09938 , 2021

    arXiv 2021

  8. [8]

    Ds-1000: A natural and reliable benchmark for data science code generation,

    Y . Lai et al., “Ds-1000: A natural and reliable benchmark for data science code generation,” arXiv preprint arXiv:2211.11501 , 2022

    arXiv 2022

Show all 9 references
  1. [9]

    Modeling the performance prediction problem in industrial and organizational psychology,

    J. P. Campbell, “Modeling the performance prediction problem in industrial and organizational psychology,” in Handbook of Industrial and Organizational Psychology, 2nd ed., M. D. Dunnette and L. M. Hough, Eds. Palo Alto, CA: Consulting Psychologists Press, 1990, vol. 1, pp. 687–732

    1990

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.