Pith. sign in

REVIEW 1 major objections 6 minor 24 references

CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation

T0 review · 1 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CodeAssay claims that benchmark ground truth is a measurement instrument, not neutral scaffolding: auditing its own tasks changed 9.0% of correctness labels and widened the measured best-to-worst model spread from 11.9 to 23.7 percentage po

desk verdict The audit-effect measurement is the real contribution here—170/1890 label changes (9%) and a spread that widens from 11.9 to 23.7 points—and the paper treats it carefully; worth a serious referee, though the internal audit and 74.8% hidden-test mutation score are real limits. read the letter →

arxiv 2608.03535 v1 pith:KSLJQKRB submitted 2026-08-04 cs.SE

Shahbaz Siddeeq , Muhammad Waseem , Umar Subhan Malhi , Pekka Abrahamsson This is my paper · ORCID
classification cs.SE
keywords LLMcodegenerationbenchmarkauditinggroundtruthvalidationmutationtestinghiddentestsuitesfunctionalcorrectnesssoftwarequalitymetricssecurity-focusedprompting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a code-generation benchmark's reference solutions and tests are part of the measurement instrument, not neutral scaffolding. To show this, the authors build CodeAssay, a 185-task Python benchmark whose ground truth was independently audited before any model was scored, with tests split into public (shown to the model) and hidden (used for grading). Regrading fixed model outputs against the pre-audit and audited versions of the suite changed 170 of 1,890 correctness labels (9.0%) and increased the measured best-to-worst model spread from 11.9 to 23.7 percentage points, while aggregate accuracy stayed nearly the same. The intended consequence: benchmark scores should be treated as evidence that requires validation, not as ground truth.

What carries the argument

The load-bearing mechanism is the audit-regrading comparison: 1,890 stored model programs, generated before the audit from audited descriptions, are re-scored against the pre-audit and post-audit test suites, so any label change must come from the repaired ground truth rather than from new model output. Around this core, CodeAssay's construction combines a taxonomy-first task pool (185 Python tasks in ten software-engineering categories), an independent five-dimension audit of references, descriptions, tests, and realism, a deterministic alternating split of test functions into public and hidden subsets, and mutation testing that reports 82.6% mutation score for the full suites and 74.8% for

What would settle it

Take a widely used un-audited benchmark, apply the same audit protocol to its references and tests without changing the tasks, and regrade a fixed set of stored model outputs; if the best-to-worst spread changes by less than the 11.8-point shift CodeAssay reports, the audit effect is corpus-specific rather than general. Alternatively, strengthen CodeAssay's hidden suites until their mutation score exceeds 95% and rerun the seven models; if the 12-of-21 pairwise separations and the security-prompt outcome persist, the conclusions do not depend on the 74.8% hidden-suite mutation score.

Watch

Extended reading notes

Core claim

The central claim is that unaudited benchmark ground truth can distort model comparisons in ways that are invisible to anyone who accepts the benchmark as given. An independent audit of the authors' own tasks found defects across all rubric dimensions—wrong references, tests requiring unspecified behavior, weak tests, and a few reproduced security flaws—and repairing them changed 170 of 1,890 correctness labels when regrading a fixed set of stored model programs. The direction of the corrections is symmetric (83 programs flipped from pass to fail, 87 from fail to pass), so aggregate correctness barely moves, but the effect is not uniform across models: the best-to-worst spread widens from 11

Load-bearing premise

The load-bearing assumption is that passing the hidden test suite is a valid measure of functional correctness; the suite detects only 74.8% of artificially injected bugs, so if those missed cases reflect real usage, the reported model rankings and the security-prompt result could change.

Editorial extensions

If this is right

  • If ground truth is part of the measurement instrument, then any benchmark that has not been audited can report model gaps that are partly artifacts of its own errors; the 9.0% label change and widening spread are direct evidence.
  • Correctness labels should be read together with the mutation score of the hidden suite; a 74.8% score means the tests leave 969 mutants alive, so passing all hidden tests is a necessary, not sufficient, condition for functional correctness.
  • Quality comparisons should be restricted to tasks jointly solved by all configurations and reported per property; on the 120-task common set, no model is best on every measure, and style results invert when whitespace-only violations are excluded.
  • A general security-focused prompt cannot be assumed to improve security: it produced no significant correctness change and no consistent reduction in selected static-analysis findings, while increasing program length and cyclomatic complexity for all seven models.
  • Generation-and-repair evaluation must keep grading tests hidden from the repair loop, since some public-test-driven repairs improved public-test performance but lowered hidden-test correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the audit effect generalizes, then widely used un-audited benchmarks may harbor comparable or larger ground-truth error rates; applying the same audit protocol to them and regrading fixed model outputs would test this directly.
  • Because the hidden suite's mutation score is only 74.8%, the reported absolute correctness numbers (77.3–98.9%) are likely upper-bound estimates for 'passes all hidden tests' and may overstate true functional correctness; a stronger hidden suite might compress or rerank models.
  • The security-prompt finding is consistent with the view that prompt-level security instructions trade structural cost for unmeasured benefit, but the paper's own static analyzers are too weak to settle the benefit question; exploit-based testing with adversarial inputs per task would be a natural extension.
  • The audit's symmetric label flips (83 vs 87) obscure the fact that the two directions are not equally distributed across models; this is why aggregate accuracy stayed flat while spread changed—a caution against reporting only mean scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 6 minor

Summary. CodeAssay is a benchmark of 185 Python code-generation tasks built taxonomy-first from ten SWEBOK-informed categories, with author-written reference solutions, a deterministic public/hidden test split, mutation-based validation of the test suites, and selected code-quality metrics (style, cyclomatic complexity, static-analysis findings, length). The central empirical claim is that auditing the benchmark's ground truth changes evaluation outcomes: regrading a fixed set of stored model programs against the pre-audit and audited full suites changed 170 of 1,890 correctness labels (9.0%) and widened the measured best-to-worst model spread from 11.9 to 23.7 percentage points. Across seven proprietary LLMs under a standard prompt, hidden-test correctness ranges from 77.3% to 98.9%, with 12 of 21 model pairs separating after Holm-corrected exact McNemar tests. A security-focused prompt produced no significant correctness change for any model and no consistent reduction in the selected static-analysis findings, but it increased program length and cyclomatic complexity for all models. The paper also reports category-level variation and a detailed discussion of threats to validity, including the hidden suite's 74.8% mutation score, non-uniform temperature settings, and the internal nature of the audit.

Significance. If the findings hold, CodeAssay makes a valuable methodological contribution: it provides concrete, quantitative evidence that benchmark ground truth is part of the measurement instrument and that unaudited test suites can distort model rankings. The statistical protocol is a strength: exact McNemar tests, Cochran's Q, Holm correction within each family, and paired comparisons on the jointly solved task set. The authors are unusually transparent about limitations—mutation score, run-to-run variability, temperature non-uniformity, and the internal audit—and the replication package with stored outputs and scripts supports exact reproducibility. The benchmark itself is likely to be useful for researchers and practitioners who want a multi-metric, audited evaluation resource. The paper does not overclaim that mutation analysis or auditing establishes complete test adequacy, and it carefully scopes the security-related conclusions.

major comments (1)
  1. [Section 3.2.3 and Abstract] The regrading analysis that produces the headline numbers (170/1,890 label changes, 9.0%, spread 11.9→23.7 points) is explicitly based on the full test suite ('using the full suite in each case'), but the benchmark's primary metric is defined in Section 3.3 as correctness on the hidden tests. The abstract and conclusion present these regrading results without qualifying that they are full-suite grading effects, not hidden-test grading effects. Since the benchmark's final scores use the hidden subset, the audit's effect on the actual measured metric is not shown and could differ. Please either compute and report the same regrading analysis restricted to the hidden subsets (applying the Section 3.1.2 partition to the pre-audit suite as well as the audited suite) or prominently qualify the abstract/conclusion claims as describing full-suite grading effects.
minor comments (6)
  1. [Section 3.2.1 / Section 1 contributions] The contribution list says 'independently audited ground truth,' but the protocol describes a second author of the same team performing the audit. This is not independent in the usual external sense. Suggest rewording to 'second-author audited' or explicitly adding an external audit as future work.
  2. [Section 3.3] Typo: 'Style is Style is measured as' should read 'Style is measured as'.
  3. [Section 8] Missing period in the final paragraph: 'increasing cyclomatic complexity For practitioners' should be 'increasing cyclomatic complexity. For practitioners'.
  4. [Table 3] The table header formatting is confusing: 'Style /100 LLOC' followed by 'all excl. ws' is hard to parse. Please introduce a clearer two-row header or column labels such as 'Flake8 per 100 LLOC (all)' and 'Flake8 per 100 LLOC (excl. whitespace)'.
  5. [References] Reference [1] is just a URL with no title or repository name. Add a descriptive title, e.g., 'CodeAssay benchmark and replication package (GitHub repository)'.
  6. [Section 3.2.5] The contamination audit reports maximum observed similarities of 0.48 and 0.64 against the prespecified threshold of 0.70. It would be helpful to state explicitly that both are below the threshold and to comment on what the gap means (e.g., no candidate overlaps found).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: results are measurements against an audited instrument, not derivations from the instrument's own construction.

full rationale

The paper's central claims are empirical measurements: model correctness is evaluated against hidden tests, and the audit-effect analysis compares two versions of the test suites on the same stored programs. The pre-audit and audited grading are independent conditions; the programs were fixed, so the only change is the test suite. No parameter is fitted to the outcome it predicts, and no equation reduces to an input. The internal audit (same research group) is a threat to external validity, not circularity, and the paper explicitly scopes the regrading to the 135-task pre-audit snapshot. The public/hidden partition is deterministic and content-independent, mutation testing is an independent procedure, and no load-bearing argument relies on a self-citation. The reported limitations (e.g., mutation score, leakage audit, internal audit) are honestly stated and do not constitute circular reasoning.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the test-suite definition of correctness, the mutation-testing proxy, and standard statistics. The only hand-chosen numeric constants are the contamination similarity threshold and the public-test excerpt length; neither is fitted to data.

free parameters (2)
  • difflib similarity threshold = 0.70
    Hand-chosen cutoff for flagging candidate overlap with HumanEval and MBPP in the contamination audit (Section 3.2.5). It is a design choice, not fitted to data, and affects only the scope of the leakage claim.
  • public-test excerpt length = 80 lines (first 80 lines of public file)
    Protocol constant for what tests are shown to models during generation and repair (Section 4.2.2). Chosen by the authors; could influence repair behavior and hence final correctness.
assumptions (4)
  • domain assumption Passing the hidden test suite is a valid operationalization of functional correctness
    Every correctness label is defined by passing all hidden tests (Section 3.3). The benchmark excludes tasks whose behavior is not deterministically assertable. However, hidden-test mutation score is only 74.8%, so a quarter of injected faults survive.
  • domain assumption Mutation testing scores estimate test-suite fault-detection ability
    Paper cites Just et al. [11] for the correlation with real faults and reports mutation scores as lower bounds because equivalent mutants were not identified (Section 3.2.4).
  • standard math Statistical tests (Cochran's Q, exact McNemar, Wilcoxon signed-rank with Pratt) are valid for these repeated binary and paired measurements
    Used in Section 4.3; p-values are exact or asymptotic above 120 pairs.
  • domain assumption The SWEBOK v4 taxonomy is an appropriate basis for task category coverage
    Authors state the mapping is a positioning device rather than a validated classification (Section 7).

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation." pith.science (2026). https://pith.science/paper/KSLJQKRB

@misc{pith2026260803535,
  author       = {Pith},
  title        = {Pith review of: CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSLJQKRB}},
  note         = {Machine review of arXiv:2608.03535}
}
read the original abstract

Large Language Models are increasingly evaluated for code generation using test-based benchmarks. The validity of such evaluations depends on the reliability of their references and tests, while test-based correctness captures only part of the observable properties of generated code. We present CodeAssay, a taxonomy-first benchmark of 185 Python tasks across ten software-engineering categories. It combines audited ground truth, public tests for generation and repair, hidden tests for grading, mutation-based test-suite validation, and selected code-property measures. Regrading fixed model outputs after the audit changed 170 of 1,890 correctness labels (9.0%) and increased the measured best-to-worst model spread from 11.9 to 23.7 percentage points, although aggregate correctness remained nearly unchanged. The complete and hidden test suites achieved mutation scores of 82.6% and 74.8%, respectively. Across seven proprietary LLMs, standard-prompt correctness ranged from 77.3% to 98.9%, with significant differences in 12 of 21 model pairs. On the 120 tasks solved by all 14 model-prompt configurations, no model performed best across all selected code properties. A security-focused prompt produced no significant change in correctness or consistent reduction in the selected static-analysis findings, while increasing program length and cyclomatic complexity across all models. These findings show that reliable evaluation of LLM-generated code requires validated ground truth, protected tests, and multiple explicitly interpreted measures. CodeAssay provides a reproducible basis for evidence-based model evaluation in AI-augmented software development.

Figures

Figures reproduced from arXiv: 2608.03535 by the authors.

Figure 1
Figure 1. CodeAssay in three phases: benchmark design and audit, experimental setup, and results [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Per-model quality measures under the standard (solid) and security-focused (hatched) prompts, as means over the 120 jointly solved tasks. Lower is better; style uses a log axis in (a) and excludes whitespace-only codes in (b). only codes, nearly all W293, whitespace on a blank line, with a small W291 contribution from trailing whitespace on lines that carry code. Excluding those codes, the two models with the highes… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 21 canonical work pages

  1. [1]

    (2026), https://github.com/Code-Assay/CodeAssay

    work page 2026

  2. [2]

    arXiv preprint arXiv:2108.07732 (2021)

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  3. [3]

    In: 2025 IEEE/ACM 2nd International Conference on AI Foundation Models and Software Engineering (FORGE)

    Bruni, M., Gabrielli, F., Ghafari, M., Kropp, M.: Benchmarking prompt engineer- ing techniques for secure code generation with GPT models. In: 2025 IEEE/ACM 2nd International Conference on AI Foundation Models and Software Engineering (FORGE). pp. 93–103. IEEE (2025)

    work page 2025

  4. [4]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  5. [5]

    Biometrika 37(3–4), 256–266 (1950) 16 S

    Cochran, W.G.: The comparison of percentages in matched samples. Biometrika 37(3–4), 256–266 (1950) 16 S. Siddeeq et al

    work page 1950

  6. [6]

    In: Proc

    Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., Lou, Y.: ClassEval: A manually-crafted benchmark for evaluating LLMs on class-level code generation. In: Proc. 46th IEEE/ACM International Conference on Software Engineering (ICSE) (2024)

    work page 2024

  7. [7]

    ACM Transactions on Software Engineering and Methodology33(8), 1–79 (2024)

    Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., Wang, H.: Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology33(8), 1–79 (2024)

    work page 2024

  8. [8]

    IEEE Computer Society: Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), Version 4.0 (2024)

    work page 2024

Show all 24 references
  1. [9]

    In: Proc

    Inozemtseva, L., Holmes, R.: Coverage is not strongly correlated with test suite effectiveness. In: Proc. 36th International Conference on Software Engineering (ICSE). pp. 435–445. ACM (2014)

    2014

  2. [10]

    In: International Conference on Learning Representations (2025)

    Jain, N., Han, K., Gu, A., Li, W.D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., Stoica, I.: LiveCodeBench: Holistic and contamination free evalua- tion of large language models for code. In: International Conference on Learning Representations (2025)

    2025

  3. [11]

    22nd ACM SIG- SOFT International Symposium on Foundations of Software Engineering (FSE)

    Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G.: Are mu- tants a valid substitute for real faults in software testing? In: Proc. 22nd ACM SIG- SOFT International Symposium on Foundations of Software Engineering (FSE). pp. 654–665. ACM (2014)

    2014

  4. [12]

    In: Advances in Neural Information Processing Systems (NeurIPS)

    Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. In: Advances in Neural Information Processing Systems (NeurIPS). vol. 36, pp. 21558–21572 (2023)

    2023

  5. [13]

    IEEE Transactions on Software Engineering SE-2(4), 308–320 (1976)

    McCabe, T.J.: A complexity measure. IEEE Transactions on Software Engineering SE-2(4), 308–320 (1976)

    1976

  6. [14]

    In: Proc

    Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R.: Asleep at the key- board? Assessing the security of GitHub Copilot’s code contributions. In: Proc. IEEE Symposium on Security and Privacy (S&P). pp. 754–768 (2022)

    2022

  7. [15]

    2023 ACM SIGSAC Conference on Computer and Communications Security (CCS)

    Perry, N., Srivastava, M., Kumar, D., Boneh, D.: Do users write more insecure code with AI assistants? In: Proc. 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS). pp. 2785–2799 (2023)

    2023

  8. [16]

    In: Proc

    Riddell, M., Ni, A., Cohan, A.: Quantifying contamination in evaluating code gen- eration capabilities of language models. In: Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL). pp. 14116–14137 (2024)

    2024

  9. [17]

    In: Proc

    Siddiq, M.L., Santos, J.C.S.: SecurityEval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In: Proc. 1st Int. Workshop on Mining Software Repositories Applications for Privacy and Security (MSR4P&S). pp. 29–33. ACM (2022)

    2022

  10. [18]

    Journal of Systems and Software238, 112885 (2026)

    Sun, X., St ˚ ahl, D., Sandahl, K., Kessler, C.: Quality assurance of LLM-generated code: Addressing non-functional quality characteristics. Journal of Systems and Software238, 112885 (2026)

    2026

  11. [19]

    42nd International Conference on Machine Learning (ICML)

    Vero, M., M¨ undler, N., Chibotaru, V., Raychev, V., Baader, M., Jovanovi´ c, N., He, J., Vechev, M.: BaxBench: Can LLMs generate correct and secure backends? In: Proc. 42nd International Conference on Machine Learning (ICML). PMLR, vol. 267, pp. 61344–61390 (2025)

    2025

  12. [20]

    Biometrics Bulletin 1(6), 80–83 (1945) CodeAssay: A Multi-Metric, Audited Code-Generation Benchmark 17

    Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80–83 (1945) CodeAssay: A Multi-Metric, Audited Code-Generation Benchmark 17

    1945

  13. [21]

    In: Proc

    Xue, H., Aleithan, R., Enan, N., Uddin, G., Wang, S.: SWE-Bench+: Enhanced LLM coding benchmark. In: Proc. 3rd ACM International Conference on AI- Powered Software (AIware). pp. 332–339. ACM (2026)

    2026

  14. [22]

    In: Proc

    Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., Xie, T.: CoderEval: A benchmark of pragmatic code generation with genera- tive pre-trained models. In: Proc. 46th IEEE/ACM International Conference on Software Engineering (ICSE) (2024)

    2024

  15. [23]

    arXiv preprint arXiv:2407.11470 (2024)

    Zheng, J., Cao, B., Ma, Z., Pan, R., Lin, H., Lu, Y., Han, X., Sun, L.: Beyond correctness: Benchmarking multi-dimensional code generation for large language models. arXiv preprint arXiv:2407.11470 (2024)

    2024 arXiv

  16. [24]

    In: Proc

    Zhuo, T.Y., Vu, M.C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I.N.B., Zhan, H., He, J., Paul, I., et al.: BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In: Proc. International Con- ference on Learning Representations (...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.