Pith. sign in

REVIEW 5 major objections 6 minor 3 cited by

Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility

T0 review · 5 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that nearly 70% of code benchmarks for LLMs skip data quality assurance, and offers a 55-criteria checklist to fix construction, evaluation, and release.

desk verdict Useful checklist and a large audit, but the prevalence statistics are not as solid as the packaging suggests — treat them as upper bounds on reported practice, not measured practice. read the letter →

arxiv 2501.10711 v5 pith:DXOIU5VT submitted 2025-01-18 cs.SE cs.AIcs.CL

Jialun Cao , Yuk-Kit Chan , Zixuan Ling , Wenxuan Wang , Shuqing Li , Mingwei Liu , Ruixi Qiao , Yuting Han
show 7 more authors
Chaozheng Wang Boxi Yu Pinjia He Shuai Wang Zibin Zheng Michael R. Lyu Shing-Chi Cheung
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords codebenchmarkslargelanguagemodelsbenchmarkqualitylifecyclereproducibilitytestcoveragedataassuranceHOW2BENCH
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code-related benchmarks are the yardsticks by which the community judges large language models' coding ability, and this paper's central claim is that the current yardsticks are largely uncalibrated. Profiling 274 code benchmarks released from 2014 to 2024, the authors find that almost 70% performed no data quality assurance, over 90% ignored code coverage when passing test cases was the oracle, more than half omitted prompts or environment details needed for reproduction, and over 10% were not fully open source. The remedy they propose is HOW2BENCH, a 55-criteria checklist covering the full benchmark lifecycle, and a human study with 49 researchers indicates that many builders are not aware these criteria matter. If the claim is right, many published LLM coding scores are built on tools whose quality is unmeasured, and the field needs the kind of standardised auditing the paper provides.

What carries the argument

The load-bearing object is HOW2BENCH, a 55-criteria checklist organised around a five-phase lifecycle of benchmark development: Phase 0 design, Phase 1 construction, Phase 2 evaluation, Phase 3 analysis, and Phase 4 release. It does two jobs: it tells future builders what to check at each stage (traceability, deduplication, denoising, test coverage, prompt validation, environment logging, licensing, and so on), and it gives auditors a uniform rubric with which to profile existing benchmarks. The profiling statistics are produced by applying this checklist to 274 benchmarks through manual review of papers and released artifacts.

What would settle it

Re-profile a random sample of the 274 benchmarks against a subset of the 55 criteria with independent coders and check whether the roughly 70% no-quality-assurance, 90% no-coverage, and 50% no-reproducibility figures reproduce; a large drop in those rates would undercut the paper's prevalence claims.

Watch

Extended reading notes

Core claim

The paper's central discovery is a systematic quality gap: across 274 code-related benchmarks released from 2014 to 2024, nearly 70% have no data quality assurance, over 90% ignore code coverage when passing tests is the oracle, more than half omit prompts or environment details needed for reproduction, and over 10% are closed or partially closed. Even highly cited benchmarks contain duplicated samples, incorrect reference solutions or tests, obscure result presentations, and leaked sensitive information. Because 18% of the benchmarks feed data into later benchmarks, these flaws can propagate. The paper argues that the path forward is HOW2BENCH, a 55-criteria checklist spanning design, construction, evaluation, analysis, and release, and reports a 49-participant study showing that awareness of these issues is often missing.

Load-bearing premise

The headline percentages rest on the authors' manual judgments about whether each of 274 benchmarks satisfies each of the 55 criteria, and on that sample standing in for code benchmarks as a whole.

Editorial extensions

If this is right

  • Benchmark scores from the surveyed corpus should be read as provisional until data-quality and coverage checks are reported.
  • Future code benchmark releases can be audited against the 55 criteria before adoption, making compliance a publishable artifact.
  • Because over 18% of benchmarks feed data into later benchmarks, fixing quality at the source propagates improvements downstream.
  • Reproducibility failures (missing prompts, environments, parameters) can be avoided by treating release-phase criteria as mandatory, not optional.
  • The human study suggests awareness gaps, so providing a checklist addresses a knowledge problem, not just an effort problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 18% inheritance rate among benchmarks is representative, a single low-quality source benchmark could inflate scores across a large descendant tree; measuring whether correcting a source's flawed tests changes descendant scores would test this directly.
  • The manual audit could be automated by applying the 55 criteria with an LLM profiler, which would let the same quality census be run continuously on newly released benchmarks.
  • Since most criteria are stated in task-neutral terms, the same audit protocol could be run on non-code benchmarks to see whether similar violation rates hold in math, QA, or multimodal evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes HOW2BENCH, a 55-criteria checklist for code-related benchmark development spanning design, construction, evaluation, analysis, and release. The authors report a profiling study of 274 code-related benchmarks published from 2014 to 2024, claim that nearly 70% of these benchmarks have no data quality assurance measures, that over 90% of test-case-based benchmarks do not report code coverage, that over half omit reproducibility-critical information such as prompts, and that over 10% are not fully open-source. They also present focused case studies of 30 highly cited or recent benchmarks, a propagation analysis showing 18% of benchmarks are reused as data sources, and a questionnaire study of 49 researchers on awareness of benchmark quality issues. The core position is that benchmark construction should prioritize rigor, reliability, and reproducibility.

Significance. If its prevalence statistics were auditable and reproducible, this paper would be an important contribution to LLM code evaluation practice. The HOW2BENCH checklist is concrete, lifecycle-oriented, and plausibly generalizable to other benchmark families. The paper's strongest evidence is qualitative and case-based: duplicated samples in MBPP, an incorrect ground-truth solution in HumanEval, wrong example tests in MBPP, API-key leakage in XSemPLR, and email leakage in CrossVul are concrete, independently checkable problems. The propagation graph (Figure 8) is a useful and underappreciated point, showing how low-quality source benchmarks can contaminate downstream evaluations. However, the headline percentages rest on manual coding of papers and artifacts, with no released coding sheets, no inter-annotator agreement, and no confidence intervals; the paper itself concedes in the Limitations section that the manual analysis 'could lead to oversight and discrepancies in the statistical results.' The study is a solid position paper with a usable checklist, but its central empirical claims need more evidence before they can be treated as established measurements.

major comments (5)
  1. [Section 3.2 Step 2 / Appendix A.3, Figures 19, 21, 27] The headline statistics conflate 'not mentioned' with 'not done'. For example, Figure 27 reports that 87.8% 'did not mention' test coverage, yet the Abstract states that 'over 90% did not consider code coverage'; Figure 19 reports that 'more than half' of benchmarks 'did not mention' deduplication, while Section 4.2 interprets this as 62% not deduplicating. A benchmark may deduplicate, measure coverage, or manually review data without documenting those steps in the scanned paper or repository. Because process-level criteria such as manual review (criterion 18) cannot be reliably inferred from absence of documentation, the reported rates of 67.9%, 8.7%, and 52.6% are not yet supported. I ask the authors to recode with explicit categories (done, not done, not reported/unknown), to report the 'not reported' category separately, to release the coding sheets and per-benchmark judgments, and to report inter-annotator agreement and confidence intervals.
  2. [Section 3.2 Step 2 / Limitations, first bullet] The sample of 274 benchmarks is obtained by snowballing, but the paper does not document the search sources, inclusion and exclusion criteria, screening process, or how the final set relates to the population of code-related benchmarks. The strong year-over-year growth, with 149 benchmarks assigned to 2024 alone, suggests the sample is not a uniform snapshot of a decade. Without a PRISMA-style flow or a clear sampling frame, claims such as 'over 90% of benchmarks' cannot be generalized beyond the collected set. The Limitations section acknowledges that 'the collection of code-related benchmarks may be incomplete'; this limitation should be reflected in the phrasing of every aggregate percentage, not only in the appendix.
  3. [Abstract vs. full text, Sections 1 and Appendix D] The Abstract states that this is 'a decade-scale (2014-2025) survey over 672 code benchmarks' and that 'in 2025 alone' the number of benchmarks ignoring code coverage nearly matches the previous decade's total, while the full text consistently reports 274 benchmarks and a 2014-2024 window. This is a direct factual inconsistency about the corpus size and time span. The authors should correct the Abstract and ensure every occurrence of the corpus size and date range agrees with the body of the paper.
  4. [Section 5 / Appendix B] The human study's awareness-gap claims are presented with strong quantifiers ('All participants concurred', 'over 40% were not aware') based on 49 self-selected respondents, mostly graduate students with at least one publication, recruited through the authors' academic networks. The paper reports no inferential statistics, confidence intervals, or analysis of nonresponse bias, and the participant pool is not described as representative of benchmark developers. The finding is suggestive and worth reporting, but the claims should be softened or supplemented with a margin of error and a discussion of selection bias. Minor: the demographics report '57.14%' for SE participants, an unjustified precision for a sample of 49; an integer percentage is more appropriate.
  5. [Section 3.2 Steps 2-3 / Data availability] The paper's own subject is reproducibility, yet the data underlying its main quantitative claims is not released: there is no benchmark list with per-criterion coding decisions, no coding protocol, no raw survey responses, and no analysis scripts. The abstract and introduction present precise numbers (67.9%, 8.7%, 52.6%, etc.) that cannot be audited without these artifacts. Releasing the coding data and a detailed protocol is a minimal requirement for a paper that argues other benchmarks should be reproducible, and it is needed before the prevalence statistics can be accepted.
minor comments (6)
  1. [Appendix A.1] The sentence 'In total, 724 programming languages are studied by these 274 benchmarks' should be rephrased to indicate that 724 is the total number of benchmark-language pairs, not the number of distinct languages.
  2. [Figure 8] The inheritance graph appears to be a screenshot with garbled labels and a local file path ('graph_sample.html') visible; this is not publication quality and should be redrawn with readable text.
  3. [Section 4.3] The phrase 'transparency and matriculated documents' should be 'transparency and meticulous documentation'.
  4. [Section 4.1 / Appendix A.1] The text refers to 'Defeat Detection' where 'Defect Detection' is intended; check for this typo in both places.
  5. [Figure 26] The heading 'Test Orcale' should be 'Test Oracle'.
  6. [Figure 7 / Appendix E, criterion 48] The criterion begins with a lowercase 'review' after the numbered item; this should be capitalized for consistency with the other items.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: HOW2BENCH is derived from literature and practitioner interviews, then applied to profile benchmarks; the few self-citations are not load-bearing.

full rationale

The paper's derivation chain is descriptive and non-circular. HOW2BENCH is constructed in Step 1 from literature (Suppes et al., 1962; Zheng et al., 2023b; Schaefer et al., 2024; Reuel et al., 2024) and stakeholder interviews, before any profiling occurs. Step 2 applies the finalized 55 criteria to the 274 collected benchmarks; the checklist is not fitted to the profiled data, and no quantity measured during profiling is used to define or retroactively adjust the criteria. Step 3's focused case study and Step 4's human study are separate validations of the checklist's usefulness, not derivations of the prevalence statistics. The paper's own Limitations section concedes that "the study involved substantial manual analysis, which could lead to oversight and discrepancies in the statistical results," which is a measurement-validity caveat rather than evidence that a result is defined in terms of its input. Self-citations such as JavaBench, CRUXEval-X, DomainEval, and Cao et al. 2024a/b appear only as benchmarks in the collection or as supporting prior observations; none is load-bearing for the central claim that most benchmarks lack quality-assurance, coverage, and reproducibility practices. There is no equation, fitted parameter, or imported uniqueness theorem whose output reduces to its input, so no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no fitted parameters and no new physical or conceptual entities whose existence needs external evidence. Its empirical claims rest on the representativeness of the benchmark sample and the accuracy of manual annotation, both acknowledged as limitations.

assumptions (2)
  • domain assumption The 274 collected benchmarks are representative of code-related benchmarks released between 2014 and 2024.
    The paper's prevalence statistics (e.g., nearly 70% lack quality assurance) generalize to the population only if the snowballing-based collection is unbiased; the paper itself notes the collection may be incomplete in the Limitations section.
  • domain assumption Manual profiling judgments about whether each benchmark satisfies each criterion are accurate.
    The statistics depend on binary coding decisions made manually by the authors; the paper acknowledges potential oversight and discrepancies in the Limitations section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility." pith.science (2026). https://pith.science/paper/DXOIU5VT

@misc{pith2026250110711,
  author       = {Pith},
  title        = {Pith review of: Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibility},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DXOIU5VT}},
  note         = {Machine review of arXiv:2501.10711}
}
read the original abstract

Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 672 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance.

Figures

Figures reproduced from arXiv: 2501.10711 by the authors.

Figure 1
Figure 1. Lifecycle of Benchmark Development mark evaluation, analysis, and release), and objec￾tives (scoring benchmarks vs. offering comprehen￾sive guidelines for future benchmark development). Additionally, the study in this paper was conducted on a much larger scale (24 vs. 274 benchmarks), statistically highlighting the prevalent issues in ex￾isting benchmarks. 3 Design 3.1 The Lifecycle of Benchmark Development Code-rel… view at source ↗
Figure 2
Figure 2. Workflow of study process 2024; Reuel et al., 2024) and brainstorming. After that, we refined the guidelines through a series of interviews with various stakeholders, including model developers and benchmark builders, allow￾ing for the addition, deletion, or modification of criteria based on expert feedback and practical in￾sights. This phase concludes with the finalization of our guidelines, HOW2BENCH. This detaile… view at source ↗
Figure 3
Figure 3. Guideline for Benchmark Design 4.1 Guideline for Benchmark Design Explanation – For benchmark design, we listed four essential criteria, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (33 more)
Figure 4
Figure 4. Figure 4: Guideline for Benchmark Construction specific checks to ensure the benchmark’s scope is strictly adhered to, such as making sure every data point falls within the targeted scope and that the data can cover all studied capabilities, domain knowledge, and application sce…
Figure 5
Figure 5. Figure 5: Guideline for Benchmark Evaluation ☞ Explanation – Guidelines for benchmark evaluation focus on the rigorousness and reliabil￾ity of the evaluation. HOW2BENCH provides 12 criteria for benchmark evaluation, as shown in Fig￾ure 5. It mainly focuses on the comprehensive e…
Figure 6
Figure 6. Figure 6: Guideline for Evaluation Analysis ☞ Explanation – The analysis of the experiment results is expected to be objective and comprehen￾sive, hopefully providing insights or actionable ad￾vice. So, we listed 10 criteria for the evaluation analysis phase, as shown in [PITH_…
Figure 7
Figure 7. Figure 7: Guideline for Benchmark Release ☞ Explanation – Finally, releasing a benchmark for open access also needs careful consideration. We offered 10 suggestions for this step, as shown in [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Relationships between Benchmarks A.1 Profile of Studied Benchmarks We first show the trend in the development of benchmarks from 2014 to 2024. As shown in Fig￾ure 9, the data shows a modest beginning, with only a handful of benchmarks created annually un￾til 2017. From…
Figure 10
Figure 10. Figure 10: Citation Distribution of Benchmarks Coding Task. Tasks like Code Summarization and Text2SQL are similarly significant, each with 25 and 22 benchmarks. These tasks focus on mak￾ing code more understandable and converting nat￾ural language queries into SQL queries. Othe…
Figure 11
Figure 11. Figure 11: Benchmark Distribution over Tasks Programming Languages [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 13
Figure 13. Figure 13: Benchmark Distribution over Natural Lan [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 15
Figure 15. Figure 15: Benchmark Distribution over Granularity [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]
Figure 14
Figure 14. Figure 14: Benchmark Distribution over Modal in Prompt Granularity. The code snippet in a code-related benchmark varies from statement-level (i.e., one line of code. For example, CoNaLa (Yin et al., 2018) and Math-QA (Amini et al., 2019)), function￾level (i.e., a function unit o…
Figure 16
Figure 16. Figure 16: Benchmark Distribution Over Capabilities [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 19
Figure 19. Figure 19: Benchmark Distribution over Deduplication [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: A Counterexample of Rule 16 from MBPP (Austin et al., 2021). The significance of data preprocessing, such as deduplication, is frequently overlooked by benchmark builders, leading to data duplication even in highly cited benchmarks. Data Quality Assurance. Ensuring da…
Figure 18
Figure 18. Figure 18: Benchmark Distribution Over Expected Ap [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 21
Figure 21. Figure 21: Benchmark Distribution over Quality Assur [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: An Example from MBPP (Austin et al., 2021) that failed to be executed. Data Contamination Resolution. Data con￾tamination (Golchin and Surdeanu, 2023; Cao et al., 2024b) threat has been widely discussed. A benchmark with contaminated data may yield overclaimed results…
Figure 27
Figure 27. Figure 27: Benchmark Distribution over Test Coverage [PITH_FULL_IMAGE:figures/full_fig_p027_27.png]
Figure 26
Figure 26. Figure 26: Benchmark Distribution over Test Orcale Code test coverage (Ivankovic et al. ´ , 2019), as a common oracle for code-related benchmarks, has been widely adopted to determine the output cor￾rectness. It should be considered if a benchmark uses test case passing as a cri…
Figure 29
Figure 29. Figure 29: Benchmark Distribution over LLM Experi￾mented Additionally, we listed the top-10 LLMs by the number of code-related benchmarks they have been evaluated, as shown in [PITH_FULL_IMAGE:figures/full_fig_p028_29.png]
Figure 32
Figure 32. Figure 32: Benchmark Distribution over Recording Ex [PITH_FULL_IMAGE:figures/full_fig_p028_32.png]
Figure 30
Figure 30. Figure 30: Top-10 Studied LLMs for Code-related Benchmarks Experiment Environments. The experimental environment (such as the operating system and hardware) is important for the reproduction of the experiment. However, [PITH_FULL_IMAGE:figures/full_fig_p028_30.png]
Figure 35
Figure 35. Figure 35: Benchmark Distribution over Repeating the [PITH_FULL_IMAGE:figures/full_fig_p029_35.png]
Figure 34
Figure 34. Figure 34: Benchmark Distribution Over Validation of [PITH_FULL_IMAGE:figures/full_fig_p029_34.png]
Figure 39
Figure 39. Figure 39: Availability of Prompts Logging Info Accessibility. Providing detailed logging information, including comprehensive ex￾perimental results, is essential for ensuring trans￾parency, verifiability, and reproducibility in bench￾marking research. However, as shown in [PIT…
Figure 38
Figure 38. Figure 38: Benchmark Data Availability Prompt Accessibility. Detailed prompts are es￾sential for ensuring the reproducibility and trans￾parency of code-related benchmarks. However, as shown in [PITH_FULL_IMAGE:figures/full_fig_p030_38.png]
Figure 42
Figure 42. Figure 42: Availability of Convenient Evaluation Inter [PITH_FULL_IMAGE:figures/full_fig_p031_42.png]
Figure 41
Figure 41. Figure 41: Availability of User Manual Convenient Evaluation Interface Availability. Providing convenient evaluation interfaces is es￾sential for enhancing the usability and accessibility of benchmarks, enabling researchers to easily re￾produce results and compare models. As sho…
Figure 44
Figure 44. Figure 44: reveals that 19.3% of benchmarks do not provide a license, limiting their usability and distribution. Encouragingly, 80.7% of benchmarks do include a license, but the lack of licensing in nearly one-fifth of the benchmarks raises concerns about widespread adoption and…
Figure 45
Figure 45. Figure 45: An Example of API Key Leakage in Bench￾mark Release from XSemPLR (Zhang et al., 2023b). CrossVul (Nikitopoulos et al., 2021), where per￾sonal names and email addresses were uninten￾tionally disclosed. Such leakage poses risks of unauthorized access and resource misuse…
Figure 46
Figure 46. Figure 46: An Example of Name & Email Leakage in Benchmark Release from CrossVul (Nikitopoulos et al., 2021). Usability. Clear and comprehensive documenta￾tion is crucial for ensuring the usability of bench￾marks, as poorly written instructions can signif￾icantly hinder adoption…
Figure 49
Figure 49. Figure 49: Geographical Distribution of Interviewees [PITH_FULL_IMAGE:figures/full_fig_p033_49.png]
Figure 50
Figure 50. Figure 50: Demography of Interviewees [PITH_FULL_IMAGE:figures/full_fig_p034_50.png]
Figure 52
Figure 52. Figure 52: Results of Human Study (Questions 29 - 55 [PITH_FULL_IMAGE:figures/full_fig_p034_52.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. To Add Is Machine, To Delete Is Human: Measuring and Mitigating Deletion Avoidance in LLM Code Editing

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Even test-passing LLM patches keep code developers deleted; a 200-task deletion-only benchmark and 0.7% post-training data show the behavior is measurable and learnable.

  2. Establishing Best Practices for Building Rigorous Agentic Benchmarks

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Agentic benchmarks frequently mis-grade agents, and the new ABC checklist helps identify and correct such errors in ten popular benchmarks.

  3. A Conceptual Framework for AI Capability Evaluations

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A descriptive conceptual framework with seven elements (target, task, subject, inputs, instance, measurement, result analysis) for systematizing analysis of AI capability evaluations.

Reference graph

Works this paper leans on

36 extracted references · 16 canonical work pages · cited by 3 Pith papers

  1. [4]

    In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019

    code2seq: Generating sequences from struc- tured representations of code. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenRe- view.net. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. 2019. MathQA: Towards interpretable math word prob...

    arXiv 2019

  2. [7]

    arXiv preprint arXiv:2401.03065

    Cruxeval: A benchmark for code reason- ing, understanding and execution. arXiv preprint arXiv:2401.03065. Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018. Deep code search. In Proceedings of the 40th Inter- national Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018, pages 933–944. ACM. Jiawei Guo, Ziming Li, Xuelin...

    arXiv 2018

  3. [8]

    Preprint, arXiv:2406.11927

    On the impacts of contexts on repository-level code generation. Preprint, arXiv:2406.11927. Patrick Haller, Jonas Golde, and Alan Akbik. 2024. PECC: problem extraction and coding challenges. In Proceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Re- sources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Tori...

    arXiv 2024

  4. [9]

    In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024

    Infiagent-dabench: Evaluating agents on data analysis tasks. In Forty-first International Confer- ence on Machine Learning, ICML 2024, Vienna, Aus- tria, July 21-27, 2024. OpenReview.net. Yang Hu, Umair Z. Ahmed, Sergey Mechtaev, Ben Leong, and Abhik Roychoudhury. 2019. Re- factoring based program repair applied to program- ming assignments. In 2019 34th ...

    arXiv 2024

  5. [15]

    Deep learning-based vulnerable function de- tection: A benchmark. In Information and Commu- nications Security - 21st International Conference, ICICS 2019, Beijing, China, December 15-17, 2019, Revised Selected Papers, volume 11999 of Lecture Notes in Computer Science, pages 219–232. Springer. Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Olivier Y . de Vel, ...

    arXiv 2019

  6. [16]

    did you miss my comment or what?

    Nl2type: inferring javascript function types from natural language information. In Proceedings of the 41st International Conference on Software En- gineering, ICSE 2019, Montreal, QC, Canada, May 25-31, 2019, pages 304–315. IEEE / ACM. Courtney Miller, Sophie Cohen, Daniel Klug, Bogdan Vasilescu, and Christian KaUstner. 2022. "did you miss my comment or w...

    arXiv 2019

  7. [19]

    In 3rd IEEE/ACM International Work- shop on Automated Program Repair, APR@ICSE 2022, Pittsburgh, PA, USA, May 19, 2022 , pages 69–75

    Can openai’s codex fix bugs?: An evaluation on quixbugs. In 3rd IEEE/ACM International Work- shop on Automated Program Repair, APR@ICSE 2022, Pittsburgh, PA, USA, May 19, 2022 , pages 69–75. IEEE. Julian Aron Prenner and Romain Robbes. 2023. Runbu- grun - an executable dataset for automated program repair. CoRR, abs/2304.01102. Ruizhong Qiu, Weiliang Will...

    arXiv 2022

  8. [20]

    Preprint, arXiv:2411.12990

    Betterbench: Assessing ai benchmarks, uncov- ering issues, and establishing best practices. Preprint, arXiv:2411.12990. Niklas Risse and Marcel Böhme. 2024. Uncovering the limits of machine learning for automatic vulnerabil- ity detection. In 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, Au- gust 14-16, 2024. USENIX Associat...

    arXiv 2024

Show all 36 references
  1. [21]

    ACM Comput

    Qa dataset explosion: A taxonomy of nlp resources for question answering and reading com- prehension. ACM Comput. Surv., 55(10). Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. In Advances ...

    2020

  2. [22]

    In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022

    Leveraging automated unit tests for unsuper- vised code translation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Rebecca L. Russell, Louis Y . Kim, Lei H. Hamilton, Tomo Lazovich, Jacob Harer, ...

    2022

  3. [23]

    In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore

    NLP evaluation in trouble: On the need to mea- sure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Sin- gapore. Association for Computational Linguistics. Irina Saparina and Mirella Lapata...

    2023 arXiv

  4. [24]

    In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    Learning performance-improving code edits. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024. Design2code: How far are we fr...

    2024 arXiv

  5. [26]

    Preprint, arXiv:1907.10701

    Benchmarking tpu, gpu, and cpu platforms for deep learning. Preprint, arXiv:1907.10701. Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023b. Mconala: A benchmark for code generation from multiple natural languages. In Findings of the Association for C...

    1907 arXiv

  6. [27]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H

    Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V . Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in ...

    2022

  7. [28]

    Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo

    OpenReview.net. Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. 2024a. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. CoRR, abs/2405.07990. Tongt...

    2024 arXiv

  8. [29]

    CoRR, abs/2408.13001

    Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution. CoRR, abs/2408.13001. Ankit Yadav, Himanshu Beniwal, and Mayank Singh. 2024a. Pythonsaga: Redefining the benchmark to evaluate code generating llms. In Findings of the Association for Computa...

    2024 arXiv

  9. [30]

    CoRR, abs/2406.06918

    Towards more realistic evaluation of llm-based code generation: an experimental study and beyond. CoRR, abs/2406.06918. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023a. Codeg...

    2021 arXiv

  10. [32]

    0 1000 2000 3000 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 Citations Figure 10: Citation Distribution of Benchmarks Coding Task

    From Figure 10, we can see a clear long- tail trend of the citations, from the highest 2735 (HumanEval (Chen et al., 2021a)) to the lowest 0. 0 1000 2000 3000 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 Citations Figure 10: Citation Distribution of Ben...

    2000

  11. [33]

    For example, Hu- manEval (Chen et al., 2021a) and MBPP (Austin et al., 2021)), class-level (i.e., a class with mul- tiple function units of code

    and Math-QA (Amini et al., 2019)), function- level (i.e., a function unit of code. For example, Hu- manEval (Chen et al., 2021a) and MBPP (Austin et al., 2021)), class-level (i.e., a class with mul- tiple function units of code. For example, Clas- sEval (Du et al., 2023b)) and...

    2019

  12. [35]

    "" if len(dict.keys()) == 0: return False else: state =

    that failed to be executed. Data Contamination Resolution. Data con- tamination (Golchin and Surdeanu, 2023; Cao et al., 2024b) threat has been widely discussed. A benchmark with contaminated data may yield overclaimed results, misleading the understand- ing of the LLMs’ capab...

    2023

  13. [36]

    assert upper_ctr('PYthon') == 1

    where the test is incorrect. It alerts us that both the quality of the test and the test adequacy (e.g., code coverage) should be considered. Wrong Example Tests { 'source_file': 'charlessutton@: Benchmark Questions Verification V2.ipynb', 'task_id’: 461, 'prompt': 'Write a py...

    2021

  14. [294]

    Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J

    ACM. Pengyu Nie, Rahul Banerjee, Junyi Jessy Li, Raymond J. Mooney, and Milos Gligoric. 2023. Learning deep semantics for test completion. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 2111–2123. IEEE...

    2023 arXiv

  15. [516]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E

    Association for Computational Linguistics. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of...

    2023 arXiv

  16. [1442]

    Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen

    IEEE. Mingsheng Jiao, Tingrui Yu, Xuan Li, Guanjie Qiu, Xiaodong Gu, and Beijun Shen. 2023. On the eval- uation of neural code translation: Taxonomy and benchmark. In 38th IEEE/ACM International Con- ference on Automated Software Engineering, ASE 2023, Luxembourg, September 11...

    2023 arXiv

  17. [1569]

    Wonseok Oh and Hakjoo Oh

    ACM. Wonseok Oh and Hakjoo Oh. 2022. Pyter: effective pro- gram repair for python type errors. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November ...

    2022 arXiv

  18. [2014]

    In 30th IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014, pages 476–480

    Towards a big data curated benchmark of inter-project code clones. In 30th IEEE International Conference on Software Maintenance and Evolution, Victoria, BC, Canada, September 29 - October 3, 2014, pages 476–480. IEEE Computer Society. Xiangru Tang, Bill Qian, Rick Gao, Jiakan...

    2014 arXiv

  19. [2017]

    CoRR, abs/1709.00103

    Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. Yaqin Zhou, Shangqing Liu, Jing Kai Siow, Xiaoning Du, and Yang Liu. 2019. Devign: Effective vul- nerability identification by learning comprehensive program semant...

    2019 arXiv

  20. [2018]

    Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li

    The Internet Society. Dianshu Liao, Shidong Pan, Xiaoyu Sun, Xiaoxue Ren, Qing Huang, Zhenchang Xing, Huan Jin, and Qinying Li. 2024. A 3-codgen: A repository-level code generation framework for code reuse with local- aware, global-aware, and third-party-library-aware. IEEE Tr...

    2024

  21. [2019]

    Juice: A large scale distantly supervised dataset for open domain context-based code gener- ation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing, EMNLP-IJCNL...

    2019

  22. [2020]

    In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020 , pages 26–38

    Building hierarchically disentangled lan- guage models for text generation with named enti- ties. In Proceedings of the 28th International Confer- ence on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020 , pages 26–38. International Commit...

    2020

  23. [2021]

    measure the ability of these models to synthesize short Python programs from natural language descriptions

    aims to “measure the ability of these models to synthesize short Python programs from natural language descriptions” for “entry-level program- mers”. As we can see from Figure 17, the prompt requires LLMs to “ Write a function to calculate the dogs’ years .” Simply from this d...

    2021

  24. [2022]

    Science, 378(6624):1092–1097

    Competition-level code generation with alpha- code. Science, 378(6624):1092–1097. Zehan Li, Jianfei Zhang, Chuantao Yin, Yuanxin Ouyang, and Wenge Rong. 2024e. Procqa: A large- scale community-based programming question an- swering dataset for code search. In Proceedings of th...

    2024 arXiv

  25. [2023]

    In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 1430–

    Impact of code language models on automated program repair. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pages 1430–

    2023

  26. [2024]

    O’Reilly Media, Inc

    Long code arena: a set of benchmarks for long-context code models. CoRR, abs/2406.11612. Jialun Cao, Zhiyong Chen, Jiarong Wu, Shing-Chi Che- ung, and Chang Xu. 2024a. Can AI beat under- graduates in entry-level java assignments? bench- marking large language models on javaben...

    2022 arXiv

  27. [5445]

    Lakshya A

    Association for Computational Linguistics. Lakshya A. Agrawal, Aditya Kanade, Navin Goyal, Shuvendu K. Lahiri, and Sriram K. Rajamani. 2023. Monitor-guided decoding of code lms with static anal- ysis of repository context. In Advances in Neural Information Processing Systems 3...

    2023

  28. [8474]

    Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D

    Association for Computational Linguistics. Ramakrishna Bairi, Atharv Sonwane, Aditya Kanade, Vageesh D. C., Arun Iyer, Suresh Parthasarathy, Sri- ram K. Rajamani, Balasubramanyan Ashok, and Shashank Shet. 2024. Codeplan: Repository-level coding using llms and planning. Proc. A...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.