{"id":"e3252897-4600-4e15-b39e-6282f3cd8bf0","arxiv_id":"2501.10711","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 55-criteria guideline and audit of 274 code benchmarks finds that most benchmarks skip data quality checks, prompting calls for more rigorous, reproducible benchmark construction.","lead":"This paper proposes a 55-point checklist called HOW2BENCH for building high-quality code benchmarks, and uses it to audit 274 code benchmarks from the past decade, finding that most skip basic quality checks. It matters because flawed benchmarks can distort claims about what AI coding models can actually do.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline percentages are built on treating 'not mentioned' as 'not done'; the coding protocol needs an independent audit before statistics like 'over 90% did not consider code coverage' can be trusted.","rationale":"The reader's weakest assumption was that the manual profiling judgments are reliable and the sample representative. My concern sharpens the first part: the protocol appears to code 'did not mention' as 'did not satisfy,' which is a systematic bias rather than random noise. This is the single most load-bearing issue because every headline percentage—roughly 70%, over 90%, over half, over 10%—is a direct output of that coding. The paper's own appendix language supports this reading, and the Limitations section concedes that the study involved substantial manual analysis that could lead to oversight and discrepancies in the statistical results. The paper deserves credit for a genuinely useful 55-criterion checklist, for concrete verified bad examples (MBPP duplicate tasks, HumanEval incorrect solution, leaked API key in XSemPLR), and for aligning with prior work such as Liu et al. 2023a; those items are not in question. What is in question is only the precision of the aggregate statistics. A small independent re-annotation with an explicit 'unknown' category would settle whether the percentages survive. Given that the central practical recommendation to use HOW2BENCH is unaffected and the qualitative direction is plausible, the reader's CONDITIONAL verdict remains the right call; no verdict change is needed.","tokens_in":43168,"tokens_out":6209,"duration_ms":63727,"concrete_test":"Independently re-annotate a random sample of 50 of the 274 benchmarks. Two coders, blind to the paper's results, label each of the 55 criteria as (a) explicit evidence of satisfaction in paper or repository, (b) explicit evidence of violation, or (c) unknown or not mentioned. Recompute the four headline statistics (data quality assurance, code coverage, reproducibility information, open-source status) twice: once with 'unknown' excluded and once with 'unknown' counted as violation. If the two estimates differ by more than 10 percentage points, or if Cohen's kappa between coders on the binary satisfied-versus-violation coding is below 0.6, the paper's percentages are not robust and should be re-reported with an explicit 'unknown' fraction. Releasing the original 274-benchmark coding sheets would allow the same check to be done directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing quantitative claims are produced by manual coding of whether each benchmark did or did not satisfy each criterion, but the paper's protocol repeatedly treats absence of documentation as absence of practice. Appendix A.3 makes this move explicitly: Figure 19 says 'only 38% benchmarks have deduplicated... More than half of them didn't mention this process'; Figure 27 says 'only 8.7% considered and reported test coverage explicitly in their papers, while 87.8% did not mention the test coverage'; Section 4.5 reports that prompts are 'not disclosed in 52.6%'. The headline 'over 90% did not consider code coverage' is therefore really 'over 90% did not report code coverage in the scanned paper or artifact.' A benchmark may perform deduplication or coverage analysis without documenting it, or may document it in a repository location the scan did not cover. Process-level criteria, such as whether data were manually reviewed, are especially hard to infer from a paper. No inter-annotator agreement is reported, and the coding sheets are not released, so the precision implied by statistics like 67.9%, 8.7%, and 52.6% cannot be audited. This does not undermine the qualitative direction—the paper's concrete bad cases and prior work support that many benchmarks are weak—but it does undermine the specific prevalence statistics that are the paper's central empirical contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HOW2BENCH, a 55-criteria checklist for code-related benchmark development spanning design, construction, evaluation, analysis, and release. The authors report a profiling study of 274 code-related benchmarks published from 2014 to 2024, claim that nearly 70% of these benchmarks have no data quality assurance measures, that over 90% of test-case-based benchmarks do not report code coverage, that over half omit reproducibility-critical information such as prompts, and that over 10% are not fully open-source. They also present focused case studies of 30 highly cited or recent benchmarks, a propagation analysis showing 18% of benchmarks are reused as data sources, and a questionnaire study of 49 researchers on awareness of benchmark quality issues. The core position is that benchmark construction should prioritize rigor, reliability, and reproducibility.","tokens_in":43399,"tokens_out":4401,"duration_ms":45990,"significance":"If its prevalence statistics were auditable and reproducible, this paper would be an important contribution to LLM code evaluation practice. The HOW2BENCH checklist is concrete, lifecycle-oriented, and plausibly generalizable to other benchmark families. The paper's strongest evidence is qualitative and case-based: duplicated samples in MBPP, an incorrect ground-truth solution in HumanEval, wrong example tests in MBPP, API-key leakage in XSemPLR, and email leakage in CrossVul are concrete, independently checkable problems. The propagation graph (Figure 8) is a useful and underappreciated point, showing how low-quality source benchmarks can contaminate downstream evaluations. However, the headline percentages rest on manual coding of papers and artifacts, with no released coding sheets, no inter-annotator agreement, and no confidence intervals; the paper itself concedes in the Limitations section that the manual analysis 'could lead to oversight and discrepancies in the statistical results.' The study is a solid position paper with a usable checklist, but its central empirical claims need more evidence before they can be treated as established measurements.","major_comments":[{"comment":"The headline statistics conflate 'not mentioned' with 'not done'. For example, Figure 27 reports that 87.8% 'did not mention' test coverage, yet the Abstract states that 'over 90% did not consider code coverage'; Figure 19 reports that 'more than half' of benchmarks 'did not mention' deduplication, while Section 4.2 interprets this as 62% not deduplicating. A benchmark may deduplicate, measure coverage, or manually review data without documenting those steps in the scanned paper or repository. Because process-level criteria such as manual review (criterion 18) cannot be reliably inferred from absence of documentation, the reported rates of 67.9%, 8.7%, and 52.6% are not yet supported. I ask the authors to recode with explicit categories (done, not done, not reported/unknown), to report the 'not reported' category separately, to release the coding sheets and per-benchmark judgments, and to report inter-annotator agreement and confidence intervals.","section":"Section 3.2 Step 2 / Appendix A.3, Figures 19, 21, 27"},{"comment":"The sample of 274 benchmarks is obtained by snowballing, but the paper does not document the search sources, inclusion and exclusion criteria, screening process, or how the final set relates to the population of code-related benchmarks. The strong year-over-year growth, with 149 benchmarks assigned to 2024 alone, suggests the sample is not a uniform snapshot of a decade. Without a PRISMA-style flow or a clear sampling frame, claims such as 'over 90% of benchmarks' cannot be generalized beyond the collected set. The Limitations section acknowledges that 'the collection of code-related benchmarks may be incomplete'; this limitation should be reflected in the phrasing of every aggregate percentage, not only in the appendix.","section":"Section 3.2 Step 2 / Limitations, first bullet"},{"comment":"The Abstract states that this is 'a decade-scale (2014-2025) survey over 672 code benchmarks' and that 'in 2025 alone' the number of benchmarks ignoring code coverage nearly matches the previous decade's total, while the full text consistently reports 274 benchmarks and a 2014-2024 window. This is a direct factual inconsistency about the corpus size and time span. The authors should correct the Abstract and ensure every occurrence of the corpus size and date range agrees with the body of the paper.","section":"Abstract vs. full text, Sections 1 and Appendix D"},{"comment":"The human study's awareness-gap claims are presented with strong quantifiers ('All participants concurred', 'over 40% were not aware') based on 49 self-selected respondents, mostly graduate students with at least one publication, recruited through the authors' academic networks. The paper reports no inferential statistics, confidence intervals, or analysis of nonresponse bias, and the participant pool is not described as representative of benchmark developers. The finding is suggestive and worth reporting, but the claims should be softened or supplemented with a margin of error and a discussion of selection bias. Minor: the demographics report '57.14%' for SE participants, an unjustified precision for a sample of 49; an integer percentage is more appropriate.","section":"Section 5 / Appendix B"},{"comment":"The paper's own subject is reproducibility, yet the data underlying its main quantitative claims is not released: there is no benchmark list with per-criterion coding decisions, no coding protocol, no raw survey responses, and no analysis scripts. The abstract and introduction present precise numbers (67.9%, 8.7%, 52.6%, etc.) that cannot be audited without these artifacts. Releasing the coding data and a detailed protocol is a minimal requirement for a paper that argues other benchmarks should be reproducible, and it is needed before the prevalence statistics can be accepted.","section":"Section 3.2 Steps 2-3 / Data availability"}],"minor_comments":[{"comment":"The sentence 'In total, 724 programming languages are studied by these 274 benchmarks' should be rephrased to indicate that 724 is the total number of benchmark-language pairs, not the number of distinct languages.","section":"Appendix A.1"},{"comment":"The inheritance graph appears to be a screenshot with garbled labels and a local file path ('graph_sample.html') visible; this is not publication quality and should be redrawn with readable text.","section":"Figure 8"},{"comment":"The phrase 'transparency and matriculated documents' should be 'transparency and meticulous documentation'.","section":"Section 4.3"},{"comment":"The text refers to 'Defeat Detection' where 'Defect Detection' is intended; check for this typo in both places.","section":"Section 4.1 / Appendix A.1"},{"comment":"The heading 'Test Orcale' should be 'Test Oracle'.","section":"Figure 26"},{"comment":"The criterion begins with a lowercase 'review' after the numbered item; this should be capitalized for consistency with the other items.","section":"Figure 7 / Appendix E, criterion 48"}],"recommendation":"major_revision","confidential_remarks":"The paper's direction is worthwhile and the checklist is likely to be useful to the community. The main risk is that the headline statistics, which are the paper's most visible contribution, are not presently auditable because they rest on unshipped manual coding that treats absence of documentation as absence of practice. The 672-versus-274 discrepancy between the abstract and the body is also concerning from an editorial standpoint and should be resolved before any resubmission. If the authors release their coding data, recode with an explicit 'not reported' category, and temper the prevalence claims accordingly, I would view the paper as suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the HOW2BENCH paper. The core deliverable — the 55-criteria checklist organized by lifecycle phase — is genuinely useful. It is a practical instrument that benchmark builders can actually apply, and the printable appendix matters. The audit of 274 benchmarks is also a real contribution; the concrete case studies (MBPP duplicated tasks and broken tests, HumanEval incorrect reference solution, API key leakage in XSemPLR) are specific and believable, and the propagation statistics are a nice observation.\n\nThe soft spots are where you'd expect. The headline percentages are built from manual coding that treats 'did not mention' as 'did not do.' Appendix A.3 says 'only 8.7% considered and reported test coverage explicitly... while 87.8% did not mention' — that is an admission that the number is about reporting, not practice. A benchmark could routinely deduplicate or check coverage without writing it in the paper. The protocol also lacks inter-annotator agreement, and the coding sheets are not shipped. So 'over 90% did not consider code coverage' should be 'over 90% did not report code coverage in the scanned paper or artifact.' That distinction matters because those percentages are the central empirical claim.\n\nAlso, the abstract says 672 benchmarks surveyed; the full text says 274. That needs fixing before publication. The human study is small (49 participants) and self-selected, but it is supplementary and the authors don't overclaim it.\n\nNone of this sinks the paper. The checklist stands on its own, the bad cases are independently checkable, and the directional story — that many code benchmarks are sloppier than they should be — is consistent with prior work. What needs work is the precision of the language and the transparency of the coding protocol.\n\nRecommendation: send it to review. It deserves referee time; just make sure the reviewers ask for the coding sheets, an inter-annotator agreement measure, and a rewording of the prevalence claims from 'did not' to 'did not report.' I'd bring it to our reading group — there's a good discussion here about what counts as evidence in meta-scientific audits.","headline":"Useful checklist and a large audit, but the prevalence statistics are not as solid as the packaging suggests — treat them as upper bounds on reported practice, not measured practice.","tokens_in":43980,"tokens_out":2427,"would_cite":true,"duration_ms":24159,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that nearly 70% of code benchmarks for LLMs skip data quality assurance, and offers a 55-criteria checklist to fix construction, evaluation, and release.","keywords":["code benchmarks","large language models","benchmark quality","benchmark lifecycle","reproducibility","test coverage","data quality assurance","HOW2BENCH"],"falsifier":"Re-profile a random sample of the 274 benchmarks against a subset of the 55 criteria with independent coders and check whether the roughly 70% no-quality-assurance, 90% no-coverage, and 50% no-reproducibility figures reproduce; a large drop in those rates would undercut the paper's prevalence claims.","tokens_in":42970,"feed_emoji":"🧪","tokens_out":5735,"duration_ms":52217,"temperature":0.7,"pith_summary":"Code-related benchmarks are the yardsticks by which the community judges large language models' coding ability, and this paper's central claim is that the current yardsticks are largely uncalibrated. Profiling 274 code benchmarks released from 2014 to 2024, the authors find that almost 70% performed no data quality assurance, over 90% ignored code coverage when passing test cases was the oracle, more than half omitted prompts or environment details needed for reproduction, and over 10% were not fully open source. The remedy they propose is HOW2BENCH, a 55-criteria checklist covering the full benchmark lifecycle, and a human study with 49 researchers indicates that many builders are not aware these criteria matter. If the claim is right, many published LLM coding scores are built on tools whose quality is unmeasured, and the field needs the kind of standardised auditing the paper provides.","feed_headline":"Nearly 70% of code benchmarks lack quality checks","feed_subtitle":"A 55-point checklist targets rigor, reliability, and reproducibility in LLM code evaluation.","key_machinery":"The load-bearing object is HOW2BENCH, a 55-criteria checklist organised around a five-phase lifecycle of benchmark development: Phase 0 design, Phase 1 construction, Phase 2 evaluation, Phase 3 analysis, and Phase 4 release. It does two jobs: it tells future builders what to check at each stage (traceability, deduplication, denoising, test coverage, prompt validation, environment logging, licensing, and so on), and it gives auditors a uniform rubric with which to profile existing benchmarks. The profiling statistics are produced by applying this checklist to 274 benchmarks through manual review of papers and released artifacts.","core_discovery":"The paper's central discovery is a systematic quality gap: across 274 code-related benchmarks released from 2014 to 2024, nearly 70% have no data quality assurance, over 90% ignore code coverage when passing tests is the oracle, more than half omit prompts or environment details needed for reproduction, and over 10% are closed or partially closed. Even highly cited benchmarks contain duplicated samples, incorrect reference solutions or tests, obscure result presentations, and leaked sensitive information. Because 18% of the benchmarks feed data into later benchmarks, these flaws can propagate. The paper argues that the path forward is HOW2BENCH, a 55-criteria checklist spanning design, construction, evaluation, analysis, and release, and reports a 49-participant study showing that awareness of these issues is often missing.","pith_inferences":["If the 18% inheritance rate among benchmarks is representative, a single low-quality source benchmark could inflate scores across a large descendant tree; measuring whether correcting a source's flawed tests changes descendant scores would test this directly.","The manual audit could be automated by applying the 55 criteria with an LLM profiler, which would let the same quality census be run continuously on newly released benchmarks.","Since most criteria are stated in task-neutral terms, the same audit protocol could be run on non-code benchmarks to see whether similar violation rates hold in math, QA, or multimodal evaluation."],"forward_implications":["Benchmark scores from the surveyed corpus should be read as provisional until data-quality and coverage checks are reported.","Future code benchmark releases can be audited against the 55 criteria before adoption, making compliance a publishable artifact.","Because over 18% of benchmarks feed data into later benchmarks, fixing quality at the source propagates improvements downstream.","Reproducibility failures (missing prompts, environments, parameters) can be avoided by treating release-phase criteria as mandatory, not optional.","The human study suggests awareness gaps, so providing a checklist addresses a knowledge problem, not just an effort problem."],"supporting_citations":[{"why":"HumanEval, the most-cited source benchmark and recurring case of flawed reference code and tests in the focused case study.","marker":"(Chen et al., 2021a)"},{"why":"MBPP, source of duplicated samples, out-of-scope prompts, and incorrect tests that the paper uses as examples.","marker":"(Austin et al., 2021)"},{"why":"Prior finding that HumanEval and MBPP contain bugs and insufficient test coverage, motivating the coverage criterion.","marker":"(Liu et al., 2023a)"},{"why":"Concurrent BetterBench assessment of AI benchmarks, contrasted with this paper's larger code-specific study.","marker":"(Reuel et al., 2024)"},{"why":"Classical measurement theory used to ground the analysis-phase criteria of difficulty, stability, and differentiability.","marker":"(Suppes et al., 1962)"},{"why":"The oracle problem in software testing, used to justify the test-quality and coverage criteria.","marker":"(Barr et al., 2014)"},{"why":"Code coverage practice at Google cited for why coverage matters in evaluation.","marker":"(Ivanković et al., 2019)"}],"fun_headline_variants":["Most code benchmarks fail basic quality checks","70% of code benchmarks cut quality corners","Code evals need a 55-point reality check","Rigor, reliability, reproducibility: the code benchmark fix","Why 9 in 10 code benchmarks ignore test coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline percentages rest on the authors' manual judgments about whether each of 274 benchmarks satisfies each of the 55 criteria, and on that sample standing in for code benchmarks as a whole.","fun_headline_variants_meta":{"raw":{"variants":["Most code benchmarks fail basic quality checks","70% of code benchmarks cut quality corners","Code evals need a 55-point reality check","Rigor, reliability, reproducibility: the code benchmark fix","Why 9 in 10 code benchmarks ignore test coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000328,"raw_usage":{"total_tokens":1792,"prompt_tokens":862,"completion_tokens":930,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":857}},"tokens_in":478,"tokens_out":930,"duration_ms":9076,"temperature":1.0,"reasoning_tokens":857,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:00:48.444826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-profile a random sample of the 274 benchmarks against a subset of the 55 criteria with independent coders and check whether the roughly 70% no-quality-assurance, 90% no-coverage, and 50% no-reproducibility figures reproduce; a large drop in those rates would undercut the paper's prevalence claims.","supporting_citations":[],"review_version":1}