Pith. sign in

REVIEW 3 major objections 4 minor 31 references

Spaghetti Architect mints oracle-verified code datasets with two independent difficulty axes and contamination-resistant re-minting.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:45 UTC pith:L7BINUWK

load-bearing objection A genuinely useful generator with refreshingly honest reporting; the only load-bearing flaw is that "correct by construction" is fixture-checked, not proven — a reframing, not a fatal problem. the 3 major comments →

arxiv 2607.18642 v1 pith:L7BINUWK submitted 2026-07-21 cs.LG cs.SE

Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

Yuxiang Ji This is my paper
classification cs.LG cs.SE
keywords data-centric machine learningcode dataset generationdata contaminationby-construction labelsmulti-language codebenchmark resourcedataset documentation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents Spaghetti Architect, a generator that creates code datasets for evaluating code models, with three properties mined corpora cannot offer. First, every instance is graded against a known-optimal clean reference—a simple JSON IR—so there is ground truth for correctness and quality. Second, difficulty is split into two orthogonal, independently controllable axes: intrinsic problem size and incidental presentation messiness. Third, the test split is re-minted from a private held-out seed, so scored items stay fresh after a model's training cutoff. The central claim is that this combination lets researchers attribute a model's failure to problem size versus surface mess, and detect memorization rather than assume it.

Core claim

Spaghetti Architect is an anti-optimization transpiler: given a clean, language-agnostic JSON intermediate representation (IR) built from four operation primitives, it renders each operation into deliberately redundant, fully-flattened programs in Python, JavaScript, Go, Java, and C++. Every emitted program is compiled, executed, and checked against a reference oracle, so the released instances are correct by construction. Because the clean IR serves as a known-optimal reference and messiness is dialed by a strictly-nested set of anti-pattern profiles (clean, minimal, light, standard, heavy, max), each instance carries two orthogonal by-construction labels: intrinsic difficulty (problem size

What carries the argument

The central mechanism is the anti-optimization transpiler pipeline: a Parser validating a typed JSON IR, a Planner selecting anti-pattern profiles from a data-driven database, five language Generators emitting idiomatic spaghetti code with always-on safety scaffolds, and a Validator that compiles, runs, and compares every output against a reference oracle. The two labelled axes—intrinsic size (N/L/W/T knobs) and incidental messiness (the nested clean⊂minimal⊂light⊂standard⊂heavy⊂max profile chain)—are carried through the IR and the profile database, making correctness and difficulty properties of the construction rather than of measurement.

Load-bearing premise

The central claim assumes that running the oracle on the fixed input fixtures shipped with each instance is enough to guarantee that the deliberately messy renderings behave identically to the clean reference on every possible input—if an injected transform diverges on any input outside those fixtures, the by-construction correctness label and the semantic-equivalence gate lose their guarantee.

What would settle it

Take a released max-profile program (e.g., the manual-index membership check with opaque predicate and try/except fallback) and run it on inputs external to the shipped fixtures—empty lists, duplicate targets, non-list collections, or mismatched types—and check whether the output always equals the clean IR's result; a single divergence, or a randomized differential test across many such inputs, would falsify the 'semantics are identical' claim.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the central claim holds, code-benchmark studies can finally separate 'model fails because the problem is bigger' from 'model fails because the code is messier,' a decomposition no static corpus offers.
  • The held-out re-mint protocol turns contamination from a hope into a measured quantity: public-split scores matching private fresh-minted scores within |Δ|≤0.012 makes memorization detectable rather than assumed.
  • The annotated-vs-unannotated ablation shows that a benchmark's own comments are not neutral: they inflate weak models more than strong ones, so any leaderboard built on annotated prompts overstates the ladder's resolution.
  • The intrinsic knob provides a tunable arithmetic-aggregation probe that floors even the strongest tested model by width W≥64, giving a controlled, regenerable probe for reasoning limits.
  • The construction offers a known-optimal reference for refactoring, enabling semantic-equivalence-gated quality scoring with a clean ceiling (0.83 Python) and a non-trivial floor, instead of grader-judged approximations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same generator could be pointed at other semantics (floats, strings, richer control flow) once cross-language oracle equivalence is handled, extending the controlled stimulus to a wider class of reasoning probes.
  • The two-axis design invites an adaptive-testing protocol: mint items inside a model's transition region (e.g., W between 8 and 64 for agg_stats) rather than past a ceiling every model fails, yielding sharper difficulty estimates.
  • The differential-annotation result is a general caution for synthetic benchmark builders: any self-documenting output (comments, docstrings, even descriptive variable names) can differentially inflate weak models and distort leaderboard resolution, not just in code.
  • The private-seed re-mint protocol could become a standard for other procedurally-generated benchmarks: making test items regenerable from a held-out seed is a general antidote to training-set leakage across modalities.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents Spaghetti Architect, an open-source generator that synthesizes code datasets from a clean, language-agnostic JSON IR. An anti-optimization transpiler renders each IR into deliberately redundant, fully flattened programs in Python, JavaScript, Go, Java, and C++, at multiple 'messiness' levels defined by strictly nested sets of SPAGH_* anti-pattern transforms. A validator compiles, runs, and compares each emitted program against a reference oracle on shipped input fixtures. The dataset labels each instance on two orthogonal axes—intrinsic problem size (N/L/W/T) and incidental presentation (the messiness knob)—and ships a contamination protocol with a public dev split, a private held-out seed, a canary GUID, and A/B/C regeneration tiers. Reference baselines on four LLMs show comprehension exact match rising with model scale, intrinsic scale collapsing arithmetic-aggregation accuracy, and an annotation ablation in which the generator's own comments differentially help weaker models on refactoring. Construct-validity evidence from static complexity and readability metrics is reported with explicit caveats.

Significance. If the central guarantees held, Spaghetti Architect would be a genuinely useful resource: a regenerable, fully labelled, contamination-controlled, multi-language code dataset with a known-optimal reference and two independently controllable difficulty axes—precisely what mined corpora cannot supply. The paper is unusually honest: it reports the inert messiness rung, the weak readability signal (ρ=0.39), the unstable rung-separation statistic, the uncontrolled time gap in the annotation ablation, and the partly tautological nature of the complexity-metric agreement. The artifact is open source, deterministic, and includes non-learned baselines, a canary, and a datasheet. However, the central 'correct by construction' and 'semantics are identical' claims are currently supported only by finite fixture-level oracle checks, not by a semantic-equivalence argument. This gap is load-bearing for the incidental-messiness axis and the refactoring equivalence gate, and must be addressed before the resource's core labelling claims are valid.

major comments (3)
  1. [Abstract, §2.1, §2.2, Listing 2] The claims 'correct by construction' (Abstract) and 'The semantics are identical' (§2.2) are stronger than what the validator establishes. §2.1 says every target is 'compiled, run, and checked against the oracle'; this is a finite check on the shipped fixtures, not a proof of behavior preservation over all valid inputs. The always-on try/except fallback in Listing 2 silently converts any unexpected exception into `is_found = False`; a model rewrite that diverges only off-fixture would also be falsely graded `semantic_ok` by the refactoring gate. Because the incidental axis is defined as 'presentation at fixed semantics' (§2.3), and because refactoring equivalence is graded by the same oracle, the labels and the gate both presuppose universal equivalence. Please either (a) prove—or at least carefully argue from a precise semantic model of the IR and each SPAGH_* transform—that every trans
  2. [§4, Table 6] The headline annotation-ablation result—that the generator's comments inflate the weakest model by an order of magnitude more than the strongest (−0.173 vs −0.017)—rests on a comparison with an uncontrolled two-week time gap between the annotated and unannotated arms, as the paper itself notes. The k=1 re-grading argument in the Table 6 caption is an indirect justification, not a contemporaneous paired replication. Since this finding is highlighted in the Abstract ('A benchmark's own commentary is not free'), the claim should either be strengthened by re-running the annotated arm at the same time as the unannotated one, or explicitly presented as suggestive rather than conclusive in the abstract and conclusions. The paper's own caveats are a credit, but they do not remove the confounding.
  3. [§3, Tables 2–3] The construct-validity section is transparent, but the evidence is weaker than the phrase 'demonstrated metric-mover' in the Abstract suggests. The complexity-metric agreement is partly tautological (the knob injects the constructs that cyclomatic and cognitive complexity count), and the less tautological readability signal is ρ=0.39 with a CI clearly below the complexity metrics. The paper already states this, and it is appropriately scoped. The concern is that the resource's 'by-construction quality order' is, as it stands, a definitional ordering of transform inclusion, not an independently validated quality order; the paper's own plan for human calibration is the right remedy, but until it is run, the 'quality' wording in the Abstract and §1 should be softened further.
minor comments (4)
  1. [§2.2] The relation between 'six transform sets' (clean, minimal, light, standard, heavy, max) and 'five distinct rungs' is confusing, especially since Table 3 reports 'five engine profiles' excluding clean. A small table listing which transforms are active at each rung, and which rungs are byte-identical, would help.
  2. [§4, Table 7] The caption reports 'Per-cell n=15 items for W∈{8,64,128} and n=90 for W∈{16,32}' with 'each over k=8 draws'. This is puzzling: if n is the number of items, the W=16/32 cells have six times more items than W=8/64/128; if n includes draws, the wording is inconsistent. Please clarify the counting.
  3. [§4, Table 4 caption] The phrase 'each item scores the fraction of its unordered candidate pairs ranked correctly (an exact complexity tie is credited 0.5)' is slightly self-contradictory: a tie is not 'ranked correctly'. Suggest 'correctly ordered, with a tie credited 0.5'.
  4. [Listing 2 comment] The comment 'manual index loop instead of ‘in‘' uses curly quotes around `in`; use straight formatting for code identifiers.

Circularity Check

0 steps flagged

No significant circularity: the generator's labels and results are either definitional by explicit design (and disclosed as such) or empirically measured; the fixture-only oracle gap is a correctness/verification issue, not a circular reduction.

full rationale

The paper's chain of claims does not reduce to its own inputs. The by-construction labels are generator settings, not fitted parameters: incidental messiness is literally defined as a strictly-nested anti-pattern inclusion chain, and intrinsic size is the generator's own N/L/W/T knob. The paper never fits a parameter to a subset and then 'predicts' a closely related quantity. The main reported numbers are measured baseline scores on a fixed four-model panel, with no fitted values reused as predictions. The places where the argument is definitional are explicitly identified and discounted by the authors. In Section 2.3 the fully-crossed design forces the messiness-vs-size Spearman correlation to \rho=0.00 'by construction rather than by measurement,' and the paper says it is reported 'as a consistency check on the design, not as evidence about the data.' In Section 3 the authors concede that the complexity-metric agreement is 'partly tautological: the incidental knob is engineered to inject redundant control flow, dead branches, and nested scaffolding, which is what cyclomatic and cognitive complexity count,' and they therefore treat the less-tautological Buse-Weimer surface-density proxy (\rho=+0.39) as their conservative quality bound rather than the larger complexity correlations. In Section 4 they explicitly refuse to report the non-Python uniform_clean_ceiling because the recovered fraction is '(s-c)/(s-c)=1 by algebraic identity,' calling it 'a statement about the metric’s definition, not a measurement.' These are honest disclosures of definitional artifacts, not hidden circular steps. There is no load-bearing self-citation: the cited prior work (Collberg-Thomborson, Fowler, Buse-Weimer, McCabe, etc.) is external, and no uniqueness claim is imported from the same authors' prior work. The paper's 'to our knowledge no prior generator delivers...' assertion is a literature claim, not a mathematical uniqueness theorem. The one substantive epistemic gap is the oracle checker. Section 2.1 claims 'Provable correctness. Every target is compiled, run, and checked against the oracle,' while Section 2.2 concludes 'The semantics are identical; only the incidental complexity differs.' The validator runs each emitted program on the shipped input fixtures only, so it establishes finite input/output agreement, not universal semantic equivalence over all valid inputs. This is an omitted equivalence proof / correctness overclaim. It is not, however, circular: the pa

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 4 invented entities

This is a resource paper, so its 'free parameters' are hand-chosen generator settings rather than fitted values. The load-bearing assumptions are the fixture-relative meaning of semantic equivalence and the tool-relative meaning of 'known optimal'; both are explicit and partly acknowledged by the authors.

free parameters (4)
  • Public generation seed = 20260619
    Hard-coded seed used to draw all RNG literals in the public dev split; arbitrary but fixed for reproducibility.
  • Intrinsic scale ranges (N/L/W/T) = W up to 160; other family ranges as shipped
    Hand-chosen knob ranges for the difficulty sweep, not fitted; their values affect where models collapse (e.g., W=64) and are design choices.
  • Messiness profiles (5 effective rungs) = minimal < light=standard < heavy < max (plus clean for Python)
    Hand-designed anti-pattern composition; one rung is inert and disclosed in Section 3, so the ladder has five distinct rungs on the released engine.
  • Input variants v0-v4 = 5 variants
    Number of fixture perturbations chosen by hand for differential grading.
axioms (4)
  • domain assumption Oracle execution on the shipped input fixtures establishes semantic equivalence of the rendered program to the clean IR.
    The validator runs each target only on the fixed inputs (Section 2.1), yet Section 2.2 states 'the semantics are identical'; equivalence over all possible inputs is not proven.
  • ad hoc to paper The tool's own clean IR is the 'known-optimal reference' for each instance.
    The IR is the generator's canonical form, not a proven minimal program; 'known-optimal' is defined relative to the tool's own design (Section 2.3).
  • domain assumption Strict set inclusion of anti-pattern profiles is a quality order (more nested transforms imply less maintainability).
    Per Sections 2.3 and 3, the messiness ladder is assumed a priori to be a maintainability order; the paper explicitly notes no human calibration is provided.
  • domain assumption The private seed (BENCH_HELDOUT_SEED) remains secret.
    The contamination protocol depends on the seed being never-committed (Section 2.4); the paper discloses the current test literals are recoverable from released completions, so only future re-mints are protected.
invented entities (4)
  • SPAGH_* anti-pattern transforms (11 transforms) independent evidence
    purpose: Inject deliberately messy but behavior-preserving constructs
    Implemented in the released open-source tool; visible in emitted programs and configurable via profiles.
  • Canary GUID (spaghetti-architect-bench:canary:9f1d4e7a-...) independent evidence
    purpose: Audit probe for training-set contamination
    Embedded in the public release so an auditor can test whether a model reproduces it.
  • Graded contamination-resistance tiers A/B/C independent evidence
    purpose: Regenerable held-out splits against verbatim, structural, and scale/depth memorization
    Shapes are public in the generator; effectiveness is measured via dev/test agreement and operation-count standardization.
  • Private held-out seed (BENCH_HELDOUT_SEED) no independent evidence
    purpose: Enables minting fresh unseen test splits after training cutoffs
    Secrecy is a promise; the paper discloses that current test literals are recoverable from committed completions, and the seed itself is never released.

pith-pipeline@v1.3.0-alltime-deepseek · 25824 in / 14026 out tokens · 117121 ms · 2026-08-01T14:45:17.329293+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator." pith.science (2026). https://pith.science/paper/L7BINUWK

@misc{pith2026260718642,
  author       = {Pith},
  title        = {Pith review of: Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7BINUWK}},
  note         = {Machine review of arXiv:2607.18642}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Mined code corpora are abundant but uncontrolled: a snippet's semantics, surface "messiness," and difficulty are whatever the wild contained; there is no known-optimal reference to grade against; and any public sample may already sit in a model's training set. We present Spaghetti Architect, a tool that mints code datasets with the control such corpora lack. An anti-optimization transpiler maps a clean, language-agnostic JSON intermediate representation to deliberately redundant, fully-flattened programs in five languages (Python, JavaScript, Go, Java, C++); every program is compiled, run, and checked against a reference oracle, so each instance is correct by construction. The clean IR is a known-optimal reference, messiness is dialed by strictly-nested anti-pattern profiles, each instance is labelled along two orthogonal difficulty axes, intrinsic (problem size) and incidental (presentation at fixed semantics), and contamination is resisted by minting fresh variants from a private held-out seed. We give construct-validity evidence that the quality order moves established complexity and readability metrics, and report baselines on a four-model open ladder: exact match rises with scale, and the intrinsic knob collapses arithmetic-aggregation accuracy of even the strongest model to zero. Further, development-set scores equal freshly re-minted held-out counterparts within $|\Delta|\le 0.012$ (comprehension) and $\le 0.011$ (refactoring); on identical programs, refactoring equivalence ($0.73 \rightarrow 0.99$) is scale-invariant while output prediction collapses; and ablating the generator's self-annotations shows they inflate the weakest model an order of magnitude more than the strongest ($-0.173$ vs $-0.017$): the annotated ladder resolves one of three adjacent pairs where the unannotated resolves all three. Open source (MIT), dependency-free, archived under a persistent DOI.

Figures

Figures reproduced from arXiv: 2607.18642 by Yuxiang Ji.

Figure 1
Figure 1. Figure 1: The Spaghetti Architect pipeline. A clean, language-agnostic JSON IR, built from four operation primitives (MEMBERSHIP_CHECK, KEY_VALUE_LOOKUP, AGGREGATE, CONDITIONAL_SELECT) and serving as the known-optimal reference, is validated by the Parser into a typed model, so a malformed IR can never reach a generator. The Planner then selects an anti-pattern profile from a data-driven database via the nested six-… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 2 canonical work pages

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models.arXiv preprint arXiv:2108.07732,

    Pith/arXiv arXiv

  2. [6]

    Maintenance.The dataset is maintained by the author via the GitHub repository (issues / pull requests); contact is the corresponding author inCITATION.cff

    A persistent Zenodo DOI is assigned to the tagged release; the concept DOIhttps://doi.org/10.5281/zenodo.21033174always resolves to the latest version. Maintenance.The dataset is maintained by the author via the GitHub repository (issues / pull requests); contact is the corresponding author inCITATION.cff. It is versioned (dataset_versionin the manifest; ...

    work page doi:10.5281/zenodo.21033174always

  3. [8]

    Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation bench- marks

    Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation bench- marks. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5075–5084,

    2023

  4. [9]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica

    doi: 10.18653/v1/2023.emnlp-main.308. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and con- tamination free evaluation of large language models for code. InInternational Conference on Learning Representations (ICLR),

    doi:10.18653/v1/2023.emnlp-main.308 2023

  5. [10]

    Martin Fowler.Refactoring: Improving the Design of Existing Code

    1109/C-M.1978.218136. Martin Fowler.Refactoring: Improving the Design of Existing Code. Addison-Wesley, 2nd edition,

    arXiv 1978

  6. [14]

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar

    arXiv:2306.09479. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models. InInternational Conference on Learning Representa- tions (ICLR),

    Pith/arXiv arXiv

  7. [15]

    Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru

    arXiv:2410.05229. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. InConference on Fairness, Accountability, and Transparency (FAT*), pages 220–229,

    Pith/arXiv arXiv

  8. [17]

    Ruchir Puri, David S

    arXiv:2402.14992. Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, SaurabhPujar, ShyamRamji, UlrichFinkler, SusanMalaika, andFrederickReiss. CodeNet: A large-scale AI for code dataset for learning a diversity of coding tasks. I...

    Pith/arXiv arXiv

  9. [19]

    Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample

    doi: 10.18653/v1/2024.acl-long.761. Baptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. Un- supervised translation of programming languages. InAdvances in Neural Information Processing Systems (NeurIPS),

    work page doi:10.18653/v1/2024.acl-long.761 2024

  10. [21]

    Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk

    doi: 10.18653/v1/2023.findings-emnlp.722. Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. A com- prehensive model for code readability.Journal of Software: Evolution and Process, 30(6),

    doi:10.18653/v1/2023.findings-emnlp.722 2023

  11. [22]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer

    doi: 10.1002/smr.1958. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. InInternational Conference on Learning Representations (ICLR),

    doi:10.1002/smr.1958 1958

  12. [24]

    doi: 10.18653/v1/2023.acl-long

    doi:10.18653/v1/2023.acl-long 2023

  13. [25]

    Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov

    Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks.arXiv preprint arXiv:1502.05698,

    Pith/arXiv arXiv

  14. [26]

    Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang

    Spotlight; arXiv:2406.19314. Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. Top leaderboard ranking=top cod- ing proficiency, always? EvoEval: Evolving coding benchmarks via LLM. InConference on Language Modeling (COLM),

    Pith/arXiv arXiv

  15. [27]

    CodeTransOcean: A comprehensivemultilingualbenchmarkforcodetranslation

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. CodeTransOcean: A comprehensivemultilingualbenchmarkforcodetranslation. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 5067–5089,

    2023

  16. [28]

    Zhou Yang, Jieke Shi, Junda He, and David Lo

    doi: 10.18653/v1/ 2023.findings-emnlp.337. Zhou Yang, Jieke Shi, Junda He, and David Lo. Natural attack for pre-trained models of code. InProceedings of the 44th International Conference on Software Engineering (ICSE), pages 1482–1493,

    doi:10.18653/v1/ 2023

  17. [29]

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie

    doi: 10.1145/3510003.3510146. Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. DyVal: Dynamic evaluation of large language models for reasoning tasks. InInternational Conference on Learning Representations (ICLR),

    arXiv

  18. [30]

    Datasheets for Datasets

    Appendix A. Datasheet We follow the “Datasheets for Datasets” template of Gebru et al. (2021). The answers describe the publicdevsplit frozen inbench/data/dev/and indexed by bench/data/manifest.json(dataset version2.0). Motivation.The dataset was created to evaluate and train code models under conditions that mined corpora cannot provide: aknown-optimal r...

    2021

  19. [1976]

    doi: 10.1109/TSE.1976.233837. Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse scaling: When bigger isn’t better.Transactions on Machine Learning Research (TMLR),

    arXiv 1976

  20. [2010]

    doi: 10.1109/TSE.2009.70. G. Ann Campbell. Cognitive complexity: An overview and evaluation. InProceedings of the 2018 International Conference on Technical Debt (TechDebt), pages 57–58,

    doi:10.1109/tse.2009.70 2009

  21. [2011]

    doi: 10.1109/TSE. 2010.62. 27 Ji Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zit- nick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. InIEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 1988–1997,

    doi:10.1109/tse 2010

  22. [2014]

    doi: 10.1109/ICSME.2014.77. Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Kr- ishna Ramanathan, Dan Roth, and Bing Xiang. ReCode: Robustness evaluation of code generation models. InProceedings of the 61st Annual Meeting of the Association fo...

    doi:10.1109/icsme.2014.77 2014

  23. [2017]

    Brenden M

    doi: 10.1109/CVPR.2017.215. Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compo- sitional skills of sequence-to-sequence recurrent networks. InInternational Conference on Machine Learning (ICML),

    doi:10.1109/cvpr.2017.215 2017

  24. [2018]

    Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song

    doi: 10.1145/3194164.3194186. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In28th USENIX Security Symposium (USENIX Security),

    arXiv

  25. [2019]

    Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B

    doi: 10.1145/3287560.3287596. Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black-box language models. InInternational Conference on Learning Representations (ICLR),

    arXiv

  26. [2020]

    NLP evaluation in trouble: On the need to measure LLM data contam- ination for each benchmark

    Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure LLM data contam- ination for each benchmark. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787,

    2023

  27. [2021]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I

    doi: 10.1145/3458723. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. InInternational Conference on Machine Learning (ICML),

    doi:10.1145/3458723

  28. [2022]

    doi: 10.1145/3531146. 3533231. Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation capabilities of language models. InProceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (ACL), pages 14116–14137,

    doi:10.1145/3531146

  29. [2023]

    Mark Chen, Jerry Tworek, et al

    doi: 10.1109/TSE.2023.3267446. Mark Chen, Jerry Tworek, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv 2023

  30. [2024]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644,

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644,

    Pith/arXiv arXiv

  31. [2025]

    Yue Jia and Mark Harman

    arXiv:2403.07974. Yue Jia and Mark Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649–678,

    Pith/arXiv arXiv